liveBook · Manning

Foundation model

As explained in Building Specialized AI Systems

A machine-learning model trained on data before being adapted to a downstream task or domain.

A foundational model is a pretrained language model that supplies broad language understanding, reasoning capability, and general knowledge before adaptation to a particular task or domain. Pretraining builds this base by learning to predict tokens from massive raw-text corpora; later instruction tuning, reinforcement learning, or fine-tuning directs its behavior toward useful tasks such as classification, summarization, and generation. section 1.1 section 1.4.1

Figure 1.1 Training process for general-purpose LLMs. Phase 1 (pre-training) ingests massive raw text corpora to build foundational language understanding; Phase 2 (instruction tuning/RL) uses curated labeled datasets to align the model toward helpful, accurate behavior.

The training pipeline separates broad pretraining from the later alignment stage that teaches more complex task behavior. section 1.1

A foundation is valuable because specialization can reuse knowledge that would be costly to recreate from scratch. Fine-tuning supplies curated examples for a target behavior and adjusts the model toward domain-specific outputs. Its success depends on the starting model: fine-tuning can refine and align existing knowledge, but limited domain grounding makes specialization inefficient and can yield unstable results. section 1.4.1 section 2.3.3

Select a foundational model by weighing the starting point against the operating constraints:

  • Match baseline task capabilities to the work, such as reading comprehension, salient-information selection, and coherent generation for summarization. section 2.5.1
  • Check domain exposure to relevant terminology, standards, reasoning patterns, and task structures; closer exposure reduces required training data, time, and compute. section 2.3.1 section 2.3.3
  • Fit model size and deployment requirements to available compute and infrastructure; larger models may offer stronger capabilities, while smaller models can suffice for well-defined tasks. section 1.4.2
  • Account for privacy, security, and regulatory constraints when sensitive financial, legal, or health data cannot be sent to third-party cloud services. section 1.1.2
  • Verify that a public evaluation dataset was not already present in the model’s training data, or fine-tuning results may be misleading. section 2.5.1

Financial analyst-report summarization

A general web-trained model may summarize an analyst report yet miss which findings are material or how to contextualize earnings guidance. Qwen2.5-7B-Instruct is selected as a starting point because of reported exposure to financial reports, filings, and market commentary, plus summarization ability and practical 7B-scale infrastructure requirements. Fine-tuning can then sharpen that existing foundation for the target workflow. section 2.5.1

General-purpose foundations trade breadth for specialization. They can provide adequate results across many domains, but specialized models can outperform them on focused financial, medical, or legal work. Frontier-scale development and local deployment also impose high cost and infrastructure burdens, making adaptation of an existing model a practical alternative to training a new one. section 1.1.2 section 1.4.1

Figure 2.2 LoRA adapter swapping for different domains or use cases. A single frozen foundation model serves as the shared backbone for multiple lightweight domain adapters. Each adapter adds a small set of trainable weights alongside the frozen layers — less than 0.1% of the base model's parameters — and can be swapped in or out without modifying the underlying model.

Parameter-efficient fine-tuning preserves one frozen foundation model while small swappable adapters provide distinct domain behaviors. section 2.2

Parameter-efficient fine-tuning extends the foundation-model approach by keeping base weights frozen and training small adapters instead. A single model can support separate finance and customer-support adaptations, reducing checkpoint storage and enabling experimentation. LoRA follows this pattern by learning low-rank updates in targeted layers rather than modifying the pretrained weights. section 2.2 section 3.5.1

How Building Specialized AI Systems covers it

How deep it goes
Explains pretraining, selection criteria, and adapter-based adaptation, while deferring detailed training mechanics to fine-tuning methods.
Examples
Uses financial report summarization, biomedical question answering, customer-service replies, and swappable finance and customer-support adapters.
What it assumes
Assumes practical familiarity with target tasks while introducing pretraining and fine-tuning through transfer-learning analogies.
What you can do after
Evaluate and select a starting model by task fit, domain exposure, compute requirements, and deployment constraints.
How it connects
Connects foundation models with pretraining, instruction tuning, transfer learning, supervised fine-tuning, PEFT, LoRA, and domain knowledge.

Also explained in

Compare how 30 books explain foundation model →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Building Specialized AI Systems. The section links point to the parts of the book it draws on.