liveBook · Manning

Foundation model

As explained in LLM Customization and Fine-Tuning

The original pretrained model whose weights are kept fixed while LoRA learns an additive adaptation.

A foundational model is the starting model for adaptation. In LoRA, its pretrained weights remain fixed while an additive adaptation is learned. Choosing the starting model sets limits on task capability, training capacity, serving latency, and the effort needed to deploy it. section 2.7

The term base model also has a narrower meaning: a model pretrained on large text corpora but not tuned to follow instructions. It tends to continue text rather than answer a request. An instruct model has already been fine-tuned to follow instructions and can itself serve as the starting model for further adaptation. section 3.3.1

Choose a starting model against four practical constraints:

  • Task complexity and headroom: a model that is too small may remain limited even after extensive domain training. section 2.7
  • Inference latency: the model must fit the application’s response-time budget. section 2.7
  • Training and serving hardware: a model that is too large can exceed available GPU capacity or slow iteration. section 2.7
  • Tooling maturity: published adaptation recipes and support in the serving stack can save substantial engineering time. When benchmarks are similar, prefer the ecosystem the team can use. section 2.7

Starting with an instruct model can reduce data needs because instruction-following is already established. With fewer than 300 examples, consider an instruct model; a pretrained-only base generally needs enough data to teach both instruction-following and the domain. Training prompts should match the prompts used at inference, including the instruction and chat template. section 3.3.1

A starting model for ticket triage

Qwen3-4B-Instruct-2507 is used as the starting model for hands-on adaptation. For ticket triage, each training example includes the same classification instruction used at inference, followed by the ticket; the assistant turn carries the label. Omitting the instruction during training creates a prompt mismatch that can degrade accuracy. section 2.7 section 3.3.1

Treat the starting model as a versioned dependency. LoRA adapters and other adapted artifacts are tied to the specific model on which they were trained, so a base-model upgrade requires rerunning the fine-tuning pipeline. Record the base model alongside the data hash, checkpoint path, technique, and evaluation metrics to support reproducibility and rollback. section 9.11

A stronger or larger starting model does not make adaptation the right answer for every problem. If the task depends on newly arrived documents, retrieval-augmented generation may be a better fit than fine-tuning. Establish a baseline and consider data quality, cost, and the pace of base-model improvement before adapting. section 2.0 section 1.11

How LLM Customization and Fine-Tuning covers it

How deep it goes
Explains selection across task, latency, hardware, and tooling constraints, and distinguishes pretrained-only from instruction-tuned starting models.
Examples
Uses Qwen3-4B-Instruct-2507 and a ticket-triage classifier to illustrate the starting model and prompt consistency.
What you can do after
Evaluate starting models against task headroom, latency, hardware, tooling, and available training data.
How it connects
Connects starting models to LoRA, supervised fine-tuning, retrieval-augmented generation, and model versioning.

Also explained in

Compare how 30 books explain foundation model →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Customization and Fine-Tuning. The section links point to the parts of the book it draws on.