liveBook · Manning

Foundation model

As explained in Rearchitecting LLMs

A foundation model is a broadly pretrained model intended to support adaptation to many downstream tasks.

A foundational model is a broadly pretrained model that can be adapted for downstream tasks. As a base model, it can also serve as the starting point for creating a smaller, faster model while preserving as much of its general capability as possible. section 1.1.2 section 2.0

For general-purpose optimization, first establish the base model’s capabilities and inference metrics. Then remove layers through depth pruning, measure the impact, and use knowledge distillation to transfer knowledge from the base model to the pruned model. Task-specific optimization is a separate phase. section 2.0 section 2.2

Baseline a Gemma model

Load google/gemma-3-270m and its tokenizer, inspect the parameter count and layer structure, and evaluate the unmodified model before pruning. The inspected model has 268,098,176 parameters and 18 layers. Baseline benchmark scores are 0.59 on ARC Easy, 0.34 on HellaSwag, 0.43 on LAMBADA OpenAI, and 0.54 on WinoGrande. section 2.2

MODEL_NAME = "google/gemma-3-270m"
 
model = AutoModelForCausalLM.from_pretrained(
    MODEL_NAME,
    device_map="auto"
)
 
tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME)
if tokenizer.pad_token is None:
    tokenizer.pad_token = tokenizer.eos_token

Load the base model and tokenizer; if the tokenizer has no padding token, use its end-of-sequence token. section 2.2

Full benchmark and recovery runs are computationally heavy. For a shorter exploratory run, set BENCHMARK_LIMIT to 100 and reduce RECOVERY_SAMPLES and EPOCHS; expect the numbers to differ from the full run. section 2.2

Fine-tuning adapts a base model using a task-specific dataset, but access matters. With a closed model, fine-tuning uses provider tools and produces a hosted artifact whose weights cannot be exported. Pricing can change, and a base-model upgrade or retirement may require repeating fine-tuning and evaluation. section 1.1.2

How Rearchitecting LLMs covers it

How deep it goes
Outlines baseline evaluation, depth pruning, impact measurement, and knowledge recovery; defers task-specific optimization.
Examples
Uses Gemma 3 270M, model inspection, and benchmark evaluations to establish a baseline before pruning.
What you can do after
Evaluate/understand a base model against a baseline before measuring the effects of structural changes.
How it connects
Connects base models to fine-tuning, retrieval-augmented generation, depth pruning, and knowledge distillation.

Also explained in

Compare how 30 books explain foundation model →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.