A foundational model is a broadly pretrained model that can be adapted for downstream tasks. As a base model, it can also serve as the starting point for creating a smaller, faster model while preserving as much of its general capability as possible. section 1.1.2 section 2.0
For general-purpose optimization, first establish the base model’s capabilities and inference metrics. Then remove layers through depth pruning, measure the impact, and use knowledge distillation to transfer knowledge from the base model to the pruned model. Task-specific optimization is a separate phase. section 2.0 section 2.2
Baseline a Gemma model
Load google/gemma-3-270m and its tokenizer, inspect the parameter count and layer structure, and evaluate the unmodified model before pruning. The inspected model has 268,098,176 parameters and 18 layers. Baseline benchmark scores are 0.59 on ARC Easy, 0.34 on HellaSwag, 0.43 on LAMBADA OpenAI, and 0.54 on WinoGrande. section 2.2
MODEL_NAME = "google/gemma-3-270m"
model = AutoModelForCausalLM.from_pretrained(
MODEL_NAME,
device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME)
if tokenizer.pad_token is None:
tokenizer.pad_token = tokenizer.eos_token
Load the base model and tokenizer; if the tokenizer has no padding token, use its end-of-sequence token. section 2.2
Full benchmark and recovery runs are computationally heavy. For a shorter exploratory run, set BENCHMARK_LIMIT to 100 and reduce RECOVERY_SAMPLES and EPOCHS; expect the numbers to differ from the full run. section 2.2
Fine-tuning adapts a base model using a task-specific dataset, but access matters. With a closed model, fine-tuning uses provider tools and produces a hosted artifact whose weights cannot be exported. Pricing can change, and a base-model upgrade or retirement may require repeating fine-tuning and evaluation. section 1.1.2





























