Chapter 15: Test-time compute and small language models

Introduction

The chapter introduction previews test-time compute, inference tools, and a commodity-hardware example of specializing an SLM with GRPO.

15.1 Test-time compute

The section explains how test-time compute can improve difficult responses, what it costs, why closed commercial implementations are hard to validate, and why domain-specific SLMs are relevant.

• Test-time candidate exploration: Explain that test-time compute gives a model more computation during inference to explore candidate responses, evaluate them, and choose an answer within a compute budget.
  Why it matters: This is the basic process behind the section’s claims about better answers and higher inference costs.
• Accuracy and compute tradeoff: Explain why extra inference-time processing may help on difficult tasks, and state its cost and limits.
  Why it matters: The potential accuracy gain has to be weighed against the additional resources spent on each answer.
• Commercial model validation risks: Describe the cost and transparency concerns when commercial closed-source models use more test-time compute.
  Why it matters: A better final answer may still be unsuitable if its price or hidden refinement process is unacceptable.
• Domain-specific SLM advantage: State why a small model can outperform a larger one on the same domain-specific task.
  Why it matters: It explains why the chapter considers small models alongside commercial large models.

Key terms: test-time compute, inference time, compute budget, candidate outputs, reasoning, compute costs, artificial general intelligence, closed-source models, refining process, validation

15.2 The OptiLLM inference proxy

This section explains how OptiLLM adds iterative reasoning to model inference, how AutoThink allocates and steers reasoning, and what a small-model evaluation reveals about overthinking and answer formatting.

• OptiLLM proxy request flow: Trace how a client request passes through OptiLLM to an OpenAI API–compatible service and how the proxy refines responses before returning one.
  Why it matters: The proxy flow explains where the additional inference work happens.
• Local inference support limits: Describe the local-model options OptiLLM supports and the limitation noted for external local servers.
  Why it matters: The way a local model is served affects which refinement approaches can be used.
• AutoThink adaptive reasoning phases: Explain how AutoThink chooses a reasoning budget and steers a model’s reasoning across its preprocessing and inference phases.
  Why it matters: AutoThink illustrates how extra inference work can be adapted to the query rather than applied uniformly.
• Small-model evaluation setup: Describe how the section evaluates a small model with AutoThink on selected math-reasoning questions.
  Why it matters: The setup establishes what the reported successes and failures are being compared against.
• Reasoning output evaluation pitfalls: Distinguish the evaluation problems illustrated by overthinking, added assumptions, and failure to follow the required answer format.
  Why it matters: A generated reasoning trace and the answer an evaluator can score are not necessarily the same thing.

Key terms: OptiLLM, inference-optimizing proxy, static inference, refinement cycle, local inference, LoRA weights, llama.cpp, Ollama, AutoThink, preprocessing

15.3 SLMs with embedded test-time compute

This section describes the availability of open-source small reasoning models and how to access them for local use.

• Open reasoning model availability: Explain that reasoning capabilities are now available in open models of different sizes, including SLMs, rather than only in closed commercial LLMs. Describe how this gives smaller organizations and individuals more room to innovate without proprietary-model costs and constraints.
  Why it matters: Reasoning-capable models are an option even when proprietary models are impractical.
• Domain tuning and local access: Describe the section’s two practical claims about open reasoning models: reinforcement learning combined with process-reward models supports transparent fine-tuning for domain-specific tasks, and the models can be accessed for local use through Transformers or, for most, quantized GGUF versions in Ollama.
  Why it matters: These are the practical routes the section identifies for adapting or running open reasoning models.

Key terms: reflective models, reasoning capabilities, open source SLMs, thinking mode, reinforcement learning, process-reward models, domain-specific tasks, Transformers, GGUF, Ollama model catalog

15.4 Building a reasoning domain-specific SLM

This section walks through specializing a small language model for additive-manufacturing reasoning with GRPO, from memory-conscious setup and reward design to training and evaluation.

• GRPO compute advantage: Explain why GRPO is presented as a less resource-intensive alternative to PPO for specializing an SLM, and how that makes the example feasible on a consumer GPU.
  Why it matters: This explains why reasoning-focused specialization is practical under a limited GPU budget.
• Memory-conscious model setup: Describe how the example sets up Qwen 2.5 3B Instruct for GRPO tuning on a single 16 GB GPU, including the purposes of 4-bit loading, LoRA, and fast inference.
  Why it matters: The setup shows which practical choices keep the training example within its hardware budget.
• Question and answer training data: Describe how the additive-manufacturing dataset is turned into prompts and ground-truth answers and divided for training and testing.
  Why it matters: Separating prompts, answer targets, and held-out examples makes the reward and evaluation steps understandable.
• Reasoning reward design: Explain what the example’s reward functions encourage in generated responses and why reward engineering matters for GRPO training.
  Why it matters: The model needs a training signal for both correct domain answers and the intended structured reasoning response.
• Training and held-out evaluation: Trace how the configured GRPO trainer is run, how progress is checked, and how the saved LoRA weights are used to inspect an answer on unseen data.
  Why it matters: This connects the training setup to an observable check of what the specialized model generates.

Key terms: GRPO, PPO, policy model, value model, LoRA, model quantization, Qwen 2.5 3B Instruct, Unsloth, FastLanguageModel, QLoRA

15.5 Summary