A small language model (SLM) is a compact model chosen when a task needs lower cost, latency, or resource use more than broad general-purpose capability. It may run on an edge device, handle a high-volume task, or serve as one specialized component in a larger AI system. Small does not mean interchangeable: choose the architecture and training target for the job. section 7.0 section 7.6
In a multi-model pipeline, SLMs can route inputs, retrieve information, screen unsafe requests, or check outputs. A larger model can retain the orchestration and broad reasoning role. This avoids paying the cost of a large-model call at every step, though a multi-agent workflow still adds operational overhead. section 7.1 section 7.2
Figure 7.2 SLMs collaborating in an agentic system. Lightweight models serve as retrievers and domain experts, while a larger model orchestrates reasoning and produces the final output.
The diagram shows lightweight retrieval and domain specialists contributing to a workflow orchestrated by a larger model. section 7.2
Match the model to the output the task requires:
- For sentiment, compliance, or intent classification, use a compact encoder when no generated response is needed. ModernBERT is used for three-class financial-news sentiment classification. section 7.3 section 7.3.3
- For a specialized conversational response, fine-tune an instruction-tuned model. Gemma 3 270M is adapted with ProsocialDialog examples to encourage empathetic, prosocial replies. section 7.4
- For a bounded generation task, fine-tune on task-specific pairs. English–Spanish OPUS-100 examples turn Gemma 3 270M into a dedicated translation model rather than a broad multilingual system. section 7.5
Build a compact sentiment classifier
Start with stratified train, validation, and test splits of Financial PhraseBank. Its neutral class is much larger than its positive and negative classes, so cap neutral examples in training while retaining the other classes. Load ModernBERT for sequence classification, tokenize with a short maximum length and dynamic padding, then train with Hugging Face Trainer. Select the best checkpoint by macro F1 and inspect accuracy, precision, recall, F1, and the test confusion matrix; accuracy alone can obscure weak performance on a minority class. section 7.3.3 section 7.7
Adapt a small generator
For prosocial dialogue, format system, user, and assistant messages with a chat template, check a baseline reply, run supervised fine-tuning with TRL SFTTrainer, and compare base and tuned generations and perplexity. For translation, put the English source and Spanish target into the same task-directed chat format, filter and sample OPUS-100, then compare translations before and after fine-tuning. Improved specialization is the goal; generated samples still need inspection. section 7.4 section 7.5
The practical trade-off is breadth versus specialization. Compact models can be fine-tuned with modest hardware and deployed locally for privacy or latency-sensitive work, but a narrowly tuned model should not be assumed to replace a larger general-purpose reasoner. Evaluate it on the specific task and deployment constraints it is meant to serve. section 7.5 section 7.6
