Chapter 1: Small language models

Introduction

The introduction explains why LLM hype makes it important to assess models against real capabilities, risks, and business needs rather than expectations.

• LLM hype and business fit: Explain how hype around LLMs can make it harder to judge what they can do and whether they fit a business need.
  Why it matters: A realistic assessment helps you decide whether an LLM suits a particular business need.

Key terms: large language models (LLMs), hype, business needs

1.1 What are small language models?

This section defines small language models and explains why their scale makes them useful for local deployment, domain-specific work, and some agentic systems.

• SLM scale and architecture: Distinguish SLMs from LLMs by their typical scale and resource needs while explaining that both use the same core Transformer technology.
  Why it matters: This distinction explains why SLMs can tackle similar kinds of tasks under tighter resource limits.
• Local deployment advantages: Explain how an SLM’s smaller resource footprint supports deployment on constrained hardware and why running locally can be valuable.
  Why it matters: The hardware and data-location benefits help explain when a smaller model is a practical choice.
• Low-cost domain specialization: Explain why SLMs are attractive for domain-specific tasks, particularly when private data and privacy requirements matter.
  Why it matters: Specialization offers a reason to choose an SLM beyond simply fitting it onto smaller hardware.
• SLMs in agentic systems: Describe the section’s argument for using SLMs in agentic systems and the proposed role of mixed-model systems.
  Why it matters: Agentic systems provide another context in which model size and cost can influence model choice.

Key terms: small language models, SLMs, LLMs, parameters, Transformer architecture, memory footprint, computational power, CPUs, consumer GPUs, offline

1.2 The Transformer architecture

This section explains why Transformers train more efficiently than RNNs, how embeddings support text representation, and how encoder- and decoder-based architectures suit different tasks.

• Self-attention and parallel processing: Explain why RNNs train slowly and how a Transformer’s self-attention and lack of recurrence allow more parallel processing.
  Why it matters: This is the section’s main explanation for the Transformer’s training advantage over RNNs.
• Word embeddings in context: Describe what word embeddings represent and why they help a Transformer work with words in context.
  Why it matters: Embeddings explain how words become numerical representations the model can use.
• Encoder and decoder task fit: Distinguish the encoder and decoder roles in the original Transformer and relate the BERT and GPT families to the text tasks described here.
  Why it matters: The task determines which Transformer family is the more appropriate starting point.
• RLHF in GPT evolution: State how the section characterizes reinforcement learning from human feedback in relation to OpenAI’s GPT-based models.
  Why it matters: It distinguishes a later fine-tuning technique from the Transformer architecture itself.

Key terms: RNN, self-attention, parallelism, recurrent structure, word embeddings, semantic, syntactic, numerical space, encoder–decoder, encoder

1.3 Areas of application

The section explains that, despite their name, LLMs can work with symbolic content beyond natural-language text and support a wide range of tasks.

• Breadth of LLM applications: Explain why “language” in LLM can be misleading and describe the kinds of tasks and content LLMs can handle beyond natural-language translation.
  Why it matters: It helps you recognize applications for LLMs beyond ordinary prose and translation.

Key terms: LLM, natural-language processing (NLP), code generation, symbolically represents

1.4 The open source revolution

This section explains how open source pretrained LLMs give organizations a lower-cost starting point for custom models, while leaving deployment and inference costs to address.

• Open source model choices: Explain that publicly available LLMs include generalist and domain-specific models under different licenses, and that an organization can adapt a pretrained model rather than build one from scratch.
  Why it matters: The available model and its usage terms shape whether it is a practical starting point for your organization.
• From-scratch cost drivers: Describe why development can cost more than training an LLM from scratch, and why deployment and inference bring substantial, hard-to-predict costs afterward.
  Why it matters: A realistic budget must account for development and serving, not just the training run.
• Pretrained model cost trade-off: Explain which costs fall when starting from a pretrained open source model and which costs remain significant.
  Why it matters: Starting with a pretrained model saves money, but it does not eliminate the work of preparing data or serving the result.

Key terms: open source model, pretrained open source model, generalist, specialized, licenses, development, training, deployment, inference serving, pretrained OS model

1.5 Risks and challenges with generalist LLMs

This section weighs the convenience of commercial generalist LLMs against privacy, control, transparency, reliability, and safety risks that can make them unsuitable for important business uses.

• External hosting and data exposure: Explain why sending prompts to a third-party LLM can put company information at risk, despite the convenience of a managed service.
  Why it matters: Convenient access to a hosted model can come at the cost of control over sensitive information.
• Provider changes and costs: Describe how reliance on a commercial provider can make application behavior and spending unpredictable.
  Why it matters: An application can change or become more expensive even when its own team has changed nothing.
• Closed model transparency limits: Explain what a customer cannot verify about a closed source LLM and why that matters.
  Why it matters: A model can be hard to audit when its construction and training data are hidden.
• Hallucinations and sectoral compliance: Distinguish the two hallucination types described here and explain why generalist commercial models can be problematic for sector-specific use.
  Why it matters: A fluent answer is not necessarily supported by evidence or suitable for a regulated business task.
• Unsafe outputs and guardrails: Describe the harmful-content and code-generation risks, including the limits of provider safeguards.
  Why it matters: Provider safeguards do not remove the need to assess generated content and code.

Key terms: deployment outside your organization’s network, data leakage risks, model or version drift, cost unpredictability, token-based billing, closeness, transparency, reproducibility, interpretability, intrinsic hallucination

1.6 Domain-specific vs. generalist LLMs for business value

This section weighs when a domain-specific model offers more business value than a generalist LLM, considering specialization, privacy, resources, sustainability, and alignment.

• Generalist versus specialized models: Explain when a generalist pretrained LLM may be sufficient and when a specialized task calls for tuning on domain-specific data.
  Why it matters: The task’s need for specialized knowledge is a starting point for choosing a model.
• Transfer learning for specialization: Describe transfer learning as applying a model trained on one task or domain to a related one, and distinguish it from necessarily training the entire model.
  Why it matters: Specialization does not mean starting over with full model training.
• Privacy and compliance considerations: Explain how regulated-domain requirements and private-data leakage concerns can favor a private domain-specific model, while creating an infrastructure consideration.
  Why it matters: A model choice must account for data protection and operational constraints, not just output quality.
• Smaller models and sustainability: Explain the section’s sustainability argument for choosing a smaller open source baseline model for domain-specific tuning.
  Why it matters: Model size affects the environmental side of the business choice.
• Domain-specific user alignment: Distinguish the common principles of LLM alignment from the author’s emphasis on matching user intent through domain specialization.
  Why it matters: A model can become more capable without necessarily becoming better matched to what its users intend.

Key terms: generalist pretrained LLMs, domain-specific data, domain specialization, transfer learning, full training, SLM, regulated domains, domain-specific training data, private data leakage, private, offline, domain-specific LLM

1.7 Summary