liveBook · Manning

Language model

As explained in Building Specialized AI Systems

A large language model trained to perform a broad range of tasks rather than being specialized for one domain.

A large language model (LLM) is a broad-purpose neural model trained on enormous token collections to predict language and support tasks such as classification and summarization. Its scale enables versatile behavior, including few-shot learning, but can create limitations for specialized applications. section 1.1 section 1.1.1

Core characteristics include:

  • Stores learned knowledge in billions or trillions of neural-network parameters represented by numerical weights. section 1.1.1
  • Processes text as tokens, typically subword units, and learns statistical relationships across very large corpora. section 1.1.1
  • Builds foundational language understanding during pre-training, then improves task behavior through instruction tuning or reinforcement learning. section 1.1
  • Provides a foundation for adaptation methods ranging from supervised and full-parameter fine-tuning to parameter-efficient approaches such as LoRA and QLoRA. section 3.8
Figure 1.1 Training process for general-purpose LLMs. Phase 1 (pre-training) ingests massive raw text corpora to build foundational language understanding; Phase 2 (instruction tuning/RL) uses curated labeled datasets to align the model toward helpful, accurate behavior.

Follow the training pipeline from raw-text pre-training to instruction tuning and reinforcement learning. section 1.1

Scale and adaptation

GPT-4 is estimated to contain more than one trillion total parameters, while Meta’s LLaMA 2 70B required approximately 1.7 million GPU-hours to train. These models can later be adapted with methods such as LoRA, which trains fewer than 1% of the parameters. section 1.1.1 section 3.8

How Building Specialized AI Systems covers it

How deep it goes
Explains parameters, tokens, training phases, and adaptation methods, but stops short of detailed architecture implementation.
Examples
Uses GPT-4, Claude 3 Opus, LLaMA 2 70B, tokens, and a two-phase training pipeline as concrete examples.
What you can do after
Evaluate model scale, generality, training stages, and trade-offs before selecting specialization or fine-tuning.
How it connects
Connects Transformer architecture, self-attention, fine-tuning, prompt engineering, retrieval-augmented generation, and specialized models.

Also explained in

Compare how 89 books explain language model →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Building Specialized AI Systems. The section links point to the parts of the book it draws on.