liveBook · Manning

Generation

As explained in Build a DeepSeek Model (From Scratch)

Generation is the process of producing a sequence of tokens from a language model, typically by sampling or selecting successive outputs.

Generation produces text by repeatedly using a fixed, pretrained language model to predict the next token. Each new token extends the sequence and becomes context for the next prediction. section 2.1.1 section 2.3.1

A practical generation loop must handle more than selecting tokens:

  • Tokenize prompts, call the policy model's generate function with sampling enabled, and record generated tokens. section 7.10
  • Store old-policy log probabilities for generated tokens when completions will support PPO or GRPO updates. section 7.10
  • Batch variable-length responses with padding and masks so early termination and padding do not affect scoring or gradients. section 7.10

Naive autoregressive generation repeatedly processes the entire growing sequence, making attention work quadratic in sequence length. A KV cache avoids recomputing prior key and value states, but shifts pressure toward GPU memory and bandwidth. section 2.3.1 section 2.5

Extending a short prompt

Given the prompt "The next day," generation processes the prompt and produces "is"; it then processes the extended sequence "The next day is" and produces "bright," followed by "and." section 2.3.1

Toy function
Real trainer equivalent
sample_group
Tokenizer + policy.generate + storage of sampled tokens and old log probabilities
verify_math_answer
Task-specific verifier, unit tests, compiler, symbolic checker, or reward model
group_advantages
Prompt-group reward normalization with numerical safeguards
sequence_logprobs
Forward pass that gathers token log probabilities under policy/reference models
grpo_loss
Masked clipped surrogate objective with KL control and optimizer integration

Maps a toy sampling function to a real trainer's tokenizer, policy.generate call, sampled-token storage, and old-log-probability recording. section 7.10

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Explains autoregressive repetition, KV-cache trade-offs, batching, and log-probability recording, but defers distributed production machinery.
Examples
Uses the prompt "The next day" and a trainer mapping from sampling functions to policy.generate implementations.
What it assumes
Assumes a pretrained model with fixed parameters and teaches the distinction between training and inference from that starting point.
What you can do after
Apply it via a policy model's generate function and diagnose sequence-length, padding, and KV-cache bottlenecks.
How it connects
Connects generation with inference, autoregressive modeling, tokenization, KV caching, sampling, log probabilities, and reinforcement learning.

Also explained in

Compare how 29 books explain generation →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.