Beam search is a decoding method for generating a token sequence. Instead of committing to one next token as greedy search does, it keeps a fixed number of promising partial sequences, called beams. This can improve fluency, though the resulting sequences may be similar to one another. section 4.1.2 section 4.3
At each generation step:
- Expand each retained sequence with possible next tokens from the vocabulary. section 4.1.2
- Compare the resulting sequences and keep only the most likely ones, up to the chosen beam width. section 4.1.2
This resembles breadth-first search over a tree of possible sentences, but prunes that tree at every level. Beam width limits how many paths remain under consideration; it does not make the search exhaustive. Unlike top-k or nucleus sampling, beam search here selects likely sequences without sampling. section 4.1.2 section 4.3
Figure 4.1 A simplified example of beam search in action, focusing on a scenario with two beams (i.e., beam width equals two). We start by initializing the process with “LLMs” as the first word. From here, the beam search algorithm evaluates the two most probable next words.
The two-beam diagram makes expansion and pruning visible: starting from “LLMs,” only two promising continuations are carried forward. section 4.1.2
Generate with five beams
A model generation call can set `num_beams=5` to retain five candidates and `num_return_sequences=3` to request three results. The illustrated call also sets `max_new_tokens=128`, `early_stopping=True`, `pad_token_id=tokenizer.eos_token_id`, and `do_sample=False`. section 4.1.2
