Conditional probability describes how likely an event is when another event or condition is already known. In language generation, it gives the probabilities of possible next words based on the words that came before. section 10.2
Choosing the next word
Starting with <SOS> and “LLMs,” a generative model ranks possible next tokens, chooses one, then conditions the next choice on the expanded sequence. Temperature controls how often it explores less-probable options. section 10.2
A small autocomplete experiment makes the calculation concrete:
- Fit scikit-learn’s CountVectorizer to bigrams from sentences in 100 Wikipedia articles, calculate conditional probabilities for possible second words, and use Python’s random.choice to autocomplete sentences. section 10.4
Figure 10.2 Stochastic chameleons deciding words one at a time
Trace one path through the branching probability tree used to select words from left to right. section 10.2
