liveBook · Manning

Language modeling

As explained in Fight Fraud with Machine Learning

Language modeling is the task of assigning probabilities to or predicting tokens in a sequence based on preceding context.

Language modeling predicts what word or word sequence is likely to come next after a preceding sequence. It assigns probabilities to candidate continuations: after “French is a beautiful …,” a trained model should give “language” a higher probability than “country.” section 5.2.1

A language model learns from a text corpus by observing which words occur together and which do not. From these patterns it forms probabilistic rules for continuations in different contexts. Training on very large volumes of text produces a large language model (LLM), whose writing capabilities can support conversation and other text tasks. section 5.2.1 section 5.0

Language modeling provides the underlying prediction capability behind several LLM uses:

  • Generate human-like text by repeatedly predicting a next word from preceding words. section 5.0 section 5.2.1
  • Support translation, summarization, question answering, and sentiment analysis. section 5.0
  • Bring broad text knowledge to phishing detection, alongside a progression from rules and classical machine learning to deep learning. section 5.0

Choosing the next word

Given the prefix “It will,” a model scores possible next words with a probability distribution. A simple greedy approach selects the most likely candidate; repeating that choice extends the generated text one word at a time. section 5.2.1

Figure 5.7 Toy example of how an LLM predicts the next word (given the preceding words) by sampling from a distribution of likely words. In this example, the LLM always chooses the most likely next word – called the greedy search approach.

This toy prediction sequence makes the probability distribution and greedy next-word selection concrete. section 5.2.1

How Fight Fraud with Machine Learning covers it

How deep it goes
Defines next-word probabilities and corpus-learned co-occurrence patterns, while deferring detailed LLM construction elsewhere.
Examples
Uses “French is a beautiful …” and “It will” to illustrate probability-ranked next-word prediction and greedy search.
What you can do after
Evaluate/understand how an LLM selects likely next words from preceding context.
How it connects
Connects language modeling to large language models, GPT, ChatGPT, text generation, and phishing detection.

Also explained in

Compare how 7 books explain language modeling →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Fight Fraud with Machine Learning. The section links point to the parts of the book it draws on.