liveBook · Manning

Language modeling

Language modeling is the task of learning or applying probability distributions over language sequences to estimate or predict units from context.

How 7 Manning books explain it

Fight Fraud with Machine Learning

Language modeling is the task of assigning probabilities to or predicting tokens in a sequence based on preceding context.

How it is used
Motivates how GPT-style LLMs generate text and contribute knowledge for phishing detection.
How deep it goes
Defines next-word probabilities and corpus-learned co-occurrence patterns, while deferring detailed LLM construction elsewhere.
Examples
Uses “French is a beautiful …” and “It will” to illustrate probability-ranked next-word prediction and greedy search.
What you can do after
Evaluate/understand how an LLM selects likely next words from preceding context.

Read how Fight Fraud with Machine Learning explains language modeling →

LLMs in Production

Language modeling is the task of learning the statistical structure of language so a system can predict or generate text.

How it is used
Frames language modeling as the foundation for understanding LLM behavior, architecture choices, and production outcomes.
How deep it goes
Traces modeling from statistical methods through neural networks and transformers, while stopping short of formal derivations.
Examples
Uses abstraction layers, historical model families, and IBM Watson’s commercialization failure to illustrate capabilities and limits.
What you can do after
Evaluate language-modeling approaches and anticipate how linguistic abstractions, architecture, and production constraints affect results.

Read how LLMs in Production explains language modeling →

The Art of AI Product Development

Language modeling is the task of predicting or generating text units from surrounding or preceding textual context.

How it is used
Frames language modeling as the raw objective underlying text generation and as a basis for comparing model behavior.
How deep it goes
Defines context-based prediction and distinguishes three objectives, but does not fully elaborate autoencoding in the supplied treatment.
Examples
Uses Mistral, ChatGPT, machine translation, summarization, code generation, and context-prediction illustrations.
What you can do after
Evaluate language-model outputs by distinguishing raw continuation behavior from conversational behavior shaped by fine-tuning.

Read how The Art of AI Product Development explains language modeling →

Machines that Think

Language modeling is the computational process of learning or applying statistical structure in language to assess or generate sequences of text.

How it is used
Introduces probabilistic sequence prediction as a foundation for representing language and understanding generative model reliability.
How deep it goes
Explains conditional frequency estimates and limited-context assumptions, then discusses evaluation and reliability constraints.
Examples
Uses “the sun rises in the east” and the frequency of “in the east” to illustrate sequence and continuation probabilities.
What you can do after
Evaluate and understand language-model predictions without treating fluency as proof of factual accuracy.

Read how Machines that Think explains language modeling →

Deep Learning with Python, Third Edition

A next-token prediction training setup in which the model learns to predict the next token from previous tokens.

How it is used
Uses next-token prediction to train sequence models and generate open-ended text from prompts.
How deep it goes
Explains token-level training, recurrent state, causal leakage, and token-by-token inference, but does not derive the optimization mathematics.
Examples
Demonstrates character-level Shakespeare generation using a prompt, vocabulary mappings, recurrent state, greedy selection, and detokenization.
What you can do after
Build it from scratch with a recurrent model, then implement prompt conditioning and autoregressive token generation.

Read how Deep Learning with Python, Third Edition explains language modeling →

Deep Learning with R, Third Edition

Language modeling is the task or technique of learning a probability distribution over sequences of linguistic symbols.

How it is used
Frames incomplete-sentence prediction as a practical NLP task and supports sequential text generation.
How deep it goes
Defines next-token prediction, probability distributions, and sampling strategies while treating architecture history comparatively.
Examples
Uses incomplete-sentence next-word prediction and greedy, random, and top-k text generation.
What you can do after
Apply language models to predict tokens and generate text using greedy, random, or top-k sampling.

Read how Deep Learning with R, Third Edition explains language modeling →

Transfer Learning for Natural Language Processing

Language modeling is the task of modeling probability distributions over sequences of language.

How it is used
Models probabilities over word sequences to support next-word prediction and language generation.
How deep it goes
Defines sequence probabilities and highlights next-word prediction without detailing algorithms or implementation.
Examples
Uses predicting the next word or sentence as examples of language generation.
What you can do after
Understand how sequence probabilities support next-word or sentence prediction.

Read how Transfer Learning for Natural Language Processing explains language modeling →

Also covered in

What to know first

Related concepts

Questions about language modeling

What is language modeling?

Language modeling is the task of learning or applying probability distributions over language sequences to estimate or predict units from context.

Which Manning books explain language modeling?

Fight Fraud with Machine Learning, LLMs in Production, The Art of AI Product Development, Machines that Think, Deep Learning with Python, Third Edition, Deep Learning with R, Third Edition and Transfer Learning for Natural Language Processing.

What should I know before learning language modeling?

Manning books that teach language modeling usually introduce language model, machine learning, machine translation and attention first.

What is the difference between language modeling and character-based modeling?

Language modeling is the task of learning or applying probability distributions over language sequences to estimate or predict units from context. Character-based modeling is language modeling in which text is represented and predicted as sequences of individual characters.

What kinds of language modeling are there?

Kinds of language modeling include Continuous Bag of Words.

What is language modeling part of?

Language modeling is part of language model. A language model is a computational model that represents patterns in linguistic sequences to assign probabilities, predict, generate, or evaluate language.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.