Language modeling
Language modeling is the task of learning or applying probability distributions over language sequences to estimate or predict units from context.
Fight Fraud with Machine Learning
Language modeling is the task of assigning probabilities to or predicting tokens in a sequence based on preceding context.
- How it is used
- Motivates how GPT-style LLMs generate text and contribute knowledge for phishing detection.
- How deep it goes
- Defines next-word probabilities and corpus-learned co-occurrence patterns, while deferring detailed LLM construction elsewhere.
- Examples
- Uses “French is a beautiful …” and “It will” to illustrate probability-ranked next-word prediction and greedy search.
- What you can do after
- Evaluate/understand how an LLM selects likely next words from preceding context.
Read how Fight Fraud with Machine Learning explains language modeling →
LLMs in Production
Language modeling is the task of learning the statistical structure of language so a system can predict or generate text.
- How it is used
- Frames language modeling as the foundation for understanding LLM behavior, architecture choices, and production outcomes.
- How deep it goes
- Traces modeling from statistical methods through neural networks and transformers, while stopping short of formal derivations.
- Examples
- Uses abstraction layers, historical model families, and IBM Watson’s commercialization failure to illustrate capabilities and limits.
- What you can do after
- Evaluate language-modeling approaches and anticipate how linguistic abstractions, architecture, and production constraints affect results.
The Art of AI Product Development
Language modeling is the task of predicting or generating text units from surrounding or preceding textual context.
- How it is used
- Frames language modeling as the raw objective underlying text generation and as a basis for comparing model behavior.
- How deep it goes
- Defines context-based prediction and distinguishes three objectives, but does not fully elaborate autoencoding in the supplied treatment.
- Examples
- Uses Mistral, ChatGPT, machine translation, summarization, code generation, and context-prediction illustrations.
- What you can do after
- Evaluate language-model outputs by distinguishing raw continuation behavior from conversational behavior shaped by fine-tuning.
Read how The Art of AI Product Development explains language modeling →
Machines that Think
Language modeling is the computational process of learning or applying statistical structure in language to assess or generate sequences of text.
- How it is used
- Introduces probabilistic sequence prediction as a foundation for representing language and understanding generative model reliability.
- How deep it goes
- Explains conditional frequency estimates and limited-context assumptions, then discusses evaluation and reliability constraints.
- Examples
- Uses “the sun rises in the east” and the frequency of “in the east” to illustrate sequence and continuation probabilities.
- What you can do after
- Evaluate and understand language-model predictions without treating fluency as proof of factual accuracy.
Deep Learning with Python, Third Edition
A next-token prediction training setup in which the model learns to predict the next token from previous tokens.
- How it is used
- Uses next-token prediction to train sequence models and generate open-ended text from prompts.
- How deep it goes
- Explains token-level training, recurrent state, causal leakage, and token-by-token inference, but does not derive the optimization mathematics.
- Examples
- Demonstrates character-level Shakespeare generation using a prompt, vocabulary mappings, recurrent state, greedy selection, and detokenization.
- What you can do after
- Build it from scratch with a recurrent model, then implement prompt conditioning and autoregressive token generation.
Read how Deep Learning with Python, Third Edition explains language modeling →
Deep Learning with R, Third Edition
Language modeling is the task or technique of learning a probability distribution over sequences of linguistic symbols.
- How it is used
- Frames incomplete-sentence prediction as a practical NLP task and supports sequential text generation.
- How deep it goes
- Defines next-token prediction, probability distributions, and sampling strategies while treating architecture history comparatively.
- Examples
- Uses incomplete-sentence next-word prediction and greedy, random, and top-k text generation.
- What you can do after
- Apply language models to predict tokens and generate text using greedy, random, or top-k sampling.
Read how Deep Learning with R, Third Edition explains language modeling →
Transfer Learning for Natural Language Processing
Language modeling is the task of modeling probability distributions over sequences of language.
- How it is used
- Models probabilities over word sequences to support next-word prediction and language generation.
- How deep it goes
- Defines sequence probabilities and highlights next-word prediction without detailing algorithms or implementation.
- Examples
- Uses predicting the next word or sentence as examples of language generation.
- What you can do after
- Understand how sequence probabilities support next-word or sentence prediction.
Read how Transfer Learning for Natural Language Processing explains language modeling →
Also covered in
- Understanding AIIn early access (MEAP)
- Hugging Face in Action
What to know first
- language modelA language model is a computational model that represents patterns in linguistic sequences to assign probabilities, predict, generate, or evaluate language.
- machine learningMachine learning is a field of computational methods in which systems learn patterns or relationships from data to make predictions, decisions, or other outputs.
- machine translationMachine translation is the automated computational conversion of text or speech from one natural language into another.
- attentionAttention is a neural-network mechanism that computes input-dependent relevance weights to combine information from representations for a context-dependent computation.
Related concepts
- Bayesian inferenceuses · Bayesian inference updates probability distributions over hypotheses or unknown quantities by combining prior information with observed evidence and a likelihood model.
- language modelpart of · A language model is a computational model that represents patterns in linguistic sequences to assign probabilities, predict, generate, or evaluate language.
- character-based modelingcontrasts with · Character-based modeling is language modeling in which text is represented and predicted as sequences of individual characters.
- Continuous Bag of Wordsincludes · Continuous Bag of Words is a Word2Vec training architecture that predicts a target word from its surrounding context words.
- convolutional neural networkused by · A convolutional neural network is a neural-network architecture that processes structured inputs with convolutional operations, typically using learned local filters to capture patterns.
Questions about language modeling
What is language modeling?
Language modeling is the task of learning or applying probability distributions over language sequences to estimate or predict units from context.
Which Manning books explain language modeling?
Fight Fraud with Machine Learning, LLMs in Production, The Art of AI Product Development, Machines that Think, Deep Learning with Python, Third Edition, Deep Learning with R, Third Edition and Transfer Learning for Natural Language Processing.
What should I know before learning language modeling?
Manning books that teach language modeling usually introduce language model, machine learning, machine translation and attention first.
What is the difference between language modeling and character-based modeling?
Language modeling is the task of learning or applying probability distributions over language sequences to estimate or predict units from context. Character-based modeling is language modeling in which text is represented and predicted as sequences of individual characters.
What kinds of language modeling are there?
Kinds of language modeling include Continuous Bag of Words.
What is language modeling part of?
Language modeling is part of language model. A language model is a computational model that represents patterns in linguistic sequences to assign probabilities, predict, generate, or evaluate language.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.






