Language modeling
Language modeling is the task of learning or applying statistical structure or probabilities over language sequences to predict or generate language units from context.
LLMs in Production
Language modeling is the task of learning the statistical structure of language so a system can predict or generate text.
- How it is used
- Frames language modeling as the foundation for understanding LLM behavior, architecture choices, and production outcomes.
- How deep it goes
- Traces modeling from statistical methods through neural networks and transformers, while stopping short of formal derivations.
- Examples
- Uses abstraction layers, historical model families, and IBM Watson’s commercialization failure to illustrate capabilities and limits.
- What you can do after
- Evaluate language-modeling approaches and anticipate how linguistic abstractions, architecture, and production constraints affect results.
The Art of AI Product Development
Language modeling is the task of predicting or generating text units from surrounding or preceding textual context.
- How it is used
- Frames language modeling as the raw objective underlying text generation and as a basis for comparing model behavior.
- How deep it goes
- Defines context-based prediction and distinguishes three objectives, but does not fully elaborate autoencoding in the supplied treatment.
- Examples
- Uses Mistral, ChatGPT, machine translation, summarization, code generation, and context-prediction illustrations.
- What you can do after
- Evaluate language-model outputs by distinguishing raw continuation behavior from conversational behavior shaped by fine-tuning.
Read how The Art of AI Product Development explains language modeling →
Deep Learning with Python, Third Edition
A next-token prediction training setup in which the model learns to predict the next token from previous tokens.
- How it is used
- Uses next-token prediction to train sequence models and generate open-ended text from prompts.
- How deep it goes
- Explains token-level training, recurrent state, causal leakage, and token-by-token inference, but does not derive the optimization mathematics.
- Examples
- Demonstrates character-level Shakespeare generation using a prompt, vocabulary mappings, recurrent state, greedy selection, and detokenization.
- What you can do after
- Build it from scratch with a recurrent model, then implement prompt conditioning and autoregressive token generation.
Read how Deep Learning with Python, Third Edition explains language modeling →
Deep Learning with R, Third Edition
Language modeling is the task or technique of learning a probability distribution over sequences of linguistic symbols.
- How it is used
- Frames incomplete-sentence prediction as a practical NLP task and supports sequential text generation.
- How deep it goes
- Defines next-token prediction, probability distributions, and sampling strategies while treating architecture history comparatively.
- Examples
- Uses incomplete-sentence next-word prediction and greedy, random, and top-k text generation.
- What you can do after
- Apply language models to predict tokens and generate text using greedy, random, or top-k sampling.
Read how Deep Learning with R, Third Edition explains language modeling →
Related concepts
- bayesian statisticsuses · Bayesian statistics is a framework for statistical inference that combines prior probability distributions with observed-data likelihoods to update probabilities over hypotheses or unknown quantities.
- language modelpart of · A language model is a computational model that represents patterns in linguistic tokens to estimate sequence probabilities and predict or generate language from context.
- convolutional neural networkused by · A convolutional neural network is a neural-network architecture that applies learned convolutional operations to extract patterns from structured data.
- LSTMused by · An LSTM is a recurrent neural network architecture with gated memory mechanisms that control information flow and help retain or update information across sequences.
- character-based modelingcontrasts with · Character-based modeling is language modeling in which text is represented and predicted as sequences of individual characters.
- perplexitymeasured by · Perplexity is a language-model metric, typically the exponential of average cross-entropy, that measures how well a model predicts a sequence, with lower values indicating better predictive fit.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.