How deeply does Deep Learning with R, Third Edition cover language modeling?

Defines next-token prediction, probability distributions, and sampling strategies while treating architecture history comparatively.

Language modeling

As explained in Deep Learning with R, Third Edition

Language modeling is the task or technique of learning a probability distribution over sequences of linguistic symbols.

Language modeling learns a probability distribution over linguistic symbol sequences. Its core task is predicting the next token from the tokens that came before it. section 16.6 section 14.1

Predicting the next word

Given an incomplete sentence, a language model estimates which word is most likely to come next. Repeating this prediction produces a longer string of text. section 14.1 section 16.6

Once a probability distribution is available, generation can choose tokens in several ways:

Greedy search selects the most likely token at every step. section 16.6
Random sampling draws directly from the predicted categorical distribution. section 16.6
Top-k sampling limits choices to the k most likely candidates. section 16.6

Language models capture statistical regularities rather than human-like understanding. LSTMs improved text modeling but struggled with long dependencies; Transformers addressed major scalability and parallelization problems. section 14.1 section 14.1

How Deep Learning with R, Third Edition covers it

Cover of Deep Learning with R, Third Edition

liveBook manning.com

Read Deep Learning with R, Third Edition on liveBook Get the book at manning.com

Also explained in

liveBook manning.com

LLMs in ProductionLanguage modeling is the task of learning the statistical structure of language so a system can predict or generate text.
liveBook manning.com

The Art of AI Product DevelopmentLanguage modeling is the task of predicting or generating text units from surrounding or preceding textual context.
liveBook manning.com

Deep Learning with Python, Third EditionA next-token prediction training setup in which the model learns to predict the next token from previous tokens.

Compare how 4 books explain language modeling →

Related concepts

bayesian statisticsuses · Bayesian statistics is a framework for statistical inference that combines prior probability distributions with observed-data likelihoods to update probabilities over hypotheses or unknown quantities.
language modelpart of · A language model is a computational model that represents patterns in linguistic tokens to assign probabilities to, predict, or generate sequences of language.
character-based modelingcontrasts with · Character-based modeling is language modeling in which text is represented and predicted as sequences of individual characters.
Continuous Bag of Wordsincludes · Continuous Bag of Words is a word-embedding training model that predicts a target word from surrounding context words.
convolutional neural networkused by · A convolutional neural network is a neural network that applies learned convolutional operations to structured data to detect local patterns and build representations.
LSTMused by · An LSTM is a recurrent neural network architecture with gated memory mechanisms that control information flow and help retain or update information across sequences.

This explainer was generated by Manning's knowledge graph from the text of Deep Learning with R, Third Edition. The section links point to the parts of the book it draws on.