Language modeling

As explained in The Art of AI Product Development

Language modeling is the task of predicting or generating text units from surrounding or preceding textual context.

Language modeling predicts missing or following words from textual context, then generates successive tokens into a complete text that is hopefully coherent and correct. Its raw objective does not necessarily produce a helpful conversational response. section 5.1.2

The objective appears in several forms:

• Sequence-to-sequence transduction translates one representation into another, supporting machine translation, summarization, and code generation. section 5.1.2
• Autoregression predicts the next token from preceding tokens, with unidirectionality limiting the available learning signal. section 5.1.2
• Autoencoding is named as another language-modeling objective, alongside sequence-to-sequence transduction and autoregression. section 5.1.2

Raw continuation versus conversation

Alex receives correct English from Mistral but not a helpful conversational turn, while ChatGPT responds conversationally and completes its sentences. Different training objectives explain the contrast. section 5.1.2

Figure 5.3 The objective of language modeling is to predict words based on their context.

Use the context illustration to see language modeling as predicting words from surrounding textual information. section 5.1.2

How The Art of AI Product Development covers it

How deep it goes

Defines context-based prediction and distinguishes three objectives, but does not fully elaborate autoencoding in the supplied treatment.

Examples

Uses Mistral, ChatGPT, machine translation, summarization, code generation, and context-prediction illustrations.

What you can do after

Evaluate language-model outputs by distinguishing raw continuation behavior from conversational behavior shaped by fine-tuning.

How it connects

Connects language modeling to autoregression, autoencoding, sequence-to-sequence transduction, transformers, and fine-tuning.

liveBookmanning.com

Read The Art of AI Product Development on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  LLMs in ProductionLanguage modeling is the task of learning the statistical structure of language so a system can predict or generate text.
• 

  liveBookmanning.com
  Deep Learning with Python, Third EditionA next-token prediction training setup in which the model learns to predict the next token from previous tokens.
• 

  liveBookmanning.com
  Deep Learning with R, Third EditionLanguage modeling is the task or technique of learning a probability distribution over sequences of linguistic symbols.

Compare how 4 books explain language modeling →

Related concepts

• bayesian statisticsuses · Bayesian statistics is a framework for statistical inference that combines prior probability distributions with observed-data likelihoods to update probabilities over hypotheses or unknown quantities.
• language modelpart of · A language model is a computational model that represents patterns in linguistic tokens to estimate sequence probabilities and predict or generate language from context.
• convolutional neural networkused by · A convolutional neural network is a neural-network architecture that applies learned convolutional operations to extract patterns from structured data.
• LSTMused by · An LSTM is a recurrent neural network architecture with gated memory mechanisms that control information flow and help retain or update information across sequences.
• character-based modelingcontrasts with · Character-based modeling is language modeling in which text is represented and predicted as sequences of individual characters.
• perplexitymeasured by · Perplexity is a language-model metric, typically the exponential of average cross-entropy, that measures how well a model predicts a sequence, with lower values indicating better predictive fit.

This explainer was generated by Manning's knowledge graph from the text of The Art of AI Product Development. The section links point to the parts of the book it draws on.