liveBook · Manning

Training dataset

As explained in Build an Advanced RAG Application (From Scratch)

Training data is the collection of examples from which a machine-learning model learns its parameters and patterns.

A training dataset is the collection of examples from which a machine-learning model learns parameters and patterns. For language models, it supplies sequences used to predict the next word and develop language understanding. section 1.1 section 1.1.2

During training, the model compares predictions with outcomes in the dataset and repeatedly adjusts weights and biases to reduce the difference. Larger, more diverse datasets expose it to broader language patterns, topics, styles, contexts, and ambiguities. section 1.1.3 section 1.1.2

A substantial and diverse training dataset helps a language model:

  • Learn general patterns that apply across contexts. section 1.1.3
  • Capture semantic nuances, contextual cues, and dependencies. section 1.1.3
  • Handle ambiguous expressions by learning context-dependent meanings. section 1.1.3
  • Generalize to unseen examples rather than overfit the training data. section 1.1.3

Common Crawl

Common Crawl illustrates the scale used for large-language-model training: its archive contains petabytes of web data collected since 2008, with crawls generally completed each month. ChatGPT was largely trained on this text corpus. section 1.1.3

How Build an Advanced RAG Application (From Scratch) covers it

How deep it goes
Explains dataset scale, diversity, generalization, and iterative weight-and-bias adjustment, without detailing data preparation.
Examples
Uses Common Crawl and diverse human-language examples to illustrate training-data scale and coverage.
What it assumes
Assumes basic familiarity with language models while grounding training in examples, predictions, weights, and biases.
What you can do after
Understand how dataset scale and diversity influence language-model prediction, robustness, ambiguity handling, and generalization.
How it connects
Connects training datasets with large language models, next-word prediction, weights, biases, semantic understanding, contextual awareness, and overfitting.

Also explained in

Compare how 50 books explain training dataset →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build an Advanced RAG Application (From Scratch). The section links point to the parts of the book it draws on.