liveBook · Manning

Dataset

As explained in LLMs in Production

A dataset is an organized collection of data used for analysis, evaluation, or machine-learning development.

A chatbot cannot learn an organization’s answers from an empty folder, and a model cannot be judged without questions to answer. A dataset is an organized collection of data prepared for a purpose. For LLMs, training data helps shape model behavior; an evaluation dataset supplies tasks whose results can be compared with ground truth using a metric. section 4.0 section 4.2

Choose and prepare the dataset for the job it must do:

  • For training or finetuning, gather relevant data, clean it, and tokenize it. Quality and variety matter; a larger collection is not automatically better. section 5.2.1 section 4.3
  • For evaluation, use tasks that resemble the application. Multiple-choice benchmarks are easy to score but may poorly represent users’ freeform questions. section 4.2 section 12.2.4
  • For retrieval-augmented generation, prepare data the system can retrieve as context. A coding assistant, for example, needs relevant code data if it is to respond in the context of an organization’s codebase. section 10.2 section 10.5
  • Check what the collection leaves in or out. Unfiltered text can carry harmful biases, while underrepresented languages can limit a model’s usefulness. section 3.2.8 section 12.2.4

From messages to a usable dataset

For an HR-help chatbot, Slack messages offer a possible source of recurring questions. A Python example pulls messages through slack_sdk, keeps the selected user’s data, organizes timestamps, users, and text with pandas, and saves the result as a Parquet file. Collecting messages is only the start: the resulting data still needs to fit the intended task. section 4.5 section 4.6

Listing 9.2 Loading and preparing the data
dataset = load_dataset(      #1
    "text",
    data_files={
        "train": ["../../data/TinyStoriesv1andv2-train.txt"],
        "val": ["../../data/TinyStoriesv1andv2-valid.txt"],
    },
    streaming=True,
)

The TinyStories loading example names separate training and validation files and enables streaming, so the entire dataset need not be loaded into memory at once. section 9.1.2

After loading, even a small demonstration pipeline filters short records, tokenizes text, shuffles data, and prepares batches. Keep training and validation splits from leaking into each other. Streaming saves memory but sacrifices some speed; careful collection, evaluation, and cleaning are still the work that makes the dataset useful. section 9.1.2

How LLMs in Production covers it

How deep it goes
Works through extraction, streaming, filtering, tokenization, and splitting, while shortcutting extensive dataset gathering and cleaning in the training example.
Examples
Uses Slack messages for a chatbot, TinyStories for streamed training data, and a Python coding dataset for retrieval.
What it assumes
Uses Python examples and defines evaluation datasets as task lists, without requiring a formal account of evaluation beforehand.
What you can do after
Apply it via Python and Hugging Face datasets to extract or stream data and prepare it for training and validation.
How it connects
Links datasets to foundation models, tokenizers, evaluation metrics, train–validation splits, and retrieval-augmented generation.
Cover of LLMs in Production
liveBookmanning.com
Read LLMs in Production on liveBook Get the book at manning.com

Also explained in

Compare how 24 books explain dataset →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLMs in Production. The section links point to the parts of the book it draws on.