liveBook · Manning

TinyStories

As explained in LLMs in Production

TinyStories is a small text dataset of short synthetic stories intended for training and experimenting with language models.

TinyStories is a compact dataset of short, single-paragraph stories for toy language-model projects. Its more than 30 million rows remain substantial on consumer hardware, making streaming preferable to loading everything into memory. section 9.1.2

A practical TinyStories preparation workflow:

  • Load training and validation files with streaming, perform minimal cleaning, tokenize prompts and stories, store attention masks, shuffle splits, and create iterators. section 9.1.2
  • Sample context-window batches for next-token training, then evaluate separately on training and validation data. section 9.1.2
Listing 9.2 Loading and preparing the data
dataset = load_dataset(      #1
    "text",
    data_files={
        "train": ["../../data/TinyStoriesv1andv2-train.txt"],
        "val": ["../../data/TinyStoriesv1andv2-valid.txt"],
    },
    streaming=True,
)

Shows the minimal streaming setup for separate TinyStories training and validation files. section 9.1.2

How LLMs in Production covers it

How deep it goes
Treats loading, cleaning, tokenization, batching, evaluation, and generation procedurally, while shortening dataset gathering and quality evaluation.
Examples
Uses TinyStories training and validation text files, a short-story prompt, tokenized columns, sampled context windows, and generated stories.
What you can do after
Apply via streaming dataset loaders and PyTorch-style iterators to prepare data for language-model training and evaluation.
How it connects
Connects streaming, tokenization, attention masks, batching, train-validation splits, next-token prediction, and text generation.
Cover of LLMs in Production
liveBookmanning.com
Read LLMs in Production on liveBook Get the book at manning.com

This explainer was generated by Manning's knowledge graph from the text of LLMs in Production. The section links point to the parts of the book it draws on.