liveBook · Manning

Autoregressive dataset

As explained in Inside Deep Learning: Math, Algorithms, Models

A dataset construction for next-character prediction in which each input chunk is paired with the same chunk shifted by one position as labels.

Constructs next-character training pairs from one long sequence. For a chunk x = text[start:start+chunk_size], the label sequence y is shifted one character: text[start+1:start+1+chunk_size]. Each position therefore teaches the model to predict the following character. section 7.5.1

A practical construction follows these steps:

  • Represent the corpus as one long string and map each character to a vocabulary index. section 7.5.1

Character-level Shakespeare data

A lowercased corpus of about 100,000 Shakespeare characters can use chunks of 250 characters. The input contains positions start through start+249; the labels contain the immediately following 250 positions. Fixed lengths avoid padded inputs and simplify GPU batching. section 7.5.1

How Inside Deep Learning: Math, Algorithms, Models covers it

How deep it goes
Defines chunk slicing and indexing in PyTorch, but defers model internals and generation details.
Examples
Uses lowercased Shakespeare text, fixed character chunks, shifted labels, and shuffled PyTorch batches.
What it assumes
Assumes a long source sequence and vocabulary mapping while teaching the one-position label shift from zero.
What you can do after
Apply it via PyTorch Dataset and DataLoader to produce aligned next-token inputs and labels.
How it connects
Connects character-level language modeling with self-supervision, recurrent neural networks, DataLoader, and cross-entropy loss.
Cover of Inside Deep Learning: Math, Algorithms, Models
liveBookmanning.com
Read Inside Deep Learning: Math, Algorithms, Models on liveBook Get the book at manning.com

This explainer was generated by Manning's knowledge graph from the text of Inside Deep Learning: Math, Algorithms, Models. The section links point to the parts of the book it draws on.