liveBook · Manning

Wikitext

As explained in Natural Language Processing in Action, Second Edition

WikiText2 is a corpus of Wikipedia article text distributed as a benchmark dataset for language modeling.

WikiText2 is a tokenized corpus of Wikipedia article text used to train and benchmark word-level language models and word embeddings. Its text preserves headings, paragraph boundaries, and special tokens, making preprocessing visible rather than hidden. section 6.3.3 section 8.5

A practical WikiText2 workflow typically includes:

  • Batchify long sequences for efficient RNN, LSTM, or GRU training. section 8.4.1

Inspecting masked tokens

The PyTorch version replaces about 2.7% of tokens with <unk>. This supports prediction experiments but can limit accuracy and confuse a language model, so raw text can provide a cleaner comparison. section 8.5

How Natural Language Processing in Action, Second Edition covers it

How deep it goes
Covers loading, tokenization, vocabulary IDs, batching, masking, and model training, but defers corpus collection and original annotation.
Examples
Uses Wikipedia paragraphs, headings, <eos> and <unk> tokens, skip-gram windows, batches, and generated text.
What it assumes
Assumes Python, PyTorch, tokenization, and vocabulary-building skills while demonstrating dataset preparation from text.
What you can do after
Train word embeddings or recurrent language models and inspect how tokenization and masking affect predictions.
How it connects
Connects tokenization, vocabulary building, word embeddings, Word2Vec, RNNs, LSTMs, GRUs, and language modeling.
Cover of Natural Language Processing in Action, Second Edition
liveBookmanning.com
Read Natural Language Processing in Action, Second Edition on liveBook Get the book at manning.com

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Natural Language Processing in Action, Second Edition. The section links point to the parts of the book it draws on.