liveBook · Manning

Windowizer

As explained in Natural Language Processing in Action, Second Edition

A windowizer partitions a token sequence into overlapping local context windows.

A windowizer partitions each tokenized sequence into overlapping local contexts, then converts nearby tokens into center-context pairs for skip-gram training. section 6.3.3

WikiText2 sliding windows

With WINDOW_WIDTH = 10, the function visits each token, collects valid positions from ten places before or after it, excludes the token itself, and emits integer token-ID pairs.

How Natural Language Processing in Action, Second Edition covers it

How deep it goes
Implements sliding index ranges, boundary checks, and center-token exclusion, then defers model training to the downstream pipeline.
Examples
Uses lowercase whitespace tokenization on WikiText2 and a configurable ten-token context width.
What it assumes
Assumes tokenized rows and token-to-ID mappings prepared before window construction.
What you can do after
Build it from scratch by mapping tokenized rows into center-context integer pairs.
How it connects
Links tokenization, vocabulary indexing, skip-gram pairs, DataLoader, and Word2Vec.
Cover of Natural Language Processing in Action, Second Edition
liveBookmanning.com
Read Natural Language Processing in Action, Second Edition on liveBook Get the book at manning.com

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Natural Language Processing in Action, Second Edition. The section links point to the parts of the book it draws on.