liveBook · Manning

Context window

As explained in Timeless Algorithms, The Foundational Ideas

The finite amount of input context a model can consider when producing an output.

A context window is the finite amount of information a model can use when producing an output. A transformer can draw on relationships among tokens throughout this available context, but information outside it is unavailable. Even within the window, including a relevant detail does not guarantee the model will use it effectively. section 12.4.2 section 12.3.2

A larger window can include more material, but it has computational costs. In standard self-attention, the number of pairwise relationships grows with the square of sequence length: doubling the tokens can quadruple attention calculations. Optimized and sparse attention methods can reduce this demand. section 12.4.2

The window is one limit among several. Each token has a finite-dimensional representation, and attention forms weighted summaries of context. Transformers make distant relationships more directly accessible than recurrent networks, but they do not remove compression or memory limits. section 12.3.2 section 12.4.2

How Timeless Algorithms, The Foundational Ideas covers it

How deep it goes
Defines the processing limit and explains attention-cost and reliability implications, without specifying window sizes or selection strategies.
Examples
Illustrates attention cost with sequence length: doubling the tokens can quadruple standard self-attention calculations.
What you can do after
Evaluate how context limits affect available information, reliability, and computational demand.
How it connects
Connects context windows to transformers, self-attention, token representations, and recurrent neural networks.

Also explained in

Compare how 38 books explain context window →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Timeless Algorithms, The Foundational Ideas. The section links point to the parts of the book it draws on.