A context window is the finite amount of information a model can use when producing an output. A transformer can draw on relationships among tokens throughout this available context, but information outside it is unavailable. Even within the window, including a relevant detail does not guarantee the model will use it effectively. section 12.4.2 section 12.3.2
A larger window can include more material, but it has computational costs. In standard self-attention, the number of pairwise relationships grows with the square of sequence length: doubling the tokens can quadruple attention calculations. Optimized and sparse attention methods can reduce this demand. section 12.4.2
The window is one limit among several. Each token has a finite-dimensional representation, and attention forms weighted summaries of context. Transformers make distant relationships more directly accessible than recurrent networks, but they do not remove compression or memory limits. section 12.3.2 section 12.4.2





































