iRoPE, or interleaved RoPE, extends transformer context windows by alternating positional and non-positional layers. It addresses the difficulty of preserving both local detail and global access across extremely long sequences. section 9.0 section 9.5.2
The architecture combines two complementary layer types:
- RoPE layers enforce positional structure and capture short-range dependencies, helping the model track details within nearby text. section 9.5.2
- NoPE layers omit positional encoding, giving attention an unbiased view across the full sequence and supporting global connectivity. section 9.5.2
- A repeated three-to-one RoPE-to-NoPE pattern combines fine-grained local awareness with unrestricted long-range access. section 9.5.2
Long-document retrieval
A document spanning millions of tokens can require both sentence-level tracking and retrieval of a distant fact. iRoPE is designed to preserve local sentence details while maintaining access to overarching themes and remote information. section 9.5.2
Figure 9.11 Needle in a haystack evaluation of Llama 4. Maverick supports text contexts up to 1 million tokens, while Scout extends text contexts to 10 million tokens. Filled cells indicate successful retrieval across different context lengths and depths, while the few white cells mark retrieval failures. The overall pattern shows how iRoPE enables robust long-context performance, maintaining accuracy even at extreme scales.
The needle-in-a-haystack evaluation illustrates long-context retrieval: Llama 4 Maverick reaches 1 million tokens, while Scout reaches 10 million tokens, with mostly successful retrieval across lengths and depths. section 9.5.2
