Rotary Position Embedding (RoPE) encodes position by rotating the query and key vectors used in attention instead of adding a position vector to each token embedding. Because the rotation angle grows with token position, the attention dot product reflects relative distance: nearby tokens have smaller angular differences, while distant tokens have larger ones. This represents both sequence order and relational structure, supporting context lengths beyond those seen during training. section 9.5.1
Practical characteristics of RoPE include:
- Contrasts with fixed sinusoidal embeddings and learned position vectors, which can struggle to generalize from shorter training sequences to much longer inputs. section 9.5.1
- Scales a model’s effective context by rescaling RoPE frequencies; a linear factor of 4 extends a 32,000-token context to 128,000 tokens. section 9.5.1
- Factorizes multimodal positions into temporal, height, and width components, using time-aligned rotary embeddings for video and synchronized audio grids. section 6.3.3
Scaling RoPE during inference
Inference libraries expose RoPE scaling at model load time. For example: model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-8B", trust_remote_code=True, rope_scaling={"type": "linear", "factor": 4.0}). The model can then generate from a longer-context input. section 9.5.1
Figure 9.9 RoPE encodes position by rotating the Q and K vectors. Left: two token positions are mapped to angles on a circle; the relative angle ΔΘ grows with position difference. Right: the attention dot product varies with relative distance as cos(ΔΘ), which enables robust length extrapolation.
Visualizes RoPE as angular positions on a circle and relates the attention dot product to the cosine of the relative angle. section 9.5.1
