Malhar Ujawane

Blog

Explainer·(updated October 31, 2026)

What is RoPE? Rotary position embeddings, explained

RoPE (rotary position embeddings) encodes token position by rotating query and key vectors, so attention sees relative distance. How it works, in Python.

What is RoPE? Rotary position embeddings, explained

RoPE (rotary position embedding) is the way most modern LLMs tell attention where each token sits: it rotates every query and key vector by an angle proportional to the token’s position. Because rotations compose, the dot product between a query at position m and a key at position n ends up depending only on the distance m minus n, so the model gets relative position for free without adding anything to the token embeddings.

I ran the code below on 2026-10-31 with Python 3.13.11 and no packages.

Why do transformers need position information at all?

Attention on its own is order-blind. It compares every query with every key by dot product, and a dot product doesn’t care where the two vectors came from. Shuffle the input tokens and, without position information, each token gets the same attention scores as before, just in a different order. “Dog bites man” and “man bites dog” would look alike.

The original transformer fixed this by adding a fixed sinusoidal vector to each token embedding before the first layer. Learned absolute position embeddings (one trained vector per position) came next. Both encode absolute position, and both have to be carried through the network for the model to work out how far apart two tokens are.

RoFormer (Su et al., April 2021) introduced RoPE as a different approach. The abstract puts it in one sentence: RoPE “encodes the absolute position with a rotation matrix and meanwhile incorporates the explicit relative position dependency in self-attention formulation.”

How does RoPE work?

Take a query or key vector with d dimensions and split it into d/2 pairs. Treat each pair as a point in a 2D plane. For a token at position pos, rotate pair i by the angle pos × θ_i, where θ_i = 10000^(-2i/d). That’s the frequency schedule in the RoFormer paper, the same base the original sinusoidal encoding used.

Three things follow from that.

  1. Rotation is applied to queries and keys inside attention, not to the embedding at the bottom of the network. Every layer gets position information where it uses it.
  2. The score depends on distance. Rotating q by angle a and k by angle b and taking their dot product gives the same result as rotating q by a minus b and leaving k alone. So the attention score between positions m and n depends on m minus n.
  3. Length doesn’t change. A rotation only changes direction, so RoPE doesn’t distort the size of the vectors.

The pairs rotate at very different speeds. Pair 0 turns fast and completes a turn every few tokens, which captures fine local order. The last pairs turn so slowly that they barely move across thousands of tokens, which carries coarse long-range position.

What does RoPE look like in code?

Here is RoPE from scratch, with checks for all three claims.

# rope.py: rotary position embeddings from scratch.
# Python 3 standard library only.
import math, random

def rope(x, pos, base=10_000.0):
    """Rotate each pair (x[2i], x[2i+1]) by angle pos * base**(-2i/d)."""
    d, out = len(x), []
    for i in range(d // 2):
        theta = pos * base ** (-2 * i / d)
        a, b = x[2 * i], x[2 * i + 1]
        out += [a * math.cos(theta) - b * math.sin(theta),
                a * math.sin(theta) + b * math.cos(theta)]
    return out

dot = lambda u, v: sum(a * b for a, b in zip(u, v))
norm = lambda u: math.sqrt(dot(u, u))

random.seed(0)
d = 64
q = [random.gauss(0, 1) for _ in range(d)]
k = [random.gauss(0, 1) for _ in range(d)]

# 1. The score depends only on the distance between positions, not on where they are.
for m, n in [(5, 2), (105, 102), (5_005, 5_002), (9, 2)]:
    print(f"q at {m:>5}, k at {n:>5} (distance {m - n}): score {dot(rope(q, m), rope(k, n)):+.6f}")

# 2. Rotation changes direction, never length.
print(f"\nlength of q before {norm(q):.6f}, after rope at pos 777: {norm(rope(q, 777)):.6f}")

# 3. Each pair spins at its own speed. Wavelength = tokens for one full turn.
print("\nwavelength in tokens of pair i (d=128)")
for base in (10_000, 500_000):
    w = [2 * math.pi / base ** (-2 * i / 128) for i in (0, 16, 32, 48, 63)]
    print(f"  base {base:>7,}: " + "  ".join(f"{x:,.0f}" for x in w))
q at     5, k at     2 (distance 3): score -1.106825
q at   105, k at   102 (distance 3): score -1.106825
q at  5005, k at  5002 (distance 3): score -1.106825
q at     9, k at     2 (distance 7): score -4.060352

length of q before 8.795825, after rope at pos 777: 8.795825

wavelength in tokens of pair i (d=128)
  base  10,000: 6  63  628  6,283  54,410
  base 500,000: 6  167  4,443  118,143  2,559,196

Same query, same key, distance 3: the score is identical at positions 5, 105 and 5,005. Change the distance to 7 and the score changes. The vector length is untouched after a rotation at position 777.

The last block is the part that matters for long context. With base 10,000 and head size 128, the slowest pair completes one turn in about 54,000 tokens. With base 500,000 the slow pairs stretch out into the millions, so far-apart positions still get distinguishable angles.

Why did RoPE become the default?

Relative position is what language mostly needs. Whether a verb is 3 tokens after its subject matters more than whether it’s token 4,012 of the document. RoPE gives attention that signal directly, adds no parameters, and costs a few multiplies per query and key.

It also works with the KV cache. A key is rotated once, by its own position, when it’s computed, and stored rotated. Later queries get rotated by their positions, and the dot product works out to relative distance without touching the cache. Schemes that add a learned bias per pair of positions don’t fit as neatly.

Adoption followed. The Llama 2 paper lists rotary embeddings among its architecture choices, Llama 3 kept them, and the DeepSeek-V3 report applies RoPE to a decoupled part of its queries and keys.

How does RoPE extend the context window?

This is where RoPE became a practical topic. A model trained on 4,096 tokens has never seen the rotation angles for position 20,000. Feed it longer inputs and attention scores go wrong, because the slow pairs reach angles they never saw in training.

Three approaches, all from primary papers:

  • Position Interpolation (Chen et al., June 2023) scales positions down to fit the trained range: position 8,000 in an 8k context gets treated like position 2,000 in a 2k one. The paper extended LLaMA models to 32,768 tokens “within 1000 steps” of fine-tuning and argues that extrapolating beyond the trained length “may lead to catastrophically high attention scores”.
  • YaRN (Peng et al., August 2023) scales different frequency pairs differently. It builds on “NTK-by-parts” interpolation: pairs whose wavelength is much shorter than the context aren’t interpolated at all, and slow pairs are. The paper reports needing “10x less tokens and 2.5x less training steps than previous methods”.
  • A bigger base. Meta’s Llama 3 paper raised the RoPE base frequency to 500,000 and cites Xiong et al. (2023) showing that value works for contexts up to 32,768 tokens. My script shows the effect: slow pairs rotate far less per token.

You’ll see these as fields in a model’s config.json on Hugging Face. Qwen2.5-7B-Instruct sets "rope_theta": 1000000.0, and DeepSeek-V3 has a rope_scaling block with "factor": 40 and YaRN’s beta_fast and beta_slow (both checked 2026-10-31). Transformers v5 groups these under rope_parameters. If you serve a model at a longer context than its default, these fields are what the serving engine reads.

The caveat from my Gemini 1.5 long context post still applies: a model that accepts a million tokens isn’t the same as a model that uses them well. Position encoding is one requirement for long context; training data with long documents is another. I’ll compare the trade-offs in RAG vs long context.

RoPE vs absolute position embeddings: what’s the difference?

Absolute (sinusoidal or learned) RoPE
Where it’s applied Added to token embeddings once Rotates q and k in every attention layer
What attention sees Absolute positions, relative must be learned Relative distance, directly in the score
Parameters None (sinusoidal) or one vector per position (learned) None
Longer than trained Learned: no vectors exist past the limit Possible with interpolation or a larger base

What I would do

Mostly, nothing: RoPE is a model-internal choice and you inherit it. Where it matters to an application builder is context length. Before I trust a model at a long context, I check its config for the RoPE base and any scaling block, and I check whether the long context came from training or from a scaling setting applied afterwards. Then I run my own needle-style test at the length I plan to use, since passing at 8k says little about 128k.

If I fine-tune a model, I keep the RoPE settings exactly as the base model shipped them. Changing the base or scaling factor during fine-tuning without matching training data is an easy way to quietly break a model.

FAQ

Is RoPE used in Llama?

Yes. The Llama 2 paper lists rotary positional embeddings (RoPE) as part of its architecture, and the Llama 3 paper raises the RoPE base frequency to 500,000 for longer contexts.

Does RoPE add parameters to the model?

No. The angles are fixed by position and the frequency schedule. RoPE adds a small amount of computation per query and key, but nothing is learned.

Why does RoPE use a base of 10,000?

RoFormer reused the base from the original transformer’s sinusoidal encoding, θ_i = 10000^(-2i/d), and shows it gives attention a decay with distance. Later models raised the base, Llama 3 to 500,000, to support longer contexts.

Is RoPE applied to the value vectors?

No. RoPE rotates queries and keys, because position only needs to affect the attention scores. Values are mixed using those scores and aren’t rotated.

Sources

Get new posts in your inbox

© 2026 Malhar Ujawane. Built with Astro.