Positional encoding and RoPE: how a transformer knows word order

A transformer turns every word (strictly, every token) into a vector, its embedding, and then lets every word look at every other through attention: each word scores how relevant every other word is, by comparing vectors. But a comparison of two vectors doesn’t care where in the sentence they came from. Shuffle the words and each one still gets the same scores, just in a different order. To attention alone, “dog bites man” and “man bites dog” are the same bag of words. Position has to be added in.

The first fix: add a position signal

The original transformer paper, Attention Is All You Need (2017), added a positional encoding to each word’s vector before anything else happens: a fixed pattern of sine and cosine waves at different frequencies, different for every position. Fast waves change from one word to the next, slow ones only over long stretches, so together they give every position a unique fingerprint, a bit like a clock with many hands. Models like BERT and GPT-2 instead learned one vector per position during training.

Both mark a word’s absolute position: “word number 7”. But what usually matters is relative position: an adjective two words before its noun means the same thing at the start of a document or in the middle of it. With absolute positions the model has to learn that, separately, for every pair of positions. And it has never seen positions beyond its training length, so it tends to break down there.

RoPE: rotate instead of add

Rotary position embedding (RoPE, from the RoFormer paper, 2021) puts position in exactly where attention compares words. Attention compares a word’s query vector with other words’ key vectors through a dot product. RoPE splits those vectors into pairs of numbers, treats each pair as an arrow in a plane, and rotates it by an angle proportional to the word’s position: position m turns it by m × θ.

The trick is what happens to the comparison. The dot product of two arrows depends only on their lengths and the angle between them. Rotate the query by its position and the key by its position, and the angle between them changes by the difference of the positions. So the score depends on how far apart the two words are, not on where they sit:

Like the original sine waves, every pair rotates at a different speed: θ is large for the first pairs and shrinks geometrically for later ones. Fast pairs separate neighbouring words sharply; slow pairs barely move between neighbours but still tell positions thousands of words apart from each other.

Why it won

• Relative position for free. It falls out of the maths of the dot product; the model doesn’t have to learn it.
• No extra parameters, and nothing is added to the word’s meaning itself: rotation keeps a vector’s length, it only turns it.
• Stretchable. Because position is just an angle, there are tricks to read past the training length, by slowing down the rotations so longer texts fit into the range of angles the model already knows.

RoPE is what the Llama and Mistral families use, and so does Solar 10.7B, the model behind my thesis agent Lumen.