Su et al. (2021), "RoFormer: Enhanced Transformer with Rotary Position Embedding," introduce rotary position embedding (RoPE) as an alternative to adding sinusoidal positional vectors to token embeddings. How does RoPE encode positional information?
- It concatenates a learned position index as an extra dimension appended to every query and key vector before the dot product is computed
- It rotates each query and key vector by an angle that depends on the token's position and a per-dimension frequency, so that the dot product between a rotated query and key naturally depends on their relative distance rather than their absolute positions alone
- It replaces the dot-product attention score with a lookup table indexed by the absolute position of the query, discarding the key vector's position entirely
- It multiplies the attention output by a fixed sinusoidal mask after the softmax has already been applied, without altering the query or key vectors themselves
Why B? And why not the others?
Correct answer: B. It rotates each query and key vector by an angle that depends on the token's position and a per-dimension frequency, so that the dot product between a rotated query and key naturally depends on their relative distance rather than their absolute positions alone
Su et al. encode position by treating pairs of dimensions within each query and key vector as coordinates and rotating them by an angle that is a function of the token's absolute position and a frequency that varies by dimension pair, before the query and key are used to compute attention scores. Because rotating both a query and a key vector by an amount tied to their own positions leaves the angle between them dependent only on the difference between those positions, the resulting dot product ends up encoding relative distance even though each vector was rotated using only its own absolute position, with the paper additionally noting that this relative dependency decays as tokens grow further apart. The option describing an appended learned position index is wrong because RoPE does not add any extra dimension to the vectors; it transforms the existing dimensions through rotation. The option describing a lookup table indexed only by the query's absolute position is wrong because RoPE's dot product depends on both vectors' positions through the same rotation mechanism, not a table keyed to one side only. The option describing a post-softmax sinusoidal mask is wrong because the rotation is applied directly to the query and key vectors before the dot product and softmax are computed, not to the attention output afterward.
Source: Su et al., "RoFormer: Enhanced Transformer with Rotary Position Embedding" (2021), arXiv:2104.09864