Press, Smith and Lewis (2021), "Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation," introduce ALiBi as a way to let a model trained on short sequences generalize to much longer ones at inference. How does ALiBi represent position, and how does this differ from adding sinusoidal or learned positional embeddings to the input?
- It adds a learned positional embedding to each token exactly as the original Transformer does, but recomputes those embeddings after every training step so they extrapolate to unseen lengths
- It removes positional information entirely, relying only on the order in which tokens are fed into the model during training to implicitly teach it sequence order
- It replaces the query and key projections with position-specific weight matrices, so each position in the sequence uses a completely separate set of learned parameters
- It adds no positional embeddings to the word embeddings at all; instead, it subtracts a static, non-learned penalty from each attention score that grows in proportion to the distance between the query and key positions, with the penalty's steepness set differently per attention head
Why D? And why not the others?
Correct answer: D. It adds no positional embeddings to the word embeddings at all; instead, it subtracts a static, non-learned penalty from each attention score that grows in proportion to the distance between the query and key positions, with the penalty's steepness set differently per attention head
Press, Smith and Lewis describe ALiBi as adding no positional embeddings to the word embeddings whatsoever; instead, after computing the raw query-key dot product, a static bias is subtracted that is proportional to the distance between the two positions, with each attention head using its own fixed slope for that penalty, so nearby tokens are penalized less than distant ones and the mechanism naturally extends to sequence lengths never seen during training. The option describing recomputed learned positional embeddings is wrong because ALiBi's bias is a fixed, non-learned function of distance, not a learned embedding table at all. The option describing reliance purely on feed order with no explicit signal is wrong because ALiBi does inject an explicit, though non-learned, positional signal directly into the attention score computation. The option describing position-specific query/key weight matrices is wrong because ALiBi keeps the same query and key projections at every position and instead modifies only the attention scores after the dot product, adding no new per-position parameters.
Source: Press, Smith & Lewis, "Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation" (2021), arXiv:2108.12409