The original Transformer architecture adds sinusoidal positional encodings to the input token embeddings before the first attention layer. Why is this necessary given how self-attention itself processes a sequence?
- Because the feed-forward sublayers in each block are recurrent and require explicit position indices to update their hidden state between positions
- Because self-attention computes a weighted sum over all positions regardless of order, so without added positional information the model would treat the input as an unordered set of tokens
- Because the token embedding layer cannot represent more than a fixed vocabulary size unless positional offsets are added to each embedding
- Because positional encodings replace the residual connections that would otherwise be required around each sublayer
Why B? And why not the others?
Correct answer: B. Because self-attention computes a weighted sum over all positions regardless of order, so without added positional information the model would treat the input as an unordered set of tokens
Self-attention computes, for each position, a weighted sum over the value vectors at every position in the sequence, and that computation is inherently permutation-invariant: reshuffling the input tokens and their positions would reshuffle the output in exactly the same way, with no built-in sense of order. Because the architecture contains no recurrence or convolution to encode sequence order implicitly, Vaswani et al. add sinusoidal positional encodings to the embeddings so the model can distinguish 'token A followed by token B' from 'token B followed by token A.' The option about recurrent feed-forward sublayers is wrong because the position-wise feed-forward network is a simple non-recurrent transformation applied independently to each position. The option about vocabulary size is wrong because vocabulary capacity is determined by the embedding table's size, unrelated to position information. The option about replacing residual connections is wrong because residual connections around each sublayer serve a separate purpose (easing gradient flow) and remain present alongside positional encoding, not instead of it.
Source: Vaswani et al., "Attention Is All You Need" (2017), arXiv:1706.03762, Section 3.5 (Positional Encoding)