Extending a transformer-based language model's context window to accept much longer input sequences is architecturally expensive without further modifications to the standard attention mechanism. What property of self-attention, as defined by Vaswani et al. (2017), causes this?
- Compute and memory for self-attention scale linearly with sequence length, so extending the context window costs proportionally the same amount regardless of length
- The context window is limited only by the size of the token vocabulary, not by any property of the attention computation itself
- Self-attention's cost depends solely on the number of stacked layers in the model and is unaffected by how many tokens are in the input sequence
- Self-attention computes a pairwise compatibility score between every two positions in the sequence, so both compute and memory scale quadratically with sequence length, making much longer context windows substantially more expensive
Why D? And why not the others?
Correct answer: D. Self-attention computes a pairwise compatibility score between every two positions in the sequence, so both compute and memory scale quadratically with sequence length, making much longer context windows substantially more expensive
Standard self-attention, as defined by Vaswani et al., computes an attention score between every pair of positions in the input sequence by taking the dot product of each query with every key. For a sequence of length n, this means roughly n-squared score computations and an n-by-n matrix of attention weights must be produced and stored, so both the compute required and the memory needed for the attention matrix grow quadratically as sequence length increases. This is why simply increasing context length without architectural changes (such as sparse or approximate attention variants) becomes disproportionately expensive as sequences get longer. The option claiming linear scaling is wrong because it describes the cost profile that alternative, non-standard attention mechanisms aim to achieve, not the standard mechanism from the original paper. The option tying context length purely to vocabulary size is wrong because vocabulary size determines how many distinct tokens can be represented, not how many positions attention can efficiently process. The option claiming cost depends only on the number of layers is wrong because within each layer, cost still grows with sequence length regardless of depth.
Source: Vaswani et al., "Attention Is All You Need" (2017), arXiv:1706.03762, Section 4 (comparison of computational complexity per layer)