Vaswani et al. (2017) use multi-head attention instead of a single attention function operating on the full-dimensional queries, keys, and values. What was the main motivation for splitting attention into multiple heads?
- It lets the model jointly attend to information from different representation subspaces at different positions, since a single attention head would average this into one weighted combination
- It doubles the effective context window length by processing two halves of the input sequence in parallel
- It removes the need for positional encoding, because each head learns to represent a different position offset
- It ensures that every attention head produces an identical set of attention weights, providing an ensembling effect through redundancy
Why A? And why not the others?
Correct answer: A. It lets the model jointly attend to information from different representation subspaces at different positions, since a single attention head would average this into one weighted combination
The paper motivates multi-head attention by noting that a single attention function averages over all the information relevant to a query into one weighted combination, which can wash out distinct relationships that exist at different representation subspaces and different positions. By projecting the queries, keys, and values into several smaller subspaces and running attention independently in each, then concatenating the results, the model can capture several different kinds of relationships simultaneously rather than blending them into one. The option about context window length is wrong because multi-head attention changes how a fixed-length sequence is processed, not how many tokens can be included in the sequence. The option about eliminating positional encoding is wrong because positional information is added separately to the input embeddings regardless of how many heads are used. The option describing identical weights across heads is wrong because the heads are given separate learned projection matrices specifically so they can learn to attend differently, not so they replicate each other.
Source: Vaswani et al., "Attention Is All You Need" (2017), arXiv:1706.03762, Section 3.2.2 (Multi-Head Attention)