In addition to the self-attention sublayer, each Transformer block (Vaswani et al., 2017) contains a position-wise feed-forward network. How is this feed-forward network applied across the sequence?
- It is applied identically and independently to each position in the sequence, typically expanding to a larger inner dimension through a nonlinearity before projecting back down
- It combines information across all positions simultaneously in the same way self-attention does, making it functionally redundant with the attention sublayer
- It only operates on the final position of the sequence, producing a single pooled representation for the entire input
- It shares its weights with the token embedding layer so that no additional parameters are introduced by including it
Why A? And why not the others?
Correct answer: A. It is applied identically and independently to each position in the sequence, typically expanding to a larger inner dimension through a nonlinearity before projecting back down
Vaswani et al. describe the position-wise feed-forward network as consisting of two linear transformations with a nonlinearity between them, applied separately and identically to each position of the sequence -- the same learned weights are reused at every position, but each position's vector is transformed independently of the others. It typically expands the representation to a larger inner dimension before projecting it back down to the model's original dimension. This is distinct from self-attention, which is the sublayer responsible for mixing information across positions; the feed-forward sublayer instead adds per-position nonlinear transformation capacity. The option describing cross-position combination is wrong because that describes what self-attention does, not the feed-forward sublayer, which processes each position on its own. The option about operating only on the final position is wrong because it is applied at every position, not just the last one. The option about sharing weights with the embedding layer is wrong because the feed-forward network has its own separate learned parameters.
Source: Vaswani et al., "Attention Is All You Need" (2017), arXiv:1706.03762, Section 3.3