Each sublayer in the Transformer encoder and decoder (Vaswani et al., 2017) is wrapped with a residual connection before layer normalization is applied. What role do these residual connections play in training deep stacks of transformer blocks?
- They reduce the total number of attention heads needed by allowing heads to share weights across different layers
- They eliminate the need for a softmax normalization step within the self-attention mechanism
- They replace the position-wise feed-forward sublayer with a simple identity function during inference, to speed up generation
- They add each sublayer's input directly to its output before normalization, giving gradients a direct path backward through the addition and easing optimization of many stacked layers
Why D? And why not the others?
Correct answer: D. They add each sublayer's input directly to its output before normalization, giving gradients a direct path backward through the addition and easing optimization of many stacked layers
In the Transformer, each sublayer's output is computed as the input added to the sublayer's own transformation of that input, with layer normalization applied to that sum; this addition is the residual (skip) connection. Because the input is added directly to the output, gradients computed at a later layer can flow backward through that addition largely unimpeded, rather than being forced entirely through the sublayer's transformation, which mitigates vanishing gradients and makes it practical to stack many such blocks and train them successfully. The option about sharing attention head weights across layers is wrong because residual connections operate on sublayer inputs and outputs, not on how heads are parameterized. The option about eliminating softmax is wrong because softmax remains a required step inside self-attention regardless of residual connections. The option about replacing the feed-forward sublayer with an identity function at inference is wrong because the feed-forward sublayer is still computed in full; the residual connection only adds its output to its input rather than bypassing the computation.
Source: Vaswani et al., "Attention Is All You Need" (2017), arXiv:1706.03762, Section 3.1