Decoder-only autoregressive language models such as GPT (Radford et al., 2018) use masked self-attention during training. What does this masking do, and why is it necessary?
- It prevents the model from attending to tokens more than a fixed window size away, purely to reduce the compute cost of attention
- It forces every attention head to attend only to the current token itself, ignoring the representations at all other positions
- It sets the attention scores for future positions to negative infinity before the softmax, so a token's representation depends only on itself and earlier tokens, matching the left-to-right generation process
- It removes the value projection for positions beyond the current one, while still letting their key vectors influence the attention weights
Why C? And why not the others?
Correct answer: C. It sets the attention scores for future positions to negative infinity before the softmax, so a token's representation depends only on itself and earlier tokens, matching the left-to-right generation process
Causal (masked) self-attention works by setting the attention scores between a given position and every later position to negative infinity before the softmax is computed, so after the softmax those later positions receive essentially zero weight. This means each position's output representation is built only from itself and the positions before it. This is necessary because these models are trained to predict the next token from only the preceding tokens, matching how the model must generate text left-to-right at inference time -- without this masking, the model could trivially 'see' the answer it is supposed to predict during training, which would not reflect real generation conditions. The option describing a fixed nearby window is wrong because causal masking blocks only future positions, not distant past ones. The option about attending only to the current token is wrong because earlier positions remain fully visible. The option about removing only the value projection while keeping keys visible is wrong because masking blocks the attention score itself, not one specific projection.
Source: Radford et al., "Improving Language Understanding by Generative Pre-Training" (2018); Vaswani et al., "Attention Is All You Need" (2017), arXiv:1706.03762, Section 3.2.3 (masked multi-head attention)