When an autoregressive Transformer decoder generates text one token at a time, naively recomputing self-attention from scratch at every step would repeat a large amount of work. What does key-value (KV) caching do to avoid this, and what does it not need to recompute?
- It stores the entire attention-score matrix from the very first generation step and reuses those exact scores unchanged for every later token, regardless of what new token is generated
- It skips computing new query vectors for later tokens, reusing the query vector from the first generated token for every subsequent generation step
- It discards the key and value vectors after each step and instead caches only the final output logits, replaying them directly for the next step
- It stores the key and value vectors computed for every previously generated token so that, at each new step, only the new token's query, key, and value need to be computed, with that new query then attended over the cached keys and values, rather than recomputing keys and values for the whole sequence so far
Why D? And why not the others?
Correct answer: D. It stores the key and value vectors computed for every previously generated token so that, at each new step, only the new token's query, key, and value need to be computed, with that new query then attended over the cached keys and values, rather than recomputing keys and values for the whole sequence so far
Because each previously generated token's key and value vectors do not change as generation proceeds -- they depend only on that token's own representation, which is fixed once it has been produced -- an autoregressive decoder can cache those key and value vectors after they are first computed. At each new decoding step, only the newest token needs a fresh query, key, and value computed; that new query then attends over the cached keys and values from every earlier position plus its own, avoiding the need to recompute keys and values for the entire sequence from scratch at every step. The option describing reused unchanged attention scores from the first step is wrong because the attention scores themselves depend on the current query, which is different at every new step, so the scores cannot simply be replayed. The option describing reuse of the first token's query vector is wrong because every new token still needs its own freshly computed query to attend correctly; only the keys and values of earlier tokens are cached, not any query. The option describing caching only final output logits is wrong because caching logits alone would provide no way to compute attention scores for new tokens against earlier positions at all.
Source: Shazeer, "Fast Transformer Decoding: One Write-Head is All You Need" (2019), arXiv:1911.02150, Section 1 (Introduction)