How LLMs Work: Transformers & Training

Why C? And why not the others?

Correct answer: C. It counteracts dot products growing large in magnitude for larger key dimensions, which would push the softmax into regions with extremely small gradients

Vaswani et al. (2017) scale the raw dot products by the square root of the key dimension because, for larger key dimensions, the dot products between query and key vectors tend to grow large in magnitude; feeding these large values into the softmax pushes it into saturated regions where the gradient with respect to its inputs becomes extremely small, hurting learning. Dividing by the square root of the dimension keeps the values in a range where the softmax remains well-behaved. The option about producing a probability distribution describes what the softmax function itself does, not what the scaling step contributes before the softmax is applied. The option about reducing parameter counts is wrong because scaling is a fixed arithmetic operation on activations, not a change to the learned projection matrices. The option about reusing weights across heads is wrong because each attention head computes its own independent set of attention weights from its own query and key projections; scaling does not link heads together in any way.

Source: Vaswani et al., "Attention Is All You Need" (2017), arXiv:1706.03762, Section 3.2.1 (Scaled Dot-Product Attention)

Why A? And why not the others?

Correct answer: A. It lets the model jointly attend to information from different representation subspaces at different positions, since a single attention head would average this into one weighted combination

The paper motivates multi-head attention by noting that a single attention function averages over all the information relevant to a query into one weighted combination, which can wash out distinct relationships that exist at different representation subspaces and different positions. By projecting the queries, keys, and values into several smaller subspaces and running attention independently in each, then concatenating the results, the model can capture several different kinds of relationships simultaneously rather than blending them into one. The option about context window length is wrong because multi-head attention changes how a fixed-length sequence is processed, not how many tokens can be included in the sequence. The option about eliminating positional encoding is wrong because positional information is added separately to the input embeddings regardless of how many heads are used. The option describing identical weights across heads is wrong because the heads are given separate learned projection matrices specifically so they can learn to attend differently, not so they replicate each other.

Source: Vaswani et al., "Attention Is All You Need" (2017), arXiv:1706.03762, Section 3.2.2 (Multi-Head Attention)

Why B? And why not the others?

Correct answer: B. Because self-attention computes a weighted sum over all positions regardless of order, so without added positional information the model would treat the input as an unordered set of tokens

Self-attention computes, for each position, a weighted sum over the value vectors at every position in the sequence, and that computation is inherently permutation-invariant: reshuffling the input tokens and their positions would reshuffle the output in exactly the same way, with no built-in sense of order. Because the architecture contains no recurrence or convolution to encode sequence order implicitly, Vaswani et al. add sinusoidal positional encodings to the embeddings so the model can distinguish 'token A followed by token B' from 'token B followed by token A.' The option about recurrent feed-forward sublayers is wrong because the position-wise feed-forward network is a simple non-recurrent transformation applied independently to each position. The option about vocabulary size is wrong because vocabulary capacity is determined by the embedding table's size, unrelated to position information. The option about replacing residual connections is wrong because residual connections around each sublayer serve a separate purpose (easing gradient flow) and remain present alongside positional encoding, not instead of it.

Source: Vaswani et al., "Attention Is All You Need" (2017), arXiv:1706.03762, Section 3.5 (Positional Encoding)

Why D? And why not the others?

Correct answer: D. It starts from individual characters and repeatedly merges the most frequent adjacent pair of symbols into a new symbol, learning a fixed number of merge operations to build a subword vocabulary

Sennrich et al. describe BPE as starting with each word represented as a sequence of characters plus an end-of-word symbol, with the initial symbol vocabulary consisting of all individual characters. The algorithm then repeatedly counts all adjacent symbol pairs across the training corpus, merges the single most frequent pair into a new symbol, and adds that symbol to the vocabulary; this repeats for a chosen number of merge operations, gradually building larger subword units out of frequent character sequences and even whole common words. This lets rare or unseen words be represented as sequences of known subwords instead of a single out-of-vocabulary token. The option describing whole-word dictionary codes is wrong because that is exactly the closed-vocabulary approach BPE was designed to avoid. The option about random sampling is wrong because merges are chosen deterministically by frequency, not randomly. The option describing pure whitespace/punctuation splitting is wrong because BPE explicitly performs further segmentation within words based on learned merges.

Source: Sennrich, Haddow & Birch, "Neural Machine Translation of Rare Words with Subword Units" (2016), arXiv:1508.07909, Section 3.2

Why C? And why not the others?

Correct answer: C. It sets the attention scores for future positions to negative infinity before the softmax, so a token's representation depends only on itself and earlier tokens, matching the left-to-right generation process

Causal (masked) self-attention works by setting the attention scores between a given position and every later position to negative infinity before the softmax is computed, so after the softmax those later positions receive essentially zero weight. This means each position's output representation is built only from itself and the positions before it. This is necessary because these models are trained to predict the next token from only the preceding tokens, matching how the model must generate text left-to-right at inference time -- without this masking, the model could trivially 'see' the answer it is supposed to predict during training, which would not reflect real generation conditions. The option describing a fixed nearby window is wrong because causal masking blocks only future positions, not distant past ones. The option about attending only to the current token is wrong because earlier positions remain fully visible. The option about removing only the value projection while keeping keys visible is wrong because masking blocks the attention score itself, not one specific projection.

Source: Radford et al., "Improving Language Understanding by Generative Pre-Training" (2018); Vaswani et al., "Attention Is All You Need" (2017), arXiv:1706.03762, Section 3.2.3 (masked multi-head attention)

Why A? And why not the others?

Correct answer: A. For a fixed training compute budget, model parameters and training tokens should be scaled at roughly the same rate, implying substantially larger training datasets relative to model size than many earlier large models had used

Hoffmann et al. found that for compute-optimal training, model size and the number of training tokens should be scaled at roughly equal rates -- for every doubling of model size, training tokens should also roughly double, yielding a compute-optimal ratio of around 20 training tokens per parameter. Applying this, their 70-billion-parameter Chinchilla model, trained on about 1.4 trillion tokens, outperformed several much larger contemporary models that had been trained on comparatively fewer tokens relative to their size, showing that many earlier large models were undertrained relative to their parameter count. The option claiming performance becomes independent of token count past a few billion parameters is wrong because the paper's whole point is that additional training tokens keep mattering as models scale. The option claiming model size should scale much faster than data is wrong because it describes the pre-Chinchilla assumption the paper revised, not its finding. The option claiming the optimal token-to-parameter ratio decreases with scale is wrong because Hoffmann et al. found this ratio to be roughly constant across the compute budgets they studied.

Source: Hoffmann et al., "Training Compute-Optimal Large Language Models" (2022), arXiv:2203.15556

Why B? And why not the others?

Correct answer: B. Supervised fine-tuning on human-written demonstrations, followed by training a reward model on human rankings of sampled model outputs, followed by reinforcement learning (PPO) that optimizes the policy against that reward model

Ouyang et al. describe reinforcement learning from human feedback (RLHF) as a three-stage process applied on top of an already-pretrained language model. First, the model is supervised fine-tuned on a dataset of human-written demonstrations of desired outputs. Second, human labelers rank multiple model outputs for the same prompts by quality, and this comparison data is used to train a separate reward model that predicts which output humans would prefer. Third, the fine-tuned model is further optimized using Proximal Policy Optimization (PPO), treating the reward model's score as the reward signal to maximize. The option describing pretraining alone followed by direct deployment is wrong because it skips both the demonstration-based fine-tuning and the preference-based reward modeling entirely. The option starting with a reward model trained on raw internet text is wrong because the reward model is trained on human comparison rankings of model outputs, not used to pre-filter the pretraining corpus. The option describing pure rule-based reinforcement learning with no human data is wrong because both the demonstration data and the comparison rankings are human-generated and essential to the pipeline.

Source: Ouyang et al., "Training Language Models to Follow Instructions with Human Feedback" (2022), arXiv:2203.02155, Section 3

Why D? And why not the others?

Correct answer: D. They add each sublayer's input directly to its output before normalization, giving gradients a direct path backward through the addition and easing optimization of many stacked layers

In the Transformer, each sublayer's output is computed as the input added to the sublayer's own transformation of that input, with layer normalization applied to that sum; this addition is the residual (skip) connection. Because the input is added directly to the output, gradients computed at a later layer can flow backward through that addition largely unimpeded, rather than being forced entirely through the sublayer's transformation, which mitigates vanishing gradients and makes it practical to stack many such blocks and train them successfully. The option about sharing attention head weights across layers is wrong because residual connections operate on sublayer inputs and outputs, not on how heads are parameterized. The option about eliminating softmax is wrong because softmax remains a required step inside self-attention regardless of residual connections. The option about replacing the feed-forward sublayer with an identity function at inference is wrong because the feed-forward sublayer is still computed in full; the residual connection only adds its output to its input rather than bypassing the computation.

Source: Vaswani et al., "Attention Is All You Need" (2017), arXiv:1706.03762, Section 3.1

Why C? And why not the others?

Correct answer: C. It computes the mean and variance across the feature dimension for each individual training example, making it independent of batch size, whereas batch normalization computes statistics across the batch for each feature

Ba et al. define layer normalization as computing the mean and variance across the feature dimension for a single training example, then using those statistics to normalize that example's activations; because the computation only ever looks within one example, it does not depend on how many examples are in a mini-batch. This differs from batch normalization, which instead computes the mean and variance for each feature across all examples in the current mini-batch. This batch-size independence is why layer normalization suits transformers and recurrent networks, where batch statistics can be unstable or awkward to define. The option describing batch-wide statistics is wrong because that describes batch normalization, not layer normalization. The option describing a post-training calibration pass is wrong because layer normalization's statistics are computed on the fly for every forward pass, during both training and inference. The option restricting normalization to only the query and key projections is wrong because layer normalization is applied to entire sublayer outputs, not to specific attention projections.

Source: Ba, Kiros & Hinton, "Layer Normalization" (2016), arXiv:1607.06450, Section 3

Why A? And why not the others?

Correct answer: A. It is applied identically and independently to each position in the sequence, typically expanding to a larger inner dimension through a nonlinearity before projecting back down

Vaswani et al. describe the position-wise feed-forward network as consisting of two linear transformations with a nonlinearity between them, applied separately and identically to each position of the sequence -- the same learned weights are reused at every position, but each position's vector is transformed independently of the others. It typically expands the representation to a larger inner dimension before projecting it back down to the model's original dimension. This is distinct from self-attention, which is the sublayer responsible for mixing information across positions; the feed-forward sublayer instead adds per-position nonlinear transformation capacity. The option describing cross-position combination is wrong because that describes what self-attention does, not the feed-forward sublayer, which processes each position on its own. The option about operating only on the final position is wrong because it is applied at every position, not just the last one. The option about sharing weights with the embedding layer is wrong because the feed-forward network has its own separate learned parameters.

Source: Vaswani et al., "Attention Is All You Need" (2017), arXiv:1706.03762, Section 3.3

Why B? And why not the others?

Correct answer: B. A trainable gating network routes each input to a sparse subset of expert feed-forward sub-networks, letting total parameter count scale far beyond what would be affordable if every expert were computed for every input

Shazeer et al. introduce a sparsely-gated mixture-of-experts layer consisting of up to thousands of feed-forward expert sub-networks, paired with a trainable gating network that, for each input, selects and combines only a small sparse subset of those experts rather than running all of them. Because only the selected experts do any computation for a given input, the total number of parameters in the layer can grow enormously -- the paper reports capacity increases of over 1000x -- while the compute cost per input stays close to that of a much smaller dense network. The option describing activation of every expert with averaged outputs is wrong because that describes a dense ensemble, which is exactly the proportional-compute-cost approach this sparse gating avoids. The option describing experts as full copies of the entire transformer stack is wrong because the experts in this layer are feed-forward sub-networks, not complete duplicated models. The option describing an untrained, randomly fixed gate is wrong because the gating network's parameters are learned jointly with the rest of the model.

Source: Shazeer et al., "Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer" (2017), arXiv:1701.06538

Why D? And why not the others?

Correct answer: D. Self-attention computes a pairwise compatibility score between every two positions in the sequence, so both compute and memory scale quadratically with sequence length, making much longer context windows substantially more expensive

Standard self-attention, as defined by Vaswani et al., computes an attention score between every pair of positions in the input sequence by taking the dot product of each query with every key. For a sequence of length n, this means roughly n-squared score computations and an n-by-n matrix of attention weights must be produced and stored, so both the compute required and the memory needed for the attention matrix grow quadratically as sequence length increases. This is why simply increasing context length without architectural changes (such as sparse or approximate attention variants) becomes disproportionately expensive as sequences get longer. The option claiming linear scaling is wrong because it describes the cost profile that alternative, non-standard attention mechanisms aim to achieve, not the standard mechanism from the original paper. The option tying context length purely to vocabulary size is wrong because vocabulary size determines how many distinct tokens can be represented, not how many positions attention can efficiently process. The option claiming cost depends only on the number of layers is wrong because within each layer, cost still grows with sequence length regardless of depth.

Source: Vaswani et al., "Attention Is All You Need" (2017), arXiv:1706.03762, Section 4 (comparison of computational complexity per layer)