How LLMs Work: Transformers & Training

Correct answer: .

Vaswani et al. (2017) scale the raw dot products by the square root of the key dimension because, for larger key dimensions, the dot products between query and key vectors tend to grow large in magnitude; feeding these large values into the softmax pushes it into saturated regions where the gradient with respect to its inputs becomes extremely small, hurting learning. Dividing by the square root of the dimension keeps the values in a range where the softmax remains well-behaved. The option about producing a probability distribution describes what the softmax function itself does, not what the scaling step contributes before the softmax is applied. The option about reducing parameter counts is wrong because scaling is a fixed arithmetic operation on activations, not a change to the learned projection matrices. The option about reusing weights across heads is wrong because each attention head computes its own independent set of attention weights from its own query and key projections; scaling does not link heads together in any way.

Source: Vaswani et al., "Attention Is All You Need" (2017), arXiv:1706.03762, Section 3.2.1 (Scaled Dot-Product Attention)

Correct answer: .

The paper motivates multi-head attention by noting that a single attention function averages over all the information relevant to a query into one weighted combination, which can wash out distinct relationships that exist at different representation subspaces and different positions. By projecting the queries, keys, and values into several smaller subspaces and running attention independently in each, then concatenating the results, the model can capture several different kinds of relationships simultaneously rather than blending them into one. The option about context window length is wrong because multi-head attention changes how a fixed-length sequence is processed, not how many tokens can be included in the sequence. The option about eliminating positional encoding is wrong because positional information is added separately to the input embeddings regardless of how many heads are used. The option describing identical weights across heads is wrong because the heads are given separate learned projection matrices specifically so they can learn to attend differently, not so they replicate each other.

Source: Vaswani et al., "Attention Is All You Need" (2017), arXiv:1706.03762, Section 3.2.2 (Multi-Head Attention)

Correct answer: .

Self-attention computes, for each position, a weighted sum over the value vectors at every position in the sequence, and that computation is inherently permutation-invariant: reshuffling the input tokens and their positions would reshuffle the output in exactly the same way, with no built-in sense of order. Because the architecture contains no recurrence or convolution to encode sequence order implicitly, Vaswani et al. add sinusoidal positional encodings to the embeddings so the model can distinguish 'token A followed by token B' from 'token B followed by token A.' The option about recurrent feed-forward sublayers is wrong because the position-wise feed-forward network is a simple non-recurrent transformation applied independently to each position. The option about vocabulary size is wrong because vocabulary capacity is determined by the embedding table's size, unrelated to position information. The option about replacing residual connections is wrong because residual connections around each sublayer serve a separate purpose (easing gradient flow) and remain present alongside positional encoding, not instead of it.

Source: Vaswani et al., "Attention Is All You Need" (2017), arXiv:1706.03762, Section 3.5 (Positional Encoding)

Correct answer: .

Sennrich et al. describe BPE as starting with each word represented as a sequence of characters plus an end-of-word symbol, with the initial symbol vocabulary consisting of all individual characters. The algorithm then repeatedly counts all adjacent symbol pairs across the training corpus, merges the single most frequent pair into a new symbol, and adds that symbol to the vocabulary; this repeats for a chosen number of merge operations, gradually building larger subword units out of frequent character sequences and even whole common words. This lets rare or unseen words be represented as sequences of known subwords instead of a single out-of-vocabulary token. The option describing whole-word dictionary codes is wrong because that is exactly the closed-vocabulary approach BPE was designed to avoid. The option about random sampling is wrong because merges are chosen deterministically by frequency, not randomly. The option describing pure whitespace/punctuation splitting is wrong because BPE explicitly performs further segmentation within words based on learned merges.

Source: Sennrich, Haddow & Birch, "Neural Machine Translation of Rare Words with Subword Units" (2016), arXiv:1508.07909, Section 3.2

Correct answer: .

Causal (masked) self-attention works by setting the attention scores between a given position and every later position to negative infinity before the softmax is computed, so after the softmax those later positions receive essentially zero weight. This means each position's output representation is built only from itself and the positions before it. This is necessary because these models are trained to predict the next token from only the preceding tokens, matching how the model must generate text left-to-right at inference time -- without this masking, the model could trivially 'see' the answer it is supposed to predict during training, which would not reflect real generation conditions. The option describing a fixed nearby window is wrong because causal masking blocks only future positions, not distant past ones. The option about attending only to the current token is wrong because earlier positions remain fully visible. The option about removing only the value projection while keeping keys visible is wrong because masking blocks the attention score itself, not one specific projection.

Source: Radford et al., "Improving Language Understanding by Generative Pre-Training" (2018); Vaswani et al., "Attention Is All You Need" (2017), arXiv:1706.03762, Section 3.2.3 (masked multi-head attention)

Correct answer: .

Hoffmann et al. found that for compute-optimal training, model size and the number of training tokens should be scaled at roughly equal rates -- for every doubling of model size, training tokens should also roughly double, yielding a compute-optimal ratio of around 20 training tokens per parameter. Applying this, their 70-billion-parameter Chinchilla model, trained on about 1.4 trillion tokens, outperformed several much larger contemporary models that had been trained on comparatively fewer tokens relative to their size, showing that many earlier large models were undertrained relative to their parameter count. The option claiming performance becomes independent of token count past a few billion parameters is wrong because the paper's whole point is that additional training tokens keep mattering as models scale. The option claiming model size should scale much faster than data is wrong because it describes the pre-Chinchilla assumption the paper revised, not its finding. The option claiming the optimal token-to-parameter ratio decreases with scale is wrong because Hoffmann et al. found this ratio to be roughly constant across the compute budgets they studied.

Source: Hoffmann et al., "Training Compute-Optimal Large Language Models" (2022), arXiv:2203.15556

Correct answer: .

Ouyang et al. describe reinforcement learning from human feedback (RLHF) as a three-stage process applied on top of an already-pretrained language model. First, the model is supervised fine-tuned on a dataset of human-written demonstrations of desired outputs. Second, human labelers rank multiple model outputs for the same prompts by quality, and this comparison data is used to train a separate reward model that predicts which output humans would prefer. Third, the fine-tuned model is further optimized using Proximal Policy Optimization (PPO), treating the reward model's score as the reward signal to maximize. The option describing pretraining alone followed by direct deployment is wrong because it skips both the demonstration-based fine-tuning and the preference-based reward modeling entirely. The option starting with a reward model trained on raw internet text is wrong because the reward model is trained on human comparison rankings of model outputs, not used to pre-filter the pretraining corpus. The option describing pure rule-based reinforcement learning with no human data is wrong because both the demonstration data and the comparison rankings are human-generated and essential to the pipeline.

Source: Ouyang et al., "Training Language Models to Follow Instructions with Human Feedback" (2022), arXiv:2203.02155, Section 3

Correct answer: .

In the Transformer, each sublayer's output is computed as the input added to the sublayer's own transformation of that input, with layer normalization applied to that sum; this addition is the residual (skip) connection. Because the input is added directly to the output, gradients computed at a later layer can flow backward through that addition largely unimpeded, rather than being forced entirely through the sublayer's transformation, which mitigates vanishing gradients and makes it practical to stack many such blocks and train them successfully. The option about sharing attention head weights across layers is wrong because residual connections operate on sublayer inputs and outputs, not on how heads are parameterized. The option about eliminating softmax is wrong because softmax remains a required step inside self-attention regardless of residual connections. The option about replacing the feed-forward sublayer with an identity function at inference is wrong because the feed-forward sublayer is still computed in full; the residual connection only adds its output to its input rather than bypassing the computation.

Source: Vaswani et al., "Attention Is All You Need" (2017), arXiv:1706.03762, Section 3.1

Correct answer: .

Ba et al. define layer normalization as computing the mean and variance across the feature dimension for a single training example, then using those statistics to normalize that example's activations; because the computation only ever looks within one example, it does not depend on how many examples are in a mini-batch. This differs from batch normalization, which instead computes the mean and variance for each feature across all examples in the current mini-batch. This batch-size independence is why layer normalization suits transformers and recurrent networks, where batch statistics can be unstable or awkward to define. The option describing batch-wide statistics is wrong because that describes batch normalization, not layer normalization. The option describing a post-training calibration pass is wrong because layer normalization's statistics are computed on the fly for every forward pass, during both training and inference. The option restricting normalization to only the query and key projections is wrong because layer normalization is applied to entire sublayer outputs, not to specific attention projections.

Source: Ba, Kiros & Hinton, "Layer Normalization" (2016), arXiv:1607.06450, Section 3

Correct answer: .

Vaswani et al. describe the position-wise feed-forward network as consisting of two linear transformations with a nonlinearity between them, applied separately and identically to each position of the sequence -- the same learned weights are reused at every position, but each position's vector is transformed independently of the others. It typically expands the representation to a larger inner dimension before projecting it back down to the model's original dimension. This is distinct from self-attention, which is the sublayer responsible for mixing information across positions; the feed-forward sublayer instead adds per-position nonlinear transformation capacity. The option describing cross-position combination is wrong because that describes what self-attention does, not the feed-forward sublayer, which processes each position on its own. The option about operating only on the final position is wrong because it is applied at every position, not just the last one. The option about sharing weights with the embedding layer is wrong because the feed-forward network has its own separate learned parameters.

Source: Vaswani et al., "Attention Is All You Need" (2017), arXiv:1706.03762, Section 3.3

Correct answer: .

Shazeer et al. introduce a sparsely-gated mixture-of-experts layer consisting of up to thousands of feed-forward expert sub-networks, paired with a trainable gating network that, for each input, selects and combines only a small sparse subset of those experts rather than running all of them. Because only the selected experts do any computation for a given input, the total number of parameters in the layer can grow enormously -- the paper reports capacity increases of over 1000x -- while the compute cost per input stays close to that of a much smaller dense network. The option describing activation of every expert with averaged outputs is wrong because that describes a dense ensemble, which is exactly the proportional-compute-cost approach this sparse gating avoids. The option describing experts as full copies of the entire transformer stack is wrong because the experts in this layer are feed-forward sub-networks, not complete duplicated models. The option describing an untrained, randomly fixed gate is wrong because the gating network's parameters are learned jointly with the rest of the model.

Source: Shazeer et al., "Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer" (2017), arXiv:1701.06538

Correct answer: .

Standard self-attention, as defined by Vaswani et al., computes an attention score between every pair of positions in the input sequence by taking the dot product of each query with every key. For a sequence of length n, this means roughly n-squared score computations and an n-by-n matrix of attention weights must be produced and stored, so both the compute required and the memory needed for the attention matrix grow quadratically as sequence length increases. This is why simply increasing context length without architectural changes (such as sparse or approximate attention variants) becomes disproportionately expensive as sequences get longer. The option claiming linear scaling is wrong because it describes the cost profile that alternative, non-standard attention mechanisms aim to achieve, not the standard mechanism from the original paper. The option tying context length purely to vocabulary size is wrong because vocabulary size determines how many distinct tokens can be represented, not how many positions attention can efficiently process. The option claiming cost depends only on the number of layers is wrong because within each layer, cost still grows with sequence length regardless of depth.

Source: Vaswani et al., "Attention Is All You Need" (2017), arXiv:1706.03762, Section 4 (comparison of computational complexity per layer)

Correct answer: .

Vaswani et al. tie a single learned weight matrix to both roles: converting input and output tokens into d_model-dimensional vectors and, transposed, converting the decoder's final vectors into logits over the vocabulary before the softmax; because one matrix serves both purposes, the model needs fewer total parameters than if the embedding and output projection were learned separately, and the input and output token representations remain linked to each other, an idea the paper credits to prior work on tying input and output embeddings. The embedding weights are then multiplied by the square root of the model dimension to keep their scale comparable to the positional encodings added to them. The option describing shared encoder/decoder self-attention weights is wrong because the tied weight matrix here is the token embedding/output-projection matrix, not the query, key, or value projections used inside attention, which remain separate learned parameters in each layer. The option describing merging the feed-forward sublayer into the embedding matrix is wrong because the position-wise feed-forward network is a separate sublayer with its own weights that this tying scheme does not touch. The option claiming every output token receives an identical initial probability is wrong because the tied matrix is initialized with the same random values used for embeddings, not with values that force a uniform output distribution.

Source: Vaswani et al., "Attention Is All You Need" (2017), arXiv:1706.03762, Section 3.4 (Embeddings and Softmax)

Correct answer: .

In the encoder-decoder attention sublayer, Vaswani et al. take the queries from the output of the decoder's own preceding sublayer, while the keys and values come from the output of the encoder stack; this lets every position in the decoder attend over all positions in the input sequence, mirroring the query-key-value structure used in earlier sequence-to-sequence attention mechanisms. The option in which queries, keys, and values all come from the encoder is wrong because the decoder must still contribute its own queries so that each decoder position can ask a different question of the encoder output. The option that swaps queries/keys and values between encoder and decoder is wrong because Vaswani et al. specifically place the keys together with the values from the same source (the encoder) rather than splitting queries and keys across two different sources. The option describing queries, keys, and values all coming from the decoder is wrong because that describes the masked self-attention sublayer earlier in the decoder block, not the separate encoder-decoder attention sublayer that follows it.

Source: Vaswani et al., "Attention Is All You Need" (2017), arXiv:1706.03762, Section 3.2.3 (Applications of Attention in our Model)

Correct answer: .

Su et al. encode position by treating pairs of dimensions within each query and key vector as coordinates and rotating them by an angle that is a function of the token's absolute position and a frequency that varies by dimension pair, before the query and key are used to compute attention scores. Because rotating both a query and a key vector by an amount tied to their own positions leaves the angle between them dependent only on the difference between those positions, the resulting dot product ends up encoding relative distance even though each vector was rotated using only its own absolute position, with the paper additionally noting that this relative dependency decays as tokens grow further apart. The option describing an appended learned position index is wrong because RoPE does not add any extra dimension to the vectors; it transforms the existing dimensions through rotation. The option describing a lookup table indexed only by the query's absolute position is wrong because RoPE's dot product depends on both vectors' positions through the same rotation mechanism, not a table keyed to one side only. The option describing a post-softmax sinusoidal mask is wrong because the rotation is applied directly to the query and key vectors before the dot product and softmax are computed, not to the attention output afterward.

Source: Su et al., "RoFormer: Enhanced Transformer with Rotary Position Embedding" (2021), arXiv:2104.09864

Correct answer: .

Press, Smith and Lewis describe ALiBi as adding no positional embeddings to the word embeddings whatsoever; instead, after computing the raw query-key dot product, a static bias is subtracted that is proportional to the distance between the two positions, with each attention head using its own fixed slope for that penalty, so nearby tokens are penalized less than distant ones and the mechanism naturally extends to sequence lengths never seen during training. The option describing recomputed learned positional embeddings is wrong because ALiBi's bias is a fixed, non-learned function of distance, not a learned embedding table at all. The option describing reliance purely on feed order with no explicit signal is wrong because ALiBi does inject an explicit, though non-learned, positional signal directly into the attention score computation. The option describing position-specific query/key weight matrices is wrong because ALiBi keeps the same query and key projections at every position and instead modifies only the attention scores after the dot product, adding no new per-position parameters.

Source: Press, Smith & Lewis, "Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation" (2021), arXiv:2108.12409

Correct answer: .

Hendrycks and Gimpel define GELU as weighting an input by the value of the standard Gaussian cumulative distribution function evaluated at that same input, which produces a smooth, differentiable curve that, unlike ReLU, does not hard-clamp every negative input to exactly zero -- small negative inputs pass through scaled by a small positive weight rather than being zeroed out entirely. GPT-2 and GPT-3 use this activation between the two linear transformations of their position-wise feed-forward sublayers. The option describing a small fixed negative constant for every negative input is wrong because that describes Leaky ReLU's behavior, a different activation with a fixed negative slope, not GELU's Gaussian-weighted curve. The option describing removal of the nonlinearity is wrong because GELU is itself the nonlinearity inserted between the two linear transformations, not a replacement that eliminates nonlinearity. The option placing this activation after the softmax on attention scores is wrong because GELU is used inside the feed-forward sublayer's two linear transformations, entirely separate from the self-attention computation and its softmax.

Source: Hendrycks & Gimpel, "Gaussian Error Linear Units (GELUs)" (2016), arXiv:1606.08415; used in Radford et al., "Language Models are Unsupervised Multitask Learners" (GPT-2, 2019) and Brown et al., "Language Models are Few-Shot Learners" (GPT-3, 2020)

Correct answer: .

GPT-style decoder-only models are pretrained with a next-token prediction objective: at every position in a training sequence, the model produces a probability distribution over the vocabulary conditioned only on the preceding tokens (enforced by causal masking), and a cross-entropy loss compares that predicted distribution against the actual next token in the corpus, averaged across all positions and all training sequences; this simple, self-supervised setup requires no labeled data beyond raw text. The option describing next-sentence-pair classification is wrong because that describes a separate auxiliary objective used by some encoder models, not the autoregressive next-token objective decoder-only GPT-style models are pretrained on. The option describing reconstruction of scattered masked tokens is wrong because that describes the masked-language-modeling objective used by bidirectional encoder models, which is incompatible with the strictly left-to-right causal masking decoder-only models use. The option describing a scalar reward regression trained on preference comparisons is wrong because that describes the separate reward-model training stage used later in an RLHF pipeline, not the initial self-supervised pretraining objective.

Source: Radford et al., "Improving Language Understanding by Generative Pre-Training" (2018), Section 3.1; Radford et al., "Language Models are Unsupervised Multitask Learners" (2019)

Correct answer: .

Because each previously generated token's key and value vectors do not change as generation proceeds -- they depend only on that token's own representation, which is fixed once it has been produced -- an autoregressive decoder can cache those key and value vectors after they are first computed. At each new decoding step, only the newest token needs a fresh query, key, and value computed; that new query then attends over the cached keys and values from every earlier position plus its own, avoiding the need to recompute keys and values for the entire sequence from scratch at every step. The option describing reused unchanged attention scores from the first step is wrong because the attention scores themselves depend on the current query, which is different at every new step, so the scores cannot simply be replayed. The option describing reuse of the first token's query vector is wrong because every new token still needs its own freshly computed query to attend correctly; only the keys and values of earlier tokens are cached, not any query. The option describing caching only final output logits is wrong because caching logits alone would provide no way to compute attention scores for new tokens against earlier positions at all.

Source: Shazeer, "Fast Transformer Decoding: One Write-Head is All You Need" (2019), arXiv:1911.02150, Section 1 (Introduction)

Correct answer: .

Using mean field theory, Xiong et al. show that in the Post-LN Transformer -- where layer normalization is applied after the residual addition, as in the original architecture -- the expected gradients near the output layer are large at initialization, and they demonstrate that this is why a learning-rate warm-up stage has empirically been necessary to train Post-LN Transformers successfully. When layer normalization is instead placed inside each residual block, before the sublayer's transformation (Pre-LN), the authors show gradients are well-behaved at initialization, and Pre-LN Transformers can be trained without the warm-up stage while reaching comparable results in less training time. The option claiming identical gradient magnitudes for both placements is wrong because the paper's central contribution is precisely that the two placements behave very differently at initialization. The option claiming Pre-LN needs a longer warm-up is wrong because the paper's finding is the reverse: Pre-LN is the placement that removes the need for warm-up, not the one that requires more of it. The option claiming the placement only affects inference cost is wrong because the paper's analysis and experiments concern training-time gradient behavior and optimization stability, not inference-time computation.

Source: Xiong et al., "On Layer Normalization in the Transformer Architecture" (2020), arXiv:2002.04745

Correct answer: .

Brown et al. describe few-shot evaluation as conditioning GPT-3 purely through text: a handful of example input-output pairs for the task are written directly into the prompt, followed by a new input, and the model is asked to produce the corresponding output -- all of this happens through ordinary text generation at inference time, with no gradient updates, fine-tuning, or any change whatsoever to the model's weights performed for the task. The option describing a few additional gradient-descent steps is wrong because that describes traditional few-shot fine-tuning, which the paper explicitly contrasts with its in-context approach applied to GPT-3. The option describing duplicating and separately fine-tuning several model copies is wrong because only a single frozen copy of GPT-3 is used, conditioned solely through the prompt, not multiple task-specific fine-tuned copies. The option describing an attached, separately trained classifier head is wrong because no new parameters of any kind are added to or trained on top of GPT-3 for the task; the entire model remains exactly as pretrained.

Source: Brown et al., "Language Models are Few-Shot Learners" (2020), arXiv:2005.14165

Correct answer: .

Shazeer's multi-query attention keeps the multiple, separately learned query heads of standard multi-head attention, but has all of those heads share one single set of key and value projections instead of each head computing and caching its own; because the memory bandwidth needed to read the cached keys and values back from memory at every incremental decoding step is the dominant cost of autoregressive generation, shrinking the cached keys and values down to a single shared copy per layer substantially reduces that bandwidth cost, at a modest expense in model quality. The option describing a single shared query head with separate keys and values per head is wrong because it inverts the paper's actual change: multiple query heads are kept, and it is the keys and values that are shared, not the queries. The option describing removal of key and value projections entirely is wrong because keys and values are still computed and cached, just from one shared projection rather than one per head. The option describing more key/value heads than query heads is wrong because multi-query attention reduces the key/value heads down to exactly one, rather than increasing them beyond the query head count.

Source: Shazeer, "Fast Transformer Decoding: One Write-Head is All You Need" (2019), arXiv:1911.02150

Correct answer: .

Dao et al. observe that standard attention implementations are limited less by the number of floating-point operations they perform and more by how much data must be moved between the GPU's large but comparatively slow high-bandwidth memory (HBM) and its much smaller but much faster on-chip SRAM; FlashAttention restructures the computation using tiling, computing attention in blocks and using recomputation during the backward pass, so that it reads from and writes to HBM far less often, all while still computing exactly the same attention output the standard formula would produce -- it is explicitly an exact algorithm, not an approximation. The option describing a sparse or low-rank approximation is wrong because FlashAttention preserves the exact standard attention computation rather than substituting a different, approximate attention pattern. The option describing lower numerical precision as the source of the speedup is wrong because FlashAttention's gains come from reducing memory movement through tiling, not from reducing precision, and it does not sacrifice numerical exactness of the result. The option describing running entirely in CPU memory is wrong because FlashAttention's optimization is specifically about the memory hierarchy within the GPU itself (HBM versus on-chip SRAM), not about moving computation to the CPU.

Source: Dao et al., "FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness" (2022), arXiv:2205.14135

Correct answer: .

Radford et al. note that applying BPE directly at the Unicode character level would start from a base vocabulary of over 130,000 symbols before any merges are learned, far larger than the 32,000-to-64,000-token vocabularies typically used; operating on bytes instead means the base vocabulary before any merges is fixed at exactly 256 symbols, one for every possible byte value, and the byte-pair merge process is then run on top of that small, fixed base, eventually growing GPT-2's vocabulary to 50,257 tokens. Because every possible input can be represented as some sequence of bytes, the model can assign a probability to any string at all, removing the need for an unknown-token fallback. The option describing skipping the merge process entirely is wrong because GPT-2 still runs the same iterative BPE merge algorithm on top of the byte-level base vocabulary; only the starting point changes. The option claiming the base vocabulary grows beyond 100,000 symbols is wrong because the entire point of operating on bytes is to keep that base vocabulary small, fixed at 256, not to expand it. The option claiming each byte remains its own separate token throughout generation is wrong because the merge operations combine frequently co-occurring byte sequences into larger subword tokens, exactly as in ordinary BPE.

Source: Radford et al., "Language Models are Unsupervised Multitask Learners" (2019), Section 2.2 (Input Representation)

Correct answer: .

Grouped-query attention divides a layer's query heads into a fixed number of groups and has every query head within a group share one key and value head, rather than each query head owning its own key and value projection as in standard multi-head attention, or all query heads sharing a single key and value head as in multi-query attention. This group count is the tunable knob that lets GQA interpolate between the two extremes: setting the number of groups equal to the number of query heads recovers ordinary multi-head attention exactly, while setting it to one recovers multi-query attention exactly, and any intermediate group count trades off some of multi-head attention's quality for much of multi-query attention's memory-bandwidth savings during autoregressive decoding. The option describing a single shared key and value head with independently-projected query heads is simply describing multi-query attention with an invented extra detail, not the grouped design GQA actually introduces. The option describing a reduction in the number of query heads themselves is backwards: GQA leaves the query heads and their projections untouched and instead reduces the number of distinct key and value heads. The option describing a shared feed-forward network replacing the key and value projections describes a mechanism the paper does not use at all; GQA still computes ordinary key and value projections, just fewer of them, shared across groups of query heads.

Source: Ainslie et al., "GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints" (2023), arXiv:2305.13245

Correct answer: .

Kaplan et al.'s original power-law fits, based on a smaller and less systematically-swept set of training runs, recommended spending a fixed compute budget mostly on making the model larger, training on a comparatively modest amount of data and stopping well short of the point where the model's loss on that data would have converged. Hoffmann et al. later trained a far larger and more carefully controlled sweep of models and found that Kaplan et al.'s fitted curves had systematically under-weighted the value of training data: compute-optimal training instead scales model size and training-data size in roughly equal proportion as the compute budget grows. Because most large models trained under the earlier recipe followed Kaplan et al.'s guidance, Hoffmann et al.'s finding implied that many of those models, though larger, were undertrained relative to what a compute-optimal allocation of the same budget would have used, and a smaller model trained on more data could match or beat one of them. The option claiming Kaplan et al. favored equal scaling and Hoffmann et al. favored model-size-heavy scaling has the two papers' conclusions swapped. The option claiming the two papers reached identical conclusions ignores the entire reason the Chinchilla paper is considered a correction to the earlier scaling recipe. The option claiming Kaplan et al. prioritized data size over model size states the opposite of what that paper's fitted allocation curve recommended.

Source: Kaplan et al., "Scaling Laws for Neural Language Models" (2020), arXiv:2001.08361; Hoffmann et al., "Training Compute-Optimal Large Language Models" (2022), arXiv:2203.15556

Correct answer: .

SentencePiece's Unigram algorithm works in the opposite direction from byte-pair encoding: it begins with a large seed vocabulary of candidate subword pieces, then repeatedly measures how much the overall likelihood of the training corpus under a unigram language model over subwords would drop if each piece were removed, and prunes the least useful pieces until the vocabulary shrinks to the target size. BPE instead starts from individual characters and builds upward, repeatedly merging whichever adjacent pair of symbols is most frequent in the training data into a single new symbol, growing the vocabulary from the bottom up rather than shrinking it from the top down. The option describing bottom-up merging of the most frequent adjacent symbols is actually describing BPE's own mechanism, not what distinguishes Unigram from it. The option claiming SentencePiece requires text to already be split into whitespace-separated words gets the comparison backwards: a major point of the SentencePiece paper is that it can train directly on raw, undelimited sentences, unlike tools that assume pre-tokenized word boundaries. The option claiming every character keeps its own permanent token and is never combined describes neither algorithm; both BPE and Unigram build multi-character subword units.

Source: Kudo & Richardson, "SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing" (2018), arXiv:1808.06226

Correct answer: .

DPO's key move is algebraic: it takes the closed-form solution to the KL-regularized reward-maximization problem that RLHF is ultimately trying to solve, and uses it to express the reward function directly in terms of the ratio between the policy being trained and a fixed reference policy. Substituting this reparameterized reward into the standard preference-modelling loss produces a simple classification-style objective that can be optimized directly on pairs of preferred and dispreferred completions, with no separate reward model ever trained and no online reinforcement-learning sampling loop such as PPO ever run. The option describing an automated LLM-as-judge replacing human comparisons while keeping the reward model and PPO step intact misidentifies what DPO removes; DPO's contribution is eliminating the reward model and the RL step themselves, not just changing how preference labels are collected. The option describing supervised fine-tuning on the reward model's top-scoring completions describes a best-of-n-style distillation approach, not DPO, and still keeps a separate reward model, which DPO does not train at all. The option claiming DPO removes the need for preference data entirely contradicts the method's basic setup: DPO is trained directly on preference pairs; it just removes the intermediate reward-model and RL stages, not the preference data itself.

Source: Rafailov et al., "Direct Preference Optimization: Your Language Model is Secretly a Reward Model" (2023), arXiv:2305.18290

Correct answer: .

Schaeffer et al.'s central claim is a measurement critique, not a claim that emergence never occurs at all: they argue that many of the sharp, unpredictable jumps Wei et al. reported arise because the tasks in question were scored with metrics such as strict exact-match accuracy, which are nonlinear or outright discontinuous functions of a model's per-token error rate on multi-step problems, so a smoothly improving underlying model can still show what looks like a sudden jump on that metric. When Schaeffer et al. re-score the same models' outputs using smoother, partial-credit metrics, the previously sharp jumps become gradual and predictable trends instead, which supports interpreting emergence as substantially a property of the chosen metric rather than a discontinuity in the model itself. The option claiming the jumps disappear once larger models are trained mischaracterizes the argument as being about model scale rather than about metric choice. The option claiming Schaeffer et al. propose a formula that predicts, in advance and from parameter count alone, exactly when any new ability will appear overstates their contribution well beyond what the paper claims; their point is that the timing looked sharp mainly due to measurement choice, not that it becomes precisely predictable from parameter count. The option restricting the argument to decoder-only architectures invents an architectural claim that does not appear in either paper.

Source: Wei et al., "Emergent Abilities of Large Language Models" (2022), arXiv:2206.07682; Schaeffer et al., "Are Emergent Abilities of Large Language Models a Mirage?" (2023), arXiv:2304.15004

Correct answer: .

The Switch Transformer's main simplification of the earlier sparsely-gated mixture-of-experts design is to route each token to exactly one expert instead of combining the weighted outputs of several experts per token, cutting the routing computation and the cross-device communication needed to gather multiple experts' outputs; a differentiable load-balancing auxiliary loss is added to keep tokens spread roughly evenly across experts despite the router now making a single hard choice per token. The option describing every token passing through every expert and then averaging all outputs describes a dense model with no sparsity at all, the opposite of what makes a mixture-of-experts layer computationally cheap relative to its total parameter count. The option describing an increase to a larger fixed number of experts activated per token, such as sixteen, moves in the opposite direction from what the paper actually does, which is to reduce the number of activated experts per token to one. The option describing a fixed, position-based routing rule invents a mechanism the paper does not use; the router remains a learned, content-based gating network, it is simply restricted to choosing a single expert per token rather than a weighted combination of several.

Source: Fedus, Zoph & Shazeer, "Switch Transformer: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity" (2021), arXiv:2101.03961

Correct answer: .

Longformer's efficiency gain comes from replacing full attention, where every token attends to every other token at quadratic cost in sequence length, with a fixed-size sliding-window attention pattern for most tokens, so each token only attends to a bounded number of nearby neighbors and the cost grows linearly with sequence length instead; a small set of task-specific tokens, such as a classification token, are additionally given full global attention to every other token, so the model still has a channel for long-range information to flow through despite most positions only seeing a local window. The option describing lower floating-point precision with an unchanged attention pattern describes a numerical-precision optimization unrelated to Longformer's actual contribution, which is about which token pairs are attended to at all, not the numeric format used. The option describing fixed non-overlapping blocks with absolutely no mechanism for information to cross block boundaries omits the global-attention tokens that Longformer specifically adds to avoid exactly that limitation. The option describing compressing the sequence with a separate pooling network before applying full attention describes a different long-context approach entirely, not the sliding-window-plus-global-attention design Longformer introduces.

Source: Beltagy, Peters & Cohan, "Longformer: The Long-Document Transformer" (2020), arXiv:2004.05150

Correct answer: .

Position Interpolation's core idea is to linearly rescale the position indices of a longer input sequence down so that even the largest index still lands within the range of position values the model actually saw during pretraining, and then briefly fine-tune the model on sequences using these rescaled positions; because the model never has to process a position value larger than anything in its training distribution, this avoids the extremely large, out-of-distribution attention scores that naive extrapolation beyond the trained range produces, which the paper shows can badly break the attention mechanism. The option describing feeding the model position values larger than any seen in training, relying on RoPE's periodicity to cope, describes the naive extrapolation approach the paper explicitly identifies as producing catastrophic attention scores, essentially the failure mode Position Interpolation is designed to avoid, not the method itself. The option describing discarding RoPE for the original sinusoidal encoding invents a change the paper does not make; the method is specifically designed to extend RoPE-based models, not replace their position embedding scheme. The option describing an increase in embedding dimension to add positional precision is not how the paper extends context length at all; it rescales existing position indices rather than changing model dimensions.

Source: Chen et al., "Extending Context Window of Large Language Models via Positional Interpolation" (2023), arXiv:2306.15595

Correct answer: .

Speculative decoding uses a small, cheap draft model to propose several candidate next tokens at once, then runs the large target model once, in parallel, over that whole candidate sequence to obtain the target model's own probabilities for each position; a rejection-sampling procedure then accepts the longest prefix of draft tokens that is consistent with the target model's distribution and, when a draft token is rejected, resamples correctly from the target model's distribution at that position, so the tokens produced are distributed exactly as if the target model alone had generated every one of them, just computed with fewer sequential large-model forward passes. The option describing the large model only occasionally spot-checking a small model's output describes an approximate scheme that would change the output distribution, which contradicts the paper's explicit goal of matching the large model's output exactly. The option describing caching previously-computed key and value tensors so they are never recomputed describes key-value caching, a different and complementary inference optimization, not speculative decoding. The option describing IO-aware fusion of attention operations to avoid writing large intermediate matrices to slow memory describes FlashAttention, another distinct optimization that changes how attention is computed, not how tokens are proposed and verified.

Source: Leviathan, Kalman & Matias, "Fast Inference from Transformers via Speculative Decoding" (2023), arXiv:2211.17192

Correct answer: .

Knowledge distillation trains the small student network to reproduce the large teacher network's full soft-target probability distribution across all classes, not merely the single class the teacher ranks highest; computing both the teacher's and the student's output distributions at a raised softmax temperature during this training softens the probabilities enough to expose the relative likelihoods the teacher assigns to the classes it considers plausible but incorrect, information the paper calls dark knowledge that a single hard label could never convey, and after training the student is deployed using the normal temperature of one. The option describing training only on the teacher's single most-likely class, identical to training on ground-truth hard labels, misses the entire soft-target mechanism the paper introduces; that approach would give the student no more information than ordinary supervised training already provides. The option describing matching hidden-layer activations layer by layer, with temperature controlling how many layers must match, describes a different, feature-based distillation strategy, not the output-distribution matching Hinton et al. actually propose. The option describing matching raw, un-normalized output scores while claiming temperature has no effect during training gets the mechanism backwards: temperature is applied precisely during training, to both networks' distributions, and removing it removes the extra information distillation is designed to transfer.

Source: Hinton, Vinyals & Dean, "Distilling the Knowledge in a Neural Network" (2015), arXiv:1503.02531

Correct answer: .

RMSNorm keeps only the re-scaling half of LayerNorm's computation: it divides each activation by the root-mean-square of the summed inputs to that layer, without first subtracting the mean, so it never computes or removes the mean statistic that ordinary LayerNorm needs for re-centering. Zhang and Sennrich hypothesized that the re-centering invariance LayerNorm provides is not essential to its benefits, and that dropping it yields comparable model quality while cutting the normalization computation, which is why RMSNorm appears in later architectures such as LLaMA, T5, and Mistral. The option claiming RMSNorm drops the learnable gain parameter is wrong because RMSNorm still keeps a learned per-feature scale; what disappears is the centering step and its associated bias parameter, not the multiplicative gain. The option describing normalization applied only at the final block misdescribes both LayerNorm and RMSNorm, which are applied at every normalization point throughout the network, not just once at the end. The option describing normalization across tokens in a batch confuses RMSNorm with batch normalization; RMSNorm, like LayerNorm, normalizes across the feature dimension of a single token's own activations, independent of other tokens or batch members.

Source: Zhang & Sennrich, "Root Mean Square Layer Normalization" (2019), arXiv:1910.07467

Correct answer: .

Word2vec assigns each vocabulary word exactly one learned vector, fixed once training finishes, so a polysemous word such as one meaning both a river bank and a financial bank gets a single blended representation used everywhere it appears, regardless of the sentence around it. Self-attention breaks that limitation because every token's output representation at each layer is computed as a function of the other tokens it attends to, so the same word produces a different vector depending on its surrounding context, letting the model distinguish the different senses a static embedding table could never separate. The option claiming static embeddings cannot be trained on large corpora at all is factually backwards, since word2vec was specifically designed to train efficiently on very large corpora using only shallow, non-transformer neural networks. The option restricting static embeddings to nouns is invented; word2vec assigns a vector to every vocabulary token regardless of part of speech, static or contextual. The option claiming the only difference is vocabulary size ignores the actual mechanism entirely: the distinguishing feature is that contextual embeddings are computed per-occurrence from surrounding tokens, not that more words are covered.

Source: Mikolov et al., "Efficient Estimation of Word Representations in Vector Space" (2013), arXiv:1301.3781; Vaswani et al., "Attention Is All You Need" (2017)

Correct answer: .

Nucleus sampling first sorts the vocabulary by the probability the model assigns at that decoding step, then adds tokens from the top down until the cumulative probability first reaches the chosen threshold p, and samples the next token only from that dynamically sized set; because the set expands when the model spreads probability across many plausible continuations and shrinks when the model is confident in a narrow set of options, nucleus sampling avoids drawing from the long, unreliable low-probability tail that Holtzman et al. tie to greedy and beam search's tendency toward degenerate, repetitive text, without imposing an arbitrary fixed cutoff on how many tokens are eligible. The option describing sampling the single most probable token and re-ranking afterward with another model matches neither greedy decoding's own mechanism nor nucleus sampling, which never collapses to one deterministic choice by design. The option describing a fixed number k of top tokens chosen once before generation describes top-k sampling, the alternative the paper explicitly contrasts nucleus sampling against, since a constant k is either too restrictive when the model is uncertain or too permissive when it is confident. The option describing an exact, deterministic highest-log-probability search over the whole sequence describes neither sampling nor how beam search itself actually works, since beam search is an approximate search, not an exact one.

Source: Holtzman, Buys, Du, Forbes & Choi, "The Curious Case of Neural Text Degeneration" (2019), arXiv:1904.09751

Correct answer: .

Loshchilov and Hutter show that for adaptive optimizers such as Adam, adding an L2 penalty to the loss is not equivalent to true weight decay the way it is for plain SGD: the L2 term gets folded into the gradient before Adam's per-parameter adaptive scaling is applied, so parameters with large accumulated squared gradients (frequently or strongly updated ones) end up with their effective decay weakened, while parameters with small gradient history get decayed more strongly than intended, an uneven and unintended effect. AdamW fixes this by decoupling weight decay from the gradient-based update entirely, applying it as a separate multiplicative shrinkage directly to the parameters at each step, so every parameter receives the same, uniform decay regardless of its gradient history. The option claiming AdamW removes weight decay entirely is backwards, since decoupled decay is AdamW's defining feature, not its absence. The option claiming AdamW switches to a single global learning rate misdescribes the fix; AdamW keeps Adam's per-parameter adaptive learning rates for the gradient-based update and only decouples the separate decay term. The option describing decay applied only to bias terms and excluding weight matrices inverts common practice, where weight decay is typically applied to weight matrices and often excluded from biases and normalization parameters, not the reverse.

Source: Loshchilov & Hutter, "Decoupled Weight Decay Regularization" (2017/2019), arXiv:1711.05101

Correct answer: .

At the very start of training, the model's weights are close to their random initialization, gradients can be large and noisy, and an adaptive optimizer's running estimates of the gradient's mean and variance have seen very few updates and are still poorly calibrated; taking full-sized steps immediately can push the model into a badly conditioned region or cause outright divergence. A short warmup period, during which the learning rate rises gradually from near zero to its peak, lets the optimizer's statistics accumulate and the model settle into a more stable region before the largest steps are taken, after which a smooth decay (often cosine-shaped) gradually reduces the step size again as training approaches convergence. The option claiming warmup saves compute is wrong because the number of floating-point operations per step in a transformer's forward and backward pass does not depend on the learning rate at all; the learning rate only scales how large a step is taken, not how much computation the step costs. The option claiming warmup is unnecessary for adaptive optimizers is wrong, since Adam and AdamW's own moment estimates are exactly what warmup protects while they are still unreliable early on. The option describing a gradually growing context window describes a completely different technique (sequence-length curriculum), not the learning rate schedule the question asks about.

Source: Brown et al., "Language Models are Few-Shot Learners" (2020), arXiv:2005.14165, Appendix B (training details)

Correct answer: .

Micikevicius et al. keep an FP32 master copy of every weight specifically because FP16's limited precision cannot faithfully accumulate the very small per-step updates an optimizer applies over many thousands of steps; each step's update is computed and applied to this FP32 copy, which is then rounded down to FP16 for the forward and backward computation. Separately, they multiply the loss by a scaling factor before backpropagation so that gradient values, which can be very small, are shifted into FP16's representable range instead of underflowing to zero, and then divide the gradients by the same factor before the optimizer step uses them, a technique called loss scaling. The option describing FP16-only weight storage with no higher-precision copy is exactly the failure mode the FP32 master weights are designed to avoid, since updates would be lost to FP16 rounding. The option describing stochastic rounding of every value with no other change ignores the loss-scaling half of the method entirely, which is necessary because rounding alone does not address gradient underflow. The option describing two independently trained model copies averaged at inference describes model ensembling, an unrelated technique that does not address numeric precision during training at all.

Source: Micikevicius et al., "Mixed Precision Training" (2017), arXiv:1710.03740, ICLR 2018

Correct answer: .

Ordinary backpropagation through a deep network keeps every layer's forward-pass activations in memory until they are needed by the backward pass, so peak memory grows linearly with depth. Gradient checkpointing instead keeps only a sparse set of checkpointed activations during the forward pass and discards the rest; when the backward pass reaches a discarded activation, it recomputes it by re-running the forward computation from the nearest earlier checkpoint. Chen et al. show this lets memory scale roughly with the square root of the number of layers rather than linearly, at the cost of roughly one additional forward pass's worth of compute across the whole network. The option describing lower-precision activation storage describes mixed precision training, a different technique that does not change which activations are recomputed or when. The option describing discarding early layers' gradients after one use describes freezing layers, which would stop them from being trained further, not the memory-for-compute trade-off gradient checkpointing actually makes, since checkpointing still trains every layer normally. The option describing splitting the model across GPUs with no recomputation describes model parallelism, a distinct memory-reduction strategy that gradient checkpointing does not require and can be combined with rather than substitute for.

Source: Chen, Xu, Zhang & Guestrin, "Training Deep Nets with Sublinear Memory Cost" (2016), arXiv:1604.06174

Correct answer: .

A token at layer k can attend to tokens within W positions of it at layer k-1, but each of those tokens was itself computed by attending up to W positions further back at layer k-2, and so on down the stack; this recursive access means information from a token up to roughly W times the number of layers away can reach the current position by the final layer, even though no single layer ever attends beyond its own fixed window W. Because each layer only ever needs the most recent W tokens' worth of key-value pairs rather than the full history, Mistral can use a rolling buffer cache of fixed size W, overwriting the oldest cached entries as the window slides forward, instead of a cache that grows with total sequence length. The option describing randomly centered windows covering the whole sequence within one layer misdescribes sliding window attention, which is a fixed local window around each token's own position, not a randomized or global one. The option claiming SWA does not extend the receptive field and relies on a separate global-attention mechanism describes a different design (closer to Longformer's local-plus-global hybrid), not Mistral's approach, which achieves its extended range purely through layer stacking. The option claiming no key-value cache is needed at all is wrong, since a rolling cache is still required and is exactly what the fixed window size enables to be bounded rather than eliminated.

Source: Jiang et al., "Mistral 7B" (2023), arXiv:2310.06825

Correct answer: .

Encoder-only models, of which BERT is the canonical example, apply bidirectional self-attention so every position can attend to every other position in the input, which suits tasks that classify or extract information from text the model has fully in front of it, such as sentiment classification or span extraction, but does not by itself define a natural way to generate new open-ended text. Decoder-only models, of which GPT is the canonical example, use causal self-attention so each position can only attend to earlier positions, which is exactly the constraint needed to generate text one token at a time from left to right. Encoder-decoder models, including the original Transformer and later models such as T5, keep both halves: an encoder that processes the full input with bidirectional attention, and a separate decoder that attends causally over its own generated output while also attending to the encoder's output, a structure well suited to sequence-to-sequence tasks like translation or summarization where an input sequence is transformed into a different output sequence. The option swapping GPT and BERT's roles and calling encoder-decoder models deprecated inverts the correct assignment and misstates encoder-decoder models' continued use. The option claiming the families differ only by layer count and the option claiming the labels only refer to software frameworks both ignore the actual architectural and attention-pattern differences the three families are defined by.

Source: Vaswani et al., "Attention Is All You Need" (2017); Devlin et al., "BERT" (2018); Radford et al., "Improving Language Understanding by Generative Pre-Training" (GPT, 2018); Raffel et al., "Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer" (T5, 2020)

Correct answer: .

BERT's masked language modeling objective randomly selects about 15% of input tokens, of which most are replaced with a special mask token, a smaller portion are replaced with a random other token, and the remainder are left unchanged, and the model is trained to predict the original token at each of those selected positions using bidirectional self-attention over the entire surrounding sequence in both directions. GPT's causal language modeling objective instead predicts each token from only the tokens that came before it, enforced by a causal attention mask that prevents any position from attending to later positions, so nothing in the input itself needs to be corrupted or masked the way BERT's does; the restriction is purely on what each position is allowed to attend to. The option claiming both objectives are identical, using only leftward context, ignores BERT's defining bidirectional mechanism entirely. The option claiming BERT predicts every unmasked token while GPT predicts only a random 15% inverts which model actually uses the 15% masking rate, which belongs to BERT, not GPT. The option claiming both models use identical causal self-attention and differ only in training document length ignores the fundamental difference in attention direction (bidirectional versus causal) that defines the two objectives.

Source: Devlin, Chang, Lee & Toutanova, "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding" (2018), arXiv:1810.04805

Correct answer: .

The softmax function converts the raw attention scores (logits from query-key dot products) into a valid probability distribution where all weights are non-negative and sum to 1. This means each token's representation is a weighted combination of all value vectors, with the weights reflecting learned relevance. Without softmax, the raw scores could be arbitrarily large or negative, making the weighted sum unstable and uninterpretable. The option about eliminating negatives is partially true (softmax outputs are non-negative) but mischaracterises the purpose — softmax normalises, it does not simply clip negatives. The option about equal weights describes mean pooling, not attention. The option restricting softmax to training is wrong — softmax is applied at every forward pass, including inference.

Source: Vaswani et al. (2017), 'Attention Is All You Need', Section 3.2.1

Correct answer: .

Fine-tuning a smaller model on a high-quality domain-specific dataset encodes the desired behavior directly into the model's weights, producing reliable outputs for narrow tasks at lower inference cost. Research consistently shows that a well-fine-tuned small model can match or exceed a much larger general model on the specific task it was trained for. The option about maximum flexibility favours the larger general-purpose model, since fine-tuning specialises a model and can reduce its breadth. The option about no labelled data makes fine-tuning infeasible, again favouring in-context learning with the larger model. The option about broad multi-step reasoning also favours the larger model, which has more parameters to store world knowledge and more capacity for complex reasoning chains.

Source: Anthropic, 'When to Fine-tune vs. Prompt', docs.anthropic.com; OpenAI Fine-tuning Guide

Correct answer: .

Shazeer's SwiGLU sublayer computes two separate linear projections of the sublayer's input, applies the Swish (SiLU) activation function to one of these projections, and takes the element-wise product of that activated projection with the other (un-activated) projection, before a final output projection maps the result back down; this element-wise product means the un-activated projection's units are "gated" by how large the corresponding Swish-activated unit is, letting the network learn to pass through more or less information per unit rather than applying one fixed nonlinearity uniformly. The option describing removal of any nonlinearity is wrong because a nonlinearity is still applied, just to only one of the two projections before gating, and residual connections don't substitute for this role. The option describing an added third layer is wrong because SwiGLU changes what the existing feed-forward sublayer computes internally rather than adding a wholly new stacked layer. The option describing a fixed, position-derived gating function is wrong because the gating values come from a fully learned linear projection of the token's own hidden representation, not from a non-trainable function of position, which would instead describe something closer to a positional encoding scheme.

Source: Shazeer, "GLU Variants Improve Transformer" (arXiv:2002.05202, 2020); adoption confirmed in LLaMA (Touvron et al., 2023) and PaLM (Chowdhery et al., 2022) architecture sections.

Correct answer: .

Shaw et al. (2018) modify self-attention itself: for each pair of positions, they look up a learned embedding indexed by the clipped relative distance between the query and key positions (distances beyond a maximum window are clamped to the same embedding), and add this distance-dependent term into both the attention score computation and the weighted sum over value vectors at every layer, rather than injecting position information once at the input via a fixed or learned absolute positional vector. This lets attention directly represent how far apart two tokens are, at every layer, rather than requiring the network to reconstruct relative distance indirectly from absolute positions added at the bottom. The option describing reliance on recurrent connections is wrong because the Transformer, with or without this relative-position modification, contains no recurrence; sequence order must come from some explicit positional signal, and relative position representations are exactly one such signal. The option describing a concatenated one-hot absolute-position vector is wrong because it describes a naive absolute-position scheme quite different from Shaw et al.'s pairwise, relative-distance embeddings, and one-hot position encoding was never their method. The option describing multiplying a query vector by a rotation matrix whose angle depends on absolute position describes RoPE (Su et al., 2021), a distinct, later technique that encodes position through rotation rather than through Shaw et al.'s learned additive distance embeddings.

Source: Shaw, Uszkoreit & Vaswani, "Self-Attention with Relative Position Representations" (NAACL 2018), arXiv:1803.02155

Correct answer: .

Lee et al. found that large web-scraped corpora such as C4 contain many near-duplicate documents and long repeated substrings, and that models trained on this duplicated data emit memorized text copied verbatim from training data at a measurably higher rate. After deduplicating the training data with their near-duplicate and exact-substring detection tools, they found models emit memorized text roughly ten times less often, and reach the same or better validation accuracy using fewer training steps overall -- deduplication is not just a memorization fix but also improves training efficiency. The option claiming no measurable effect is wrong because the paper documents concrete, large duplication rates (for example, over a percent of C4 being near-duplicate) and a corresponding measurable reduction in memorization after deduplication. The option claiming deduplication increases memorization is wrong and inverts the paper's actual finding; concentrating training on fewer, non-duplicated examples did not increase verbatim memorization of what remained. The option restricting the effect to evaluation alone is wrong because, while the paper separately notes that deduplication reduces train-test overlap and thus makes evaluation more accurate, it also reports the training-side effects on memorization rate and training efficiency described above, not just an evaluation-side correction.

Source: Lee et al., "Deduplicating Training Data Makes Language Models Better" (ACL 2022), arXiv:2107.06499

Correct answer: .

Petrov et al. measured tokenized length of parallel text (the same content translated across many languages) using tokenizers from widely deployed LLMs, and found disparities of up to roughly 15 times as many tokens for the same content depending on the language, a disparity that persists even for tokenizers explicitly trained with multilingual support in mind. Because commercial LLM usage is commonly billed and rate-limited per token, and because a model's fixed context window holds a fixed number of tokens rather than a fixed amount of text, this means speakers of more heavily-tokenized languages pay more, wait longer, and can fit less content into the same context window for equivalent content, which the paper frames as a fairness concern. The option describing only a trivial, sub-10%-variation is wrong and understates the paper's actual, much larger measured disparities. The option attributing the disparity entirely to character-set size and claiming it vanishes when measured in bytes is wrong; the paper's point is specifically about token counts under real deployed tokenizers, and byte-level measurement is a different, tokenizer-independent yardstick that doesn't resolve the token-based cost and context disparities users actually face. The option claiming multilingual training from scratch eliminates the disparity is wrong because the paper found the unfairness persists even in tokenizers already built for multilingual support, not only in ones retrofitted after the fact.

Source: Petrov, La Malfa, Torr & Bibi, "Language Model Tokenizers Introduce Unfairness Between Languages" (NeurIPS 2023), arXiv:2305.15425

Correct answer: .

Vaswani et al.'s Table 1 reports that a self-attention layer connects all positions with a constant, O(1), number of sequentially executed operations (all pairwise interactions are computed in parallel within one layer), and that its maximum path length between any two input and output positions is likewise O(1), the same short distance regardless of how far apart the two positions are in the sequence; a recurrent layer, in contrast, needs O(n) sequential operations and has a maximum path length of O(n), since information must pass step-by-step through every intervening position. A shorter, position-independent path length matters because it makes it easier for gradients and information to flow between distant positions during learning, which the paper cites as one motivation for self-attention over recurrence for capturing long-range dependencies. The option describing O(n) sequential operations for self-attention is wrong and instead describes the recurrent layer's own property; self-attention's sequential-operation count in the table is constant, not linear in sequence length. The option treating self-attention and convolutional layers as identical is wrong because the table gives convolutional layers a different, kernel-size-dependent maximum path length (logarithmic in sequence length for stacked dilated convolutions, not constant) and a different per-layer complexity expression. The option claiming self-attention requires more sequential operations than a recurrent layer is wrong and reverses the table's actual comparison; the paper's caveat about sequence length exceeding representation dimensionality applies to the separate complexity-per-layer comparison (self-attention's per-layer complexity is favorable when n is smaller than d), not to the sequential-operations or path-length comparisons this question asks about.

Source: Vaswani et al., "Attention Is All You Need" (2017), arXiv:1706.03762, Table 1 and Section 4

Correct answer: .

Xiao et al. found that in autoregressive language models, the first few tokens of a sequence consistently absorb a disproportionately large share of attention across many layers and attention heads, regardless of whether those tokens carry any relevant semantic content for later text -- a phenomenon they term "attention sinks," which likely exists because the softmax over attention scores must sum to one, and the model learns to dump unneeded attention mass onto these easily-identified early positions rather than distributing it meaningfully. Once a naive sliding-window cache evicts these first tokens to make room for new ones, this stabilizing target disappears and the model's perplexity and output quality collapse. StreamingLLM fixes this cheaply, without any fine-tuning, by always retaining a handful of these initial "sink" tokens in the KV cache in addition to the usual sliding window of recent tokens, which keeps attention well-behaved indefinitely. The option about positional encodings is wrong because the paper's fix does not recompute positional encodings for the whole cache; it instead keeps positions consistent within the retained window and sink tokens. The option about vocabulary shrinkage is wrong because a sliding window doesn't restrict which tokens the model's vocabulary contains, only which positions remain visible to attention, and StreamingLLM's fix doesn't involve randomly reinserting evicted tokens. The option attributing the collapse purely to a training/inference length mismatch and fixing it via fine-tuning on very long sequences is wrong because StreamingLLM's headline contribution is precisely that it avoids any such fine-tuning, extending models to effectively unbounded stream lengths using only this fixed-size cache change.

Source: Xiao, Tian, Chen, Han & Lewis, "Efficient Streaming Language Models with Attention Sinks" (ICLR 2024), arXiv:2309.17453

Correct answer: .

Power et al. observed that on small algorithmic datasets, after a network's training accuracy has already reached (or nearly reached) 100% while its validation accuracy remains stuck near chance level, continuing to train far past this point can eventually produce a sudden, delayed rise in validation accuracy up to near-perfect generalization, well after the point where the training loss had already appeared to converge; they named this delayed transition "grokking." They also found smaller training datasets required substantially more optimization steps before this transition occurred. The option claiming validation accuracy never improves is wrong and describes the opposite of the paper's central, titular finding. The option claiming training accuracy declines and calling this "double descent" is wrong on two counts: it misdescribes what happens to training accuracy (which stays high, not declining) and misapplies "double descent," which is a different, well-known phenomenon about test error non-monotonically depending on model or dataset size, not this training-duration-based transition. The option describing endless oscillation and calling it "catastrophic forgetting" is wrong; catastrophic forgetting refers to a trained model losing previously learned capability when trained on new data or tasks, an unrelated phenomenon to the single-task, delayed-generalization pattern this paper documents.

Source: Power, Burda, Edwards, Babuschkin & Misra, "Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets" (2022), arXiv:2201.02177

Correct answer: .

Berglund et al. fine-tuned GPT-3 and Llama-1 on invented statements of the form "[Name] is the [role] of [thing]" and found that, despite the models correctly answering questions about the original direction they were trained on, they failed to answer the reversed direction (e.g., asking who holds that role for that thing) at a rate no better than chance, a failure they term the "Reversal Curse." Crucially, they also found this failure specifically applies to recalling the relationship purely from parameters learned during fine-tuning: if both directions of the same relationship are given together within the model's context at inference time, the model can read off the reverse relationship directly from what's in front of it rather than needing to recall it from training, so the reversal failure specifically concerns a gap in what fine-tuning generalizes into the model's weights, not a gap in the model's in-context reasoning ability generally. The option claiming symmetric generalization is wrong and states the opposite of the paper's central finding. The option about real versus fictitious facts is wrong; the paper deliberately used fictitious statements to rule out the possibility that models were merely recalling the reverse direction from separate exposure during pretraining, and this is a control, not the condition under which reversal succeeds. The option restricting the failure to non-numerical, named-entity statements is wrong; the paper's core claim is about the general direction-dependence of what fine-tuning teaches, not a distinction between quantitative and named-entity content.

Source: Berglund et al., "The Reversal Curse: LLMs Trained on 'A is B' Fail to Learn 'B is A'" (ICLR 2024), arXiv:2309.12288

Correct answer: .

Yang et al. formalize the problem of predicting the next word given a context as a matrix factorization problem: the ideal matrix of true log-probabilities, over all contexts and all vocabulary words, has some true rank, but a standard softmax layer's context-to-word log-probability matrix is mathematically constrained to have rank no higher than the size of the model's hidden dimension, since it's produced from a single linear projection of a single hidden vector per context; when the true distribution's effective rank exceeds this bound, no amount of additional training data or optimization can let a single softmax represent it exactly, a limitation they call the "softmax bottleneck." Their Mixture of Softmaxes (MoS) works around this by computing several distinct softmax distributions from the same underlying hidden state (each using its own linear projection), then combining these distributions in a learned, context-dependent weighted average, which can represent a higher-rank matrix than any single softmax term alone, improving perplexity on standard benchmarks. The option about softmax computation speed is wrong; the paper's bottleneck is about the softmax's representational capacity (its output rank), not its computational cost, and MoS doesn't approximate any normalizing constant. The option about floating-point precision is wrong and misidentifies an unrelated numerical-precision issue as the paper's actual, rank-based argument. The option describing the bottleneck as overfitting from weight tying, fixed by shrinking the hidden dimension, is wrong and inverts the paper's proposed fix, which combines multiple softmaxes to increase expressiveness rather than reducing the hidden dimension to reduce parameters.

Source: Yang, Dai, Salakhutdinov & Cohen, "Breaking the Softmax Bottleneck: A High-Rank RNN Language Model" (ICLR 2018), arXiv:1711.03953

Correct answer: .

Chinchilla's scaling analysis (Hoffmann et al., 2022) answers a specific, narrower question: for a fixed training compute budget, what model size and token count minimizes training loss. It says nothing about inference cost, which for a widely-deployed model can dwarf training cost over the model's lifetime, since inference cost scales with the number of parameters served times the number of requests, not with how the model was trained. Touvron et al.'s LLaMA paper explicitly reasoned about this: given that a smaller model is cheaper and faster to run at inference time, it can be worth spending more training compute than Chinchilla-optimal to train that smaller model on many more tokens than Chinchilla's ratio would recommend, continuing to see loss improve well past the Chinchilla-optimal point for that smaller size, in exchange for a model that is both strong and cheap to serve at scale. The option claiming Chinchilla contained a corrected arithmetic error is wrong; Touvron et al. did not claim to have found an error in Hoffmann et al.'s analysis, only to be optimizing for a different objective (inference cost included) than pure training-compute-optimality. The option claiming a smaller, longer-trained model always strictly beats a larger Chinchilla-optimal model on training loss for the same compute is wrong; Chinchilla's whole point is that its ratio is loss-minimizing for a given training compute budget, so deliberately deviating from it to train a smaller model longer means using more total training compute, not matching it. The option attributing the change to now-larger available datasets making Chinchilla's analysis obsolete is wrong; the actual argument for training beyond Chinchilla-optimal is about weighing inference cost against training cost, not about dataset size having outgrown the original analysis's assumptions.

Source: Touvron et al., "LLaMA: Open and Efficient Foundation Language Models" (2023), arXiv:2302.13971, Section 1; Hoffmann et al., "Training Compute-Optimal Large Language Models" (2022), arXiv:2203.15556

Correct answer: .

WordPiece scores each candidate merge by how much it would increase the likelihood of the training corpus under a language model, computed as the probability of the merged unit divided by the product of the probabilities of its two parts considered separately. This means a pair that is only moderately frequent can still be merged ahead of a more frequent pair if combining it captures a strong statistical dependency between its parts, letting WordPiece favor subwords that are informative rather than merely common. BPE, by contrast, always merges whichever adjacent pair has the highest raw frequency, with no notion of how much that merge actually helps predict the data. The option claiming the two algorithms always produce identical vocabularies is wrong because it collapses WordPiece's likelihood criterion into BPE's frequency criterion, erasing the exact distinction the question asks about. The option describing a fixed-size character n-gram model is wrong because WordPiece, like BPE, builds its vocabulary through iterative merging of symbols into longer subwords rather than using static n-grams. The option requiring human-labeled word segmentations is wrong because WordPiece's likelihood criterion is computed directly from unlabeled training text using a language model fit to that text, the same kind of raw corpus BPE itself trains on.

Source: Schuster & Nakajima, "Japanese and Korean Voice Search" (2012); Devlin et al., "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding" (2018), arXiv:1810.04805, Section 3.2

Correct answer: .

Sparse Transformer's factorized attention splits the full attention pattern across heads into a local pattern, where each position attends to a fixed-size window of nearby positions, and a strided pattern, where each position attends to positions spaced at regular fixed intervals further back in the sequence. Neither pattern alone directly connects every pair of positions within one layer, but because the model stacks several such layers, information from a given position can reach a far-away position by hopping through intermediate positions reached by local connections in some layers and strided connections in others, much like how stacking several layers of short-range dilated convolutions builds a long receptive field without any single layer looking at the whole input. This is what lets Child et al. reduce complexity from the quadratic cost of full attention to O(n√n) while still approximating dense attention's effective connectivity. The option describing full attention at reduced precision is wrong because it keeps every position attending to every other position, which is exactly the quadratic cost the sparse factorization is designed to avoid. The option claiming the strided pattern is discarded during training is wrong because both patterns are used throughout training, not added only at inference. The option claiming the strided pattern alone connects every pair of positions within a single layer is wrong because the strided pattern only reaches positions at fixed stride intervals, not arbitrary positions, which is precisely why the local pattern and depth are both needed for full reachability.

Source: Child, Gray, Radford & Sutskever, "Generating Long Sequences with Sparse Transformers" (2019), arXiv:1904.10509, Section 4

Correct answer: .

MLA's low-rank joint compression projects each token's keys and values down into a single compact latent vector using a learned down-projection matrix, and only this latent vector is written to the KV cache, rather than separate full-dimensional keys and values for every head. At attention time, learned up-projection matrices expand that one stored latent vector back out into the distinct per-head keys and values each head needs, so the heads still attend using different effective keys and values even though only one shared low-dimensional vector was ever cached per token. Because cache size scales with the latent dimension rather than with the number of heads times the per-head dimension, DeepSeek-V2 reports a far smaller KV cache than standard multi-head attention at the same model size, reported as roughly a 93% reduction relative to its predecessor. The option describing 8-bit quantization of full per-head keys and values is wrong because MLA's saving comes from storing a lower-dimensional latent representation, not from quantizing already-full-dimensional cached tensors, and it still requires up-projection to reconstruct per-head values, not none at all. The option describing a single shared key head and value head across all heads is wrong because that describes multi-query attention, which MLA is explicitly contrasted against; MLA reconstructs distinct per-head keys and values from the shared latent rather than having every head literally use the same key and value. The option describing full recomputation at every step is wrong because MLA still caches something, the compressed latent vector, rather than caching nothing and recomputing everything from scratch.

Source: DeepSeek-AI, "DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model" (2024), arXiv:2405.04434, Section 2.1 (Multi-Head Latent Attention)

Correct answer: .

Tensor parallelism, as Shoeybi et al. implement it in Megatron-LM, splits the individual weight matrices inside a layer, such as the matrices in a feed-forward or attention sublayer, into column-wise or row-wise partitions distributed across GPUs, so that no single GPU holds a complete copy of every weight matrix; computing that layer's output requires the participating GPUs to exchange partial results, typically via an all-reduce operation, before the computation can proceed. This differs from data parallelism, where every GPU already holds a full copy of the whole model and parallelism instead comes from splitting the training batch across GPUs. The option describing data parallelism itself, with the model fully replicated and only the batch split, is wrong because it describes the baseline approach tensor parallelism is introduced as an alternative to, not what tensor parallelism changes. The option describing different layers assigned to different GPUs that process data sequentially describes pipeline (inter-layer) parallelism, a distinct distributed-training strategy that splits the model by depth rather than splitting individual weight matrices within a layer. The option describing training several independent models and averaging their final weights is wrong because tensor-parallel GPUs are jointly computing the output of one single model's layers in real time during every forward and backward pass, not training separate models that are only reconciled at the end.

Source: Shoeybi, Patwary, Puri, LeGresley, Casper & Catanzaro, "Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism" (2019), arXiv:1909.08053, Section 3

Correct answer: .

Olsson et al. identify induction heads as attention heads that implement a simple copy-and-complete pattern: given the current token, the head searches back through the context for an earlier occurrence of that same token, finds whatever token immediately followed it that earlier time, and boosts the probability that the same token will follow again now. This single mechanism accounts for a large share of a transformer's ability to pick up patterns from examples placed in its prompt, with no gradient update needed. The authors further report a sharp phase change early in training, a narrow window during which induction heads form and in-context learning ability jumps abruptly, a pattern they observed consistently from small, attention-only toy models up through 13-billion-parameter language models. The option describing retrieval of the single most similar pretraining example is wrong because induction heads operate entirely within the current context window by matching tokens already present in the prompt, not by searching pretraining data, and the paper's central finding is an abrupt phase change, not smooth, gradual improvement. The option restricting induction heads to numerical and tabular data is wrong because the copy-and-complete pattern they implement operates on tokens generally, including the ordinary natural-language few-shot completion behavior GPT-3 was shown to exhibit. The option claiming induction heads only appear after a separate fine-tuning stage is wrong because Olsson et al. study them forming during ordinary pretraining itself, well before any instruction-tuning or fine-tuning stage would occur.

Source: Olsson, Elhage, Nanda, Joseph, DasSarma, Henighan et al. (Anthropic), "In-context Learning and Induction Heads" (2022), Transformer Circuits Thread

Correct answer: .

Mamba's selection mechanism makes the state space model's transition and input-projection matrices depend on the current input token, rather than holding them fixed as constants shared across every token the way linear-time-invariant (LTI) SSMs such as S4 do. This restores content-based reasoning: the model can selectively propagate, compress, or discard information in its recurrent state depending on what each token actually is, something an LTI system mathematically cannot do since it must treat every input the same way. Making the matrices input-dependent breaks the convolutional, FFT-parallelizable view that LTI SSMs rely on for efficient training, so Mamba instead introduces a hardware-aware parallel scan algorithm to keep training efficient, while still achieving linear-time scaling in sequence length, unlike attention's quadratic cost. The option describing a switch to full quadratic self-attention is wrong because it would surrender exactly the subquadratic scaling Mamba is designed to preserve, and the paper's whole motivation is to match attention's modeling power without paying its quadratic cost. The option describing a fixed, content-independent LSTM-style gate added on top of unchanged constant matrices is wrong because it does not make the transition dynamics themselves depend on the input at all; Mamba's change is to the matrices' values, not an added fixed gating layer around still-constant matrices. The option describing enlarged but still fixed matrices is wrong because the paper attributes the modeling weakness specifically to the matrices being input-independent, not to insufficient size; scaling up fixed constants cannot make them respond to a given token's content.

Source: Gu, A. & Dao, T. (2023), "Mamba: Linear-Time Sequence Modeling with Selective State Spaces," arXiv:2312.00752

Correct answer: .

GPipe splits each training mini-batch into a number of smaller micro-batches and feeds them one after another into the layer-partitioned pipeline, so that as soon as an accelerator finishes processing one micro-batch's forward or backward computation and passes its output to the next (or previous) stage, it can immediately begin the next micro-batch rather than sitting idle waiting for the whole batch to clear the pipeline. GPipe schedules all micro-batches' forward passes first, then runs all their backward passes; the remaining idle time at the start and end of each step, the pipeline 'bubble,' shrinks as the number of micro-batches grows relative to the number of pipeline stages, though it is never eliminated entirely. The option describing a full model copy on every accelerator is wrong because it describes data parallelism, the opposite setup from the layer-partitioned scheme the question describes, and it would not even fit if no single accelerator can hold the whole model. The option describing splitting individual weight matrices within each layer across accelerators is wrong because it describes tensor (intra-layer) parallelism, the Megatron-LM style of splitting a layer's own matrices rather than pipelining whole layers across micro-batches; it solves a different problem (fitting one oversized layer) via a different mechanism. The option describing discarding and recomputing activations is wrong because that is GPipe's separate activation-recomputation (re-materialization) technique for reducing memory footprint, not its mechanism for keeping pipeline stages busy during a training step.

Source: Huang, Y., Cheng, Y., Bapna, A. et al. (2018), "GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism," arXiv:1811.06965

Correct answer: .

BPE-dropout keeps the same learned BPE vocabulary and merge rules, but during training it randomly drops out, with a small fixed probability, some of the merges that would otherwise apply at each step of the standard BPE procedure; when a merge is dropped, the algorithm is forced to leave that position split into smaller subwords instead, so the same word can come out segmented differently from one training pass to the next, all while remaining fully compatible with ordinary deterministic BPE used unchanged at inference. Training on this variety of segmentations makes the model less dependent on any single fixed split of a word and more robust to segmentation variation, which Provilkov et al. report improves translation quality by up to 3 BLEU over standard BPE. The option describing a switch to a Unigram language model is wrong because that describes Kudo's distinct SentencePiece algorithm, which the paper explicitly contrasts as the prior way of obtaining multiple segmentations; BPE-dropout instead stays within the BPE merge procedure itself rather than switching algorithms. The option describing a separate neural segmentation model is wrong because no additional model is trained at all; the randomness is injected directly into the existing deterministic BPE merge steps. The option describing reordering merges is wrong because BPE-dropout does not reorder anything; it stochastically omits certain merges outright, which changes the final token boundaries, rather than merely processing an unchanged set of merges in a different sequence.

Source: Provilkov, I., Emelianenko, D. & Voita, E. (2020), "BPE-Dropout: Simple and Effective Subword Regularization," ACL 2020, arXiv:1910.13267

Correct answer: .

By replacing softmax(QK^T) with phi(Q)phi(K)^T for a kernel feature map phi, attention's output can be rewritten, via the associativity of matrix multiplication, as phi(Q) multiplied by a running sum of phi(K_i) outer-product V_i terms accumulated across positions, without ever forming the full N-by-N pairwise similarity matrix; this drops both the time and memory cost of computing attention over a sequence of length N from quadratic to linear. For autoregressive generation, that running sum plays the same role as an RNN's hidden state: at each new token the model updates the single accumulated sum with that position's contribution and reads off the next output in constant time per step, instead of recomputing attention over the whole growing sequence, which the authors report yields speedups of up to 4000x at long sequence lengths. The option describing an unchanged quadratic computation with the similarity matrix formed first is wrong because the entire point of the kernel reformulation is to avoid ever forming that matrix, not to compute it and then discard it from memory. The option describing a fixed sparse subset of key positions is wrong because that describes sparse or windowed attention approaches, such as the Sparse Transformer or Longformer, which change which positions get attended to; linear attention still lets every query attend to every prior key, just computed through the kernel trick rather than restricted. The option describing a fused GPU kernel that reduces memory reads and writes while keeping the same quadratic computation is wrong because that describes FlashAttention's exact, IO-aware approach, the opposite strategy from Katharopoulos et al.'s, who change the mathematical form of attention itself to eliminate quadratic cost rather than only optimizing its memory-access pattern.

Source: Katharopoulos, A., Vyas, A., Pappas, N. & Fleuret, F. (2020), "Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention," ICML 2020, arXiv:2006.16236

Correct answer: .

Adaptive softmax exploits the highly unbalanced, roughly Zipfian frequency distribution of words in natural-language corpora: it partitions the vocabulary into a small head cluster containing the most frequent words plus a compact set of placeholder symbols, one per lower-frequency cluster, and one or more much larger tail clusters of progressively rarer words. Computing the head cluster's full softmax is cheap since it is small, and each much larger tail cluster's own full softmax over its member words only needs to be computed on the comparatively rare training examples whose actual target word falls in that cluster; this lowers the average computational cost per position far below that of one full softmax over the entire vocabulary every single time, and the clustering is deliberately structured around matrix-matrix operations efficient on GPUs. The option describing a sigmoid applied independently to each word is wrong because that describes a different strategy, closer to candidate sampling or noise-contrastive estimation, which discards the shared normalization structure entirely rather than adaptive softmax's actual approach of keeping normalized mini-softmaxes within a cluster hierarchy. The option describing merging rare words into a shared unknown-word token is wrong because adaptive softmax does not discard or collapse any vocabulary word; every word remains individually distinguishable, just scored through a cheaper cluster hierarchy rather than removed from the vocabulary. The option describing moving computation to the CPU is wrong because the method's explicit motivation, named in its own title, is to be efficient specifically on GPUs through its clustering structure, not to avoid the GPU altogether.

Source: Grave, E., Joulin, A., Cissé, M., Grangier, D. & Jégou, H. (2017), "Efficient softmax approximation for GPUs," ICML 2017, arXiv:1609.04309