Fine-tuning & Model Customisation

Why B? And why not the others?

Correct answer: B. It freezes all of the pretrained weight matrices and, for selected layers, injects a pair of much smaller trainable matrices whose low-rank product approximates the update those weights would otherwise need, so only that pair is trained

LoRA leaves the pretrained weight matrices frozen and, for the layers it targets, adds a new pair of much smaller matrices -- one that projects down to a low intermediate rank and one that projects back up -- whose product forms a low-rank approximation of whatever weight update the task would otherwise require; only the two small matrices are trained, which is why the trainable parameter count drops so sharply while the frozen base weights carry over unchanged. The option describing an ordinary full fine-tune with a smaller learning rate is wrong because a smaller learning rate does not shrink the number of parameters being updated, only the size of each update, so it does nothing to reduce the underlying memory and compute cost. The option describing deleting and retraining the attention layers from scratch is wrong because LoRA never removes or replaces any pretrained component; it augments the existing frozen weights alongside a small addition rather than discarding learned structure. The option describing full-parameter backpropagation on 8-bit integer weights is wrong because that describes a form of quantized full fine-tuning, not LoRA's low-rank decomposition, and still requires updating every original parameter rather than a small injected pair.

Source: Hu et al., 'LoRA: Low-Rank Adaptation of Large Language Models' (arXiv:2106.09685, 2021)

Why D? And why not the others?

Correct answer: D. It quantizes the frozen base model's weights to a 4-bit NormalFloat data type for storage while computing in a higher-precision type, trains LoRA adapters on top of those frozen quantized weights, and adds double quantization of the quantization constants plus paged optimizers to absorb memory spikes

QLoRA's actual recipe is to store the frozen base model's weights in a 4-bit NormalFloat (NF4) format -- an information-theoretically tuned datatype for normally distributed weights -- while dequantizing on the fly to a higher-precision compute type for the actual forward and backward math, then training ordinary LoRA adapters on top of those frozen quantized weights; it further saves memory by quantizing the quantization constants themselves (double quantization) and uses paged optimizers built on unified GPU memory to absorb the memory spikes that occur when processing long sequences. The option describing full 4-bit computation throughout both passes is wrong because QLoRA's 4-bit format is used only for storing the frozen base weights, not for the arithmetic itself, which is why it can match 16-bit fine-tuning quality. The option describing permanently zeroing small-magnitude weights describes a pruning technique, not quantization, and QLoRA does not delete any of the base model's weights. The option describing training a smaller distilled student from scratch describes knowledge distillation, an entirely different technique that discards the original model's exact parameters rather than adapting them.

Source: Dettmers, Pagnoni, Holtzman & Zettlemoyer, 'QLoRA: Efficient Finetuning of Quantized LLMs' (arXiv:2305.14314, 2023)

Why A? And why not the others?

Correct answer: A. It uses the closed-form relationship between an optimal KL-regularized policy and its implied reward under the Bradley-Terry preference model to rewrite the reward directly in terms of the policy itself, so the policy can be trained straight from the preference pairs with a single supervised-style loss, with no separate reward model and no online RL sampling loop

DPO's key mathematical move is to observe that, under the Bradley-Terry model of preferences, the reward implied by the optimal KL-regularized policy can be expressed purely in terms of that policy's own probabilities relative to a reference model; substituting this expression back into the standard preference loss produces a single supervised classification-style loss on the preference pairs themselves, which means the policy is trained directly, with no separate reward model ever fit and no online rollout-and-update RL loop like PPO required at all. The option describing still fitting a full reward model and then applying a closed-form update is wrong because DPO's entire point is to avoid fitting any separate reward model; the reward is never estimated as its own object. The option describing discarding human preference pairs in favor of self-critique against written principles describes a Constitutional-AI-style approach, not DPO, which is built specifically to consume labeled preference pairs. The option describing keeping the full RLHF pipeline intact and only swapping the reward model's architecture is wrong because it retains exactly the separate-reward-model-plus-RL-loop structure that DPO was designed to eliminate.

Source: Rafailov, Sharma, Mitchell, Ermon, Manning & Finn, 'Direct Preference Optimization: Your Language Model is Secretly a Reward Model' (arXiv:2305.18290, 2023)

Why C? And why not the others?

Correct answer: C. This is catastrophic forgetting: fine-tuning updates the model's weights to optimize performance on the new, narrow task distribution, and in doing so can overwrite representations the model relied on for previously learned capabilities; this effect tends to become more pronounced the further the fine-tuning data diverges from the pretraining distribution and can worsen with more fine-tuning steps or a higher learning rate

Catastrophic forgetting occurs because gradient updates during fine-tuning reshape the same shared weights the model uses for every capability; optimizing those weights hard toward a new, narrow task distribution can overwrite or distort the representations the model was previously relying on for unrelated skills like arithmetic, even though no arithmetic examples appeared in the new training data. This effect is documented to become more severe as the fine-tuning distribution diverges further from the original pretraining distribution, and can be worsened by training for more steps or with a higher learning rate, both of which push the weights further from their pretrained configuration. The option attributing this to memorized label noise is wrong because forgetting is a structural side effect of weight updates, not a symptom of bad labels in the fine-tuning set. The option attributing this to a tokenizer mismatch is wrong because fine-tuning as described does not alter the model's vocabulary or tokenization scheme. The option calling it a benchmark artifact is wrong because the described capability loss is a real, well-studied consequence of how fine-tuning updates shared parameters, not a measurement error.

Source: Luo et al., 'An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuning' (arXiv:2308.08747, 2023)

Why B? And why not the others?

Correct answer: B. Instruction tuning substantially improved zero-shot performance on the held-out, unseen tasks, in some cases surpassing a much larger model's zero-shot performance and even beating that larger model's few-shot performance on several benchmarks; the gains depended on tuning across a large number of diverse task types, not just scaling up the amount of data for a single task

FLAN fine-tuned a 137-billion-parameter pretrained model on over 60 NLP tasks verbalized as natural-language instructions and then tested it, with no examples given, on tasks held out from that tuning set entirely; the tuned model's zero-shot performance improved substantially, surpassing a much larger untuned model's zero-shot results on most evaluated tasks and even beating that larger model's few-shot performance on several benchmarks, with ablations showing the number of distinct task types used in tuning -- not just raw data volume -- was central to the gain. The option claiming the benefit required examples at test time is wrong because the whole point of the finding is that the gains showed up under a strictly zero-shot evaluation. The option claiming the model only improved on the exact tasks it was tuned on is wrong because the evaluated tasks were specifically held out and unseen during tuning. The option attributing the gain purely to increased parameter count is wrong because the ablations isolate task diversity, not model scale, as the key driver.

Source: Wei et al., 'Finetuned Language Models Are Zero-Shot Learners' (arXiv:2109.01652, 2021)

Why D? And why not the others?

Correct answer: D. Bottleneck adapters insert new down-projection and up-projection feed-forward layers with a nonlinearity directly into the forward pass, adding extra sequential computation and inference latency, while LoRA's trainable matrices form a parallel low-rank update to an existing weight matrix that can be merged back into that weight after training, adding no extra inference latency

Houlsby et al.'s bottleneck adapters are new modules -- a down-projection, a nonlinearity, and an up-projection -- spliced directly into the transformer's forward pass, so every inference call must sequentially compute through that extra module, adding a small but real amount of latency and depth that cannot be removed without retraining. LoRA instead represents its trainable low-rank matrices as an additive update running in parallel to an existing weight matrix rather than as a new sequential layer, and because the update and the original weight are simply added together, the low-rank product can be folded back into the original weight matrix once training is finished, leaving a model that runs at the original architecture's inference speed with no separate module left in the forward path. The option reversing which method merges and which inserts a sequential layer states the opposite of both papers' actual designs. The option claiming the two methods are mathematically identical aside from initialization ignores that one changes the network's computational graph with a new sequential module and the other only changes an existing weight matrix's value. The option claiming both modify the same attention weight matrices identically ignores that adapters are typically inserted as standalone modules after the attention and feed-forward sublayers rather than modifying attention's own weight matrices at all.

Source: Houlsby et al., 'Parameter-Efficient Transfer Learning for NLP' (arXiv:1902.00751, 2019); Hu et al., 'LoRA: Low-Rank Adaptation of Large Language Models' (arXiv:2106.09685, 2021)

Why A? And why not the others?

Correct answer: A. In a first, supervised phase, the model critiques and revises its own sampled responses against the constitution's principles and is fine-tuned on the revised outputs; in a second, reinforcement-learning phase, the model itself judges which of two sampled responses better satisfies the constitution to build an AI-generated preference dataset, which trains a preference model used as the reward signal

Constitutional AI runs in two stages: a supervised-learning stage where the model generates a response, then critiques and revises that response against a written set of principles, and is fine-tuned directly on those self-revised outputs; followed by a reinforcement-learning stage where the model -- rather than a human -- compares pairs of sampled responses against the same constitution to build a dataset of AI-generated preference judgments, which is used to train a preference model that then supplies the reward signal for reinforcement learning, a process the paper calls reinforcement learning from AI feedback (RLAIF). The option describing a single supervised phase trained purely on human-labeled harmful-response comparisons is wrong because it omits the entire point of the method, which is to replace human labeling of harmful comparisons with the model's own judgments. The option describing a static inference-time filter is wrong because both phases involve actually updating the model's weights through training, not filtering outputs after the fact. The option describing a gradient-free rule-injection procedure is wrong because both phases described in the paper use ordinary gradient-based supervised fine-tuning and reinforcement learning, not direct weight editing.

Source: Bai et al., 'Constitutional AI: Harmlessness from AI Feedback' (arXiv:2212.08073, 2022)

Why C? And why not the others?

Correct answer: C. RLAIF achieved performance comparable to RLHF as judged by human evaluators across the tasks tested, including cases where the labeling model was the same size as the policy being trained or even the exact same checkpoint, and RLAIF outperformed the supervised-fine-tuned baseline in these comparisons

Lee et al. found that a reward model trained on preference labels generated by an off-the-shelf LLM, rather than by human annotators, produced policies that human evaluators rated comparably to policies trained with standard human-labeled RLHF across the summarization and dialogue tasks tested, and this held even in the more demanding case where the model generating the labels was the same size as the policy being trained, or the identical checkpoint as the policy's own starting point; RLAIF also beat a supervised-fine-tuned baseline in these comparisons. The option requiring the labeler to be strictly larger and more capable is wrong because the study specifically highlights comparable performance even at equal size or an identical checkpoint. The option claiming RLAIF substantially underperformed both RLHF and the supervised baseline is the opposite of the paper's finding. The option claiming the reward model and the RL step were eliminated entirely in every tested configuration describes a separate variant the paper calls direct-RLAIF, not the core RLAIF setup evaluated in most of the study's main comparisons, which still trains a reward model and runs RL against it, just with AI-generated rather than human-generated preference labels.

Source: Lee et al., 'RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback' (arXiv:2309.00267, 2023)

Why B? And why not the others?

Correct answer: B. Gold-standard performance initially rises alongside the proxy reward model's score but eventually plateaus and then declines even as the proxy score keeps climbing, an instance of Goodhart's law where the imperfect proxy is increasingly over-exploited; the relationship follows scaling patterns that shift with the reward model's size and with whether optimization is done via reinforcement learning or best-of-n sampling

Gao, Schulman & Hilton's synthetic setup used a large gold-standard reward model as a stand-in for true human judgment while a smaller proxy reward model, trained on the gold model's own labels, supplied the actual optimization signal; as they pushed optimization against that fixed proxy further, the policy's true quality, measured against the gold model, rose initially but then plateaued and eventually fell even while the proxy's own score kept climbing, which is the paper's central demonstration of Goodhart's law: an imperfect proxy stops being a reliable stand-in once it is exploited hard enough, and this pattern's exact shape shifted predictably with the reward model's size and with whether the optimization method was reinforcement learning or best-of-n sampling. The option claiming gold-standard performance rises without any ceiling is wrong because the decline after a peak is exactly what the paper documents and calls overoptimization. The option restricting the effect to best-of-n only is wrong because the paper studied and found the same qualitative overoptimization pattern under RL optimization as well, just following a different functional form. The option claiming a sufficiently large proxy reward model eliminates the decline entirely is wrong because the paper found scale shifts where the decline sets in, not that it disappears.

Source: Gao, Schulman & Hilton, 'Scaling Laws for Reward Model Overoptimization' (arXiv:2210.10760, 2022)

Why D? And why not the others?

Correct answer: D. GRPO removes PPO's separate learned value/critic network entirely; for a given prompt it samples a group of outputs from the current policy, uses that group's own average reward as a baseline, and computes each output's advantage as how far its reward deviates from that group's baseline, normalized by the group's reward variability

GRPO's defining change from standard PPO is eliminating the separately learned value or critic network that PPO normally trains to estimate a baseline for computing advantages; instead, for each prompt, GRPO samples a whole group of candidate outputs from the current policy, treats that group's own average reward as the baseline, and computes each individual output's advantage as the deviation of its reward from that group average, normalized using the group's own reward standard deviation, which removes the memory and compute cost of maintaining a separate critic model. The option claiming GRPO still trains a critic network and only removes the reward model is backwards; GRPO keeps a reward signal but removes the critic. The option describing a single fixed, hand-chosen baseline is wrong because GRPO's baseline is computed dynamically per prompt from the sampled group's own rewards, not set once in advance. The option requiring a human preference judgment for every sampled output is wrong because GRPO's advantage computation works with whatever reward signal the setup provides (which in DeepSeekMath is often an automatic correctness check), not a per-output human label.

Source: Shao et al., 'DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models' (arXiv:2402.03300, 2024)