Fine-tuning & Model Customisation

Correct answer: .

LoRA leaves the pretrained weight matrices frozen and, for the layers it targets, adds a new pair of much smaller matrices -- one that projects down to a low intermediate rank and one that projects back up -- whose product forms a low-rank approximation of whatever weight update the task would otherwise require; only the two small matrices are trained, which is why the trainable parameter count drops so sharply while the frozen base weights carry over unchanged. The option describing an ordinary full fine-tune with a smaller learning rate is wrong because a smaller learning rate does not shrink the number of parameters being updated, only the size of each update, so it does nothing to reduce the underlying memory and compute cost. The option describing deleting and retraining the attention layers from scratch is wrong because LoRA never removes or replaces any pretrained component; it augments the existing frozen weights alongside a small addition rather than discarding learned structure. The option describing full-parameter backpropagation on 8-bit integer weights is wrong because that describes a form of quantized full fine-tuning, not LoRA's low-rank decomposition, and still requires updating every original parameter rather than a small injected pair.

Source: Hu et al., 'LoRA: Low-Rank Adaptation of Large Language Models' (arXiv:2106.09685, 2021)

Correct answer: .

QLoRA's actual recipe is to store the frozen base model's weights in a 4-bit NormalFloat (NF4) format -- an information-theoretically tuned datatype for normally distributed weights -- while dequantizing on the fly to a higher-precision compute type for the actual forward and backward math, then training ordinary LoRA adapters on top of those frozen quantized weights; it further saves memory by quantizing the quantization constants themselves (double quantization) and uses paged optimizers built on unified GPU memory to absorb the memory spikes that occur when processing long sequences. The option describing full 4-bit computation throughout both passes is wrong because QLoRA's 4-bit format is used only for storing the frozen base weights, not for the arithmetic itself, which is why it can match 16-bit fine-tuning quality. The option describing permanently zeroing small-magnitude weights describes a pruning technique, not quantization, and QLoRA does not delete any of the base model's weights. The option describing training a smaller distilled student from scratch describes knowledge distillation, an entirely different technique that discards the original model's exact parameters rather than adapting them.

Source: Dettmers, Pagnoni, Holtzman & Zettlemoyer, 'QLoRA: Efficient Finetuning of Quantized LLMs' (arXiv:2305.14314, 2023)

Correct answer: .

DPO's key mathematical move is to observe that, under the Bradley-Terry model of preferences, the reward implied by the optimal KL-regularized policy can be expressed purely in terms of that policy's own probabilities relative to a reference model; substituting this expression back into the standard preference loss produces a single supervised classification-style loss on the preference pairs themselves, which means the policy is trained directly, with no separate reward model ever fit and no online rollout-and-update RL loop like PPO required at all. The option describing still fitting a full reward model and then applying a closed-form update is wrong because DPO's entire point is to avoid fitting any separate reward model; the reward is never estimated as its own object. The option describing discarding human preference pairs in favor of self-critique against written principles describes a Constitutional-AI-style approach, not DPO, which is built specifically to consume labeled preference pairs. The option describing keeping the full RLHF pipeline intact and only swapping the reward model's architecture is wrong because it retains exactly the separate-reward-model-plus-RL-loop structure that DPO was designed to eliminate.

Source: Rafailov, Sharma, Mitchell, Ermon, Manning & Finn, 'Direct Preference Optimization: Your Language Model is Secretly a Reward Model' (arXiv:2305.18290, 2023)

Correct answer: .

Catastrophic forgetting occurs because gradient updates during fine-tuning reshape the same shared weights the model uses for every capability; optimizing those weights hard toward a new, narrow task distribution can overwrite or distort the representations the model was previously relying on for unrelated skills like arithmetic, even though no arithmetic examples appeared in the new training data. This effect is documented to become more severe as the fine-tuning distribution diverges further from the original pretraining distribution, and can be worsened by training for more steps or with a higher learning rate, both of which push the weights further from their pretrained configuration. The option attributing this to memorized label noise is wrong because forgetting is a structural side effect of weight updates, not a symptom of bad labels in the fine-tuning set. The option attributing this to a tokenizer mismatch is wrong because fine-tuning as described does not alter the model's vocabulary or tokenization scheme. The option calling it a benchmark artifact is wrong because the described capability loss is a real, well-studied consequence of how fine-tuning updates shared parameters, not a measurement error.

Source: Luo et al., 'An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuning' (arXiv:2308.08747, 2023)

Correct answer: .

FLAN fine-tuned a 137-billion-parameter pretrained model on over 60 NLP tasks verbalized as natural-language instructions and then tested it, with no examples given, on tasks held out from that tuning set entirely; the tuned model's zero-shot performance improved substantially, surpassing a much larger untuned model's zero-shot results on most evaluated tasks and even beating that larger model's few-shot performance on several benchmarks, with ablations showing the number of distinct task types used in tuning -- not just raw data volume -- was central to the gain. The option claiming the benefit required examples at test time is wrong because the whole point of the finding is that the gains showed up under a strictly zero-shot evaluation. The option claiming the model only improved on the exact tasks it was tuned on is wrong because the evaluated tasks were specifically held out and unseen during tuning. The option attributing the gain purely to increased parameter count is wrong because the ablations isolate task diversity, not model scale, as the key driver.

Source: Wei et al., 'Finetuned Language Models Are Zero-Shot Learners' (arXiv:2109.01652, 2021)

Correct answer: .

Houlsby et al.'s bottleneck adapters are new modules -- a down-projection, a nonlinearity, and an up-projection -- spliced directly into the transformer's forward pass, so every inference call must sequentially compute through that extra module, adding a small but real amount of latency and depth that cannot be removed without retraining. LoRA instead represents its trainable low-rank matrices as an additive update running in parallel to an existing weight matrix rather than as a new sequential layer, and because the update and the original weight are simply added together, the low-rank product can be folded back into the original weight matrix once training is finished, leaving a model that runs at the original architecture's inference speed with no separate module left in the forward path. The option reversing which method merges and which inserts a sequential layer states the opposite of both papers' actual designs. The option claiming the two methods are mathematically identical aside from initialization ignores that one changes the network's computational graph with a new sequential module and the other only changes an existing weight matrix's value. The option claiming both modify the same attention weight matrices identically ignores that adapters are typically inserted as standalone modules after the attention and feed-forward sublayers rather than modifying attention's own weight matrices at all.

Source: Houlsby et al., 'Parameter-Efficient Transfer Learning for NLP' (arXiv:1902.00751, 2019); Hu et al., 'LoRA: Low-Rank Adaptation of Large Language Models' (arXiv:2106.09685, 2021)

Correct answer: .

Constitutional AI runs in two stages: a supervised-learning stage where the model generates a response, then critiques and revises that response against a written set of principles, and is fine-tuned directly on those self-revised outputs; followed by a reinforcement-learning stage where the model -- rather than a human -- compares pairs of sampled responses against the same constitution to build a dataset of AI-generated preference judgments, which is used to train a preference model that then supplies the reward signal for reinforcement learning, a process the paper calls reinforcement learning from AI feedback (RLAIF). The option describing a single supervised phase trained purely on human-labeled harmful-response comparisons is wrong because it omits the entire point of the method, which is to replace human labeling of harmful comparisons with the model's own judgments. The option describing a static inference-time filter is wrong because both phases involve actually updating the model's weights through training, not filtering outputs after the fact. The option describing a gradient-free rule-injection procedure is wrong because both phases described in the paper use ordinary gradient-based supervised fine-tuning and reinforcement learning, not direct weight editing.

Source: Bai et al., 'Constitutional AI: Harmlessness from AI Feedback' (arXiv:2212.08073, 2022)

Correct answer: .

Lee et al. found that a reward model trained on preference labels generated by an off-the-shelf LLM, rather than by human annotators, produced policies that human evaluators rated comparably to policies trained with standard human-labeled RLHF across the summarization and dialogue tasks tested, and this held even in the more demanding case where the model generating the labels was the same size as the policy being trained, or the identical checkpoint as the policy's own starting point; RLAIF also beat a supervised-fine-tuned baseline in these comparisons. The option requiring the labeler to be strictly larger and more capable is wrong because the study specifically highlights comparable performance even at equal size or an identical checkpoint. The option claiming RLAIF substantially underperformed both RLHF and the supervised baseline is the opposite of the paper's finding. The option claiming the reward model and the RL step were eliminated entirely in every tested configuration describes a separate variant the paper calls direct-RLAIF, not the core RLAIF setup evaluated in most of the study's main comparisons, which still trains a reward model and runs RL against it, just with AI-generated rather than human-generated preference labels.

Source: Lee et al., 'RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback' (arXiv:2309.00267, 2023)

Correct answer: .

Gao, Schulman & Hilton's synthetic setup used a large gold-standard reward model as a stand-in for true human judgment while a smaller proxy reward model, trained on the gold model's own labels, supplied the actual optimization signal; as they pushed optimization against that fixed proxy further, the policy's true quality, measured against the gold model, rose initially but then plateaued and eventually fell even while the proxy's own score kept climbing, which is the paper's central demonstration of Goodhart's law: an imperfect proxy stops being a reliable stand-in once it is exploited hard enough, and this pattern's exact shape shifted predictably with the reward model's size and with whether the optimization method was reinforcement learning or best-of-n sampling. The option claiming gold-standard performance rises without any ceiling is wrong because the decline after a peak is exactly what the paper documents and calls overoptimization. The option restricting the effect to best-of-n only is wrong because the paper studied and found the same qualitative overoptimization pattern under RL optimization as well, just following a different functional form. The option claiming a sufficiently large proxy reward model eliminates the decline entirely is wrong because the paper found scale shifts where the decline sets in, not that it disappears.

Source: Gao, Schulman & Hilton, 'Scaling Laws for Reward Model Overoptimization' (arXiv:2210.10760, 2022)

Correct answer: .

GRPO's defining change from standard PPO is eliminating the separately learned value or critic network that PPO normally trains to estimate a baseline for computing advantages; instead, for each prompt, GRPO samples a whole group of candidate outputs from the current policy, treats that group's own average reward as the baseline, and computes each individual output's advantage as the deviation of its reward from that group average, normalized using the group's own reward standard deviation, which removes the memory and compute cost of maintaining a separate critic model. The option claiming GRPO still trains a critic network and only removes the reward model is backwards; GRPO keeps a reward signal but removes the critic. The option describing a single fixed, hand-chosen baseline is wrong because GRPO's baseline is computed dynamically per prompt from the sampled group's own rewards, not set once in advance. The option requiring a human preference judgment for every sampled output is wrong because GRPO's advantage computation works with whatever reward signal the setup provides (which in DeepSeekMath is often an automatic correctness check), not a per-output human label.

Source: Shao et al., 'DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models' (arXiv:2402.03300, 2024)

Correct answer: .

Prefix-tuning learns a continuous vector, prepended to the keys and values of the self-attention computation, at every transformer layer, so subsequent tokens attend to these vectors as if they were additional 'virtual tokens' at each layer of the network; prompt tuning, described by its own authors as a simplification of prefix-tuning, instead learns vectors only at the input embedding layer and lets the frozen transformer stack propagate their influence forward from there. The option describing prompt tuning as reaching into every layer's attention and prefix-tuning as embedding-only is wrong because it swaps which method does which -- it is prefix-tuning, not prompt tuning, that touches every layer. The option describing both methods as inserting an identical number of vectors at every layer is wrong because prompt tuning's defining simplification is confining its trainable vectors to the embedding layer alone, not matching prefix-tuning's per-layer insertion. The option describing insertion only at the final output layer before the softmax is wrong because neither method works this way; both condition the model's processing from near the start of the network rather than only adjusting its final logits.

Source: Li & Liang, 'Prefix-Tuning: Optimizing Continuous Prompts for Generation' (arXiv:2101.00190, 2021); Lester, Al-Rfou & Constant, 'The Power of Scale for Parameter-Efficient Prompt Tuning' (arXiv:2104.08691, 2021)

Correct answer: .

DoRA decomposes each target pretrained weight matrix into a magnitude vector and a directional matrix, then applies a LoRA-style low-rank update only to the direction while the magnitude is fine-tuned as its own separate, small set of parameters; because full fine-tuning is free to adjust both magnitude and direction independently while plain LoRA's single low-rank update conflates the two, this decomposition lets DoRA's learning pattern resemble full fine-tuning's more closely at the same parameter budget, without adding any inference-time latency once the adapter is merged back into the base weights. The option describing an automatically growing rank is wrong because DoRA does not change the rank dynamically during training; its rank is fixed like LoRA's, and the improvement comes from the magnitude/direction decomposition, not from using more rank. The option describing 4-bit quantization of the frozen weights is wrong because that combination of quantization with low-rank adaptation describes QLoRA, a different, memory-focused technique, not DoRA's magnitude-direction decomposition. The option describing a single full-rank matrix applied only to the value projection is wrong because DoRA still uses LoRA's low-rank matrix pair; it does not replace them with a full-rank matrix, and it is not restricted to only the value projection.

Source: Liu et al., 'DoRA: Weight-Decomposed Low-Rank Adaptation' (arXiv:2402.09353, ICML 2024)

Correct answer: .

Rehearsal mitigates forgetting empirically, by literally mixing a sample of the original task's examples back into the fine-tuning batches so the ordinary training loss continues to reward good performance on that earlier task alongside the new one; EWC instead mitigates forgetting analytically, without replaying any original data during the new training run, by adding a quadratic penalty to the loss that is weighted by each parameter's Fisher information -- an estimate of how important that parameter was to the earlier task -- so parameters that mattered most for the earlier task are constrained to change less while less-important parameters remain free to adapt. The option that swaps these mechanisms, crediting EWC with replaying data and rehearsal with a Fisher-information penalty, is wrong because it reverses which technique does which. The option claiming the two are mathematically identical and differ only by optimizer is wrong because they modify entirely different parts of the training process -- one changes the training data mixture, the other changes the loss function's penalty term -- regardless of which optimizer applies the resulting gradients. The option claiming both require discarding the new dataset and training only on old data is wrong because both strategies are designed to let the model continue learning the new task while protecting the old one, not to abandon the new task altogether.

Source: Kirkpatrick et al., 'Overcoming catastrophic forgetting in neural networks' (PNAS, 2017)

Correct answer: .

Knowledge distillation trains the student against the teacher's softened output distribution -- the full probability the teacher assigns to every class or token, not just its single top pick -- obtained by raising the softmax temperature so the smaller probabilities on the wrong classes become visible instead of being rounded to near zero; those relative probabilities across the 'wrong' classes still encode real information about which mistakes the teacher considers more or less plausible, which Hinton termed 'dark knowledge,' and a hard, one-correct-label target simply cannot carry. The option describing the student copying the teacher's raw pretraining gradients is wrong because distillation trains the student on the teacher's output distribution for the actual training examples, not on a historical record of the teacher's own gradient updates. The option describing extra hard-labeled data the teacher never saw is wrong because distillation's value comes from the soft probability signal itself, not from giving the student more hard-labeled examples. The option describing the student copying the teacher's attention weight matrices is wrong because distillation transfers behavior through the training signal (the output distribution), not by directly copying internal weight matrices between differently sized architectures.

Source: Hinton, Vinyals & Dean, 'Distilling the Knowledge in a Neural Network' (arXiv:1503.02531, 2015)

Correct answer: .

LIMA's result -- strong, competitive output from only 1,000 carefully curated examples and no reinforcement learning or preference modeling -- is offered by its authors in support of what they call the Superficial Alignment Hypothesis: that a model's knowledge and capabilities are learned almost entirely during pretraining, and fine-tuning's job is mainly the comparatively narrow, learnable task of teaching the model which format and style to use when responding, which is why a small but carefully curated set of examples can be enough. The option claiming instruction tuning is how a model acquires new factual knowledge is wrong because it inverts LIMA's own claim, which is that knowledge comes from pretraining and fine-tuning contributes format and style rather than new facts. The option claiming dataset size is irrelevant and larger datasets always hurt is wrong because LIMA does not make a universal claim against scale; it demonstrates that a small, carefully curated set can suffice for its specific alignment goal, not that scale is actively harmful in every setting. The option claiming LIMA overturns FLAN's zero-shot generalization finding is wrong because the two studies test different properties -- FLAN's finding concerns zero-shot transfer to task categories never seen during instruction tuning, which LIMA's single-style curated dataset was not designed to test at all, so LIMA's result does not contradict it.

Source: Zhou et al., 'LIMA: Less Is More for Alignment' (arXiv:2305.11206, 2023); Wei et al., 'Finetuned Language Models Are Zero-Shot Learners' (arXiv:2109.01652, 2022)

Correct answer: .

Yu et al. identify conflicting gradients -- task gradients at shared parameters that point away from each other, i.e. have negative cosine similarity -- as a key optimization-level cause of multi-task interference, and propose 'gradient surgery' (PCGrad), which projects a task's gradient onto the normal plane of a conflicting gradient before applying it, so the update no longer directly fights the other task at that step. The option equating this with catastrophic forgetting is wrong because catastrophic forgetting is specifically a sequential-training phenomenon -- losing an earlier task's performance while training on a later one -- whereas this interference happens with tasks trained jointly, at the same time, which is a distinct optimization dynamic. The option blaming only an overly high learning rate is wrong because the paper attributes the effect to the geometric relationship between task gradients, not simply to step size, and shows the conflict persists as a structural property even with careful tuning. The option claiming joint training always helps every task is wrong because it directly contradicts the premise the paper investigates -- individual task performance genuinely can, and does, drop under naive joint optimization.

Source: Yu et al., 'Gradient Surgery for Multi-Task Learning' (arXiv:2001.06782, NeurIPS 2020)

Correct answer: .

KTO trains directly on a single binary desirable/undesirable label per output, without needing that output compared against an alternative for the same prompt, because its loss function is built on a Kahneman-Tversky-style utility model from prospect theory rather than on a pairwise comparison objective; DPO, by contrast, needs an actual preferred/dispreferred pair sharing a prompt, which is exactly the paired data this team lacks. The option pointing to RLAIF is wrong because RLAIF still produces paired preference comparisons, just generated by an LLM judge instead of human labelers, so it does not remove the need to construct pairs -- it only changes who constructs them. The option describing Constitutional AI as purely binary-label-based throughout is wrong because its second, reinforcement-learning phase still trains against AI-generated pairwise preference comparisons, not solely independent binary labels. The option claiming standard RLHF's reward model already works directly on binary labels is wrong because that reward model is conventionally trained with a pairwise (Bradley-Terry-style) comparison loss, not a per-output binary label.

Source: Ethayarajh et al., 'KTO: Model Alignment as Prospect Theoretic Optimization' (arXiv:2402.01306, 2024)

Correct answer: .

S-LoRA-style serving keeps a single frozen base model loaded on the GPU and stores each customer's much smaller LoRA adapter weights separately, loading and batching only the specific adapters needed for the requests currently being served against that one shared base model -- since an individual adapter is a tiny fraction of the base model's size, this lets one GPU deployment support thousands of distinct customizations instead of one full model copy per customer. The option describing merging every adapter into one shared model in advance is wrong because that would make every customization active for every request, defeating the purpose of per-customer personalization. The option describing quantizing the base model so thousands of full copies fit in memory is wrong because it still requires a full model copy per customer; quantization shrinks each copy but doesn't eliminate the fundamental one-copy-per-customer duplication. The option describing training one shared model on all customers' concatenated data is wrong because that produces a single generic model with no per-customer distinction at all, which is the opposite of what per-customer personalization requires.

Source: Sheng et al., 'S-LoRA: Serving Thousands of Concurrent LoRA Adapters' (arXiv:2311.03285, 2023)

Correct answer: .

OpenAI's supervised fine-tuning takes a JSONL training file in which each line is one training example formatted as a list of chat messages carrying roles like system, user, and assistant, mirroring the same structure used at inference time; hyperparameters such as the number of epochs, batch size, and learning-rate multiplier can be left as 'auto' for OpenAI to select or set explicitly by the developer, and the finished fine-tuned model is accessed through its own separate fine-tuned model identifier, used alongside, not instead of, the original base model. The option requiring raw plain-text files is wrong because the same chat-message format used for ordinary API calls is exactly what the training file uses, not a different plain-text format. The option claiming every hyperparameter must be manually specified is wrong because OpenAI explicitly offers an automatic 'auto' setting for these hyperparameters as an alternative to manual specification. The option claiming the base model itself gets silently overwritten account-wide is wrong because the fine-tuned result is exposed under its own distinct model identifier, leaving the original base model and its behavior for other users completely unaffected.

Source: OpenAI API documentation, fine-tuning / model optimization guide (developers.openai.com/api/docs/guides/model-optimization)

Correct answer: .

Full fine-tuning's access to every parameter generally gives it the greatest raw capacity to absorb a large distribution shift or a genuinely new capability, but that comes with the largest compute and memory bill, a full new model copy to store per task, and the greatest exposure to catastrophic forgetting of prior abilities; parameter-efficient methods restrict learning to a much smaller set of newly introduced parameters, which is far cheaper to train, store, and swap between tasks, but that same restriction can become a real capacity ceiling when the target domain requires learning a very large amount of genuinely new material. The option claiming parameter-efficient methods always match or exceed full fine-tuning's capacity for any shift size is wrong because a much smaller set of trainable parameters is a real constraint that can bind precisely in the large-distribution-shift scenario described. The option claiming full fine-tuning trains strictly fewer parameters is wrong because full fine-tuning updates every one of the model's parameters, which is categorically more than the small added parameter set any parameter-efficient method trains. The option describing the two approaches as mathematically identical, differing only by optimizer library, is wrong because they update fundamentally different sets of parameters -- the entire model versus a small added subset -- not merely different software implementations of the same operation.

Source: Hu et al., 'LoRA: Low-Rank Adaptation of Large Language Models' (arXiv:2106.09685, 2021)

Correct answer: .

IA3 does not touch any pretrained weight matrix at all. Instead, for each targeted transformer block it introduces a small set of new trainable vectors that elementwise-rescale the key, value, and feedforward-hidden activations as they flow through the frozen network, amplifying or inhibiting specific dimensions of those activations rather than approximating a weight update with a low-rank matrix product the way LoRA does. Because these are just extra multiplicative vectors rather than a full additional weight matrix, IA3 introduces even fewer trainable parameters than LoRA, and because the rescaling is a simple elementwise operation, it can also be absorbed back into the adjacent frozen weights before inference so no extra latency remains, similar in spirit to a merged LoRA adapter. The option describing a trainable classification head is wrong because that only adapts the model's output layer and leaves the transformer's internal representations exactly as pretrained, which is a much weaker and different kind of adaptation than reshaping activations throughout the network. The option describing newly inserted transformer layers is wrong because that describes an adapter-style architecture change that adds depth to the network, not IA3's activation-rescaling mechanism, and would also add real inference latency that cannot be merged away. The option describing training the difference between two half-precision copies of each weight matrix is wrong because that is simply full fine-tuning restated with a memory trick, still updating every original parameter, which is the opposite of IA3's small, fixed set of rescaling vectors.

Source: Liu et al., 'Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning' (NeurIPS 2022, arXiv:2205.05638)

Correct answer: .

Ben Zaken et al. found that on small-to-medium training data, fine-tuning only the bias terms of a pretrained transformer such as BERT is competitive with full fine-tuning, and sometimes even outperforms it, while updating only about 0.1% of the model's total parameters. The paper frames this as evidence that most of the task-relevant capability already exists in the pretrained weights, and that fine-tuning is largely about exposing and steering that existing knowledge rather than learning substantial new computation, which is why nudging just the additive bias terms in each layer can redirect the model's behavior effectively. The option claiming bias-only fine-tuning fails to learn the task is wrong because the paper's central empirical result is the opposite: it works surprisingly well, not that it fails outright. The option requiring a newly added low-rank adapter matrix is wrong because that describes combining BitFit with a LoRA-style addition, which is not what the original BitFit method does; BitFit's whole point is that touching only pre-existing bias parameters, with no new matrices added anywhere, is already sufficient. The option claiming bias-only fine-tuning needs far more training epochs is wrong because the paper does not report a slower convergence tradeoff of this kind; its headline finding is about matching full fine-tuning's accuracy with drastically fewer trainable parameters, not about needing more training time to get there.

Source: Ben Zaken, Goldberg & Ravfogel, 'BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models' (ACL 2022, arXiv:2106.10199)

Correct answer: .

Self-Instruct is an iterative bootstrapping pipeline: starting from a small, manually written seed set of instructions, the language model itself is prompted to generate new instructions along with plausible inputs and outputs for them, invalid or excessively similar generations are filtered out, and the surviving synthetic examples are added to the pool used to fine-tune that same model, letting a small seed set expand into a much larger instruction-tuning dataset without further human annotation. Wang et al. reported that applying this to GPT-3 closed most of the gap to InstructGPT despite using only model-generated data grown from a small seed. The option describing training a separate smaller model solely on the seed examples is wrong because Self-Instruct's value comes precisely from generating large amounts of new synthetic data beyond the seed set, not from training directly on the small seed alone. The option describing scraping and relabeling existing question banks is wrong because that would reuse text authored by third parties rather than having the model generate genuinely new instructions and responses, which is the defining bootstrapping step Self-Instruct performs. The option describing a longer unrelated pretraining run is wrong because Self-Instruct is specifically about generating instruction-following fine-tuning data, not about extending unsupervised pretraining on unrelated general text.

Source: Wang et al., 'Self-Instruct: Aligning Language Models with Self-Generated Instructions' (ACL 2023, arXiv:2212.10560)

Correct answer: .

In rejection sampling fine-tuning, the current policy samples K candidate responses for each prompt, the reward model scores every candidate, and the highest-scoring candidate (or candidates) becomes new supervised training data that the model is fine-tuned on directly, functioning like a best-of-N selection step folded into supervised learning rather than a policy-gradient reinforcement learning update. Meta's Llama 2 paper describes using exactly this approach alone through RLHF-V4 before layering PPO on top from RLHF-V5 onward, treating the reward-selected best sample as the new gold-standard target for the next round of supervised fine-tuning. The option describing unconditional acceptance of a single generated response regardless of reward score is wrong because the entire point of the method is selective: multiple candidates are generated and scored, and only the reward model's top pick is kept, which is what makes it 'rejection' sampling rather than plain self-training. The option describing the reward model being fine-tuned while the policy stays frozen is backwards; the reward model is the fixed scorer used to select data, while it is the policy that gets updated via supervised fine-tuning on the selected samples. The option describing a PPO clipped surrogate update is wrong because rejection sampling fine-tuning is explicitly the alternative to PPO used in the earlier rounds; PPO and rejection sampling fine-tuning are two distinct mechanisms in the paper, and 'rejection sampling' here refers to discarding lower-reward generated candidates, not malformed API responses.

Source: Touvron et al., 'Llama 2: Open Foundation and Fine-Tuned Chat Models' (arXiv:2307.09288), Section 3.2

Correct answer: .

ORPO folds preference alignment into the ordinary supervised fine-tuning objective by adding a log odds-ratio term that contrasts the model's odds of generating the chosen response against its odds of generating the rejected response for the same prompt, combined with the standard negative log-likelihood loss on the chosen response, all in one monolithic training stage. Because the odds ratio is computed from the same policy being trained rather than compared against a separately maintained frozen copy, ORPO needs neither a preceding supervised fine-tuning warm-up phase nor a frozen reference model during training, which is exactly the overhead DPO still carries and PPO carries even more of via its explicit reward model and KL penalty. The option describing training a reward model and then running standard PPO is wrong because that is the very RLHF pipeline ORPO is designed to avoid, complete with the reward model and the reinforcement-learning loop ORPO has no need for. The option describing two frozen model copies and a KL divergence signal is wrong because that describes something closer to DPO's or PPO's reliance on a fixed reference policy, whereas ORPO's defining feature is that it uses no reference model at all, frozen or otherwise. The option describing discarding the preference labels and fine-tuning only on chosen responses is wrong because ORPO's odds-ratio term specifically depends on the rejected response too, actively penalizing it, rather than ignoring it as the option suggests.

Source: Hong, Lee & Thorne, 'ORPO: Monolithic Preference Optimization without Reference Model' (EMNLP 2024, arXiv:2403.07691)

Correct answer: .

Once a LoRA adapter's low-rank matrices are merged into the base weight matrices, the resulting checkpoint has exactly the same tensor shapes and layer structure as the original base model; the forward pass no longer computes a separate adapter contribution and adds it on top, it simply runs the base model's ordinary computation with different numerical weight values. That means merged inference carries essentially zero added latency compared to the unmodified base model, which is why merging is the preferred choice for single-task, latency-sensitive deployment. Keeping the adapter separate, by contrast, requires computing and adding the extra low-rank term at every targeted layer during each forward pass, which is a real (if often small) computational overhead compared to a merged checkpoint, and is the tradeoff systems that serve many different LoRA adapters on one base model, such as S-LoRA, deliberately accept in order to avoid maintaining many separate merged copies. The option claiming merging forces double computation is wrong because merging eliminates the separate adapter computation entirely rather than duplicating it. The option claiming adapters already add zero overhead when kept separate is wrong because computing an extra matrix product at every targeted layer is not free, even though it is typically small relative to the model's overall compute. The option claiming merging permanently blocks future fine-tuning is wrong because a merged checkpoint is just an ordinary set of weights and remains fully fine-tunable, including by attaching a brand-new LoRA adapter on top of it.

Source: Hu et al., 'LoRA: Low-Rank Adaptation of Large Language Models' (arXiv:2106.09685), Section 4.2 (merging and zero inference latency)

Correct answer: .

InstructGPT's three-stage pipeline first produces a supervised fine-tuned (SFT) model trained on human demonstrations, and it is that SFT model, not the raw pretrained base model, that is used to sample multiple candidate responses per prompt for labelers to rank in the second stage. This matters because the reward model needs to learn to discriminate quality among responses that resemble what the policy actually being aligned will generate once RLHF begins; ranking outputs from a very different distribution, such as the raw pretrained model's often incoherent or off-task completions, would teach the reward model to distinguish differences that are far less relevant to the fine-tuned policy's actual output space. The option describing sampling from the original pretrained GPT-3 model before any supervised fine-tuning is wrong because that base model's outputs are not yet representative of the aligned policy's behavior, and the paper's pipeline explicitly inserts the supervised fine-tuning stage first for this reason. The option describing outputs from a separate, unrelated reward model is backwards: the reward model is the thing being trained from these rankings, not the source generating the candidates being ranked. The option describing outputs written entirely by human labelers is wrong because the reward-model stage specifically compares and ranks model-generated candidate responses; the labelers' human-written demonstrations belong to the earlier supervised fine-tuning stage, not to this ranking stage.

Source: Ouyang et al., 'Training language models to follow instructions with human feedback' (NeurIPS 2022, arXiv:2203.02155), Section 3

Correct answer: .

A reward model is only an imperfect proxy for true human judgment, and a policy optimized hard enough against it, with no other constraint, can drift into regions of output space that score highly under the reward model's specific quirks and blind spots while actually producing repetitive, incoherent, or otherwise degenerate text a human would rate poorly -- a form of reward hacking. Adding a per-token KL divergence penalty between the current policy and a frozen copy of the supervised fine-tuned model anchors the policy near its well-behaved starting distribution, so large reward gains that require straying far from realistic, fluent language are penalized even if the reward model itself would have scored them favorably. The option describing collapsing generation to always pick the single highest-probability token is wrong because the KL penalty constrains how far the whole output distribution can drift from the frozen reference, it does not force greedy, zero-randomness decoding. The option describing weight sharing between the policy and the frozen reference to save parameters is wrong because the reference model is kept as a separate frozen copy specifically so its distribution stays fixed for comparison; sharing weights would make the reference model change along with the policy, defeating the entire purpose of the penalty. The option describing the reward model being discarded after the first few updates is wrong because the reward model's score remains the primary training signal throughout PPO; the KL term is an additional regularizer alongside it, not a replacement for it.

Source: Ouyang et al., 'Training language models to follow instructions with human feedback' (arXiv:2203.02155), Section 3.3; Schulman et al., 'Proximal Policy Optimization Algorithms' (arXiv:1707.06347)

Correct answer: .

Domain-adaptive pretraining, as studied by Gururangan et al., does not introduce a new training objective or labels at all; it simply continues the model's original unsupervised pretraining objective, such as masked language modeling, on a large corpus of unlabeled text drawn from the target domain, before the usual task-specific fine-tuning stage begins. Across four domains (biomedical and computer science publications, news, and reviews) and eight classification tasks, the study found that this extra in-domain pretraining phase generally improved downstream performance in both high-resource and low-resource task settings, essentially shifting the model's learned representations toward the statistics of the target domain before it is specialized further for any specific labeled task. The option describing a supervised classification loss on labeled in-domain data is wrong because DAPT is explicitly unsupervised, using unlabeled domain text and the model's existing pretraining objective, not new labels. The option describing freezing every layer except the final classification head is wrong because that describes a linear-probing style adaptation, not the full continued-pretraining approach DAPT uses, and the study did not find that partial freezing matched full fine-tuning. The option describing discarding the original pretrained weights and training a new smaller model from scratch is wrong because DAPT explicitly continues from the existing pretrained checkpoint to retain its general-purpose knowledge, rather than throwing that knowledge away and starting over.

Source: Gururangan et al., 'Don't Stop Pretraining: Adapt Language Models to Domains and Tasks' (ACL 2020, arXiv:2004.10964)

Correct answer: .

LoRA's injected update is the product of two low-rank matrices, and without any correction, increasing the rank r adds more terms to that product and roughly increases the raw magnitude of the resulting update, meaning a change to rank alone would also change how strongly the frozen weights are perturbed and would typically force retuning the learning rate every time rank is adjusted. Scaling the raw update by alpha divided by r counteracts this: with alpha held fixed, sweeping r keeps the update's overall magnitude approximately constant, which is what lets a practitioner treat rank largely as a separate, independent knob for capacity rather than one that is entangled with how large a perturbation the adapter applies. The option describing conversion to integer precision is wrong because alpha/r is a floating-point scalar multiplier applied to the update's magnitude, not a numerical-precision or quantization mechanism at all. The option describing controlling how many layers LoRA is applied to is wrong because layer selection is a separate configuration choice (which weight matrices to target) made independently of the alpha/r scalar, which only affects the magnitude of updates at whichever layers are already selected. The option describing a separate learning rate for the frozen base weights is wrong because the base weights are frozen and never updated by gradient descent at all during LoRA fine-tuning, so no learning rate applies to them; alpha/r instead rescales the trainable adapter's own contribution to the forward pass.

Source: Hu et al., 'LoRA: Low-Rank Adaptation of Large Language Models' (arXiv:2106.09685), Section 4.1

Correct answer: .

LoRA, introduced by Hu et al. (2021), decomposes each weight update into a product of two small matrices with a low rank (e.g., rank 4 or 16). The original pre-trained weights are frozen and unchanged; only the small adapter matrices are trained. For a weight matrix of size d×d, a rank-r LoRA adapter has only 2×d×r parameters — dramatically fewer than d×d. This means gradients, optimizer states, and the trainable parameters themselves consume a fraction of the memory required for full fine-tuning. The option about removing layers describes model pruning, not LoRA. The option about lower learning rates conflates a hyperparameter choice with an architectural technique. The option about quantisation describes QLoRA (which combines LoRA with 4-bit quantisation) but confuses 'low-rank' (a linear algebra concept about matrix decomposition) with 'low precision' (a numerical representation concept).

Source: Hu et al. (2021), 'LoRA: Low-Rank Adaptation of Large Language Models', arXiv:2106.09685

Correct answer: .

AdaLoRA parameterizes each targeted weight matrix's update in the form of a singular value decomposition, splitting it into triplets of a singular value and its corresponding left and right singular vectors. As training proceeds, it computes an importance score for every triplet based on its contribution to the model's performance and prunes, meaning zeroes out, the singular values of the lowest-scoring triplets, which shrinks their effective rank contribution toward zero while leaving high-importance triplets at full strength. Because the update is already parameterized in this near-SVD form rather than as an arbitrary low-rank product, the method can approximate the effect of an exact SVD's pruning without repeatedly computing one, and the rank budget saved from unimportant layers is effectively redirected toward layers whose triplets keep proving important. The option describing full-rank dense updates with whole-layer pruning after training is wrong because AdaLoRA prunes at the fine-grained triplet level continuously during training, not whole layers after the fact, and never abandons the low-rank formulation for a dense one. The option describing a fixed rank with fewer adapted layers chosen at random is wrong because AdaLoRA still touches every targeted layer; what varies is each layer's effective rank, not whether the layer is adapted at all, and the reallocation is driven by importance scores rather than randomness. The option describing unfreezing random individual weight-matrix entries is wrong because AdaLoRA never abandons the low-rank, SVD-form update for direct entry-level training; that describes a different family of sparse fine-tuning methods entirely.

Source: Zhang et al., 'AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning' (ICLR 2023, arXiv:2303.10512)

Correct answer: .

GPTQ is a one-shot, post-training quantization method: it processes each layer's weight matrix column by column (or in small blocks), and for each step it uses second-order information, an approach adapted from the Optimal Brain Quantization solver, to decide which weights to quantize next and to update the still-unquantized weights so that the layer's output error on a small calibration dataset is minimized, since that error is quadratic and second-order information is the natural tool for minimizing it. No gradients are backpropagated through the model and no weights are retrained; the whole process runs once, using only a modest calibration set, and pushes accurate quantization down to the 4-bit range and below with far less accuracy loss than naive rounding. The option describing training new low-rank adapters and quantizing only those is wrong because GPTQ quantizes the model's existing weights directly; it does not introduce any new adapter matrices, which is instead the general idea behind LoRA-style methods, not GPTQ. The option describing a single global rounding rule with one scale factor and no calibration data is wrong because that describes naive round-to-nearest quantization, exactly the crude baseline GPTQ's per-layer, error-minimizing reconstruction was built to outperform. The option describing full retraining from scratch with quantization-aware training is wrong because GPTQ is specifically a one-shot post-training method that avoids any full retraining pass over the pretraining corpus, which is the far more expensive alternative it was designed to replace.

Source: Frantar et al., 'GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers' (arXiv:2210.17323)

Correct answer: .

A task vector is defined simply as the difference obtained by subtracting the pretrained base model's weights from the same model's weights after it has been fine-tuned on a particular task, giving a direction in weight space that points toward better performance on that task. Because this difference lives in the same weight space as the model itself, task vectors from different tasks can be added directly to the base model's weights to improve performance on multiple tasks at once, or negated and added to reduce performance on a task the fine-tuning targeted, all without touching any training data or running further gradient updates; the method's experiments also show that when four tasks are related by an analogy of the form 'A is to B as C is to D,' combining task vectors from three of them can improve performance on the fourth even though no data from that fourth task was used. The option describing separately stored full checkpoints loaded one at a time is wrong because that is simply keeping several ordinary fine-tuned models, with no arithmetic operation combining them in weight space at all. The option describing a text embedding of the task's description is wrong because a task vector is a weight-space quantity derived from actual fine-tuning, not a language embedding used inside a prompt. The option describing a single training-step gradient is wrong because a task vector is computed from the full difference after fine-tuning has completed, not from one intermediate gradient during training.

Source: Ilharco et al., 'Editing Models with Task Arithmetic' (ICLR 2023, arXiv:2212.04089)

Correct answer: .

Lightman et al. trained and compared an outcome-supervised reward model, which learns only from whether a solution's final answer was correct, against a process-supervised reward model, which learns from step-level human feedback labeling each individual step in a solution as correct, incorrect, or ambiguous, using their released PRM800K dataset of roughly 800,000 step-level labels. They found process supervision significantly outperformed outcome supervision on their representative subset of the MATH benchmark, with their best process-supervised model solving about 78% of problems in that test set, because process supervision pinpoints exactly where a reasoning chain goes wrong rather than only judging the destination, and it more directly rewards a model for producing a chain of reasoning a human would actually endorse step by step. The option claiming outcome supervision won because step-level feedback is too noisy is wrong because it states the opposite of the paper's central empirical result. The option claiming the two performed indistinguishably because a wrong answer implies a single identifiable wrong step is wrong because solutions can reach a wrong final answer through multiple compounding errors, or reach a correct final answer despite a flawed step, so outcome supervision does not capture the same signal as fine-grained step labels. The option claiming process supervision could not be evaluated because no labelers would grade individual steps is wrong because the study specifically collected and released hundreds of thousands of step-level human labels to make this exact comparison possible.

Source: Lightman et al., 'Let's Verify Step by Step' (arXiv:2305.20050)

Correct answer: .

PPO's clipped surrogate objective computes the probability ratio between the current policy and the policy that generated the sampled data, multiplies that ratio by the estimated advantage as an ordinary policy-gradient surrogate would, but also computes a second version of that product using the ratio clipped to a fixed range around 1, and then takes the minimum of the unclipped and clipped versions as the actual objective term. The effect is that once the ratio moves far enough in the direction that would increase the objective by exploiting a large advantage, the clipped term caps how much further objective value can be gained from continuing to push in that direction, removing the incentive for a single update to move the policy too far, which approximates the stabilizing effect of a trust region without requiring one to be solved explicitly. The option describing discarding whole batches based on a fixed advantage threshold is wrong because clipping is applied per sampled action directly inside the objective's math, not as a batch-level filtering rule applied before training. The option describing a KL-divergence term added to the reward is wrong because that is a separate mechanism, a per-token penalty against a frozen reference policy applied to the reward signal itself, which is distinct from the ratio-clipping operation that PPO applies directly to its policy-gradient surrogate objective. The option describing a full second-order constrained optimization exactly like TRPO is wrong because PPO's entire motivation, per Schulman et al., is replacing TRPO's expensive constrained quadratic program with this much simpler first-order clipping trick that only requires ordinary gradient-based optimization.

Source: Schulman et al., 'Proximal Policy Optimization Algorithms' (arXiv:1707.06347)

Correct answer: .

Singhal et al. ran both observational and interventional analyses across several open-domain RLHF settings, including dialogue and web question answering, and found that response length was the single largest driver of the reward gains RLHF produces: reward scores correlated strongly with length, RLHF consistently pushed output length up relative to the supervised-fine-tuned starting point, and even a reward signal based purely on length, with no other quality signal at all, reproduced most of the downstream improvement that standard RLHF achieved. They traced this to the reward models themselves being non-robust and easily swayed by length-related patterns present in the human preference data used to train them, rather than to any genuine, length-independent improvement in response quality. The option attributing the gain to improved factual accuracy that merely happens to correlate with length is wrong because the paper's interventional experiments, including a length-only reward, isolate length itself as the driver rather than treating it as an incidental correlate of some other improvement. The option claiming RLHF made responses shorter is wrong because the study found the opposite: RLHF increased length relative to the supervised-fine-tuned baseline. The option claiming no identifiable common cause was found and that length was explicitly ruled out is wrong because the paper's central, repeated finding across settings was precisely that length was a major, identifiable common driver.

Source: Singhal et al., 'A Long Way to Go: Investigating Length Correlations in RLHF' (arXiv:2310.03716)

Correct answer: .

Self-Rewarding Language Models use the same model being trained for two roles at once: it generates candidate responses to instructions, including instructions it creates for itself, and it also acts as its own judge through LLM-as-a-judge prompting, scoring and ranking its own candidate responses to build preference pairs. Those self-generated preference pairs are then used to train the model further via DPO, and the resulting, improved model is used to generate the next iteration's instructions, responses, and judgments, so that across iterations both the model's instruction-following ability and the quality of its own self-judgments improve together, without ever depending on a separately trained external reward model or fresh human labels after the first iteration. The option describing a frozen model judged purely by an unrelated external model is wrong because the defining feature of this method is that the same evolving model supplies its own rewards; introducing an external, unrelated judge model would defeat the point of self-rewarding entirely. The option describing two frozen copies debating with a fixed majority-vote rule is wrong because nothing in that setup involves any model being updated or improving across iterations, whereas Self-Rewarding Language Models specifically retrain the model each round and observe its self-judging ability improve as a result. The option describing training only on a static pre-existing human dataset is wrong because the entire mechanism depends on generating new self-created instructions, responses, and self-judgments at every iteration rather than reusing one fixed dataset.

Source: Yuan et al., 'Self-Rewarding Language Models' (arXiv:2401.10020)

Correct answer: .

Sparse upcycling reuses an already-pretrained dense checkpoint's sunk training cost by cloning its existing feed-forward layer several times to form the starting set of experts inside each new mixture-of-experts layer, and adding a freshly, randomly initialized routing network in front of each such layer to decide which tokens go to which expert. Pretraining then continues from this initialization, during which the identically-cloned experts gradually diverge and specialize on different kinds of inputs while the router learns sensible routing decisions, and the paper reports that upcycled models trained this way, using roughly half the compute of the original dense pretraining run, outperform their dense counterparts on benchmarks such as SuperGLUE and ImageNet. The option describing discarding the feed-forward layers and randomly reinitializing the mixture-of-experts layers is wrong because the entire point of upcycling is reusing, not discarding, the dense checkpoint's already-trained feed-forward weights as the experts' starting point. The option describing training a brand-new model from scratch, using the dense model only as a comparison benchmark, is wrong because that describes exactly the sunk-cost waste sparse upcycling was designed to avoid; the dense weights are directly reused as initialization, not merely referenced afterward. The option describing pruning down to a single sparse subset of weights with nothing added is wrong because upcycling creates multiple full expert copies plus a new router, the opposite of discarding weights down to one surviving subset.

Source: Komatsuzaki et al., 'Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpoints' (ICLR 2023, arXiv:2212.05055)

Correct answer: .

AdapterFusion works in two separate stages: a knowledge-extraction stage where individual task adapters are trained as usual, one per task, and a later knowledge-composition stage where all of those existing task adapters are kept completely frozen while a new attention-based fusion module is trained on top of them for the target task. That fusion module learns to dynamically weight each frozen adapter's contribution differently at different layers, as a function of the target task's own input at each point, so the composition can draw different amounts of knowledge from different adapters depending on the layer and the input, all without ever modifying the original adapters' already-learned parameters, which is why the paper describes the composition as non-destructive. The option describing averaging all the adapters' weights into one new adapter and fine-tuning that average is wrong because averaging would blend and dilute each adapter's distinct specialized knowledge into one static combination rather than dynamically weighting each one's contribution per layer and per input. The option describing selecting a single best-performing adapter and discarding the rest is wrong because the entire purpose of AdapterFusion is combining knowledge from multiple adapters together, not choosing a single winner and throwing away the others. The option describing concatenating the adapters' weight matrices into one much larger block trained jointly from scratch is wrong because that would abandon each adapter's already-learned frozen knowledge and let parameter count grow with the number of adapters, contrary to AdapterFusion's frozen, non-destructive composition approach.

Source: Pfeiffer et al., 'AdapterFusion: Non-Destructive Task Composition for Transfer Learning' (EACL 2021, arXiv:2005.00247)

Correct answer: .

ZeRO addresses the memory redundancy of standard data-parallel training by partitioning the model's state across the participating GPUs instead of replicating all of it on every GPU: its first stage partitions only the optimizer states, its second stage additionally partitions the gradients, and its third and most aggressive stage additionally partitions the model parameters themselves, so each GPU holds and updates only its own shard of whichever categories are partitioned rather than a full redundant copy of everything. This lets memory use scale down as more GPUs are added while the training still behaves as ordinary data-parallel training overall, which is what allowed models with hundreds of billions to trillions of parameters to be trained on hardware that could never hold a full redundant copy of everything on every device. The option describing inserting low-rank adapter matrices and freezing the original weights is wrong because that describes a parameter-efficient fine-tuning method like LoRA, which the scenario explicitly rules out since the team wants to fine-tune all of the model's parameters, and ZeRO itself does not touch the model's architecture at all. The option describing 4-bit quantization while still keeping a full redundant copy on every GPU is wrong on both counts: ZeRO does not quantize anything, and its defining feature is specifically eliminating the full redundant copy on every GPU, not preserving it. The option describing recomputing gradients and optimizer states from activations at every step is wrong because ZeRO stores each GPU's own partitioned shard of that state rather than discarding and recomputing it; recomputing activations to save memory is a different, unrelated technique called activation checkpointing.

Source: Rajbhandari et al., 'ZeRO: Memory Optimizations Toward Training Trillion Parameter Models' (arXiv:1910.02054, SC20)

Correct answer: .

TIES-Merging addresses two specific sources of interference that plain task arithmetic ignores: redundant, small-magnitude parameter changes, and disagreement in sign across task vectors on the same parameter. Its three-step pipeline first trims each task vector by keeping only the top fraction of entries by absolute magnitude and zeroing the rest, treating small changes as likely noise; it then elects a single dominant sign per parameter by a magnitude-weighted vote across all the task vectors being merged; and finally it performs a disjoint merge, averaging together only the entries whose sign matches the elected sign for that parameter while discarding conflicting entries entirely. This trim-elect-merge sequence is what lets TIES-Merging recover much more of joint multi-task performance than naively summing task vectors. The option describing a trained attention-based mixing module re-optimized per input is wrong because that describes AdapterFusion-style composition of adapter modules at inference time, not TIES-Merging's one-time, gradient-free operation directly on task vectors in weight space before any deployment. The option describing converting task vectors into natural-language descriptions and re-prompting the base model is wrong because TIES-Merging never leaves weight space or touches the model's text interface at all; it operates purely on the numerical difference vectors. The option describing keeping only the single best-validated task vector and discarding the rest is wrong because that abandons the entire goal of combining multiple tasks' capabilities into one model, which is exactly what TIES-Merging's sign-election and disjoint-merge steps are designed to preserve from all the inputs.

Source: Yadav, Tam, Choshen, Raffel & Bansal, 'TIES-Merging: Resolving Interference When Merging Models' (NeurIPS 2023, arXiv:2306.01708)

Correct answer: .

SimPO's implicit reward for a candidate response is the average log-probability per token that the policy being trained assigns to that response, rather than the raw summed log-probability over all of its tokens. This average, length-normalized form matters precisely because an un-normalized sum of per-token log-probabilities (which are negative numbers) tends to grow more negative as a response gets longer, meaning a policy could appear to prefer shorter responses or exploit length in ways unrelated to actual quality; normalizing by length keeps the reward comparable across responses of different lengths and keeps it aligned with the same per-token likelihood that the model actually uses to generate text, which is also what lets SimPO drop DPO's frozen reference model entirely, since the reward is computed purely from the policy's own probabilities with nothing to compare against. The option describing an un-normalized raw sum with length-normalization meaning fixed-length truncation is wrong because SimPO's normalization is a per-token averaging of the reward itself, not a preprocessing step that truncates training data to a fixed token count. The option describing a frozen reference model's probability ratio exactly as in DPO is wrong because SimPO's entire point is eliminating that reference model dependency, replacing it with a reward computed solely from the policy being trained. The option describing counting human preference votes divided by word count is wrong because SimPO's reward is a property of the model's own output probabilities computed during training, not a tally of raw human votes from data collection.

Source: Meng, Xia & Chen, 'SimPO: Simple Preference Optimization with a Reference-Free Reward' (NeurIPS 2024, arXiv:2405.14734)

Correct answer: .

Sequence-level knowledge distillation replaces the training targets themselves: the teacher model is run with beam search over the training inputs to generate its own complete output sequences, and the student is then trained with ordinary cross-entropy to reproduce those teacher-generated sequences rather than the original human reference translations. This shifts the distillation signal from matching the teacher's per-token soft probability distribution at each position (word-level distillation) to approximating the teacher's full-sequence behavior as a whole, which Kim & Rush found could be combined with weight pruning to shrink translation models by up to roughly 13x with only a small drop in quality. The option describing only a lowered softmax temperature is wrong because that is simply a variant of word-level distillation's existing soft-label mechanism, not the sequence-level approach, which changes the training data itself rather than how sharply a per-token distribution is softened. The option describing training only on human reference translations with no teacher involvement is wrong because the teacher's beam-search output sequences are exactly what the student is trained on; the original references are not used in this scheme. The option describing a single averaged corpus-wide distribution applied at every position is wrong because sequence-level distillation generates a distinct output sequence from the teacher for each individual training input, not one global distribution reused everywhere.

Source: Kim & Rush, 'Sequence-Level Knowledge Distillation' (EMNLP 2016, arXiv:1606.07947)

Correct answer: .

LongLoRA's shifted sparse attention splits the long input context into several smaller groups and computes ordinary full attention only within each group independently, which is far cheaper than attending across the entire long context at once; to prevent this from isolating each group entirely, half of the attention heads have their tokens shifted by half a group's size before grouping, so those heads' group boundaries land in different places than the unshifted heads' boundaries, letting information flow across what would otherwise be a hard cutoff between neighboring groups. This mechanism is used only to make the fine-tuning phase itself cheap (it can be switched off at inference, where normal full attention is used), and combined with making LoRA's embedding and normalization layers trainable, it let the authors extend Llama 2 7B from a 4k to a 100k context on a single 8-GPU machine. The option describing a permanently fixed sliding window with no cross-window attention ever is wrong because S2-Attn's shifting mechanism specifically exists to preserve cross-group information flow, and it is optional at inference rather than a permanent restriction. The option describing unchanged dense attention sped up purely through 8-bit quantization is wrong because S2-Attn's saving comes from restricting attention to within smaller groups, not from quantizing the attention computation itself. The option describing replacing attention with a recurrent summarization module is wrong because LongLoRA keeps the transformer's attention mechanism throughout, only changing which tokens attend to which others during the fine-tuning phase.

Source: Chen et al., 'LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models' (ICLR 2024, arXiv:2309.12307)

Correct answer: .

Activation steering is a representation-engineering technique that never touches the model's weights at all: a steering vector is typically computed in advance as the average difference in residual-stream activations between pairs of contrasting prompts (for example, prompts eliciting honest versus dishonest responses), and that fixed vector is then added into the residual stream at a chosen layer during generation. Because the base weights are untouched, the behavioral shift is entirely a property of inference-time computation, which means it can be scaled up or down, swapped for a different steering vector, or dropped entirely on a per-request basis with no retraining, unlike a fine-tuned model where the behavior change is baked permanently into the weights. The option describing a labeled preference dataset and a full RLHF-style reinforcement-learning loop is wrong because activation steering's vectors are typically derived from a small set of contrasting prompts and simple activation averaging, not from reward modeling or policy-gradient optimization. The option describing permanently overwriting weight matrices is wrong because it describes something closer to merging a fine-tuned update into the model, which is the opposite of activation steering's inference-time-only intervention. The option claiming activation steering is mathematically identical to LoRA is wrong because LoRA adds a learned low-rank update to weight matrices that persists in the deployed model, whereas activation steering adds a vector to intermediate activations at inference time without altering any weight matrix.

Source: Turner et al., 'Steering Language Models With Activation Engineering' (arXiv:2308.10248, 2023); Zou et al., 'Representation Engineering: A Top-Down Approach to AI Transparency' (arXiv:2310.01405, 2023)

Correct answer: .

LoraHub combines existing LoRA modules by searching for a set of per-module combination coefficients using a gradient-free black-box optimization procedure, specifically Covariance Matrix Adaptation Evolution Strategy (CMA-ES), which proposes candidate coefficient settings, evaluates how well the resulting combined LoRA performs directly on the handful of available examples from the new task, and iteratively refines the coefficients over a small number of inference steps, all without ever computing gradients through the LoRA modules themselves or requiring any additional model parameters. This is what makes LoraHub usable in genuinely low-resource, few-shot settings where a full gradient-based training run is not an option. The option describing standard gradient-descent fine-tuning of a new combination layer, backpropagating through all the LoRA modules, is wrong because that is exactly the gradient-based training LoraHub is designed to avoid needing for the new task. The option describing always using equal, fixed weights regardless of task is wrong because LoraHub's coefficients are searched and adapted specifically for each new task's own few examples, not held fixed. The option describing weighting modules purely from a language model reading their text descriptions, without ever evaluating on the new task's examples, is wrong because LoraHub's coefficient search is driven by actual performance on the few available labeled examples, not by a purely textual judgment of module descriptions.

Source: Huang, Liu, Lin, Pang, Du & Lin, 'LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Composition' (COLM 2024, arXiv:2307.13269)

Correct answer: .

VeRA's core idea is to stop training a distinct pair of low-rank matrices for every layer the way ordinary LoRA does, and instead share a single pair of frozen, randomly initialized low-rank matrices across all of the layers being adapted. The only parameters actually trained per layer are small scaling vectors that rescale those shared, frozen random matrices differently at each layer, which lets VeRA reach roughly an order of magnitude or more fewer trainable parameters than LoRA while reporting comparable downstream performance on benchmarks like GLUE and instruction-tuned language models, which matters directly for a team needing to store a separate lightweight adaptation per customer. The option describing a single scalar learning-rate multiplier per layer applied to frozen base weights is wrong because that would not add any task-specific directional update at all, just a uniform rescaling of the base model's existing weights, which is not what VeRA's trainable scaling vectors do. The option describing training a separate full-rank update per layer and compressing it afterward with quantization is wrong because that increases rather than decreases trainable parameters before any compression, the opposite of VeRA's design. The option describing every layer reusing LoRA's own trained matrices with no scaling is wrong because VeRA's matrices are randomly initialized and kept entirely frozen, never trained, with only the per-layer scaling vectors actually learned.

Source: Kopiczko, Blankevoort & Asano, 'VeRA: Vector-based Random Matrix Adaptation' (ICLR 2024, arXiv:2310.11454)

Correct answer: .

Diff pruning learns a task-specific 'diff' vector that is added on top of the frozen pretrained weights, and during training this diff vector is pushed toward sparsity using a differentiable approximation to the L0-norm penalty, which directly encourages most of its entries to become exactly zero rather than relying on any separately applied threshold after the fact. Because only the nonzero positions and their values need to be stored per task, while the shared pretrained weights are amortized across every task, diff pruning can match fully fine-tuned baselines on the GLUE benchmark while modifying only about 0.5% of the pretrained model's parameters for each task. The option describing hand-picked layers chosen in advance for dense fine-tuning is wrong because diff pruning's sparsity pattern is learned automatically by the L0 penalty during training, scattered across the model rather than confined to a small number of preselected layers. The option describing full fine-tuning followed by threshold-based pruning afterward is wrong because diff pruning's sparsity is shaped during training itself by the differentiable L0 penalty, not discovered by pruning a fully fine-tuned model post hoc with a fixed threshold. The option describing a newly initialized sparse network trained from scratch, discarding pretrained values, is wrong because diff pruning adds its sparse diff vector on top of the existing frozen pretrained weights, which remain intact and are never replaced.

Source: Guo, Rush & Kim, 'Parameter-Efficient Transfer Learning with Diff Pruning' (ACL 2021, arXiv:2012.07463)

Correct answer: .

Context distillation treats the same base model as both teacher and student: run with the system prompt included, it acts as the teacher, producing an output distribution already shaped by the prompt's instructions; run without the prompt, the same model acts as the student. Training the student version to match the teacher version's output distribution on matching inputs transfers the prompt's behavioral effect directly into the student's weights, so the resulting model behaves as though the prompt were still present even when it is no longer included in the input, freeing up context window space and avoiding repeated prompt tokens on every request. The option describing appending the prompt's text as a hidden prefix in the embedding table is wrong because that still keeps the prompt's content present and consuming resources in some form, rather than removing it and relying on a distilled weight change. The option describing inserting new permanent attention heads derived from the prompt is wrong because context distillation changes the model's existing weights through an ordinary training signal, not by architecturally adding new heads. The option describing supervised fine-tuning directly on the prompt's own text with no teacher-student comparison is wrong because context distillation's defining mechanism is matching the prompted (teacher) and unprompted (student) versions of the same model's output distributions, not training on the prompt text itself as a target.

Source: Snell, Klein & Zhong, 'Learning by Distilling Context' (arXiv:2209.15189, 2022)

Correct answer: .

LoRA, Biderman et al. found, substantially underperforms full fine-tuning at learning a new target domain such as programming or math, in both instruction-tuning and continued-pretraining settings, yet it preserves more of the base model's performance on tasks outside that target domain and keeps more diverse generations than full fine-tuning does, even outperforming common explicit regularizers such as weight decay and dropout at limiting this forgetting. The paper attributes part of this tradeoff to rank: full fine-tuning's learned weight perturbations sit at a rank typically ten to a hundred times higher than a typical LoRA configuration's, which the authors suggest helps explain why LoRA's more constrained update learns less of the new domain while disturbing less of what the model already knew. The option claiming LoRA matches full fine-tuning on both learning and forgetting is wrong because the paper's central finding is exactly that the two methods diverge on those two axes. The option claiming LoRA outperforms full fine-tuning on the coding benchmark itself is wrong because the paper reports the opposite: LoRA substantially underperforms full fine-tuning at learning the target domain. The option about convergence speed is wrong because the paper's comparison concerns what capability each method ultimately learns and forgets, not which one takes longer to train.

Source: Biderman et al., 'LoRA Learns Less and Forgets Less' (arXiv:2405.09673, 2024)

Correct answer: .

Wortsman et al. found that instead of running a hyperparameter sweep and keeping only the checkpoint with the best validation accuracy, the weights of several (or even all) of the independently fine-tuned checkpoints, which emerge from the same pretrained initialization and sit in a shared low-error basin, can simply be averaged together into one set of weights, a 'soup.' Because the result is still a single model with the original architecture, it costs nothing extra at inference or in memory compared to shipping one checkpoint, unlike an ensemble. The paper reports this soup can exceed the single best individual checkpoint's accuracy, including on out-of-distribution robustness and downstream transfer tasks, and even produced a new state-of-the-art ImageNet top-1 result with a souped ViT-G model. Unlike task arithmetic or TIES-merging, which combine task vectors from models fine-tuned on different tasks to grant one model several capabilities, model soups average checkpoints fine-tuned on the same task and dataset with different hyperparameters, aiming for a better single-task model rather than a multi-task one. The option describing logit averaging at inference is wrong because that is ordinary ensembling, which does cost extra inference compute and memory per model kept loaded, exactly what the paper's weight-averaging approach avoids. The option describing distillation into a larger model is wrong because the paper never trains a new model from the others' outputs; it directly averages existing weights. The option claiming weight averaging is numerically unstable at this scale is wrong because the paper's central empirical claim, demonstrated across CLIP, ALIGN, and ViT-G scale models, is exactly the opposite: averaging works and helps.

Source: Wortsman et al., 'Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time' (arXiv:2203.05482, ICML 2022)

Correct answer: .

NEFTune adds independently sampled uniform noise, each entry drawn from the range [-1, 1], to the token embedding vectors on every forward pass during training, scaling the raw noise vector by a tunable base noise scale alpha divided by the square root of sequence length times embedding dimension; everything else in the training loop, the loss, the optimizer, and the data, is left completely unchanged, so the method adds no extra data and no extra compute. The paper reports that this single change took LLaMA-2-7B fine-tuned on Alpaca from roughly 30% to roughly 65% on AlpacaEval, with further gains of around 8-10 percentage points on other instruction datasets, and that even models already refined with RLHF benefited from additional NEFTune training. The option describing label smoothing is wrong because that changes the loss function's target distribution, not the embeddings, and is a different, unrelated regularization technique. The option describing a trainable adapter after the embedding layer is wrong because NEFTune adds no new trainable parameters at all; the noise is sampled fresh each step rather than learned. The option describing duplicating shuffled-word examples is wrong because NEFTune never touches the training data itself, only the embedding vectors computed from it during the forward pass.

Source: Jain et al., 'NEFTune: Noisy Embeddings Improve Instruction Finetuning' (arXiv:2310.05914, 2023)

Correct answer: .

LoReFT leaves every one of the base model's pretrained weights frozen and instead learns an intervention, parameterized by a low-rank projection matrix with orthonormal rows plus a small learned linear map, that edits a hidden representation only within that low-rank subspace: it computes a target value for the subspace from the current representation, then moves the representation's component inside that subspace toward that target while leaving everything outside the subspace completely untouched. This intervention is applied only at a fixed, small number of prefix and suffix token positions at a chosen set of layers, which is why its added inference cost does not grow with prompt length and why the paper reports it trains roughly 15 to 65 times fewer parameters than LoRA, around 0.031% of the base model's parameters for LLaMA-7B, while matching or exceeding other parameter-efficient methods across several benchmarks. The option describing new low-rank weight matrices initialized from existing weights is wrong because LoReFT never adds trainable weight matrices into the forward computation at all; it edits the representation that comes out of a layer, not the weights that produce it. The option describing replacing the final layers with a retrained feed-forward network is wrong because LoReFT keeps the entire pretrained architecture and its representations intact everywhere outside the small intervened subspace. The option describing training a subset of attention projection matrices is wrong because LoReFT does not touch any attention projection matrix; its only trainable parameters are the intervention's own projection matrix and linear map, applied to the representation after it leaves a layer.

Source: Wu et al., 'ReFT: Representation Finetuning for Language Models' (arXiv:2404.03592, 2024)

Correct answer: .

GaLore is explicitly a full-parameter training method: unlike LoRA, which freezes the pretrained weights and adds a trainable low-rank matrix pair alongside them, GaLore lets every original weight get updated, but before the gradient computed for a weight matrix reaches the optimizer, it is projected into a low-rank subspace, so memory-hungry optimizer state such as Adam's first and second moment estimates is stored and updated at that lower rank rather than at the full matrix's size. The paper reports this cuts optimizer-state memory by up to 65.5% (82.5% for an 8-bit variant), enough to let a 7B model be pretrained on a single 24GB consumer GPU with no model parallelism, checkpointing, or offloading, while matching full-rank training's performance on pretraining and fine-tuning benchmarks. The option describing frozen weights with 8-bit low-rank matrices is wrong because that describes a LoRA/QLoRA-style parameter-efficient approach, the opposite of GaLore's full-parameter updates. The option describing sharding weights, gradients, and optimizer states across multiple GPUs is wrong because that describes ZeRO's partitioning strategy across devices, not GaLore's single-GPU, low-rank gradient-projection approach. The option describing training on a random subset of tokens each step is wrong because GaLore's memory savings come from how the gradient is projected before reaching the optimizer, not from using less training data per step.

Source: Zhao et al., 'GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection' (arXiv:2403.03507, ICML 2024)