A team wants to adapt a large pretrained language model to a new domain without updating billions of parameters or storing a full second copy of the model's weights. They choose LoRA (Low-Rank Adaptation) to do this. Under Hu et al.'s original LoRA method, how does the technique actually reduce the number of trainable parameters during fine-tuning?
- It fine-tunes every parameter in the pretrained model as usual, but uses a much smaller learning rate so the resulting weight changes stay numerically small enough to store efficiently
- It freezes all of the pretrained weight matrices and, for selected layers, injects a pair of much smaller trainable matrices whose low-rank product approximates the update those weights would otherwise need, so only that pair is trained
- It deletes the attention layers of the pretrained model and replaces them with a smaller, randomly initialized transformer block that is trained from scratch on the new domain data
- It converts the entire pretrained model to 8-bit integer weights and performs ordinary full-parameter backpropagation directly on those compressed integer weights
Why B? And why not the others?
Correct answer: B. It freezes all of the pretrained weight matrices and, for selected layers, injects a pair of much smaller trainable matrices whose low-rank product approximates the update those weights would otherwise need, so only that pair is trained
LoRA leaves the pretrained weight matrices frozen and, for the layers it targets, adds a new pair of much smaller matrices -- one that projects down to a low intermediate rank and one that projects back up -- whose product forms a low-rank approximation of whatever weight update the task would otherwise require; only the two small matrices are trained, which is why the trainable parameter count drops so sharply while the frozen base weights carry over unchanged. The option describing an ordinary full fine-tune with a smaller learning rate is wrong because a smaller learning rate does not shrink the number of parameters being updated, only the size of each update, so it does nothing to reduce the underlying memory and compute cost. The option describing deleting and retraining the attention layers from scratch is wrong because LoRA never removes or replaces any pretrained component; it augments the existing frozen weights alongside a small addition rather than discarding learned structure. The option describing full-parameter backpropagation on 8-bit integer weights is wrong because that describes a form of quantized full fine-tuning, not LoRA's low-rank decomposition, and still requires updating every original parameter rather than a small injected pair.
Source: Hu et al., 'LoRA: Low-Rank Adaptation of Large Language Models' (arXiv:2106.09685, 2021)