Both bottleneck adapter modules (Houlsby et al.) and LoRA are parameter-efficient fine-tuning methods that add a small number of new trainable parameters to a frozen pretrained model, but they differ in how those new parameters interact with the model's forward pass at inference time. Which statement correctly distinguishes them on this specific point?
- Bottleneck adapters add their new parameters as a parallel low-rank update to existing weight matrices that can be merged back into those weights after training, while LoRA inserts new down-projection and up-projection feed-forward layers with a nonlinearity directly into the forward pass
- Both methods add their new parameters in mathematically identical ways, differing only in what random distribution is used to initialize the new weights before training begins
- Both methods modify the exact same weight matrices inside the attention mechanism in an identical fashion, differing only in the name each paper's authors chose to give the technique
- Bottleneck adapters insert new down-projection and up-projection feed-forward layers with a nonlinearity directly into the forward pass, adding extra sequential computation and inference latency, while LoRA's trainable matrices form a parallel low-rank update to an existing weight matrix that can be merged back into that weight after training, adding no extra inference latency
Why D? And why not the others?
Correct answer: D. Bottleneck adapters insert new down-projection and up-projection feed-forward layers with a nonlinearity directly into the forward pass, adding extra sequential computation and inference latency, while LoRA's trainable matrices form a parallel low-rank update to an existing weight matrix that can be merged back into that weight after training, adding no extra inference latency
Houlsby et al.'s bottleneck adapters are new modules -- a down-projection, a nonlinearity, and an up-projection -- spliced directly into the transformer's forward pass, so every inference call must sequentially compute through that extra module, adding a small but real amount of latency and depth that cannot be removed without retraining. LoRA instead represents its trainable low-rank matrices as an additive update running in parallel to an existing weight matrix rather than as a new sequential layer, and because the update and the original weight are simply added together, the low-rank product can be folded back into the original weight matrix once training is finished, leaving a model that runs at the original architecture's inference speed with no separate module left in the forward path. The option reversing which method merges and which inserts a sequential layer states the opposite of both papers' actual designs. The option claiming the two methods are mathematically identical aside from initialization ignores that one changes the network's computational graph with a new sequential module and the other only changes an existing weight matrix's value. The option claiming both modify the same attention weight matrices identically ignores that adapters are typically inserted as standalone modules after the attention and feed-forward sublayers rather than modifying attention's own weight matrices at all.
Source: Houlsby et al., 'Parameter-Efficient Transfer Learning for NLP' (arXiv:1902.00751, 2019); Hu et al., 'LoRA: Low-Rank Adaptation of Large Language Models' (arXiv:2106.09685, 2021)