A general-purpose language model that previously solved multi-step arithmetic and logic problems reliably is fine-tuned exclusively on a narrow customer-support ticket dataset for several epochs. Afterward, the model handles support tickets well but its arithmetic and logic performance has dropped sharply, even though none of the fine-tuning data was arithmetic-related. What phenomenon does this describe, and what tends to make it worse?
- This is overfitting to noise in the customer-support dataset, meaning the model has memorized incorrect labels in that specific dataset; it has nothing to do with the model's previously learned capabilities, which fine-tuning cannot affect
- This is a tokenizer mismatch, meaning the fine-tuning process silently changed the model's vocabulary so that arithmetic expressions are now encoded into different tokens than during pretraining
- This is catastrophic forgetting: fine-tuning updates the model's weights to optimize performance on the new, narrow task distribution, and in doing so can overwrite representations the model relied on for previously learned capabilities; this effect tends to become more pronounced the further the fine-tuning data diverges from the pretraining distribution and can worsen with more fine-tuning steps or a higher learning rate
- This is a known evaluation artifact where the arithmetic benchmark itself becomes unreliable after any fine-tuning run, regardless of what the fine-tuning data contained, so the drop is not a real change in the model's underlying ability
Why C? And why not the others?
Correct answer: C. This is catastrophic forgetting: fine-tuning updates the model's weights to optimize performance on the new, narrow task distribution, and in doing so can overwrite representations the model relied on for previously learned capabilities; this effect tends to become more pronounced the further the fine-tuning data diverges from the pretraining distribution and can worsen with more fine-tuning steps or a higher learning rate
Catastrophic forgetting occurs because gradient updates during fine-tuning reshape the same shared weights the model uses for every capability; optimizing those weights hard toward a new, narrow task distribution can overwrite or distort the representations the model was previously relying on for unrelated skills like arithmetic, even though no arithmetic examples appeared in the new training data. This effect is documented to become more severe as the fine-tuning distribution diverges further from the original pretraining distribution, and can be worsened by training for more steps or with a higher learning rate, both of which push the weights further from their pretrained configuration. The option attributing this to memorized label noise is wrong because forgetting is a structural side effect of weight updates, not a symptom of bad labels in the fine-tuning set. The option attributing this to a tokenizer mismatch is wrong because fine-tuning as described does not alter the model's vocabulary or tokenization scheme. The option calling it a benchmark artifact is wrong because the described capability loss is a real, well-studied consequence of how fine-tuning updates shared parameters, not a measurement error.
Source: Luo et al., 'An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuning' (arXiv:2308.08747, 2023)