Researchers take a large pretrained language model and fine-tune it on a diverse collection of more than 60 different NLP tasks, each phrased as a natural-language instruction, then evaluate it on entirely new tasks it never saw during this fine-tuning, with zero examples given at test time. What did Wei et al.'s FLAN study find about this kind of instruction tuning?
- Instruction tuning only improved performance when the model was still given several worked examples of the new task at test time; with zero examples given, it performed no better than the original untuned model
- Instruction tuning substantially improved zero-shot performance on the held-out, unseen tasks, in some cases surpassing a much larger model's zero-shot performance and even beating that larger model's few-shot performance on several benchmarks; the gains depended on tuning across a large number of diverse task types, not just scaling up the amount of data for a single task
- Instruction tuning had no measurable effect on unseen tasks at all, and only helped on the exact same 60-plus tasks it was fine-tuned on, meaning the benefit was pure memorization rather than any generalization
- Instruction tuning worked only because the model's parameter count was increased at fine-tuning time; the improvement disappeared entirely once the study controlled for holding the model's size fixed
Why B? And why not the others?
Correct answer: B. Instruction tuning substantially improved zero-shot performance on the held-out, unseen tasks, in some cases surpassing a much larger model's zero-shot performance and even beating that larger model's few-shot performance on several benchmarks; the gains depended on tuning across a large number of diverse task types, not just scaling up the amount of data for a single task
FLAN fine-tuned a 137-billion-parameter pretrained model on over 60 NLP tasks verbalized as natural-language instructions and then tested it, with no examples given, on tasks held out from that tuning set entirely; the tuned model's zero-shot performance improved substantially, surpassing a much larger untuned model's zero-shot results on most evaluated tasks and even beating that larger model's few-shot performance on several benchmarks, with ablations showing the number of distinct task types used in tuning -- not just raw data volume -- was central to the gain. The option claiming the benefit required examples at test time is wrong because the whole point of the finding is that the gains showed up under a strictly zero-shot evaluation. The option claiming the model only improved on the exact tasks it was tuned on is wrong because the evaluated tasks were specifically held out and unseen during tuning. The option attributing the gain purely to increased parameter count is wrong because the ablations isolate task diversity, not model scale, as the key driver.
Source: Wei et al., 'Finetuned Language Models Are Zero-Shot Learners' (arXiv:2109.01652, 2021)