Brown et al. (2020), "Language Models are Few-Shot Learners," describe evaluating GPT-3 in a few-shot setting on many tasks. What does "few-shot" mean in this evaluation, and what happens to the model's weights during it?
- A small number of labeled examples for the task are used to fine-tune GPT-3's weights with a few additional gradient-descent steps before the model is evaluated on new examples
- A small number of task examples are included as text directly within the prompt given to the model at inference time, and GPT-3 is evaluated on new examples using this prompt alone, with no gradient updates or fine-tuning of its weights performed for the task
- GPT-3's weights are duplicated into several smaller copies, each fine-tuned on a few examples from a different task, and the copy that performs best on a validation set is kept
- A small, separate classifier head is attached to GPT-3 and trained from scratch on a few labeled examples, while the rest of the pretrained model is frozen
Why B? And why not the others?
Correct answer: B. A small number of task examples are included as text directly within the prompt given to the model at inference time, and GPT-3 is evaluated on new examples using this prompt alone, with no gradient updates or fine-tuning of its weights performed for the task
Brown et al. describe few-shot evaluation as conditioning GPT-3 purely through text: a handful of example input-output pairs for the task are written directly into the prompt, followed by a new input, and the model is asked to produce the corresponding output -- all of this happens through ordinary text generation at inference time, with no gradient updates, fine-tuning, or any change whatsoever to the model's weights performed for the task. The option describing a few additional gradient-descent steps is wrong because that describes traditional few-shot fine-tuning, which the paper explicitly contrasts with its in-context approach applied to GPT-3. The option describing duplicating and separately fine-tuning several model copies is wrong because only a single frozen copy of GPT-3 is used, conditioned solely through the prompt, not multiple task-specific fine-tuned copies. The option describing an attached, separately trained classifier head is wrong because no new parameters of any kind are added to or trained on top of GPT-3 for the task; the entire model remains exactly as pretrained.
Source: Brown et al., "Language Models are Few-Shot Learners" (2020), arXiv:2005.14165