Kojima et al. (2022), "Large Language Models are Zero-Shot Reasoners," show that a specific technique substantially improves LLM performance on multi-step reasoning benchmarks without any task-specific worked examples in the prompt. What is this zero-shot chain-of-thought technique, and how does it differ from the few-shot chain-of-thought prompting of Wei et al. (2022)?
- Fine-tuning the model on a small labeled set of step-by-step solutions before inference, whereas Wei et al.'s method requires no training at all
- Providing the correct final answer to the model up front and asking it to work backward to justify it, whereas Wei et al.'s method asks for the answer with no justification
- Appending a task-agnostic trigger phrase such as "Let's think step by step" before the answer, with no worked examples in the prompt at all, whereas Wei et al.'s method requires several few-shot exemplars that each include their own written-out reasoning steps
- Replacing the multiple-choice options with open-ended text so the model can no longer guess from answer choices, a change unrelated to Wei et al.'s few-shot method
Why C? And why not the others?
Correct answer: C. Appending a task-agnostic trigger phrase such as "Let's think step by step" before the answer, with no worked examples in the prompt at all, whereas Wei et al.'s method requires several few-shot exemplars that each include their own written-out reasoning steps
Kojima et al. (2022) show that a single, task-agnostic trigger phrase like "Let's think step by step," inserted before the model generates its answer with zero worked examples, elicits a chain of reasoning and substantially improves accuracy on arithmetic and symbolic reasoning benchmarks such as MultiArith and GSM8K, calling this Zero-shot-CoT. This differs from Wei et al.'s few-shot chain-of-thought prompting, which requires several exemplars in the prompt that each spell out intermediate reasoning steps leading to their answers. The second option is wrong because Zero-shot-CoT involves no fine-tuning or gradient updates; the trigger phrase is applied at inference time to a frozen, pretrained model. The third option is wrong because it describes working backward from a given final answer, which is the opposite of eliciting reasoning that leads to an answer the model has not yet produced. The fourth option is wrong because Zero-shot-CoT makes no change to how answer choices are presented; the entire intervention is the trigger phrase inserted before the model's output.
Source: Kojima, Gu, Reid, Matsuo, Iwasawa, "Large Language Models are Zero-Shot Reasoners" (2022), arXiv:2205.11916