Zhang et al. (2022), "Automatic Chain of Thought Prompting in Large Language Models," propose Auto-CoT to avoid the manual effort, and potential for hand-written mistakes, involved in writing few-shot chain-of-thought exemplars by hand. What two-step procedure does Auto-CoT use to build its demonstrations automatically?
- First, ask human annotators to hand-write a reasoning chain for every single question in the dataset; second, use the model to pick which of those human-written chains looks most fluent
- First, cluster the dataset's questions by similarity and pick one representative question from each cluster; second, generate a reasoning chain for each representative question automatically using zero-shot chain-of-thought (for example, "Let's think step by step"), assembling the resulting diverse set of question-plus-generated-chain pairs into the few-shot demonstrations
- First, fine-tune the model on a large labeled reasoning dataset; second, discard the few-shot examples entirely, since the fine-tuned model no longer needs any demonstrations
- First, generate one reasoning chain for the very first question in the dataset; second, reuse that exact same chain, unmodified, as the only demonstration for every other question regardless of topic
Why B? And why not the others?
Correct answer: B. First, cluster the dataset's questions by similarity and pick one representative question from each cluster; second, generate a reasoning chain for each representative question automatically using zero-shot chain-of-thought (for example, "Let's think step by step"), assembling the resulting diverse set of question-plus-generated-chain pairs into the few-shot demonstrations
Auto-CoT first partitions the dataset's questions into clusters by similarity and selects one representative question per cluster, then generates a reasoning chain for each representative question automatically using zero-shot chain-of-thought prompting, and finally assembles this diverse set of question-and-generated-chain pairs into the few-shot demonstrations used for the actual task, which the paper shows matches or exceeds manually written chain-of-thought demonstrations on ten reasoning benchmarks. Sampling diverse clusters rather than similar questions reduces the chance that a single mistaken generated chain gets repeated across many similar demonstrations. The second option is wrong because it reintroduces the manual, human-written-chain effort that Auto-CoT is explicitly designed to eliminate. The third option is wrong because Auto-CoT involves no fine-tuning; it remains a purely prompting-based method that still uses few-shot demonstrations, just auto-generated ones. The fourth option is wrong because Auto-CoT deliberately draws diverse representative questions from multiple clusters rather than repeating one chain for every question regardless of topic, which is precisely the failure mode diversity is meant to avoid.
Source: Zhang, Zhang, Li, Smola, "Automatic Chain of Thought Prompting in Large Language Models" (ICLR 2023), arXiv:2210.03493