Chia et al. (2023), "Contrastive Chain-of-Thought Prompting," start from a surprising finding in prior work: giving a model chain-of-thought demonstrations with deliberately invalid reasoning had only a small negative effect compared to fully valid demonstrations, suggesting standard CoT does not clearly teach the model what mistakes to avoid. What do the authors propose instead, and what is its rationale?
- Removing chain-of-thought demonstrations altogether and relying solely on zero-shot instructions, on the theory that any demonstration risks teaching a spurious pattern
- Providing both valid and invalid reasoning demonstrations side by side for the same or paired problems, together with an automatic method for constructing such contrastive demonstrations, so the model is explicitly shown examples of flawed reasoning alongside correct reasoning rather than only ever seeing correct chains
- Increasing the number of valid chain-of-thought demonstrations from a handful to several dozen, so that sheer repetition overwhelms any influence from invalid reasoning
- Replacing human-written demonstrations with demonstrations generated entirely by a separate, larger teacher model, without regard to whether the reasoning in them is valid or invalid
Why B? And why not the others?
Correct answer: B. Providing both valid and invalid reasoning demonstrations side by side for the same or paired problems, together with an automatic method for constructing such contrastive demonstrations, so the model is explicitly shown examples of flawed reasoning alongside correct reasoning rather than only ever seeing correct chains
Contrastive Chain-of-Thought proposes giving the model both valid and invalid reasoning demonstrations for the same or paired problems, inspired by the observation that humans learn effectively from both positive and negative examples, and it introduces an automatic method for constructing such contrastive demonstration pairs rather than requiring them to be hand-written. The rationale is that standard chain-of-thought only ever shows correct reasoning and therefore never explicitly informs the model about what kinds of mistakes to avoid, which the authors argue is why invalid demonstrations had surprisingly little negative effect in prior work; showing the flawed reasoning alongside the correct reasoning is meant to close that gap. The option describing removal of all demonstrations is wrong because contrastive chain-of-thought still relies on demonstrations, now paired with invalid counterparts, rather than eliminating them. The option describing simply adding more valid demonstrations is wrong because the paper's fix is about demonstration content and pairing, not raw demonstration count. The option describing demonstrations generated entirely by a larger teacher model is wrong because the method's automatic construction process is not defined by, or dependent on, using a separate larger model as a teacher.
Source: Chia, Chen, Luu, Poria & Bing, 'Contrastive Chain-of-Thought Prompting' (arXiv:2311.09277, 2023)