A team wants the reasoning-boosting benefit of few-shot chain-of-thought prompting on a new task, but has no labeled exemplars available and does not want to spend engineering effort hand-writing or retrieving demonstrations for every new problem type. Yasunaga et al. (2023), "Large Language Models as Analogical Reasoners," propose Analogical Prompting to address this. What does the model do under this method, and what does it avoid needing?
- The model retrieves the single most similar labeled example from a large curated exemplar database using a nearest-neighbor embedding search, then copies that example's reasoning structure exactly
- The model is fine-tuned on a small set of analogous problems drawn from a related domain before being asked to solve the target problem
- The model asks a human expert, via an interactive clarification step, to supply one worked analogy before attempting to solve the problem
- The model is prompted to recall or self-generate relevant exemplars and, where useful, relevant background knowledge related to the given problem before solving it, entirely on its own; the method avoids needing any labeled exemplars to be hand-written or retrieved from an external source
Why D? And why not the others?
Correct answer: D. The model is prompted to recall or self-generate relevant exemplars and, where useful, relevant background knowledge related to the given problem before solving it, entirely on its own; the method avoids needing any labeled exemplars to be hand-written or retrieved from an external source
Analogical Prompting is designed so the model itself recalls or generates exemplars and relevant background knowledge related to the problem at hand, inspired by how humans draw on analogous past experience when reasoning, and it does this entirely at inference time on its own. Because the exemplars are self-generated rather than sourced externally, the method avoids the engineering cost of hand-writing demonstrations or building and querying a retrieval system, while still tailoring the generated exemplars to each specific problem; the paper reports it outperforming both zero-shot chain-of-thought and manual few-shot chain-of-thought across math, code generation, and other reasoning benchmarks. The option describing retrieval of the single most similar example from a curated database is wrong because that reintroduces exactly the external retrieval and curated-labeling burden the method is designed to avoid. The option describing fine-tuning on analogous problems is wrong because Analogical Prompting requires no training step; it operates purely through prompting a frozen model. The option describing a human supplying a worked analogy is wrong because the method is meant to be fully automatic, with no human clarification step involved.
Source: Yasunaga et al., 'Large Language Models as Analogical Reasoners' (arXiv:2310.01714, 2023)