A model is asked a specific, detail-heavy physics question and, despite reasoning step by step, applies the wrong underlying formula because it dives straight into the specific numbers without first recalling which general principle governs the situation. Zheng et al. (2023), "Take a Step Back: Evoking Reasoning via Abstraction in Large Language Models," propose Step-Back Prompting to address exactly this failure mode. How does the technique change what the model is prompted to do before answering?
- It has the model answer the specific question first, and only afterward asks it to state which general principle it implicitly used, purely as a post-hoc explanation with no effect on the answer
- It has the model retrieve the original specific question from a database of similar past questions and copy the closest match's stored answer
- It has the model break the specific question into the smallest possible sub-questions and answer each sub-question independently before summing the sub-answers
- It first prompts the model to derive a higher-level, more abstract question or general principle from the specific instance, then has the model reason from that abstraction to answer the original specific question, rather than reasoning directly from the details alone
Why D? And why not the others?
Correct answer: D. It first prompts the model to derive a higher-level, more abstract question or general principle from the specific instance, then has the model reason from that abstraction to answer the original specific question, rather than reasoning directly from the details alone
Step-Back Prompting has the model perform an abstraction step before it ever engages with the specific question's details: it first prompts the model to derive a higher-level question or general principle that the specific instance is an example of, and only then has the model reason from that abstraction down to the original, detail-heavy question, rather than reasoning directly from the details alone. The paper reports this abstraction-first ordering produces substantial gains on reasoning-intensive tasks tested with PaLM-2L, GPT-4, and Llama2-70B, including double-digit percentage improvements on STEM knowledge and multi-hop reasoning benchmarks. The option describing a post-hoc statement of the general principle after already answering is wrong because doing the abstraction after the answer has already been produced cannot change how that answer was derived, which defeats the technique's purpose. The option describing retrieving a stored answer from a database of similar past questions is wrong because Step-Back Prompting generates its own abstraction through reasoning, not through looking up a previously stored answer. The option describing decomposition into the smallest sub-questions is wrong because that describes decomposition-style prompting, not the abstraction-then-reasoning structure this paper proposes.
Source: Zheng, Mishra, Chen, Cheng, Chi, Le & Zhou, 'Take a Step Back: Evoking Reasoning via Abstraction in Large Language Models' (arXiv:2310.06117, 2023)