Zhou et al. (2022), "Least-to-Most Prompting Enables Complex Reasoning in Large Language Models," describe a two-stage strategy for solving problems harder than those shown in the prompt's examples. What is that strategy?
- First prompt the model to decompose the problem into a sequence of simpler subproblems, then sequentially prompt it to solve each subproblem in order, feeding each prior subproblem's answer into the context used for the next
- First fine-tune the model on the hardest available examples, then evaluate it zero-shot on easier examples to measure generalization downward
- First ask the model to guess the final answer directly, then ask it to generate a chain-of-thought justification for that already-chosen answer after the fact
- First run the same prompt through several different LLMs from different vendors, then pick whichever vendor's answer appears most frequently
Correct answer: A. First prompt the model to decompose the problem into a sequence of simpler subproblems, then sequentially prompt it to solve each subproblem in order, feeding each prior subproblem's answer into the context used for the next
Least-to-most prompting works in two stages: a decomposition stage where the model breaks a complex problem into an ordered list of simpler subproblems, followed by a sequential problem-solving stage where the model solves each subproblem in turn, with the answers to earlier subproblems included in the context used to solve later ones; this lets the technique generalize to problems harder than the demonstrations shown, unlike standard chain-of-thought prompting. The second option is wrong because the technique involves no fine-tuning at all; it is a purely prompting-based, inference-time method applied to a frozen model. The third option is wrong because it reverses the actual order and purpose of the method -- least-to-most decomposes a problem before solving it, rather than justifying an answer that was already guessed, and post-hoc justification is not what the paper studies. The fourth option is wrong because the method uses a single model working sequentially through ordered subproblems, not a multi-vendor voting ensemble.
Source: Zhou et al., "Least-to-Most Prompting Enables Complex Reasoning in Large Language Models" (ICLR 2023), arXiv:2205.10625