A team building a math word-problem solver notices that a large language model using standard chain-of-thought prompting writes out a correct step-by-step plan in natural language but still makes an arithmetic slip when computing the final number, producing a wrong answer despite sound reasoning. Gao et al. (2022), "PAL: Program-Aided Language Models," propose an alternative prompting approach specifically to fix this class of error. What does PAL have the model do differently?
- PAL asks the model to write out the same natural-language reasoning chain twice independently and takes whichever of the two final numeric answers appears first
- PAL replaces every arithmetic step with a request for the model to look up the answer in a retrieved external knowledge base of pre-solved problems
- PAL prompts the model to translate the problem into intermediate steps expressed as runnable code (for example Python statements) and hands that generated program to an external interpreter to execute, using the interpreter's output as the final answer instead of having the model compute it itself
- PAL fine-tunes the underlying model on millions of additional arithmetic examples so it memorizes correct computations for common operation patterns
Why C? And why not the others?
Correct answer: C. PAL prompts the model to translate the problem into intermediate steps expressed as runnable code (for example Python statements) and hands that generated program to an external interpreter to execute, using the interpreter's output as the final answer instead of having the model compute it itself
PAL keeps the language model responsible only for reading the word problem and decomposing it into a short program, typically statements that mirror the natural-language reasoning steps, but hands the actual arithmetic evaluation of that program to a real interpreter rather than asking the model to carry out the computation itself. Because the interpreter executes the generated code deterministically, the kind of small arithmetic slip that undermines otherwise sound chain-of-thought reasoning disappears, since the model never has to compute the final number by reasoning about digits. The option describing two independent natural-language reasoning chains with the first answer kept is wrong because that describes neither PAL nor genuine self-consistency, which uses majority voting across many samples rather than order of arrival, and it still leaves all arithmetic to the model. The option describing lookup in a retrieved knowledge base of pre-solved problems is wrong because PAL involves no retrieval step at all; it only needs a code interpreter. The option describing additional fine-tuning is wrong because PAL is a pure prompting technique applied to a frozen, already-trained model via few-shot examples that pair problems with programs.
Source: Gao, Madaan, Zhou, Alon, Liu, Yang, Callan & Neubig, 'PAL: Program-aided Language Models' (arXiv:2211.10435, 2022)