Press et al. (2022), "Measuring and Narrowing the Compositionality Gap in Language Models," introduce the "self-ask" prompting method to address multi-hop questions whose sub-answers the model already knows individually but fails to combine correctly. How does self-ask structure the model's output?
- The model is asked to answer the multi-hop question directly in a single token with no intermediate text of any kind, which the paper shows eliminates the compositionality gap entirely
- Two separate, independently prompted models debate each other's answers over several rounds until they converge on the same final answer, which is the paper's central proposed method
- The compositionality gap is closed simply by scaling up model size, and self-ask is presented only as an unrelated historical baseline that the paper argues should be discarded
- The prompt's few-shot exemplars teach the model to explicitly decide whether a follow-up question is needed, then pose and answer that follow-up question itself within the same generation, repeating as needed before stating the final answer -- a structure into which an external search engine can optionally be plugged to answer the follow-up questions instead
Why D? And why not the others?
Correct answer: D. The prompt's few-shot exemplars teach the model to explicitly decide whether a follow-up question is needed, then pose and answer that follow-up question itself within the same generation, repeating as needed before stating the final answer -- a structure into which an external search engine can optionally be plugged to answer the follow-up questions instead
The paper defines the compositionality gap as the gap between how often a model answers all of a multi-hop question's individual sub-questions correctly versus how often it produces the correct composed final answer, and it shows this gap does not reliably shrink as GPT-3-family models scale up. Self-ask narrows it by using few-shot exemplars that teach the model to state whether a follow-up question is needed, then ask and answer that follow-up itself before giving a final answer, repeating as necessary; the same structured slot for follow-up questions also makes it easy to swap in an external search engine to answer them instead, which further improves accuracy. The second option is wrong because self-ask relies on generating intermediate follow-up questions and answers, the opposite of a single-token direct answer, and the paper does not claim any method eliminates the gap entirely. The third option is wrong because no multi-model debate scheme is described. The fourth option is wrong because the paper's key finding is that scaling alone does not reliably close the gap, and self-ask is the paper's own main proposed method, not a discarded baseline.
Source: Press, Zhang, Min, Schmidt, Smith, Lewis, "Measuring and Narrowing the Compositionality Gap in Language Models" (Findings of EMNLP 2023), arXiv:2210.03350