Liu et al. (2022), "Generated Knowledge Prompting for Commonsense Reasoning," propose a two-stage prompting pipeline for commonsense question answering. What are the two stages?
- First, retrieve documents from a fixed external knowledge base such as an encyclopedia; second, fine-tune the model's weights on those retrieved documents before it answers the question
- First, ask the model to answer the question directly with no context; second, ask a different model to translate that answer into another language to check consistency
- First, prompt a language model to generate several relevant knowledge statements about the question's topic; second, provide those generated knowledge statements as additional context alongside the original question in a separate prompt that produces the final answer
- First, generate several candidate final answers; second, average their token probabilities together to produce a single blended answer string
Why C? And why not the others?
Correct answer: C. First, prompt a language model to generate several relevant knowledge statements about the question's topic; second, provide those generated knowledge statements as additional context alongside the original question in a separate prompt that produces the final answer
Generated knowledge prompting first prompts a language model to produce several candidate knowledge statements relevant to the question's topic, using a separate knowledge-generation prompt with its own few-shot examples, and then feeds those generated statements as additional context alongside the original question into a second prompt whose job is to produce the final answer, integrating over multiple generated knowledge statements when helpful. This achieved state-of-the-art results on commonsense benchmarks like NumerSense, CommonsenseQA 2.0, and QASC. The second option is wrong because the method generates knowledge from the language model's own parametric knowledge rather than retrieving it from a fixed external knowledge base, and it involves no fine-tuning or weight updates at any stage. The third option is wrong because there is no translation-based consistency check between models in this pipeline. The fourth option is wrong because the method conditions the final answer on generated knowledge text used as context, not on averaging token-level probabilities across multiple candidate answer strings.
Source: Liu, Liu, Lu, Welleck, West, Le Bras, Choi, Hajishirzi, "Generated Knowledge Prompting for Commonsense Reasoning" (ACL 2022), arXiv:2110.08387