Zhou et al. (2022/2023), "Large Language Models Are Human-Level Prompt Engineers," propose Automatic Prompt Engineer (APE). How does APE generate and select an effective task instruction, according to the paper?
- It treats the instruction itself as a "program": an LLM proposes a pool of candidate instructions from a handful of input-output demonstrations, and each candidate is scored (for example, by how well it reproduces the demonstrations when used as a prompt) so the highest-scoring instruction can be selected or further refined
- It requires a human panel to write hundreds of candidate instructions, which the LLM then simply ranks by fluency without regard to task performance
- It fine-tunes the model's weights on many instruction/response pairs and treats the resulting model checkpoint itself as the "instruction"
- It works only for classification tasks with a fixed label set and cannot generate free-form instructions for open-ended tasks
Why A? And why not the others?
Correct answer: A. It treats the instruction itself as a "program": an LLM proposes a pool of candidate instructions from a handful of input-output demonstrations, and each candidate is scored (for example, by how well it reproduces the demonstrations when used as a prompt) so the highest-scoring instruction can be selected or further refined
APE frames instruction generation as a program-synthesis problem: given a small set of input-output demonstrations, an LLM is prompted to propose a pool of candidate natural-language instructions, and each candidate is then scored by a chosen metric, such as how well it lets the model reproduce the demonstrations when prepended as a prompt, so search and Monte Carlo-style resampling can converge on a high-scoring instruction. The paper reports this automatically generated instruction matches or beats human-written instructions on 19 of 24 tasks tested. The second option is wrong because no human panel writes the candidates; the LLM itself generates the pool of candidate instructions, and selection is driven by a task-performance score, not by fluency ranking. The third option is wrong because APE requires no fine-tuning or weight updates at all; it is a purely inference-time search over instruction text using a frozen model. The fourth option is wrong because the paper evaluates APE across 24 diverse NLP tasks, including open-ended generation and even steering models toward truthfulness, not solely fixed-label classification.
Source: Zhou, Muresanu, Han, Paster, Pitis, Chan, Ba, "Large Language Models Are Human-Level Prompt Engineers" (2023), arXiv:2211.01910