Madaan et al. (2023), "Self-Refine: Iterative Refinement with Self-Feedback," propose a prompting loop that improves a language model's own output without any additional training data, fine-tuning, or external verifier model. What is the loop, and which single model performs every role in it?
- A larger 'teacher' model grades the outputs of a separate, smaller 'student' model and returns a numeric score that the student uses to pick from several candidate answers
- A retrieval system feeds the model documents relevant to its own previous answer, and the model simply copies the most relevant retrieved sentence into its final response
- The same pretrained model first produces an initial answer, then critiques that answer as feedback, then uses its own critique to produce a refined answer, repeating this generate-feedback-refine loop for multiple rounds; no separate model or extra training is involved at any stage
- A human reviewer reads the model's first answer and writes the feedback, which is then pasted back into a second prompt for the model to revise
Why C? And why not the others?
Correct answer: C. The same pretrained model first produces an initial answer, then critiques that answer as feedback, then uses its own critique to produce a refined answer, repeating this generate-feedback-refine loop for multiple rounds; no separate model or extra training is involved at any stage
Self-Refine uses one pretrained model for every step of the loop: the model produces an initial answer, is then prompted to critique that same answer and generate feedback on its weaknesses, and is then prompted again to use its own feedback to produce a refined answer, with this generate-feedback-refine cycle repeatable for multiple rounds. No additional training data, fine-tuning, or separate verifier model is required, since the same model plays the generator, critic, and refiner roles purely through prompting; the paper reports around a 20 percent average performance gain across tasks like math reasoning and dialogue response. The option describing a separate larger teacher model grading a smaller student model is wrong because Self-Refine's central claim is that a single model can supply useful feedback about its own output, with no second model involved. The option describing a retrieval system feeding in outside documents is wrong because Self-Refine's feedback comes from the model's own critique of its prior answer, not from any retrieved external text. The option describing a human writing the feedback is wrong because Self-Refine is designed to work without any human in the loop.
Source: Madaan et al., 'Self-Refine: Iterative Refinement with Self-Feedback' (arXiv:2303.17651, 2023)