After an agent fails a coding task on its first attempt, a framework has the agent generate a written, natural-language reflection on what went wrong and why, store that reflection in a memory buffer, and then attempt the same task again with that reflection available to it -- with no change to the underlying model's weights at any point. What does this Reflexion-style approach rely on to improve performance across attempts?
- A traditional reinforcement-learning gradient update that adjusts the model's parameters after each failed attempt, using the failure as a scalar reward signal
- Fine-tuning a copy of the model on the transcript of the failed attempt before the next attempt begins, so the improvement is baked into new weights
- Verbal reinforcement -- having the agent verbally reflect on task feedback and store that self-generated reflective text in an episodic memory buffer, which is then fed back in on later attempts to induce better decision-making, entirely without updating any model weights
- Randomly perturbing the prompt's wording on each retry until one phrasing happens to produce a correct answer, independent of any feedback about the previous failure
Why C? And why not the others?
Correct answer: C. Verbal reinforcement -- having the agent verbally reflect on task feedback and store that self-generated reflective text in an episodic memory buffer, which is then fed back in on later attempts to induce better decision-making, entirely without updating any model weights
Reflexion relies on verbal reinforcement: the agent verbally reflects on feedback about its own failure, stores that self-generated reflective text in an episodic memory buffer, and that stored reflection is fed back into the agent's context on the next attempt to induce better decision-making, all without ever touching the underlying model's weights. The option describing a gradient update from a scalar reward is wrong because that describes conventional reinforcement learning acting on model parameters, which is precisely what Reflexion is designed to avoid -- the improvement here comes from language stored and reused in context, not from a parameter update. The option describing fine-tuning a copy of the model on the failed transcript is wrong for the same reason: it bakes the improvement into new weights, whereas Reflexion's whole mechanism is to leave the weights untouched and instead carry improvement forward as reusable text. The option describing random prompt perturbation is wrong because it describes a trial-and-error process disconnected from any actual feedback about why the previous attempt failed, whereas Reflexion's reflection is specifically generated from and responsive to that feedback.
Source: Shinn et al., 'Reflexion: Language Agents with Verbal Reinforcement Learning' (2023), arXiv:2303.11366 -- describes agents that 'verbally reflect on task feedback signals, then maintain their own reflective text in an episodic memory buffer to induce better decision-making in subsequent trials,' without updating model weights.