A team wants to steer a large black-box LLM they cannot fine-tune (no access to weights) toward producing summaries that reliably include certain keywords, without hand-crafting a new instruction for every single input document. Li et al. (2023), "Guiding Large Language Models via Directional Stimulus Prompting," propose a framework for exactly this setting. What mechanism does Directional Stimulus Prompting use, and how is that mechanism trained?
- It trains a separate, small, tunable policy model (such as a T5-sized model) to generate an instance-specific 'directional stimulus,' a short hint or set of keywords tailored to each input, which is inserted into the prompt sent to the black-box LLM; the policy model itself is optimized via supervised fine-tuning on labeled data and/or reinforcement learning using rewards derived from the black-box LLM's own output quality
- It fine-tunes the black-box LLM's own weights directly on a small labeled dataset of ideal summaries, despite the team's lack of weight access, by using a gradient-free zeroth-order optimization method against the model's API
- It hard-codes the same fixed list of keywords into every prompt regardless of the input document, relying on the black-box LLM to decide on its own which of the fixed keywords are actually relevant
- It relies entirely on the black-box LLM's built-in retrieval-augmented generation feature to pull in relevant keywords from a search index, with no additional model or training step involved
Why A? And why not the others?
Correct answer: A. It trains a separate, small, tunable policy model (such as a T5-sized model) to generate an instance-specific 'directional stimulus,' a short hint or set of keywords tailored to each input, which is inserted into the prompt sent to the black-box LLM; the policy model itself is optimized via supervised fine-tuning on labeled data and/or reinforcement learning using rewards derived from the black-box LLM's own output quality
Directional Stimulus Prompting sidesteps the impossibility of fine-tuning a black-box LLM by instead training a separate, small, tunable policy model, such as a model of T5's size, whose job is to generate an instance-specific directional stimulus, a short hint or set of keywords tailored to the particular input, which is then inserted into the prompt actually sent to the black-box LLM. That policy model is what gets optimized, either through supervised fine-tuning on labeled data, through reinforcement learning using rewards derived from evaluating the black-box LLM's resulting output, or both, while the black-box LLM's own weights are never touched. The option describing direct zeroth-order fine-tuning of the black-box LLM's weights is wrong because the entire premise of the framework is that the LLM's weights remain inaccessible and untouched; only the small policy model is trained. The option describing a single fixed keyword list reused for every input is wrong because the method's value comes specifically from generating a different, tailored stimulus per instance. The option describing built-in retrieval-augmented generation is wrong because the framework introduces a trained generative policy model, not a retrieval or search mechanism.
Source: Li et al., 'Guiding Large Language Models via Directional Stimulus Prompting' (arXiv:2302.11520, NeurIPS 2023)