Direction Stimulus Prompting

The core idea

Direction Stimulus Prompting is a prompting strategy in which hints about the desired direction of the response are added to the end of a prompt. These hints are called directional stimuli (DS). For summarization, they may be keywords that should appear in the summary; for dialogue generation, they may specify the role or goal of a response.

Adding such guidance can substantially improve performance. The model that generates the directional stimulus is a smaller model trained using task-specific examples.

Source: https://arxiv.org/abs/2302.11520

A policy model for black-box LLMs

When using a black-box LLM such as ChatGPT, a policy language model can be trained to write prompts appropriate for a particular task. For document summarization, for example, the policy model can extract keywords from a document and create an instruction telling the LLM to produce a summary that includes those keywords.

The policy model can first undergo supervised fine-tuning (SFT). It can then be further trained with reinforcement learning (RL), calculating rewards from the final outputs of the black-box LLM. This approach improves performance compared with using the black-box LLM directly.

Dialogue applications

Chatbots are one of the major applications of LLMs, but there are limits to using a general-purpose model for a chatbot with a specific objective. Performance improves significantly when directional stimuli are created by matching the goal of each dialogue with tags.

BLEU scores may not show a large improvement because they are affected by factors such as speaking style. However, inform and success scores increase sharply.

Instance-specific chain-of-thought prompts

Chain-of-thought research has often focused on finding a fixed, task-specific prompt. With DSP, however, it is possible to generate and insert a chain-of-thought prompt suited to each individual instance. This was the most impressive part to me.

Conclusion

• A policy model can be trained with a small number of examples to raise LLM performance on a specific task to a state-of-the-art level.
• Generated directional stimuli may help us understand LLM behavior.
• The method offers an idea for an intermediate step that guides LLM outputs.

My thoughts

For summarization, I wonder whether there should also have been a two-step approach in which the model generates keywords itself and then produces a summary based on them.

This is a strategy for improving performance on a specific task by applying SFT and RL to a small model that modifies prompts. It can be viewed as a rephrase-and-respond strategy with an adapter model added.

Ultimately, labeled data is required. If that data is available, this is a very good strategy.

If exemplars are unavailable, perhaps a policy LM could be trained through an APE-style approach instead: a self-improvement strategy that manipulates prompts.

This is not a strategy that ordinary users of AI chatbots can use.