Prompt Engineering for Claude’s Long Context Window

Why Long Context Matters

I use Claude when I need to process large amounts of information because it has a longer context window than other commercial language models: 100k tokens, compared with 2k for Bard and 8k for ChatGPT-4.

I found a report on generating and solving questions with this long context window particularly interesting. Since I often create questions from textbooks and extract necessary information from papers, this is a highly relevant topic.

The report is available at: https://www.anthropic.com/index/prompting-long-context

Challenges in Question Generation

The report identifies several common difficulties in creating questions from documents.

• Questions may ask about broad introductory material that is only indirectly related to the document.
• Correct answers tend to be detailed, while incorrect answers tend to lack detail.
• Questions may explicitly refer to the source with wording such as “in this document.”
• The wording of the question itself may contain the answer.

Prompting for Question Generation

The question-generation prompt provides passage–question examples, with multiple questions for each of two documents. It also gives concrete instructions about the common mistakes above in a “Guideline:” format. This use of guidelines resembles a prompt format I have also seen in CoD.

Fixing the correct answer as option 1 also seems like a good approach.

Answering Questions About Long Documents

For the experiment, the documents used above were concatenated into documents of 75k and 90k tokens. The model was then given questions and asked to answer them.

The experiment used Claude Instant 1.2. Claude 2 performs so well that it would be difficult to observe the effect of prompt engineering, so a lower-performing model was used instead.

Question-Answering Strategies

The study compared four prompting strategies, each also tested with a scratchpad, for a total of eight strategies.

• Simply provide the question and ask the model to solve it.
• Provide two question-answering examples unrelated to the government documents used in the experiment.
• Provide two examples involving other questions from the experimental material.
• Use the same approach as the third strategy, but with five examples.
• Add a scratchpad to each of the four strategies.

Results

Accuracy was compared according to whether the relevant document content appeared at the beginning, middle, or end of the long context. Performance differences by position are known to be substantial, including effects such as lead bias.

The four strategies generally improved in performance in the order listed above. Adding a scratchpad, in which the model first writes down a relevant quotation and then answers the question, improved results for all four strategies.

Five-shot prompting had a slight negative effect when the relevant content was at the end of the context. The suggested explanation is that the examples increase the distance between the relevant content and the question.

For Claude 2, the five-shot-plus-scratchpad strategy improved performance from 0.939 to 0.961. Although the numerical difference appears small, it represents a 36% reduction in errors.

Conclusions and My Thoughts

The main conclusion is that providing many examples and using a scratchpad are helpful. Contextual examples are more effective than generic examples that only demonstrate the desired format.

Claude 1.2 showed an inverse relationship between question performance and passage position, while Claude 2 showed a U-shaped pattern.

I had assumed that accuracy by position would naturally be U-shaped, so this linear increasing pattern was new to me. I should look into the related research further.

There has been considerable discussion that generic examples are also sufficiently helpful. However, as model performance has improved and context windows have become larger, contextual examples may have become much more effective. This is another point I need to update my understanding of.

I currently use long context windows mainly to find and compile necessary information from summaries of multiple papers, such as when extracting ideas for writing a paper’s discussion section. I should try the scratchpad approach for this work as well.