For Long LLM Documents, Put the Instruction Last

A useful paper on financial document question answering

FinanceBench is a benchmark for answering financial questions from company reports. It examines a practical question for work involving large documents: when information must be drawn from a long source, is retrieval-augmented generation (RAG) better, or is a long-context approach better? And if using long context, should the question come before or after the document?

The paper is FinanceBench: A New Benchmark for Financial Question Answering by Pranab Islam, Anand Kannappan, Douwe Kiela, Rebecca Qian, Nino Scherrer, and Bertie Vidgen. It evaluated GPT-4, GPT-4 Turbo with a 128k context window, Claude 2 with a 100k context window, and Llama 2.

I nearly skipped it because the title made it sound like it would be purely about economics. I am very glad I read it: it contains a great deal of practical material for working with long documents.

Five prompting approaches

The researchers tested five prompting approaches:

• Closed book: provide no document information at all. This is an unrealistic scenario.
• Oracle: provide only the page containing both the question and answer. This is unrealistic, but it represents the easiest possible setup.
• A single vector store.
• A shared vector store.
• Long context: provide a large portion of the document directly.

Long context and retrieval

The results were broadly what one might expect. Long-context prompting was useful, although documents that exceeded the available context window still had to be split. This needs to be kept in mind when interpreting the results. Context windows are now larger than they were during these experiments, so this approach may be more practical now.

Long context was especially effective for questions that require extracting information from different parts of a long document, combining it, and then answering with a calculated number. For example: by what percentage did revenue increase compared with ten years ago?

With RAG, the system must retrieve the material from ten years ago, retrieve this year's material as well, and then correctly combine the two values and calculate the result. That is a very difficult chain of tasks. With a long-context approach, the relevant information can be available together.

Document first, question last

The point that attracted the most interest was whether the question should appear first or last. I had already found empirically that placing it last works better, so it was satisfying to see supporting evidence here.

The answer is: show the document first, and put the question at the end.

One possible explanation is that, when the document comes first, the model can read through it and form an understanding of the important information before encountering the question. It can then generate an answer on the basis of that understanding. When the question comes first, it may be harder to retain focus on it while reading the document without being distracted by the intervening material.