From Sparse to Dense: Chain of Density Prompting

The Core Idea

The paper develops a Chain of Density (CoD) prompt. It begins with a sparse summary and progressively produces denser summaries.

Rather than increasing the summary length, CoD repeatedly identifies details that are missing from the current summary and incorporates them into the next version. The authors provide 500 annotated CoD summaries.

Missing Entities

A missing entity is a detail that should be added to the next summary. It must meet the following conditions:

• Relevant: it relates to the main story.
• Specific: it is descriptive but concise, using five words or fewer.
• Novel: it does not appear in the previous summary.
• Faithful: it is present in the article.
• Anywhere: it may be located anywhere in the article.

Prompting and Evaluation

The baseline, or vanilla, prompt is: “Write a VERY short summary of the Article. Do not exceed 70 words.” The 70-word limit was selected to match the length of CoD summaries.

The paper evaluates summaries using several measures. Abstractiveness is the average squared length of extractive fragments. Fusion is the average number of source sentences aligned with each summary sentence. Alignment uses a relative ROUGE-gain method: source sentences are aligned to a target sentence as long as adding another sentence produces a positive gain.

The authors also examine content distribution. As the process is repeated, lead bias decreases, while content from the middle and end of the article becomes incorporated.

Trade-offs in Human Evaluation

Increasing density creates a trade-off between clarity and informativeness. Human evaluators found that informativeness increased with each step, while readability and coherence declined.

Overall evaluations were strongest at intermediate stages. This suggests that the densest possible summary is not necessarily the preferred one.

Limitations

The study is limited to news summarization, so it is difficult to generalize its findings too quickly to other domains.

It does not reach a definitive conclusion about the best level of density. This is inherently a subjective question. GPT-4 is a closed model, so its model weights cannot be shared, although all of the data are released.

Further research is needed on how to balance coherence, readability, and informativeness.

My Thoughts

Good results do not come from one magical prompt. They emerge through incremental refinement, much like working with a person.

I have been experiencing how useful step-by-step improvement can be, so I found this paper particularly interesting. At the same time, I wonder whether this approach is only possible because it uses a highly capable model such as GPT-4. It would be valuable to verify whether it also works with open-source models.

I would like to apply the idea to the research I am currently preparing. Instead of finding missing entities in a summary, the process could examine whether there are missing steps needed for reasoning.