Advanced Reasoning / Summarization

Chain of Density

Advanced [4/5]
CoD Iterative densification Progressive summarization

Definition

Chain of Density (CoD) is an iterative summarization technique where the model generates increasingly entity-dense summaries while maintaining the same length. Each iteration adds more salient information by replacing filler content with key entities and facts.

This produces summaries that are information-rich without becoming longer, solving the verbosity problem in LLM summarization.

Key Concepts

  • Entity density: Number of entities per token in summary
  • Iterative refinement: Multiple passes increasing information density
  • Length constraint: Keep summary length constant across iterations
  • Missing entities: Track what important info is not yet included

Examples

Process
Chain of Density Iterations
CHAIN OF DENSITY SUMMARIZATION: ORIGINAL ARTICLE (about a tech company): "Anthropic, the AI safety company founded by former OpenAI researchers Dario and Daniela Amodei, announced Claude 3 on March 4, 2024. The model family includes Opus, Sonnet, and Haiku variants. Claude 3 Opus scored 86.8% on MMLU, surpassing GPT-4's 86.4%..." ITERATION 1 (Low density - verbose): "A technology company has released a new artificial intelligence system. The system comes in different versions for different use cases. It performs well on standard tests compared to other systems." Entities: ~2 | Words: 32 | Density: 0.06 ITERATION 2 (Medium density): "Anthropic released Claude 3, an AI model family including Opus, Sonnet, and Haiku. The flagship Opus model achieved strong benchmark results, competing with leading AI systems." Entities: ~6 | Words: 28 | Density: 0.21 ITERATION 3 (High density): "Anthropic, founded by Dario and Daniela Amodei, launched Claude 3 (Opus/Sonnet/Haiku) on March 4, 2024, with Opus scoring 86.8% on MMLU, exceeding GPT-4's 86.4%." Entities: ~10 | Words: 29 | Density: 0.34 DENSITY PROGRESSION: Iter 1: ░░░░░░░░░░ 0.06 Iter 2: ██░░░░░░░░ 0.21 Iter 3: ███████░░░ 0.34 Same length, 5x more information!
Prompt
Chain of Density Prompt Template
CHAIN OF DENSITY PROMPT: Article: {article} You will generate increasingly dense summaries. Repeat the following 2 steps 5 times: Step 1: Identify 1-3 informative entities from the article which are missing from the previous summary. Step 2: Write a new, denser summary of identical length which covers every entity and detail from the previous summary plus the missing entities. Guidelines: - First summary should be ~80 words, entity-sparse - Each summary has same length but more entities - Make space by fusion, compression, removing filler - Never drop entities from previous summary - Missing entities can appear anywhere in new summary Summary 1: [Generate sparse summary] Missing Entities: [List what's missing] Summary 2: [Denser version] Missing Entities: [List what's missing] ... Summary 5: [Final, densest version] WHAT GETS REPLACED: Low-value content removed: - "It is worth noting that..." → [delete] - "As mentioned above..." → [delete] - "This is significant because..." → [compress] Replaced with entities: - Names (people, companies, products) - Numbers (dates, percentages, amounts) - Specific facts and claims - Technical terms and jargon

Interactive Exercise

Densify a Summary

Starting summary: "A company released a new product. The product has received positive reviews and is available for purchase online."

Densify this while keeping similar length. Add: Apple, iPhone 15 Pro, September 2023, $999, titanium design, A17 Pro chip.

Pro Tips
  • Human preference peaks at medium density (iteration 3-4 of 5)
  • Too dense becomes hard to read - find the sweet spot
  • Great for news summarization and research paper abstracts
  • Use final iteration for data extraction, middle for readability

Related Terms