liveBook · Manning

Community detection

As explained in Retrieval Augmented Generation, The Foundational Ideas

Community detection is the algorithmic identification of densely connected groups of nodes in a graph.

Community detection identifies densely connected groups of nodes in a graph. In graph-based RAG, it changes retrieval from isolated text chunks toward coherent entity groups that can support global summaries and multi-hop context. section 12.1

Leiden builds communities through an iterative optimization process:

  • Start with every node in its own community. section 12.2.1
  • Move nodes between communities to improve modularity, which compares internal connectivity with a random distribution of edges. section 12.2.1
  • Refine communities by splitting disconnected sub-clusters, preserving internal connectivity. section 12.2.1
  • Aggregate each community into a super-node and repeat on the smaller graph until improvement stops. section 12.2.1

From entity graph to corpus-wide summary

After duplicate entities and relationships such as AuthService have been consolidated, Leiden groups the resulting graph into connected communities. GraphRAG can then summarize each group and repeat across higher levels, supporting questions about dominant themes or conflicts rather than only the nearest matching chunk. section 12.1 section 12.2.1

Table 12.2 The graph-based RAG taxonomy matrix
Feature
Primary goal
Mechanism
Best query
Trade-off
Type I: Global summarizers (GraphRAG)
Sense-making across the entire dataset (e.g., "What are the evolving themes in this conflict?")
Hierarchical summarization. Uses community detection (Leiden) to cluster nodes and pre-computes summaries for every cluster, creating a pyramid of context.
"What are the main conflicts in this dataset?"
High indexing cost/fast retrieval
Type II: Associative reasoners (HippoRAG)
Needle-in-a-haystack: connecting disconnected facts where there is no keyword overlap.
Probability flow. Personalized PageRank lets activation flow from query "seeds" to relevant neighbors, discovering hidden paths.
"How is the marketing director connected to the safety failure?"
Low indexing cost/high inference latency
Type III: Hybrid optimizers (LightRAG)
Efficiency and routing: balancing the precision of graph structures with the speed of vector search.
Dual-path retrieval. Routes specific queries to entity profiles and abstract queries to vector chunks, then synthesizes.
"List the safety incidents in 2023 and explain their impact on the 2024 budget."
Architectural complexity

Compare community detection's global-summarization role with associative and hybrid graph-based RAG approaches, including their query types and indexing trade-offs. section 12.1

How Retrieval Augmented Generation, The Foundational Ideas covers it

How deep it goes
Explains Leiden initialization, modularity optimization, connectivity refinement, and recursive aggregation, but does not provide implementation code.
Examples
Uses consolidated entities such as AuthService and community summaries for questions about dominant themes or conflicts.
What it assumes
Assumes graphs, nodes, edges, and traversal, while defining communities, modularity, and internally connected clusters.
What you can do after
Apply it via Leiden to partition an entity graph into connected groups suitable for hierarchical LLM summarization.
How it connects
Connects graph-based RAG with Leiden algorithm, modularity, hierarchical summarization, entity resolution, and multi-hop reasoning.

Also explained in

Compare how 5 books explain community detection →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Retrieval Augmented Generation, The Foundational Ideas. The section links point to the parts of the book it draws on.