Chapter 6: Building knowledge graphs with large language models

Introduction

The introduction recalls what the Rockefeller Archive Center documents contain and sets up the move from extracted text to a knowledge graph.

• Rockefeller archive document context: Describe what the Rockefeller Archive Center documents record and how the Rockefeller Foundation used the information.
  Why it matters: The purpose of the original records gives context to the archive being turned into a knowledge graph.

Key terms: Rockefeller Archive Center, Rockefeller Foundation, program officers

6.1 Transforming an archive to a KG

This section explains the archive’s main extraction challenges and why LLM-assisted knowledge graph building is promising for addressing them.

• Archive text and entity challenges: Explain why turning the officer diaries into usable text and consistent entities is difficult: analog pages require OCR, historical research lacks a comprehensive reference for disambiguation, and abbreviations and specialized research terms challenge existing models.
  Why it matters: These obstacles explain why extracting names from the diaries is more than a straightforward text-processing task.
• Relation schema and identity quality: Explain why dense relationships call for a carefully defined relation-extraction schema, and why varying names and descriptions make graph normalization and entity resolution substantial work.
  Why it matters: The graph’s usefulness depends on extracting the right connections without treating inconsistent references as unrelated facts.
• LLM assistance versus traditional models: Describe the section’s evidence for LLM assistance: tested off-the-shelf coreference models failed on shortened names, while GPT handled NER, entity resolution, and relation extraction in response to prompts; contrast this with the resources traditional ML approaches would have required.
  Why it matters: This contrast motivates using LLMs for the archive without claiming that every graph-quality problem is already solved.

Key terms: optical character recognition (OCR), named entity recognition (NER), disambiguation, entity resolution, Occupations, relation extraction (RE) schema, KG normalization, cleansing, GPT, prompts

6.1.1 Graph modeling

This section introduces the RAC graph’s three-layer schema and explains why LLM-extracted entities need further processing before becoming a knowledge graph.

• Three-layer graph schema: Describe what the document-level, metagraph, and KG layers contain in the RAC project.
  Why it matters: The layers distinguish the original documents, extracted information, and final graph.
• Need for intermediate processing: Explain why the section does not treat LLM output as a finished knowledge graph.
  Why it matters: It explains why extracted information is kept separate from the final KG.

Key terms: document-level layer, metagraph layer, KG layer, File node, Page node, Entity nodes, LLM, normalization, entity resolution, schema

6.1.2 Creating a metagraph

The section explains how to build a page-based metagraph that keeps entity mentions and relations tied to their source text before creating the final knowledge graph.

• Page-based text chunking: Explain why long documents are split before processing and identify the page-based approach used here.
  Why it matters: Chunking makes the archive manageable for text processing.
• Per-page mention graph: Describe what is created after each page is processed and why identified entity mentions are not merged at that stage.
  Why it matters: Keeping mentions separate preserves the context needed to resolve which ones refer to the same entity.
• Provenance and KG reconstruction: Explain how the metagraph supports the final knowledge graph while preserving where its information came from.
  Why it matters: Source links make the finished graph explainable and easier to inspect or revise.

Key terms: chunking strategy, max tokens limit, splitting by page, metagraph, entity mentions, Entity, RELATED_TO_ENTITY, normalization, entity resolution, final KG

6.1.3 Normalization and cleansing

The section shows how metagraph statistics reveal opportunities to normalize extracted entities before linking them into the knowledge graph.

• Metagraph statistics for cleansing: Explain how inspecting top entities and relation classes gives a quick overview of the extracted knowledge and can reveal opportunities to improve graph connectivity.
  Why it matters: A quick look at the metagraph can reveal name inconsistencies that would otherwise weaken connections.
• Normalized names for consistent linking: Describe why the section lowercases names when case is irrelevant and strips titles from people’s names, and state how the resulting name_normalized property will be used.
  Why it matters: Consistent names make it more likely that mentions from different documents link to the same graph entity.

Key terms: metagraph, top entities per class, top relation classes, normalization, cleansing, name_normalized, Occupations

6.1.4 Graph-based entity resolution

This section explains how to resolve entity mentions across documents by combining name similarity with graph relationships, then grouping matching mentions into final KG entities.

• Remaining entity resolution need: Explain why clean entity names produced by an LLM do not eliminate the need to resolve mentions across documents.
  Why it matters: Cleaner extraction does not guarantee that a KG has one entity for each real-world subject.
• Name and relationship signals: Describe how the metagraph uses name similarity and relationships to identify mentions that may refer to the same entity, including why similarity rules need domain-specific care.
  Why it matters: Combining name and graph evidence makes a potential match more convincing than a name match alone.
• Graph paths between mentions: Trace how the Ernest Lawrence example uses graph paths to support resolving differently written mentions.
  Why it matters: The example shows how connections can provide evidence when a shortened mention is hard to match by name alone.
• Additional graph and semantic signals: Describe the additional matching signals proposed from interaction communities and semantic similarity between topics.
  Why it matters: These signals can help when names or occupation labels alone are ambiguous.
• Resolved entity construction: Describe the baseline sequence for grouping person mentions and creating resolved entities in the final KG layer.
  Why it matters: This is the section’s concrete route from potentially matching mentions to entities in the final graph.

Key terms: entity resolution, entity disambiguation, coreference resolution, metagraph layer, META_SIMILAR, WORKS_FOR, string similarity, stopword list, WORKS_ON, Occupations

6.2 Intellectual network analysis: The value of graphs

The section uses an LLM-built knowledge graph to explore scientists’ influence networks, find useful connections, and show why graph findings need human validation.

• Intellectual network graph analytics: Explain how graph analytics and visualization help identify influential people and bridges in the scientists’ intellectual network.
  Why it matters: Graph measures can draw attention to connectors who are not obvious from a list of famous scientists.
• Cyclotron influence exploration: Describe how the cyclotron example uses a few-hop query and a graph visualization to explore people connected to the research and its funding.
  Why it matters: A focused graph query can uncover relevant people beyond a researcher’s immediate contacts.
• Validation of surprising links: Explain why the unexpected appearance of Laurence Irving in the cyclotron network calls for analyst validation of KG content.
  Why it matters: Checking unexpected results prevents extraction mistakes from being treated as discoveries.
• Cross-university connector search: Trace how the influence network helps identify people to approach about a physics project spanning Johns Hopkins and Harvard.
  Why it matters: The graph turns a broad question about whom to contact into an inspectable set of connections and viewpoints.

Key terms: intellectual network, betweenness centrality, bridges, few-hop query, two-hop distance, PageRank centrality, RE task, feedback loop, validate or invalidate, WORKS_FOR

6.3 Next steps in the Rockefeller Archive Center project

The section identifies what the Rockefeller Archive Center knowledge graph would need to move from a 150-page demonstration toward a production-quality project.

• Production extraction at archive scale: Explain why the existing graph is not yet production-quality and how the project handles diaries that exceed an LLM's token limit.
  Why it matters: A promising small graph still needs accurate extraction and a way to process the full archive.
• Resolution of occupation topics: Explain why Occupations need entity resolution and describe the approach the project found most effective.
  Why it matters: Linking differently specific topic names lets questions draw on the topic's history rather than isolated mentions.
• Conversation and grant links: Describe what information would define Conversation nodes and what questions linking conversations to grants could support.
  Why it matters: Connecting diary conversations with grant records would make it possible to investigate how recommendations relate to awards.

Key terms: prompt engineering, fine-tuning, token limit, diary entries, Occupations, entity resolution, embeddings, agglomerative hierarchical clustering, Conversation nodes, follow-up chains

6.4 The value of knowledge graphs in the LLM era

This section explains why building a knowledge graph from data can be more useful than asking an LLM questions about that data directly.

• Inspectable and verifiable insights: Explain how a well-designed KG makes answers inspectable and why using an LLM to extract facts into a KG differs from simply fine-tuning it on the data.
  Why it matters: An inspectable graph gives people a way to check an insight rather than simply trust an LLM's answer.
• Reusable graph economics: Explain how producing a KG with an LLM can make the model's capabilities useful to organizations without large training or fine-tuning budgets.
  Why it matters: Reusing a graph can reduce dependence on repeated use of an expensive model.
• Graph exploration and analysis: Describe how a KG supports exploring relational data and conducting downstream analysis.
  Why it matters: A graph is useful not just for storing extracted facts but for investigating relationships and analyzing them.

Key terms: Explainability, Demystification, de-black-boxing, conflicting sources of information, Democratization, downstream tasks and analyses, batch updates, Explorability, Advanced analytics, global views

6.5 Summary