Chapter 6: Constructing knowledge graphs with LLMs

Introduction

This chapter introduces using LLMs to extract structured information from unstructured text and turn it into a knowledge graph for queries that embeddings alone handle poorly.

• Structured extraction purpose: Explain why LLM-based structured data extraction is needed alongside text embeddings, especially for questions requiring filtering, counting, or aggregation.
  Why it matters: Structured graph data enables forms of retrieval and analysis that similarity search over text embeddings alone cannot reliably provide.
• LLM-to-graph construction workflow: Describe the chapter's end-to-end workflow: use an LLM to extract structured information from raw text, design a knowledge graph model for that information, and import the data into a graph database.
  Why it matters: This workflow converts unstructured documents into graph data that can be stored and queried as part of a GraphRAG application.
• Legal contract retrieval example: State the motivating example of retrieving information across multiple legal contracts and their terms, and describe how structured extraction is presented as a solution.
  Why it matters: The contract example makes concrete why document collections often need structured representations rather than text retrieval alone.

Key terms: text embeddings, structured data extraction, filtering, counting, aggregation, knowledge graph, unstructured text, structured information, knowledge graph model, graph database

6.1 Extracting structured data from text

This section explains why embedding-only retrieval can fail for legal-document questions and introduces using LLMs to extract structured data for a Neo4j knowledge graph.

• Embedding retrieval limitations: Explain why retrieving top-ranked text chunks by embedding similarity can produce inaccurate answers for contract questions, and why it cannot directly perform business-intelligence operations such as filtering and counting.
  Why it matters: Recognizing these limits helps you choose structured graph data when semantic chunk retrieval alone is insufficient.
• LLM structured extraction workflow: Describe the workflow for turning unstructured contract documents into graph-ready data: an LLM extracts facts such as parties, dates, and terms into structured JSON, which is stored in Neo4j for the legal chatbot.
  Why it matters: This workflow converts document content into data that can support reliable, structured retrieval for RAG applications.
• Structured extraction value: Explain why LLMs are useful for structured data extraction from large document collections: their natural-language understanding can identify requested data points when guided by prompts or finetuning, reducing labor-intensive manual organization.
  Why it matters: LLM-based extraction makes it practical to turn large volumes of unstructured business documents into actionable data.

Key terms: text embeddings, text chunks, top k chunks, semantic similarity, structured data, unstructured data, structured data extraction, structured output, JSON, Neo4j

6.1.1 Structured Outputs model definition

This section explains how to define a Pydantic-based Structured Outputs schema that guides an LLM to extract reliable contract data from text.

• Structured Outputs purpose: Explain how OpenAI Structured Outputs uses a predefined schema, supplied through function calling or schema definitions, to constrain an LLM response to a specified structure.
  Why it matters: A schema turns free-form model output into data an application can use consistently.
• Schema fields and descriptions: Describe how attribute names, supported types, and descriptions in a schema instruct the LLM what information to extract and how to format it, including specifying a date as a string in yyyy-MM-dd format.
  Why it matters: Precise field definitions reduce ambiguity and make extracted values usable without extensive post-processing.
• Contract model design: Explain how the Contract data model should be selected with domain experts and end users, then defined to capture key contract details such as type, parties, dates, and financials.
  Why it matters: An extraction system is only useful when its schema represents the information people actually need from the documents.
• Optionality, enums, and nesting: Describe how optional fields, enums, and nested custom objects shape the Contract schema: optional fields allow absent information without encouraging invented values, enums constrain known categories, and Organization and Location objects represent detailed parties and addresses without excessive nesting.
  Why it matters: These modeling choices balance reliable, standardized extraction with the flexibility needed for incomplete and variable contract language.

Key terms: Structured Outputs, function calling, schema definitions, Pydantic, attributes, description, str, date, yyyy-MM-dd, String

6.1.2 Structured Outputs extraction request

This section shows how to request contract-data extraction from an LLM using OpenAI Structured Outputs, a guiding system message, and a function that returns a contract-model dictionary.

• Structured Outputs conformance: Explain how OpenAI Structured Outputs guides an LLM to produce extracted data that strictly adheres to the predefined Contract data model in a consistent format.
  Why it matters: Reliable schema-conforming output is needed before extracted information can be imported into a graph.
• System message guidance: Describe the role of the system message in the extraction request: it directs the LLM to the task, defines the domain, and gives context about how the output will be used; explain that refining the ideal message often requires trial and error.
  Why it matters: A focused system message helps steer extraction behavior even when the output schema is enforced.
• Extraction function flow: Trace the extract function's request-and-return flow: it accepts any document text, sends the system message followed by the unmodified raw user document to a structured-output-capable GPT-4o model, and returns a dictionary formatted as the Contract data model.
  Why it matters: This function turns source contract text into structured records that can be used in the graph-construction workflow.

Key terms: OpenAI Structured Outputs, Contract data model, structured information, system message, domain, context, trial and error, extract function, GPT-4o, raw user-provided document text

6.1.3 CUAD dataset

This section demonstrates extracting a structured contract record from a CUAD legal-contract document with the previously defined data model and interpreting missing versus descriptive fields.

• CUAD contract extraction demonstration: Describe how the section uses a single contract text document from the CUAD dataset to demonstrate LLM-based extraction of structured legal-contract data.
  Why it matters: This demonstrates turning an unstructured legal document into data that can be organized and used downstream.
• Extracted contract field interpretation: Explain what the extracted contract fields show, including that populated fields capture contract type, parties, roles, locations, dates, renewal conditions, scope, and terms, while None can represent information absent or unspecified in the contract.
  Why it matters: Interpreting both populated and missing fields is necessary before treating LLM-extracted records as graph-ready data.
• Contract model refinement exercise: Describe the exercise of downloading CUAD, defining different contract data models for different contract types, and testing and refining those models based on how well they capture and categorize key legal information.
  Why it matters: Different contract types may require models that better fit the legal information they contain.

Key terms: CUAD dataset, contract data model, structured data, unstructured legal documents, end_date, total_amount, None, contract_scope, parties, renewal conditions

6.2 Constructing the graph

This section introduces the contract graph model and outlines the steps for importing, validating, and refining extracted contract data in Neo4j.

• Contract graph model: Describe the contract graph model: Contract, Organization, and Location are the main entities, with Contract connected to Organization by HAS_PARTY and Organization connected to Location by HAS_LOCATION.
  Why it matters: A clear graph model determines how extracted contract facts become connected, queryable knowledge in Neo4j.
• Location node rationale: Explain why Location is modeled as a separate node: one organization may have multiple addresses, and each Location captures address, city, state, and country.
  Why it matters: Modeling locations separately preserves the real-world possibility of multiple organizational addresses.
• Graph construction workflow: Trace the planned graph-construction workflow: define unique constraints and indexes, import structured contract data with Cypher, visualize the graph to verify entities and relationships, and refine data through entity resolution while considering structured and unstructured data.
  Why it matters: This workflow turns extracted data into an integrity-checked knowledge graph rather than merely loading records into a database.

Key terms: Contract, Organization, Location, HAS_PARTY, HAS_LOCATION, graph model, address, city, state, country

6.2.1 Data import

This section prepares and runs a Cypher import that loads extracted contract data into Neo4j, while noting integrity, performance, and duplicate-import concerns.

• Unique graph constraints: Explain why unique constraints are defined for Contract, Organization, and Location nodes before importing data into Neo4j.
  Why it matters: Constraints protect graph integrity and can make later graph queries faster.
• Schema-aligned Cypher import: Describe how an import Cypher statement takes the extracted dictionary output and loads it into Neo4j according to the intended graph schema.
  Why it matters: The import query is the step that turns LLM-extracted structured data into a usable knowledge graph.
• Non-idempotent contract imports: Explain why the import query is not idempotent: randomUUID() generates a new contract ID on each run, so repeated execution creates duplicate contract entries.
  Why it matters: Knowing that repeated runs duplicate contracts helps prevent accidental corruption of the imported graph.
• Imported contract graph structure: Describe the imported example graph as a central Licensing Agreement contract connected by HAS_PARTY to Mortgage Logic.com, Inc. and TrueLink, Inc., with each organization connected by LOCATED_AT to the US location node.
  Why it matters: Inspecting this structure verifies that the imported data matches the intended graph model.

Key terms: unique constraints, indexes, Contract, Organization, Location, graph integrity, import Cypher statement, dictionary output, Neo4j, graph schema

6.2.2 Entity resolution

This section explains entity resolution as the domain-specific process of detecting and merging duplicate representations of the same real-world entity in a knowledge graph.

• Entity resolution purpose: Explain entity resolution as identifying different representations of one real-world entity and merging them into a single graph node.
  Why it matters: Resolving duplicates keeps a knowledge graph coherent rather than scattering facts about one entity across several nodes.
• Duplicate entity variations: Describe why duplicate entities arise in LLM- or natural-language-derived graph data, using variations in spelling, naming conventions, or data formats and the UTI Asset Management Company example.
  Why it matters: Recognizing the sources of duplicate nodes helps you identify when graph cleanup is needed.
• Domain-specific matching strategy: Justify using domain-specific ontologies or rules, subject-matter-expert matching criteria, and iterative verification rather than a universal entity-resolution method.
  Why it matters: Matching rules that work in one domain can produce poor entity matches in another.
• Graph quality improvements: Explain that unifying duplicate entities improves data integrity and supports more accurate graph inferences and relationships.
  Why it matters: A graph with one coherent representation per entity can connect and reason over its data more reliably.

Key terms: entity resolution, real-world entity, duplicate, single, coherent node, spelling variations, naming conventions, data formats, UTI Asset Management Company Limited, UTI Asset Management Company Ltd, domain-specific ontologies

6.2.3 Adding unstructured data to the graph

This section explains why a knowledge graph should retain original unstructured documents alongside extracted structured data, and how chunking should preserve meaningful document structure.

• Combined graph data representation: Explain why a graph can store both the original unstructured documents and the structured data extracted from them by an LLM. Describe how this combination preserves the original data's richness while supporting more precise querying and analysis of extracted information.
  Why it matters: Keeping both forms of data lets an application use structured facts without losing the source text and its context.
• Structure-aware document chunking: Describe how unstructured text is split into manageable chunks for incorporation into a graph. Distinguish simple token-count or word-length chunking from domain-specific chunking that follows semantic structure, such as splitting legal contracts by clauses.
  Why it matters: Chunk boundaries affect whether the graph retains meaningful relationships and can support accurate downstream analysis.

Key terms: structured data, unstructured data, knowledge graph, extracted structured data, chunking, token count, word length, semantic structure, clauses

6.3 Summary

This summary explains why LLM-extracted structured data, clear schemas, entity resolution, and combined graph content improve retrieval over document chunks alone.