Chapter 5: Extracting domain-specific knowledge from unstructured data

Introduction

Introduces why unstructured documents are valuable sources for knowledge graphs and frames the work as knowledge representation and knowledge learning.

• Value of unstructured sources: Explain why emails, research papers, news articles, and other unstructured sources are worth considering when building knowledge graphs.
  Why it matters: It explains why the book turns from structured data to documents and other text.
• Knowledge representation challenge: Describe the knowledge representation challenge and what a well-designed KG contributes in this context.
  Why it matters: The graph's value depends not just on extracting information, but on organizing it for use.
• Knowledge learning from documents: Explain the knowledge learning challenge at the level introduced here, including the kinds of processing involved in transforming unstructured data into knowledge.
  Why it matters: It distinguishes finding knowledge in documents from representing that knowledge in a graph.

Key terms: unstructured data, structured data, knowledge representation, KG, knowledge learning, NLP, LLMs, data enrichment, data modeling

5.1 The archives challenge

This section introduces the Rockefeller Archive Center’s program-officer diaries as a challenging source of unstructured data for investigating how research grants were awarded.

• Challenges of archival diaries: Describe why the program-officer diaries are valuable but difficult sources for domain-specific knowledge extraction.
  Why it matters: The diaries’ richness and messiness explain why this case calls for domain-specific extraction.
• Grant-making questions in the archive: Explain what questions mining and modeling the diaries could help investigate about grant-making.
  Why it matters: These questions show why the relationships buried in the notes are worth extracting.

Key terms: program officers, diaries, domain-specific nomenclature, unstructured data, grant-making processes, recommendations, internal discussions, funding subdisciplines

5.2 Key concepts of knowledge extraction

This section identifies entity recognition and relationship extraction as the two building blocks for turning text into a structured knowledge graph.

• Two knowledge extraction building blocks: Explain that turning textual data into a structured knowledge graph involves identifying entities in the text and extracting the relationships connecting them.
  Why it matters: This distinction gives you a way to describe what must be extracted from text before it can form a knowledge graph.

Key terms: entities, relationships, knowledge extraction, structured KGs

5.2.1 Recognizing named entities

This subsection introduces named entity recognition as the first step from unstructured text toward a knowledge graph and explains why an archive may need domain-specific entity categories.

• Named entity recognition role: Explain that named entity recognition identifies mentions in raw text and assigns them to predefined categories as a first step toward building a knowledge graph.
  Why it matters: Recognizing and categorizing mentions gives unstructured text a starting point for a knowledge graph.
• Domain specific entity categories: Justify why the Warren Weaver diary calls for custom entity extraction: generic categories miss important conversation topics, including research disciplines, technologies, and diseases.
  Why it matters: The entities worth extracting depend on the questions and vocabulary of the domain.

Key terms: named entity recognition (NER), named entities, predefined categories, generic entity categories, Topic entity, research discipline, technology, disease, custom entity-extraction system

5.2.2 Extracting relations

The section explains how extracting relationships between named entities turns information in text into knowledge graph connections.

• Relation extraction for knowledge graphs: Explain that relation extraction identifies semantic relationships between pairs of entities in text, and illustrate how such a relationship becomes a knowledge graph connection. Describe why the chosen representation must be consistent across documents.
  Why it matters: A knowledge graph needs connections between entities, not just a list of entities found in text.

Key terms: relation extraction (RE), semantic relations, entity pairs, KG

5.3 Building KGs with large language models

The section explains why LLMs can help extract knowledge from text for KG building, and why their outputs still require caution.

• Transfer learning for extraction: Explain how reusing patterns and knowledge learned on a generic task reduces the labeled data needed for a task such as relation extraction.
  Why it matters: This explains why LLM-based extraction can require less task-specific labeling than traditional NLP.
• Scale quality and prompting: Describe how model complexity and training-corpus quantity and quality contribute to LLM capabilities, and contrast the section’s data-centric emphasis and natural-language prompting with traditional model-centric work.
  Why it matters: It explains the shift from building a separate model for a task toward expressing that task in a prompt.
• Hallucination risk in LLMs: Explain what the section means by hallucinations and why they matter when a system depends on LLM-generated information.
  Why it matters: Invented facts could become false knowledge if LLM output is accepted uncritically.
• LLM role in KG building: Describe the role proposed for LLMs in building KGs from text, including what kinds of knowledge they are asked to extract.
  Why it matters: It identifies the practical KG-building job assigned to LLMs without overstating what the models are.

Key terms: transfer learning, human-labeled data, RE, model complexity, training corpus, data-centric paradigm, model-centric approach, prompt engineering, hallucinations, false reasoning

5.3.1 Using LLMs

This subsection explains how to use LLM text completion to extract entities and relations for a knowledge graph, starting with prompts and fine-tuning when prompts are insufficient.

• Prompt-based knowledge extraction: Explain how an LLM can use a natural-language prompt to identify entities and their relations in text, and why prompt wording is refined iteratively.
  Why it matters: A well-formulated prompt may be enough to turn source text into material for a KG.
• Domain exploration and references: Describe why data exploration comes before designing the extraction system, using the diary entry’s references to C. G. Rossby as an example.
  Why it matters: Examining the documents first exposes references an extraction system might otherwise miss.
• Fine-tuning decision and iteration: Explain when the workflow moves from prompting to fine-tuning and how the training data and subsequent iterations are used.
  Why it matters: The choice between using a prompt and fine-tuning balances output quality against the effort and expense of specialization.

Key terms: text completion, prompt, prompt engineering, entities, relations, data exploration, domain, knowledge extraction, prompt-based approach, training dataset

5.3.2 Prompt engineering examples

This section uses successive prompts to show how LLMs extract knowledge from a diary entry and how prompt guidance improves consistency for knowledge-graph building.

• Contextual entity and title resolution: Explain how the LLM used the diary entry’s context to produce full names and titles from abbreviated names, pronouns, and shortened titles.
  Why it matters: It shows why the initial extraction looked promising before its consistency problems became apparent.
• Generic prompt extraction limitations: Explain why accurate-looking output from a generic entity-and-relation prompt was not yet reliable or normalized enough for a knowledge graph.
  Why it matters: A graph built from inconsistent or missing extractions can give incomplete answers even when individual relations look plausible.
• Iterative prompt guidance: Trace how the authors revised the prompt to guide entity and relation assignments, and describe the improvements and remaining shortcomings after each revision.
  Why it matters: The revisions demonstrate how examples and explicit categories can steer extraction toward a more usable KG.

Key terms: coreference resolution, generative nature of LLMs, traditional RE, normalization, node labels, relation types, prediction stability, KG schema, entity classes, occupation

5.3.3 Prompt engineering guidelines

This section gives practical guidelines for prompting LLMs to extract entities and relations, testing the resulting graph, and deciding when to stop refining the prompt.

• Task and class specification: Explain how a clear task description, dataset-specific guidance, and well-chosen entity and relation class names improve extraction output.
  Why it matters: Clear instructions and class names make extracted knowledge more consistent.
• Representative extraction examples: Describe how to choose a prompt example for entity and relation extraction, including difficult language, key relation types, and clean entity names.
  Why it matters: A realistic example shows the model the output you want even when the source text is messy.
• Temperature and prediction stability: Explain the trade-off between temperature and output variability, and describe how to check stability when extracting factual knowledge.
  Why it matters: A factual KG is hard to trust if identical inputs yield substantially different facts.
• Iterative prompt verification: Describe how unit tests and small, inspectable KGs support prompt iteration, and when the section recommends moving beyond prompt changes.
  Why it matters: Testing preserves earlier gains, while inspecting a mini-KG exposes failures that isolated examples may hide.
• Extraction quality evaluation: Describe the proposed quantitative evaluation of a satisfactory prompt and explain what that evaluation is for.
  Why it matters: Measured errors help you decide whether extraction is reliable enough to run across a large dataset.
• Exploratory graph for schema discovery: Explain how an initial exploratory KG can help when the dataset's contents and desired KG schema are not yet clear.
  Why it matters: A rough graph can reveal what is worth extracting before you commit to a detailed schema.

Key terms: task description, domain-specific guidance, entity and relation classes, normalized outputs, entity RE, input and expected output, key relation types, full, clean entity names, temperature, hallucinations

5.3.4 KG building: Traditional NLP or LLMs?

This section compares traditional NLP and LLMs for building knowledge graphs and explains when each approach may be preferable.

• Traditional NLP tradeoffs: Explain why traditional NLP can be attractive for fast, low-cost, secure prediction, yet expensive and difficult to set up for a specific domain.
  Why it matters: It explains why an older approach may still be practical despite its high initial investment.
• LLM benefits and costs: Contrast the quicker domain customization and combined processing offered by LLMs with their slower predictions, GPU requirements, and potential costs.
  Why it matters: It helps distinguish a faster path to an initial KG from the cost of running that path in production.
• Approach selection and fallback: Justify when traditional NLP remains a sensible choice and describe how LLMs can still help if they cannot be used in production.
  Why it matters: It turns the comparison into a practical choice rather than treating LLMs as a universal replacement.

Key terms: traditional NLP, NLP pipeline, data annotation, on-premises, LLMs, transfer learning, prompt engineering, post-processing, fine-tuning, streaming scenarios

5.4 Summary