Chapter 7: Named entity disambiguation

Introduction

The chapter introduction explains what named entity recognition contributes to building knowledge graphs from text and why it is not sufficient for precise domain understanding.

• NER’s Role and Limits: Explain that named entity recognition identifies relevant entity mentions in raw text and assigns them to predefined categories, but does not by itself provide a precise understanding of the text in an application domain.
  Why it matters: This distinction explains why recognizing an entity mention is not the same as understanding it precisely in context.

Key terms: named entity recognition (NER), knowledge graphs (KGs), unstructured data, predefined categories

7.1 From recognition to disambiguation

The section explains why recognizing medical entity mentions is not enough and how disambiguating them against a knowledge base supports an interactive healthcare advisory system.

• Contextual identity of Zika mentions: Distinguish named entity recognition from named entity disambiguation using the Zika example. Explain that NED uses each mention’s context to link it to the appropriate knowledge-base entity, including distinctions between the virus, ordinary infection, and congenital infection.
  Why it matters: In medicine, treating distinct conditions as the same entity can obscure an important distinction.
• Entity links and knowledge retrieval: Explain how mapping a textual mention to a UMLS entity supports both detecting a meaningful entity and retrieving related information from biomedical ontologies. Give an example of a connection those ontologies can reveal.
  Why it matters: Grounding a mention lets an advisory system move from identifying what a person means to finding relevant knowledge about it.

Key terms: NER, named entity disambiguation, NED, mention, knowledge base, UMLS, IAS, UMLS entity, biomedical ontologies, contextual knowledge

7.2 Understanding named entity disambiguation

This section explains how named entity disambiguation links mentions in text to knowledge-base entities and how ontology integration turns those links into a knowledge graph.

• Recognition versus disambiguation: Explain why recognizing an entity mention is not enough and describe how candidate selection and ranking identify the entity it refers to.
  Why it matters: Finding a name in text is different from identifying the specific entity the author meant.
• Ranked biomedical entity links: Describe what the scispaCy example produces and what its results illustrate about linking repeated mentions.
  Why it matters: The example makes clear that linking is done for individual mentions, not just for a word in isolation.
• Ontology links in the graph: Explain how UMLS and SNOMED information extends disambiguated text into a knowledge graph, using the Zika example.
  Why it matters: Ontology links let a text-derived entity lead to related, structured clinical information.

Key terms: NER, NED, candidate selection, candidate ranking, reference knowledge base, scispaCy, ranked candidates, UMLS entity ID, ontology integration, UMLS

7.3 Domain-based NED and LLMs

The section uses a ChatGPT experiment to show why detecting medical entities is not enough for domain-based named entity disambiguation.

• Contextual medical entity meaning: Explain why ChatGPT’s list of detected medical entities did not establish that it had disambiguated them: the two mentions of “Zika” refer to different things in context.
  Why it matters: A plausible list of entities can hide an important difference in what the text means.
• Knowledge base mapping requirement: Explain why the section tests extracted entities against UMLS identifiers and what the ChatGPT response suggests is needed for this domain-based NED task.
  Why it matters: Checking a medical mention against a domain knowledge base is the step that makes its identity verifiable.

Key terms: medical entities, Zika, virus entity, virus infection, NED, UMLS, UMLS ids, knowledge base, KGs

7.4 Business and domain understanding

The section introduces the SoHO business context and explains why knowledge graphs suit its information needs.

• SoHO business context: Describe what substances of human origin (SoHO) include and why their management matters in the scenario of standards and regulations.
  Why it matters: The medical uses explain why managing SoHO is a meaningful business problem.
• Knowledge graph suitability: Explain why KG technology is presented as useful for the SoHO scenario.
  Why it matters: These are the section’s stated reasons for using a KG in the SoHO setting.

Key terms: SoHO, standards and regulations, KG technology, representation flexibility, harmonization, unified view

7.4.1 Context

This section establishes the patient-safety and regulatory context for extracting information from documents about substances of human origin.

• BTC supply chain safety: Describe why safety matters across the blood, tissues, and cells (BTC) supply chain, from donation to patient treatment.
  Why it matters: The supply chain shows why patient safety depends on more than the final treatment.
• Need for adaptable regulation: Explain why SoHO quality and safety requirements need to remain adaptable.
  Why it matters: Changing supplies, methods, and risks make fixed safety requirements insufficient.
• Complementary sources for safety rules: Distinguish the ECDC and EDQM document roles and explain why extracting information from both matters.
  Why it matters: Both document types contribute information needed to identify and update patient-safety rules.

Key terms: blood, tissues, and cells (BTC), SoHO, traceability, biovigilance, public health crises, legal framework, European Centre for Disease Prevention and Control (ECDC), European Directorate for the Quality of Medicines & HealthCa, joint regulation, textual documents

7.4.2 Use case definition

The section defines four ways a knowledge graph can help a health policy officer find, connect, interpret, and investigate information across health documents.

• Meaning-Based Conceptual Search: Explain how conceptual search finds information by meaning rather than exact wording, including how it handles different names for the same entity and similar names for different entities.
  Why it matters: A search should find the right concept even when a document uses a different name for it.
• Ontology-Guided Document Search: Describe how structured knowledge-based search uses relationships in domain ontologies to connect topics and retrieve relevant text across documents.
  Why it matters: Ontology relationships let a search bring together relevant passages that are distributed across documents.
• Interpretation Versus Graph Discovery: Distinguish using ontology relationships to interpret why entities co-occur in text from using connections to discover an insight that enriches the text.
  Why it matters: The distinction helps a reader tell an explanation of a textual connection from an additional insight suggested by graph relationships.
• Unformalized Co-occurrence Insights: Explain how a graph-based view can surface co-occurring entities whose connection is not yet formalized in ontologies, and how stakeholders might use that view alongside health bulletins.
  Why it matters: A graph view can bring emerging connections and current public-health evidence together for a policy decision.

Key terms: conceptual search, entity, structured knowledge-based search, domain ontologies, ontology paths, interpretability, discovery, co-occur, co-occurring entities, EDQM guidelines

7.5 Understanding the data

This section introduces the two kinds of information that must be brought together to develop an intelligent system in the SoHO domain.

• SoHO information integration: Explain that developing an intelligent system in the SoHO domain requires bringing heterogeneous information from different repositories into one source, including both unstructured and semantically structured data.
  Why it matters: This identifies the kinds of information the SoHO system needs to bring together.

Key terms: IAS, SoHO domain, heterogeneous information, unstructured, semantically structured data, unified source

7.5.1 Unstructured data

This section identifies the unstructured SoHO documents to be transformed into a knowledge graph and distinguishes the information each provides.

• Policy reports and stakeholder positions: Distinguish the EC’s impact assessment reports from stakeholders’ responses to regulation proposals, using examples of the information each contains.
  Why it matters: These documents capture both the case for regulatory change and responses to proposed changes.
• Technical guidance and disease surveillance: Describe the different information supplied by EDQM guidance and ECDC communicable disease threat reports.
  Why it matters: The two sources contribute different kinds of information to the documents being transformed into a KG.

Key terms: impact assessment reports, regulation proposals, stakeholders, position papers, EDQM, ECDC, technical guidance, communicable disease threat reports, epidemic intelligence

7.5.2 Domain ontologies

This section identifies the UMLS, SNOMED, and HPO ontology files used in the medical-entity scenario and explains what each contributes to disambiguation or graph construction.

• UMLS entity IDs and types: Explain how the two selected UMLS files support medical-entity disambiguation: MRCONSO.RRF supplies names, source vocabulary identifiers, and UMLS IDs, while MRSTY.RRF supplies semantic types.
  Why it matters: These IDs and types let the scenario connect disambiguated mentions to categorized medical entities.
• SNOMED names and relationships: Describe how the selected SNOMED files represent medical entity names and relationships, including the source–relationship–target structure illustrated by IS_A examples.
  Why it matters: The two files show how SNOMED contributes both medical terminology and relationships between concepts.
• HPO phenotype ontology format: State what kind of information HPO contains and how its T1D example is presented in the section.
  Why it matters: This distinguishes HPO’s phenotype information from the terminology and relationship files used for the other ontologies.

Key terms: UMLS, MRCONSO.RRF, MRSTY.RRF, UMLS ID, semantic type, NED, SNOMED, description file, relationship file, triplets

7.6 Building a SoHO knowledge graph

This section outlines the steps for building the SoHO knowledge graph and offers two paths through the implementation material.

7.6.1 Defining the schema

This section defines a knowledge graph schema that connects document pages, entity mentions, disambiguated medical entities, and domain ontology entities.

• Document and mention structure: Describe how the schema represents ingested documents as File nodes connected to Page nodes, with each Page linked to EntityMention nodes for recognized entities in its text.
  Why it matters: This structure keeps extracted mentions connected to their source pages.
• Mentions and disambiguated entities: Explain why EntityMention nodes connect to MedicalEntity nodes through DISAMBIGUATED_TO, including how identical strings can mean different entities and different strings can mean the same entity. Describe the separate MENTIONS_ENTITY connection from a Page to a disambiguated MedicalEntity.
  Why it matters: Separating the words in a document from the entities they refer to is what makes disambiguation representable in the graph.
• Medical entity ontology mappings: Describe how the schema maps extracted MedicalEntity nodes to SnomedEntity, HpoEntity, and HpoDiseaseEntity nodes using the specified relationship types.
  Why it matters: These links place extracted medical entities alongside the domain ontology entities in the graph.

Key terms: File, Page, EntityMention, MedicalEntity, DISAMBIGUATED_TO, MENTIONS_ENTITY, SnomedEntity, HpoEntity, HpoDiseaseEntity, IS_SNOMED_ENTITY

7.6.2 Processing and ingesting documents

This section explains how SoHO documents are converted into reconstructed text and loaded into a knowledge graph.

• Document text preparation: Describe how text is prepared from PDF and DOCx documents before ingestion into Neo4j, including extraction and reconstruction across different page layouts.
  Why it matters: Reconstructing the text gives the graph usable document content rather than just the original files.
• Document content graph ingestion: Explain what is loaded into the knowledge graph after text processing and how the section says that loading is performed.
  Why it matters: This step makes the prepared document text available as graph content for subsequent work.

Key terms: PDF, DOCx, Amazon Textract, OCR, full text, Cypher, Neo4j, KG, page texts

7.6.3 Disambiguating and ingesting medical entities

This section describes how extracted medical-entity results are stored and loaded into a Neo4j graph that retains both text mentions and disambiguated entities.

• Medical entity ingestion flow: Trace how medical entities move from OCR-generated JSON through a Python dictionary into Neo4j, including the nodes and connections created when the results are loaded.
  Why it matters: The loading sequence explains what information the resulting graph makes available.
• Mention and entity detail retention: Explain what details are retained for extracted mentions and why the graph keeps both EntityMention and disambiguated MedicalEntity nodes.
  Why it matters: Retaining both the text occurrence and the resolved entity preserves detail for flexible queries.

Key terms: OCR, Python dictionary, Neo4j, scispaCy, EntityMention, Page, sentence index, start and end characters, types, aliases

7.6.4 Processing, loading, and mapping ontologies

The section explains how SNOMED and HPO are loaded into a knowledge graph, enriched and connected to UMLS and medical entities extracted from text.

• SNOMED relationship representation: Explain how the section keeps the SNOMED graph schema simple while preserving relationship types and hierarchy.
  Why it matters: This representation keeps the graph manageable while retaining the hierarchy needed for the next step.
• SNOMED semantic type propagation: Explain why and how semantic types are propagated from SNOMED’s first-level nodes to deeper entities.
  Why it matters: Propagation makes it possible to identify whether a deep SNOMED entity is, for example, a disease or a product.
• HPO features and diseases: Describe how the HPO ontology and its disease annotations become connected graph entities.
  Why it matters: The annotation links let the graph relate HPO phenotypic features to diseases.
• Ontology and text integration: Explain the role of UMLS in integrating the ontologies and how extracted medical entities are connected to ontology nodes.
  Why it matters: These connections bring extracted text entities and information from multiple medical ontologies into the same graph.

Key terms: SNOMED_RELATION, type property, SNOMED_IS_A, semantic types, first-level nodes, HPO, RDF/XML, Neosemantics, phenotypic features, annotation file

7.6.5 Generating entity co-occurrences

The section explains how sentence-level co-occurrence relationships connect medical entities in text and support analysis alongside domain ontologies.

• Sentence-level entity co-occurrence: Explain what co-occurrence means in this knowledge graph, how COOCCURR relationships are created, and why making those connections explicit is useful.
  Why it matters: These relationships let the graph connect evidence found in text with structured domain knowledge.

Key terms: co-occurrence, Page nodes, Entity nodes, COOCCURR, domain ontologies

7.7 KG-based use cases

This introduction previews KG and NED use-case examples, points to the analysis step, and notes why query results may vary.

7.7.1 Conceptual search

This section contrasts keyword-based search with entity-based conceptual search, using medical examples to show how conceptual search changes results and makes them easier to inspect.

• Keywords versus entity meanings: Distinguish full-text search from conceptual search: full-text search can match words without finding the intended disease, while conceptual search uses an identified entity to retrieve mentions expressed in different ways.
  Why it matters: Searching for the entity can find relevant documents that a search for the query words might miss or rank poorly.
• Context-sensitive search results: Explain how conceptual search can distinguish an expression’s intended meaning from an unrelated use of the same word, using “islands of Langerhans” and “pancreatic islets” as the example.
  Why it matters: The intended meaning matters both for excluding misleading matches and for finding relevant alternative expressions.
• Entity mentions and inspectability: Describe what the breakbone-fever comparison reveals about retrieved pages and what additional evidence conceptual search provides for inspecting results.
  Why it matters: The mention evidence helps a reader see why a page was returned and inspect possible entity-detection errors.

Key terms: conceptual search, full-text search, entity, alias, UMLS, islands of Langerhans, pancreatic islets, entity mentions, NED model, page index

7.7.2 Structured knowledge-based search

The section shows how ontology relationships extend document search beyond matching a named concept to finding text about related medical entities.

• Ontology-guided document retrieval: Explain how structured knowledge-based search uses relationships in a domain ontology to find text mentioning entities related to a search concept, rather than only text referring to that same concept.
  Why it matters: Ontology links can surface relevant passages even when they do not mention the concept used to start the search.
• Multi-step ontology path search: Describe how following a path of CAUSATIVE_AGENT relationships can broaden a search from Zika virus to documents mentioning other diseases associated with Togaviruses.
  Why it matters: Following longer ontology paths helps retrieve documents about related diseases even when they have no direct link to the queried entity.

Key terms: structured knowledge-based search, conceptual search, domain ontologies, FINDING_SITE, SNOMED ontology, CAUSATIVE_AGENT, Togavirus, longest paths

7.7.3 KG-based interpretability and discovery

This section shows how ontology paths can explain, validate, and extend relationships between medical entities that co-occur in text.

• Interpretation versus discovery: Distinguish ontology paths that explain why entities co-occur in a sentence from paths that reveal additional connections beyond the sentence. Explain how the AIDS and hepatitis example illustrates both uses.
  Why it matters: It separates evidence for why a text mentions two entities together from additional knowledge supplied by the ontology.
• Zika co-occurrence analysis: Trace how the section moves from entity types co-occurring with Zika virus to contextualized disease co-occurrences and then to SNOMED paths. Describe what the paths contribute for Zika virus disease and Dengue Fever.
  Why it matters: Filtering and checking context helps an analyst focus on co-occurrences whose ontology connections are worth interpreting.
• HPO phenotype–disease associations: Describe how the validation approach is extended to HPO by selecting co-occurrences between a disease entity and an indexed phenotypic feature.
  Why it matters: It shows that the same co-occurrence analysis can target a specific kind of biomedical association.

Key terms: co-occurrence, interpretability, discovery, SNOMED, ontology paths, Zika virus, Disease or Syndrome, shortest paths, Dengue Fever, HPO

7.7.4 Uncovering new knowledge

This section uses a Zika virus example to show how a text-derived knowledge graph can suggest a medical relationship missing from an ontology, and why ontology paths must be checked before treating them as evidence.

• Textual clues for ontology enrichment: Explain how co-occurrences in a knowledge graph built from text can suggest facts that have not yet been consolidated in a medical ontology. Describe why the Zika virus and Guillain–Barre syndrome co-mention warrants a closer look.
  Why it matters: It shows how the direction of enrichment can run from a text-derived graph toward an ontology, rather than only the other way around.
• Checking the proposed relationship: Trace how the section investigates the Zika virus and Guillain–Barre syndrome co-mention using the underlying text, then checks whether SNOMED encodes the relationship suggested by that text.
  Why it matters: Checking the source text and the ontology separately prevents a co-mention or an arbitrary graph path from being mistaken for an encoded medical fact.
• Hub filtering and its limits: Explain why hub nodes produce misleading ontology paths and describe how the section uses node degree to filter those paths. State what the filtered results do and do not establish.
  Why it matters: Filtering makes an overwhelming path search more useful without confusing fewer paths with proof of a direct relationship.

Key terms: co-occur, KG built from textual content, domain ontologies, co-endemicity, Guillain–Barre syndrome, Infectious neuronitis, SNOMED, CAUSATIVE_AGENT, hub nodes, graph projection

7.8 Summary