liveBook · Manning

Coreference resolution

As explained in Natural Language Processing in Action, Second Edition

Coreference resolution is the task of determining which entity or expression a referring expression, such as a pronoun, denotes in text.

Coreference resolution identifies expressions that refer to the same thing. In “Maria was born in Ukraine. Her father was a physicist. 15 years later she left there for Israel,” “Her” and “she” point to Maria, while “there” points to Ukraine. Making those links is quick for a reader but difficult to automate. section 5.2.3

In knowledge extraction, these links keep repeated mentions from becoming duplicate knowledge-graph nodes or misleading relations. Named entity recognition can find names; coreference resolution then connects mentions before later extraction steps use them. In a passage about Timnit Gebru, for example, “Gebru,” “she,” and “her” can refer to the same person. section 11.3 section 11.5

Listing 11.4 Coreference chains from Wikipedia text
>>> import spacy, coreferee                   #1
>>> nlptrf = spacy.load('en_core_web_trf')
>>> nlptrf.add_pipe('coreferee')
<coreferee.manager.CorefereeBroker at 0x...>
>>> doc_gebru = nlptrf(text_gebru)            #2
>>> doc_gebru._.coref_chains
[0: [13], [16], [26], [34],
 1: [51], [56]]
>>> doc_gebru._.coref_chains.print()
0: Gebru(13), she(16), she(26), she(34)    #3
1: advice(51), it(56)

Inspect the Coreferee output: one chain groups “Gebru” with three instances of “she,” while another links “advice” with “it.” The token numbers show where each mention occurs. section 11.5.1

When grammar is not enough

“The city councilmen refused the demonstrators a permit because they feared violence.” Resolving “they” as the city councilmen calls for commonsense reasoning, not just finding a nearby plural noun. Such sentences are used in Winograd schema challenges to test language understanding. section 11.5

A spaCy workflow can add Coreferee to a transformer-based pipeline and inspect `doc._.coref_chains`. Install it in a separate Python virtual environment: the cited Coreferee setup requires spaCy 3.0.0–3.5.4, and incompatible package versions can cause hard-to-detect failures. Treat name-based assumptions about sex or gender cautiously; they can amplify bias and may not reflect a person’s gender. section 11.5.1 section 5.2.3

How Natural Language Processing in Action, Second Edition covers it

How deep it goes
Defines the task, examines difficult pronouns, and demonstrates Coreferee chains and setup without explaining resolver internals.
Examples
Uses Maria’s pronouns, a Timnit Gebru article and Coreferee chains, and an ambiguous city-councilmen sentence.
What it assumes
Assumes some Python and spaCy pipeline familiarity, while introducing coreference through plain-language pronoun examples.
What you can do after
Apply it via Coreferee in a spaCy pipeline and inspect chains of mentions before consolidating extracted entities.
How it connects
Links named entity recognition, knowledge graphs, relation extraction, and commonsense reasoning.

Also explained in

Compare how 3 books explain coreference resolution →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Natural Language Processing in Action, Second Edition. The section links point to the parts of the book it draws on.