liveBook · Manning

Entity resolution

As explained in Learn AI Data Engineering

Entity resolution is the process of determining which records refer to the same real-world entity despite differences or errors in their attributes.

Entity resolution determines when slightly different records refer to the same real-world entity. It is common in customer data: a person may appear through separate work and personal accounts, changed names, email domains, devices, or addresses. The aim is to unify these fragments into a fuller customer view. section 7.3

Figure 7.1 A single user appears under two separate accounts—one personal, one professional. Despite differences in name and email domain, shared signals like device ID and activity reveal an underlying connection. Entity resolution helps unify these records to build a complete view of the customer.

Compare the separate personal and professional accounts as one identity problem: shared device and activity signals can outweigh superficial differences in name or email domain. section 7.3

A traditional workflow normalizes fields, combines them into a fingerprint, and scores each known candidate against an incoming record. This makes matching logic explicit, but can require bespoke rules for string cleanup, similarity scoring, field weighting, and ambiguous cases. section 7.3

Listing 7.7 Python - Entity Resolution
import re  #A
from rapidfuzz import fuzz  #A
 
# Create a fingerprint by combining and normalizing key fields  #B
def build_fingerprint(record):  #B
    combined = f"{record['name']} {record['email']} {record['device_id']} {record['ip_address']} {record['top_coding_language']}"  #B
    normalized = re.sub(r'\W+', '', combined.lower())  #B
    return normalized  #B
 
# Build normalized fingerprints for the incoming activity and all customers  #C
incoming_fingerprint = build_fingerprint(incoming_activity)  #C
customer_candidates = [  #C
    {
        "original": customer,  #C
        "fingerprint": build_fingerprint(customer)  #C
    }
    for customer in customers  #C
]
 
# Score each candidate against the incoming fingerprint  #D
matches = [  #D
    {
        "name": c["original"]["name"],  #D
        "email": c["original"]["email"],  #D
        "score": fuzz.token_sort_ratio(incoming_fingerprint, c["fingerprint"])  #D
    }
    for c in customer_candidates  #D
]
 
# Sort matches by score and print the best one  #E
top_match = sorted(matches, key=lambda m: m["score"], reverse=True)[0]  #E
 
print(f"Best match: {top_match['name']} ({top_match['email']}) - Score: {top_match['score']}")  #E
 

The Python pattern builds a normalized fingerprint from name, email, device ID, IP address, and coding language, then scores candidate records. section 7.3

An AI-assisted resolution pattern can use a prompt as matching logic:

  • Provide the incoming record and candidate customer records with the attributes available for comparison. section 7.3
  • Instruct the model to compare all signals, choose the best match, and return a structured name, email, confidence score, and reasoning. section 7.3
  • Treat exact email matches as especially strong, consider nicknames, abbreviations, typos, and address similarities, and conservatively flag duplicates only above a high-confidence threshold. section 7.6

Johnny Smith versus known customers

An incoming activity for Johnny Smith uses a Gmail address, a new device, and a new IP address, but shares Python usage and a similar name with John Smith. The reported best match is John Smith at moderate 0.60 confidence because the name and language align while email, device, and IP do not. section 7.3

For duplicate-heavy CRM data, resolution supports merging records into golden customer or account profiles. AI matching evaluates customers across multiple factors and can return confidence and reasoning, helping handle nickname and email-format variations that rigid rules may miss. section 7.5 section 7.6

class EntityMatch(BaseModel):
    customer_id: str
    similar_customers: List[str]
    match_confidence: float
    match_reasoning: str
    is_likely_duplicate: bool
entity_resolution_prompt = """
You are an entity resolution specialist. Identify if this customer appears to be the same person as any other customers:
Consider these matching factors:
1. Email similarity (exact match is strongest indicator)
2. Name similarity (nicknames, abbreviations, typos)
3. Address similarity (same location, minor formatting differences)
Rules for matching:
- Exact email matches are very strong indicators
- Similar names + similar addresses are strong indicators  
- Consider common nicknames (Alexander/Alex, Jordan/J)
- Consider typos and formatting differences
- Be conservative - only flag as duplicate if confidence is high (>0.7)
"""

The structured duplicate-detection schema captures the customer ID, similar customers, confidence, reasoning, and a duplicate flag, alongside explicit matching rules. section 7.6

How Learn AI Data Engineering covers it

How deep it goes
Contrasts normalized fingerprint scoring with multi-attribute AI matching, structured confidence, and match reasoning.
Examples
Uses John, Jonathan, Jane, and Johnny Smith records with conflicting email, device, IP, and Python signals.
What it assumes
Starts from defined Python customer records and an incoming activity record without stating broader prerequisites.
What you can do after
Apply it via Python and an AI prompt to identify duplicate CRM customers and build golden records.
How it connects
Connects string normalization, similarity scoring, regular expressions, structured output parsing, and golden record creation.

Also explained in

Compare how 6 books explain entity resolution →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Learn AI Data Engineering. The section links point to the parts of the book it draws on.