Deduplication identifies records that represent one underlying entity and decides how they should be handled. It is a data-modeling step when repeated customer details, overlapping source systems, or duplicate measurements would distort counts and analysis. Exact duplicate removal is only the simplest case; records can describe the same customer while differing in identifiers or incomplete fields. section 3.3.1 section 3.0
Start by inspecting how an entity can appear across datasets and by choosing match fields deliberately. A shared name and postcode may not prove identity, linked accounts need representation, and missing fields change the decision. Keep source identifiers when records are linked: deleting one identifier can lose data lineage and make the resulting model harder to trace and trust. section 3.3.1 section 3.3.2
A practical workflow separates obvious duplication from uncertain matching:
- Verify key assumptions and schemas before merging datasets; then remove records that are identical in every relevant field. section 3.3.1 section 3.4.3
- Preserve incomplete columns until late in the process, even when they are excluded from matching decisions. section 3.4.3
- Give guest records distinct identifiers when identifiers are needed to link accounts, while clearly recognizing that generated IDs are artificial. section 3.4.3
- Group records on selected fields to find candidate duplicates, store linked customer IDs, and designate a main record rather than silently discarding related rows. section 3.4.3
- Compare alternative matching methods and investigate disagreements rather than treating any one method as automatically correct. section 3.4.3
Customer records across databases
Customer details may repeat when purchase data, a CRM, and a customer database are combined. Exact duplicates can be dropped, but Jane Smith with different customer IDs should retain both IDs as linked accounts. For less exact matches, block candidate pairs by postcode, compare first and last names with Damerau-Levenshtein similarity, require the selected comparison criteria, then merge the resulting links back into the customer model. section 3.3.1 section 3.4.3
Traffic measurements with a composite key
Duplicate traffic rows can share a composite identifier while containing different measurement values. Replace missing grouping values with safe placeholders, group by the non-measurement columns, average measurements within each group, then restore missing values. This reduces the data to one row per unique composite-key combination while avoiding incorrect grouping caused by missing values. section 8.4.1
indexer = recordlinkage.Index() #1
indexer.block('postcode') #2
candidate_links
= indexer.index(all_customers.set_index("customer_id")) #3
Blocking customer records by postcode narrows record-linkage comparisons to plausible candidate pairs before string similarity is evaluated. section 3.4.3
