An exact duplicate is an event record whose relevant values match another record, such as user ID, session ID, event type, app, and timestamp. In app-usage data, nearly 60% of records may be exact duplicates, making data-quality checks part of analysis rather than a cleanup afterthought. section 12.5.1
Repeated app events
For user 138, an “Opened” event is followed immediately by a “Closed” event and another “Opened” event for the same app. The first and third rows are exact duplicates. Duplicates may not affect counts of unique apps, but they can inflate activity at particular times of day. section 12.5.1
Handle duplicates according to the intended analysis:
- Keep them when counting unique apps if duplication does not change the result. section 12.5.1
- Investigate or remove them when measuring phone-use timing, because uneven duplication can skew temporal results. section 12.5.1
- Avoid blindly dropping exact duplicates: different-timestamp events can remain distinct and preserve additional problems. section 12.5.1
app_data.duplicated().sum() / len(app_data)
Calculate the proportion of exact duplicate rows before deciding whether remediation is necessary. section 12.5.1
