Data cleaning
Data cleaning is the process of detecting and correcting, removing, transforming, or otherwise handling inaccurate, incomplete, inconsistent, duplicate, or unsuitable data.
Starting Data Analytics with Generative AI and Python
A data cleaning protocol is a documented procedure for detecting, correcting, and documenting problems in data.
- How it is used
- Prepares collected data for reliable analysis and business decisions before processing or evaluating results.
- How deep it goes
- Details core checks and their ordering, then demonstrates text cleaning; leaves the exact protocol dependent on the dataset.
- Examples
- Uses ingredient preparation and Portuguese customer-review keyword search, including measured sensitivity and specificity.
- What you can do after
- Apply it via Python to clean review text, then check whether keyword-search results improve.
Read how Starting Data Analytics with Generative AI and Python explains data cleaning →
Data Without Labels
Data cleaning is the process of detecting and correcting or removing inaccurate, incomplete, inconsistent, duplicate, or irrelevant data.
- How it is used
- Prepares structured and text data for reliable analysis and machine learning after discovery and before modeling.
- How deep it goes
- Details quality checks and context-dependent text-cleaning choices, but does not specify a universal treatment for every dirty value.
- Examples
- Uses sparse or constant customer attributes, incompatible date formats, and context-sensitive stop words to illustrate cleaning decisions.
- What you can do after
- Practice/perform it by checking duplicates, missing values, junk, and outliers, then choosing treatments suited to the dataset.
The Well-Grounded Data Analyst
Data cleaning is the process of detecting, correcting, standardizing, or removing inaccurate, inconsistent, incomplete, or irrelevant data.
- How it is used
- Uses cleaning to prepare address and published-statistics data for geographic or exploratory analysis.
- How deep it goes
- Explains inspection, standardization, matching, fallback categories, and exceptions, but stops short of a general automated validation framework.
- Examples
- Demonstrates address casing, delimiter matching, variable-length records, reference city lists, unmatched cases, and customer-spend summaries.
- What you can do after
- Apply it via pandas to inspect, standardize, classify, and summarize imperfect tabular data.
Read how The Well-Grounded Data Analyst explains data cleaning →
LLMs in Production
Data cleaning is the process of detecting, correcting, removing, or managing inaccurate, inconsistent, incomplete, or unsuitable data.
- How it is used
- Prepares raw text datasets for LLM training and finetuning so their contents better support the intended behavior.
- How deep it goes
- Outlines schema, distribution, bias-checking, and ingestion steps, but stops short of implementing data pipelines.
- Examples
- Uses large parquet or text dumps and web-scraped datasets to show why abundant unannotated text still needs curation.
- What you can do after
- Practice/perform a high-level cleaning pass that checks task fit, bias, anomalies, and readiness for model ingestion.
The Quick Python Book, Fourth Edition
Data cleaning is the process of identifying and correcting, removing, or otherwise handling irregular, invalid, missing, or inconvenient data values and formats.
- How it is used
- Prepares text-file and CSV values for calculations and consistent output during data processing.
- How deep it goes
- Works through missing-value choices, row-wise file cleanup, and pandas column conversions; does not prescribe one cleaning policy for every dataset.
- Examples
- Uses weather observations containing "Missing" heat-index values and percentage strings, plus a tab-delimited county-to-CSV conversion.
- What you can do after
- Apply it via Python’s csv module or pandas to handle missing markers, convert percentages, and produce usable CSV data.
Read how The Quick Python Book, Fourth Edition explains data cleaning →
Pandas Workout: 200 exercises to make you a stronger data analyst
Data cleaning is the process of detecting, correcting, removing, or transforming inaccurate, incomplete, or inconsistent data.
- How it is used
- Uses pandas to make messy datasets accurate, analyzable, and easier to understand.
- How deep it goes
- Works through missing values, transformations, judgment, and dtype conversion, including chunk-inference issues.
- Examples
- Uses parking-ticket records, BLANKPLATE values, taxi rides, missing-value counts, and dtype conversion as practical cases.
- What you can do after
- Apply it via pandas to measure missingness, filter rows, replace invalid values, reshape columns, and correct dtypes.
Read how Pandas Workout: 200 exercises to make you a stronger data analyst explains data cleaning →
D3.js in Action, Third Edition
Data cleaning is the detection and correction or removal of inaccurate, inconsistent, incomplete, or improperly formatted data.
- How it is used
- Prepares gathered information for reliable, focused data visualizations.
- How deep it goes
- Explains scoping, gathering, analysis, cleaning, formatting, and standardization, but stops short of detailed validation rules.
- Examples
- Uses Van Gogh artwork and letters, Wikipedia tables, and Facebook posts collected through CrowdTangle.
- What you can do after
- Apply it via D3.js by producing a scoped, standardized dataset ready for visualization.
Read how D3.js in Action, Third Edition explains data cleaning →
First-Party Data Activation
The process of detecting and correcting inaccurate, inconsistent, incomplete, or improperly formatted data.
- How it is used
- Supports identity resolution, unified customer profiles, CRM quality, and reliable lead scoring.
- How deep it goes
- Explains hygiene, match errors, field selection, manual review, and scope decisions without detailing cleaning algorithms.
- Examples
- Uses false-positive and false-negative matches, company fields, duplicate CRM accounts, and merged company records.
- What you can do after
- Apply data hygiene to improve matching accuracy, consolidate duplicate accounts, and reduce privacy and customer-experience risks.
Read how First-Party Data Activation explains data cleaning →
Grokking Machine Learning
Data cleaning is the process of detecting, correcting, removing, or imputing invalid, inconsistent, or missing data.
- How it is used
- Prepares real-world data for model training by inspecting and handling missing values.
- How deep it goes
- Demonstrates missing-value counts, column removal, median imputation, and filling a categorical value.
- Examples
- Uses Titanic passenger data, with missing Cabin, Age, and Embarked values, as a cleanup example.
- What you can do after
- Apply it via Pandas to count missing values, drop a column, fill gaps, and save cleaned data as a CSV.
Outlier Detection in Python
Data cleaning is the process of identifying and correcting, removing, or excluding inaccurate, corrupted, irrelevant, or anomalous data.
- How it is used
- Prepares contaminated tabular data for outlier-detector training and scoring.
- How deep it goes
- Explains preprocessing choices and gives detector-based and IQR-based procedures, but stops short of universal cleaning rules.
- Examples
- Uses null-count inspection, median imputation for Strikeouts, IQR filtering, detector ensembles, and synthetic inlier datasets.
- What you can do after
- Apply via statistical tests, machine-learning detectors, and dataframe filters while preserving caution about false exclusions and distribution mismatch.
Read how Outlier Detection in Python explains data cleaning →
The Art of AI Product Development
Data cleaning is the detection and correction or removal of inaccurate, inconsistent, duplicate, missing, or otherwise problematic data.
- How it is used
- Prepares behavioral and profile data for clustering and more trustworthy engagement insights.
- How deep it goes
- Details missing values, incorrect data, ranges, duplicates, normalization, and iterative review, but not cleaning algorithms.
- Examples
- Uses Mixpanel events, CRM records, call-center records, purchases, visits, values, and activity timestamps.
- What you can do after
- Apply it by validating, deduplicating, correcting, normalizing, and revisiting data before and during model development.
Read how The Art of AI Product Development explains data cleaning →
Demand Forecasting Best Practices
Data cleaning is the detection and correction of errors, inconsistencies, missing values, and unsuitable representations in datasets.
- How it is used
- Improves demand forecasts by making demand and demand-driver data usable at the required aggregation.
- How deep it goes
- Covers collection, root-cause correction, and error feedback, but does not prescribe specific cleaning algorithms.
- Examples
- Uses Excel files, prices, promotions, holidays, shortages, weather, and external market data as cleaning cases.
- What you can do after
- Diagnose forecast errors by tracing them to inaccurate data or missing demand drivers, then collect and clean relevant data.
Read how Demand Forecasting Best Practices explains data cleaning →
Everyday Data Visualization
Data cleaning is the process of detecting and correcting inaccurate, inconsistent, or improperly formatted data.
- How it is used
- Supports requirements gathering by checking whether available data can actually be combined and used.
- How deep it goes
- Explains formatting cleanup and mapping keys, but stops short of prescribing a detailed cleaning workflow.
- Examples
- Uses team names stored differently across two systems and a mapping table for linking unmatched values.
- What you can do after
- Diagnose/debug with it whether mismatched values need formatting cleanup or a manually maintained mapping key.
Read how Everyday Data Visualization explains data cleaning →
Financial AI in Practice
The process of detecting and correcting or removing errors, inconsistencies, and irrelevant data to improve data quality.
- How it is used
- Prepares heterogeneous news and credit data for embedding, analysis, and downstream modeling.
- How deep it goes
- Details source-specific normalization, merging, deduplication, and credit-data handling, but leaves full production orchestration outside the light example.
- Examples
- Uses four news feeds merged by URL and a Kaggle-based credit dataset with missingness, categorical-variable, and outlier handling.
- What you can do after
- Apply it via source-specific pandas functions to standardize, merge, and deduplicate news records before embedding.
Related concepts
- preprocessingpart of · Preprocessing is the preparation or transformation of data, content, or other inputs into a form suitable for a subsequent computational operation.
- pandasused by · pandas is an open-source Python library that provides labeled data structures and tools for manipulating and analyzing tabular and other structured data.
- stop-word removalhas part · Stop-word removal is a text-preprocessing technique that excludes selected common words considered uninformative for a language-processing, retrieval, or analysis task.
- lexical searchused by · Lexical search retrieves or ranks items by matching query terms or other surface-text features against indexed content, often using sparse representations and statistical weighting.
- textused by · Text is written or digitally encoded linguistic content represented as characters, words, tokens, or other textual units.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.