Data cleaning
Data cleaning is the process of detecting and correcting, removing, transforming, or otherwise handling inaccurate, incomplete, inconsistent, duplicate, or unsuitable data.
Starting Data Analytics with Generative AI and Python
A data cleaning protocol is a documented procedure for detecting, correcting, and documenting problems in data.
- How it is used
- Prepares business data early so collection and preparation artifacts do not distort analysis.
- How deep it goes
- Outlines a cleaning checklist and order-dependent risks, while leaving the protocol adaptable to the use case.
- Examples
- Uses duplicate removal before average imputation and Portuguese customer-review keyword matching to illustrate cleaning choices.
- What you can do after
- Evaluate and apply cleaning steps to a dataset while checking their order and relevance.
Read how Starting Data Analytics with Generative AI and Python explains data cleaning →
Data Without Labels
Data cleaning is the process of detecting and correcting or removing inaccurate, incomplete, inconsistent, duplicate, or irrelevant data.
- How it is used
- Prepares structured and text data for reliable analysis and machine learning after discovery and before modeling.
- How deep it goes
- Details quality checks and context-dependent text-cleaning choices, but does not specify a universal treatment for every dirty value.
- Examples
- Uses sparse or constant customer attributes, incompatible date formats, and context-sensitive stop words to illustrate cleaning decisions.
- What you can do after
- Practice/perform it by checking duplicates, missing values, junk, and outliers, then choosing treatments suited to the dataset.
The Well-Grounded Data Analyst
Data cleaning is the process of detecting, correcting, standardizing, or removing inaccurate, inconsistent, incomplete, or irrelevant data.
- How it is used
- Uses cleaning to prepare address and published-statistics data for geographic or exploratory analysis.
- How deep it goes
- Explains inspection, standardization, matching, fallback categories, and exceptions, but stops short of a general automated validation framework.
- Examples
- Demonstrates address casing, delimiter matching, variable-length records, reference city lists, unmatched cases, and customer-spend summaries.
- What you can do after
- Apply it via pandas to inspect, standardize, classify, and summarize imperfect tabular data.
Read how The Well-Grounded Data Analyst explains data cleaning →
LLMs in Production
Data cleaning is the process of detecting, correcting, removing, or managing inaccurate, inconsistent, incomplete, or unsuitable data.
- How it is used
- Prepares data for LLM training when self-supervised learning removes some traditional preparation but not the need for quality.
- How deep it goes
- Outlines preparation steps and linguistic considerations while stopping short of actual data-pipeline construction.
- Examples
- Uses large text dumps and web-scraped datasets to motivate schemas, annotation, and task-appropriate distributions.
- What you can do after
- Evaluate and understand cleaning choices for LLM training, including their implications for data quality and user experience.
The Quick Python Book, Fourth Edition
Data cleaning is the process of identifying and correcting, removing, or otherwise handling irregular, invalid, missing, or inconvenient data values and formats.
- How it is used
- Prepares text and CSV data for calculations, further processing, and export.
- How deep it goes
- Works through missing-value handling, record and column selection, percentage conversion, and output, using both row-wise and data-frame operations.
- Examples
- Uses Illinois weather records to extract Cook County data, handle 'Missing', and convert heat-index coverage percentages.
- What you can do after
- Apply it via Python’s csv module or pandas to select records, handle missing markers, convert percentages, and export cleaned data.
Read how The Quick Python Book, Fourth Edition explains data cleaning →
Pandas Workout: 200 exercises to make you a stronger data analyst
Data cleaning is the process of detecting, correcting, removing, or transforming inaccurate, incomplete, or inconsistent data.
- How it is used
- Uses pandas to make messy datasets accurate, analyzable, and easier to understand.
- How deep it goes
- Works through missing values, transformations, judgment, and dtype conversion, including chunk-inference issues.
- Examples
- Uses parking-ticket records, BLANKPLATE values, taxi rides, missing-value counts, and dtype conversion as practical cases.
- What you can do after
- Apply it via pandas to measure missingness, filter rows, replace invalid values, reshape columns, and correct dtypes.
Read how Pandas Workout: 200 exercises to make you a stronger data analyst explains data cleaning →
Building Specialized AI Systems
Data cleaning is the process of detecting and correcting or removing erroneous, inconsistent, irrelevant, or low-quality data and standardizing the remaining content.
- How it is used
- Prepares representative, privacy-conscious datasets for reliable fine-tuning and downstream evaluation.
- How deep it goes
- Details preprocessing operations and iterative cleaning cycles, but defers model training and rigorous evaluation procedures.
- Examples
- Uses transaction logs, financial reports, DOCX files, dataframes, instruction prompts, and conversational input-response pairs.
- What you can do after
- Apply it via pandas and regular expressions to produce standardized, anonymized, model-ready records.
Read how Building Specialized AI Systems explains data cleaning →
D3.js in Action, Third Edition
Data cleaning is the detection and correction or removal of inaccurate, inconsistent, incomplete, or improperly formatted data.
- How it is used
- Prepares gathered information for reliable, focused data visualizations.
- How deep it goes
- Explains scoping, gathering, analysis, cleaning, formatting, and standardization, but stops short of detailed validation rules.
- Examples
- Uses Van Gogh artwork and letters, Wikipedia tables, and Facebook posts collected through CrowdTangle.
- What you can do after
- Apply it via D3.js by producing a scoped, standardized dataset ready for visualization.
Read how D3.js in Action, Third Edition explains data cleaning →
Vibe Engineering
Data cleaning is the process of detecting, correcting, removing, or managing inaccurate, incomplete, inconsistent, or irrelevant data.
- How it is used
- Prepares business data for statistical analysis, notebooks, and machine-learning workflows.
- How deep it goes
- Details null filtering, percentile-based outlier removal, categorical encoding, and evaluation, but not broader cleaning automation.
- Examples
- Uses a real Indian medicine dataset, null dosage and salt fields, 99th-percentile price filtering, and one-hot encoding.
- What you can do after
- Apply it by preparing a price-prediction dataset and checking resulting predictions with standard error metrics.
First-Party Data Activation
The process of detecting and correcting inaccurate, inconsistent, incomplete, or improperly formatted data.
- How it is used
- Supports identity resolution, unified customer profiles, CRM quality, and reliable lead scoring.
- How deep it goes
- Explains hygiene, match errors, field selection, manual review, and scope decisions without detailing cleaning algorithms.
- Examples
- Uses false-positive and false-negative matches, company fields, duplicate CRM accounts, and merged company records.
- What you can do after
- Apply data hygiene to improve matching accuracy, consolidate duplicate accounts, and reduce privacy and customer-experience risks.
Read how First-Party Data Activation explains data cleaning →
Software Engineering for Data Scientists
Data cleaning is the detection and correction or removal of inaccurate, inconsistent, incomplete, or otherwise unsuitable data.
- How it is used
- Positions data cleaning between data exploration and feature engineering as preparation for analysis and model development.
- How deep it goes
- Defines four issue categories and treatments but stops short of implementation procedures or detailed scaling techniques.
- Examples
- Uses missing-value replacement, correlated-variable removal, skew transformation, and ad-click imbalance as concrete examples.
- What you can do after
- Apply it by minimizing dataset issues before analysis or model training and by recognizing scaling concerns in larger workflows.
Read how Software Engineering for Data Scientists explains data cleaning →
Grokking Machine Learning
Data cleaning is the process of detecting, correcting, removing, or imputing invalid, inconsistent, or missing data.
- How it is used
- Prepares real-world data for model training by inspecting and handling missing values.
- How deep it goes
- Demonstrates missing-value counts, column removal, median imputation, and filling a categorical value.
- Examples
- Uses Titanic passenger data, with missing Cabin, Age, and Embarked values, as a cleanup example.
- What you can do after
- Apply it via Pandas to count missing values, drop a column, fill gaps, and save cleaned data as a CSV.
Outlier Detection in Python
Data cleaning is the process of identifying and correcting, removing, or excluding inaccurate, corrupted, irrelevant, or anomalous data.
- How it is used
- Prepares contaminated tabular data for outlier-detector training and scoring.
- How deep it goes
- Explains preprocessing choices and gives detector-based and IQR-based procedures, but stops short of universal cleaning rules.
- Examples
- Uses null-count inspection, median imputation for Strikeouts, IQR filtering, detector ensembles, and synthetic inlier datasets.
- What you can do after
- Apply via statistical tests, machine-learning detectors, and dataframe filters while preserving caution about false exclusions and distribution mismatch.
Read how Outlier Detection in Python explains data cleaning →
Learn AI Data Engineering
Data cleaning is the process of detecting and correcting, removing, or standardizing inaccurate, incomplete, duplicated, or unsuitable data.
- How it is used
- Supports data-quality and web-scraping pipelines by making records or extracted text reliable enough for downstream processing.
- How deep it goes
- Explains duplicate removal, null-column filtering, structured AI responses, retries, and empty-content validation without detailing broader governance policies.
- Examples
- Uses a customer DataFrame with duplicate rows and a null-heavy column, plus HTML pages requiring tag removal and retrieval checks.
- What you can do after
- Apply pandas or structured Chat Completions workflows to remove duplicates, filter null-heavy columns, and validate cleaned content.
The Art of AI Product Development
Data cleaning is the detection and correction or removal of inaccurate, inconsistent, duplicate, missing, or otherwise problematic data.
- How it is used
- Prepares behavioral and profile data for clustering and more trustworthy engagement insights.
- How deep it goes
- Details missing values, incorrect data, ranges, duplicates, normalization, and iterative review, but not cleaning algorithms.
- Examples
- Uses Mixpanel events, CRM records, call-center records, purchases, visits, values, and activity timestamps.
- What you can do after
- Apply it by validating, deduplicating, correcting, normalizing, and revisiting data before and during model development.
Read how The Art of AI Product Development explains data cleaning →
Demand Forecasting Best Practices
Data cleaning is the detection and correction of errors, inconsistencies, missing values, and unsuitable representations in datasets.
- How it is used
- Improves demand forecasts by making demand and demand-driver data usable at the required aggregation.
- How deep it goes
- Covers collection, root-cause correction, and error feedback, but does not prescribe specific cleaning algorithms.
- Examples
- Uses Excel files, prices, promotions, holidays, shortages, weather, and external market data as cleaning cases.
- What you can do after
- Diagnose forecast errors by tracing them to inaccurate data or missing demand drivers, then collect and clean relevant data.
Read how Demand Forecasting Best Practices explains data cleaning →
Everyday Data Visualization
Data cleaning is the process of detecting and correcting inaccurate, inconsistent, or improperly formatted data.
- How it is used
- Supports requirements gathering by checking whether available data can actually be combined and used.
- How deep it goes
- Explains formatting cleanup and mapping keys, but stops short of prescribing a detailed cleaning workflow.
- Examples
- Uses team names stored differently across two systems and a mapping table for linking unmatched values.
- What you can do after
- Diagnose/debug with it whether mismatched values need formatting cleanup or a manually maintained mapping key.
Read how Everyday Data Visualization explains data cleaning →
Financial AI in Practice
The process of detecting and correcting or removing errors, inconsistencies, and irrelevant data to improve data quality.
- How it is used
- Prepares heterogeneous news and credit data for embedding, analysis, and downstream modeling.
- How deep it goes
- Details source-specific normalization, merging, deduplication, and credit-data handling, but leaves full production orchestration outside the light example.
- Examples
- Uses four news feeds merged by URL and a Kaggle-based credit dataset with missingness, categorical-variable, and outlier handling.
- What you can do after
- Apply it via source-specific pandas functions to standardize, merge, and deduplicate news records before embedding.
Also covered in
- Digital Twins in Action
- Fight Fraud with Machine LearningIn early access (MEAP)
- 100 C++ Mistakes and How to Avoid Them
- Data-Driven Cybersecurity
- Machine Learning System Design
- AI Agents in Action
- AI for Everyday IT
- Grokking Algorithms, Second Edition
- Python How-To
- Get Programming: Learn to code with Python
- Enterprise RAGIn early access (MEAP)
- Knowledge Graphs and LLMs in Action
What to know first
- language modelA language model is a computational model that represents patterns in linguistic sequences to assign probabilities, predict, generate, or evaluate language.
- PythonPython is a high-level, general-purpose, dynamically typed programming language with readable syntax and support for multiple programming paradigms.
- machine learningMachine learning is a field of computational methods in which systems learn patterns or relationships from data to make predictions, decisions, or other outputs.
- missing valueA missing value is an expected data value or observation that is absent, unavailable, unknown, or inapplicable.
Related concepts
- preprocessingpart of · Preprocessing is the preparation or transformation of data or other inputs into a form suitable for subsequent processing, analysis, or execution.
- pandasused by · pandas is an open-source Python library providing labeled data structures and tools for manipulating and analyzing tabular and other structured data.
- lexical searchused by · Lexical search retrieves or ranks items by matching query terms or other surface-text features against indexed content, often using sparse representations and statistical weighting.
- stop-word removalhas part · Stop-word removal is a text-preprocessing operation that excludes selected common words considered to provide little useful information for a task.
Questions about data cleaning
What is data cleaning?
Data cleaning is the process of detecting and correcting, removing, transforming, or otherwise handling inaccurate, incomplete, inconsistent, duplicate, or unsuitable data.
Which Manning books explain data cleaning?
Starting Data Analytics with Generative AI and Python, Data Without Labels, The Well-Grounded Data Analyst, LLMs in Production, The Quick Python Book, Fourth Edition, Pandas Workout: 200 exercises to make you a stronger data analyst, Building Specialized AI Systems, D3.js in Action, Third Edition and 10 more.
What should I know before learning data cleaning?
Manning books that teach data cleaning usually introduce language model, Python, machine learning and missing value first.
What is the difference between data cleaning and fairness-constrained training?
Data cleaning is the process of detecting and correcting, removing, transforming, or otherwise handling inaccurate, incomplete, inconsistent, duplicate, or unsuitable data. Training a model with constraints intended to promote fair outcomes or limit unfair disparities.
What is data cleaning part of?
Data cleaning is part of preprocessing. Preprocessing is the preparation or transformation of data or other inputs into a form suitable for subsequent processing, analysis, or execution.
What are the parts of data cleaning?
Data cleaning includes stop-word removal.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.

















