liveBook · Manning

Data cleaning

As explained in Vibe Engineering

Data cleaning is the process of detecting, correcting, removing, or managing inaccurate, incomplete, inconsistent, or irrelevant data.

Data cleaning detects, corrects, removes, or manages inaccurate, incomplete, inconsistent, or irrelevant data before analysis or machine learning. It includes parsing, normalization, handling malformed records, and deciding whether unusual values represent real cases or broken data. section 4.0 section 4.1

The appealing shortcut is to use a dataset that already looks clean, especially synthetic data. That can produce unrealistically accurate analyses or models because it avoids outliers, malformed records, and other real-world difficulties. Treat cleaning decisions as model assumptions, and verify the resulting numbers against business context: plausible output can still hide flawed logic. section 4.0 section 4.1

A practical cleaning workflow for price prediction includes:

  • Load a prepared feature file and confirm that the preceding preparation step completed; otherwise, the workflow fails. section 4.5.4
  • Remove rows with null Dosage_Value or Salt_Name values when the training process requires complete columns. section 4.5.4
  • Exclude prices at or above the 99th percentile to reduce the influence of outliers or potentially broken values. section 4.5.4
  • Separate features from the price label, then encode categorical fields such as Salt_name, Manufacturer, and Dosage_Unit numerically. section 4.5.4
  • Evaluate predictions with R², mean absolute error, and root mean squared error rather than trusting a successful run alone. section 4.5.4
Use Case
Dataset A (Indian)
Dataset B (Synthetic)
Exploratory Data Analysis (EDA)
✅ Strong — real distributions, price trends
⚠️ Limited — synthetic distributions
Supervised ML for Prediction
⚠️ Limited (no target label by default)
⚠️ Same — likely no target labels
Clustering / Unsupervised learning
✅ Yes — cluster by manufacturer, composition
⚠️ Yes but less real-world meaning
NLP/Categorization
⚠️ Yes — text fields (names/composition)
⚠️ Yes (names) but synthetic semantics
Recommender systems
⚠️ Possible but needs augmentation
❌ Limited
Benchmarking models / DEMO
⚠️ Possible but raw
✅ Ideal due to cleaner, controlled data
Aspect
Indian Medicine Dataset (A)
Kaggle Medicine Dataset (B)
Source
Scraped real data
Synthetic generated data
Size
~250 K records
~50 K records
Real-world
✅ Yes
❌ No
Cleanliness
⚠️ No (requires cleaning)
✅ Likely
Usability for Research
High
Moderate
ML Power
Good for real analysis
Good for ML practice

Compare real and synthetic datasets by cleanliness, realism, exploratory-analysis value, and machine-learning use before choosing what to clean. section 4.1

How Vibe Engineering covers it

How deep it goes
Details null filtering, percentile-based outlier removal, categorical encoding, and evaluation, but not broader cleaning automation.
Examples
Uses a real Indian medicine dataset, null dosage and salt fields, 99th-percentile price filtering, and one-hot encoding.
What it assumes
Assumes basic programming and business-domain knowledge while leaving Python expertise unnecessary for notebook creation.
What you can do after
Apply it by preparing a price-prediction dataset and checking resulting predictions with standard error metrics.
How it connects
Connects data cleaning with dataset selection, normalization, feature engineering, categorical encoding, outliers, and model evaluation.

Also explained in

Compare how 18 books explain data cleaning →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Vibe Engineering. The section links point to the parts of the book it draws on.