Data cleaning prepares records for analysis, much as a cook inspects and prepares ingredients before making a meal. It finds inaccuracies and inconsistencies, corrects or removes them, and puts data into a uniform format. Do it early: errors in representation or integrity can persist through later work and distort business conclusions. section 2.4.1 section 2.5
A practical cleaning protocol should identify the checks to run and the changes made:
- Remove duplicate records before calculations that depend on record counts. section 2.4.1 section 2.5
- Assess missing values and decide how to handle them. Filling a column with its average before removing duplicates can hide a problem. section 2.4.1 section 2.5
- Correct data-entry errors, make formats consistent, and validate the result. Inspect what changed after each step. section 2.4.1 section 2.5
Cleaning reviews before keyword search
Suppose a keyword search misses positive customer reviews. An AI assistant might suggest removing punctuation and extra spaces, lowercasing text, and stemming words. Apply only changes the data needs: lowercasing may already be done. Then rerun the search and check it against review scores rather than assuming cleaner text means better results. One basic search had about 89% specificity but only 47% sensitivity, leaving many positive reviews undetected. section 5.5.5
Listing 5.8 Data cleaning before keyword search
import string
import nltk
from nltk.stem import RSLPStemmer
nltk.download('rslp')
stemmer = RSLPStemmer()
def remove_punctuation(text):
return text.translate(str.maketrans("", "", string.punctuation))
def remove_extra_whitespace(text):
return ' '.join(text.split())
def stem_words(text):
words = text.split()
stemmed_words = [stemmer.stem(word) for word in words]
return ' '.join(stemmed_words)
def clean_text(text):
if not isinstance(text, str):
return ""
text = text.lower()
text = remove_punctuation(text)
text = remove_extra_whitespace(text)
text = stem_words(text)
return text
df['cleaned_review_text'] = df['review_text'].apply(clean_text)
This Python listing combines text-cleaning operations into one function. Inspect its handling of non-text values and test the output on sample reviews before using it for keyword search. section 5.5.5
There is no universal cleaning sequence for every dataset. First understand what the records mean and how complete they are; adapt the protocol to the data and question. AI can suggest code, but it may make hidden assumptions or skip cleaning altogether. Check its suggestions against raw records and the intended analysis. section 2.4 section 2.5 section 5.5.7
