Use pandas fillna to replace missing values instead of immediately discarding rows. It works on Series and DataFrame objects and returns a new object, so assign the result back when you want the cleaned values to persist. section 2.20
Recovering missing tens digits
After converting numbers to strings and selecting the second-to-last character, one-digit values produce NaN. Replace those missing characters with the string "0", then convert the result back to a small integer type. section 1.11
(
s
.astype(str) # get a series based on s, with dtype str
.str.get(-2) # retrieve the second-to-last character
.fillna('0') # replace NaN with '0'
.astype(np.int8) # get a new series back dtype int8
)
A readable method chain shows conversion to strings, character selection, missing-value replacement, and final conversion as separate steps. section 1.11
Imputing Titanic data
For numeric age values, calculate the non-null mean and assign the filled Series back to df['age']. For categorical home.dest values, calculate the mode, select its first value, and assign that replacement back to the column. section 5.10
Choose the replacement according to the data and the task:
- Use a literal such as '0' when every missing value needs the same explicit fallback. section 1.11
- Use a calculated statistic such as mean for numeric data or mode for categorical data, while recognizing that imputation can alter the distribution. section 5.10
- Inspect missingness first with isnull and decide whether replacement or row removal is appropriate. section 2.20 section 5.10
