NaN (“not a number”) is pandas’ traditional marker for missing numeric data. It is a floating-point value: it cannot be converted to an integer, and it is not equal to itself. Missing values arise naturally when a sensor fails, a survey response is absent, or newly combined rows lack a value from an existing column. section 2.20 section 2.10
Spot missing values after combining data
Appending products that lack sales information leaves NaN in the sales column. Select the affected row indexes and column with loc, inspect the resulting Series, then assign replacement sales values. Because NaN is a float, pandas changes the column to float64 to accommodate it. section 2.10
product_id name wholesale_price retail_price sales 0 23 computer 500.0 1000.0 100.0 1 96 Python Workout 35.0 75.0 1000.0 2 97 Pandas Workout 35.0 75.0 500.0 3 15 banana 0.5 1.0 200.0 4 87 sandwich 3.0 5.0 300.0 5 24 phone 200.0 500.0 NaN 6 16 apple 0.5 1.0 NaN 7 17 pear 0.6 1.2 NaN
The combined product table makes the gap visible: the three appended rows have NaN in sales while their other fields are present. section 2.10
Treating every missing value as zero can distort analysis, such as a mean. First decide whether the value should be removed, replaced, or estimated from nearby data. The choice depends on what the column represents and how much data would be discarded. section 2.20 section 5.1
Common pandas responses to NaN include:
- Count available observations with count(). On a Series it returns non-NaN values; on a DataFrame it returns a per-column count of non-NaN values. section 3.2
- Remove incomplete rows with dropna(). Use subset to require values only in columns that matter, and thresh when a minimum number of non-null values is enough. section 5.2 section 5.4
- Replace missing values with fillna() when a known substitute is appropriate, such as using '0' after extracting a missing tens digit from a string. section 1.11
- Estimate gaps with interpolate() for continuous measurements. Its default linear method averages surrounding values; method='nearest' uses the closest non-NaN value. section 2.22
Repairing a temperature outage
A temperature DataFrame can simulate outages by assigning NaN to temp where hour is 3 or 6. Calling interpolate() returns a new DataFrame that fills the gaps. Linear interpolation is a reasonable choice when temperatures change gradually from hour to hour, but it is not automatically suitable for every kind of data. section 2.22
NaN also affects dtype planning. Integer columns containing NaN must remain floating-point until the missing rows are removed; afterward they can be converted to a compact integer dtype. When reading a large CSV, specifying temporary float32 types avoids inconsistent chunk-level type inference that could otherwise produce an object column. section 3.10
For string-oriented columns, newer pandas guidance distinguishes NaN from pd.NA: pd.StringDtype should use pd.NA, a missing-value marker compatible with pandas dtypes. The older object-string approach can still contain float NaN. section 9.0
