A missing value marks an unavailable entry in a column. It affects more than calculations: it can determine which data type a column uses, how much memory it occupies, and whether a conversion succeeds. Inspect both the values and the inferred type before downsizing a column. section 7.1.2 section 8.3.2
A one-byte column with missing entries
A trip-data payment code has only a few numeric values, yet casting it directly from float64 to NumPy int8 fails because some entries are NA. If zero is unused, `df["payment_type"].fillna(0).astype(np.int8)` makes the conversion possible and cuts that column from about 48 MB to 6 MB. The chosen replacement must not collide with a real value; other columns may need a different strategy. section 7.1.2
Arrow uses a separate bit array with one entry per row to indicate whether a value is missing. An integer column can therefore retain an integer type while containing nulls, without reserving an integer value as a marker. This adds a modest memory cost but can make low-cardinality columns substantially more compact. Converting an Arrow table to pandas can lose that compact representation and costs time and memory. section 7.4.2
Decide what a missing entry means before processing it:
- For arithmetic, convert nonnumeric placeholders to missing values, then filter only the rows the calculation requires. A property-tax calculation replaces `X` with `np.nan` and excludes rows without property-tax amounts before dividing by total taxes. section 10.1.1
- For partitioned Parquet output, null partition keys are filtered out in the demonstrated workflow because those partitions cannot be based on nulls. Dropping such rows is an exercise-specific choice, not a general cleanup rule. section 8.2.3
- For chunked CSV conversion, specify types consistently: an integer-like column may be inferred as float in chunks containing nulls, changing the schema between chunks. section 8.3.2
