liveBook · Manning

Type inference

As explained in Fast Python: High performance techniques for large datasets

Type inference is the automatic deduction of program or expression types from available values, context, and constraints without requiring explicit type annotations.

Type inference assigns column types from loaded values and context. In pandas, failed inference can promote integers or other compact values to object or float64, expanding a 566 MB text file into a 2 GB DataFrame. section 7.1.2

Use memory evidence to validate inferred types and correct columns explicitly:

  • Inspect deep memory usage and per-column unique-value counts before changing dtypes. section 7.1.2
  • Convert date-time strings with pd.to_datetime; each example column falls from 464 MB to 48 MB. section 7.1.2
  • Choose compact numeric representations when value ranges permit them, such as int8 for payment_type. section 7.1.2

Repairing an inferred payment type

Replace missing values before converting payment_type to np.int8; this column has only six numeric values but was inferred as float64.

How Fast Python: High performance techniques for large datasets covers it

How deep it goes
Explains warnings, profiles column memory, and demonstrates conversions, but does not detail the inference algorithm internals.
Examples
Uses datetime columns, payment_type, and store_and_fwd_flag to demonstrate inferred-type diagnosis and narrowing.
What it assumes
Assumes familiarity with pandas DataFrames and basic column operations while teaching memory inspection and dtype conversion.
What you can do after
Apply it via pandas to inspect inferred dtypes and reduce DataFrame memory consumption.
How it connects
Connects type inference with pandas dtypes, missing values, cardinality, and memory usage.
Cover of Fast Python: High performance techniques for large datasets
liveBookmanning.com
Read Fast Python: High performance techniques for large datasets on liveBook Get the book at manning.com

Also explained in

Compare how 16 books explain type inference →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Fast Python: High performance techniques for large datasets. The section links point to the parts of the book it draws on.