liveBook · Manning

Data transformation

As explained in Statistics Every Programmer Needs

map is a pandas Series method that applies a mapping or function to each element and returns the transformed values.

Data transformation changes values into a form suited to analysis. For a pandas Series of categorical labels, `map()` applies a mapping to each value and returns the transformed values. Assign the result back to the column when the numeric codes should replace the original labels. section 5.2.1

Encode a binary outcome

To prepare raisin varieties for logistic regression, define `class_mapping = {'Kecimen': 0, 'Besni': 1}` and run `raisins['Class'] = raisins['Class'].map(class_mapping)`. Check selected columns with `head()` and `tail()`: the former Kecimen rows should show 0, and the former Besni rows should show 1. Either assignment of 0 and 1 works, but interpret predictions as the probability of the class coded 1. section 5.2.1

The mapping must serve the meaning of the variable, not just its data type. In fourth-down play data, categorical venue and play type become numeric inputs for decision trees and random forests. Score differential needs a different transformation: multiply it by −1 for road-team plays so its sign describes the advantage from the offense’s perspective. section 6.2.2

>>> OFFENSIVE_TEAM_VENUE_mapping = {'Road': 0, 'Home': 1}
>>> nfl_pbp['OFFENSIVE_TEAM_VENUE'] = \
>>>     nfl_pbp['OFFENSIVE_TEAM_VENUE'].map(OFFENSIVE_TEAM_VENUE_mapping)

>>> PLAY_TYPE_mapping = {'Run': 0, 'Pass': 1}
>>> nfl_pbp['PLAY_TYPE'] = \
>>>     nfl_pbp['PLAY_TYPE'].map(PLAY_TYPE_mapping)

The listing shows two mapping dictionaries and the corresponding Series `map()` assignments for venue and play type. section 6.2.2

Verify transformed columns before modeling. The football data’s `info()` output reports 881 non-null venue values among 883 rows after conversion, so numeric dtype alone does not establish that every row has a usable value. Similarly, replacing missing yards gained with zero is appropriate only when zero reflects the play; imputation can otherwise distort predictions. section 6.2.2

Numeric transformations address another problem: distribution shape. Logarithmic, reciprocal, and square-root transformations can compress right-skewed values before linear regression, depending on the distribution. A fitted relationship then applies on the transformed scale, not automatically on the original scale; consider whether a different model better represents a nonlinear relationship. section 4.1.3

How Statistics Every Programmer Needs covers it

How deep it goes
Demonstrates dictionary-based Series mapping and output checks, then distinguishes encoding from conditional and distribution-shaping transformations.
Examples
Uses raisin-variety labels mapped to 0 and 1, plus football venue and play-type labels mapped to numeric codes.
What it assumes
Assumes basic DataFrame column access while demonstrating how to define a mapping dictionary and apply it to a Series.
What you can do after
Apply it via pandas `map()` to encode categories, then inspect the resulting values and non-null counts before modeling.
How it connects
Links pandas Series mapping to binary outcomes, logistic regression, decision trees, random forests, and missing-data checks.

Also explained in

Compare how 20 books explain data transformation →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Statistics Every Programmer Needs. The section links point to the parts of the book it draws on.