Data exploration investigates a tabular dataset before modeling so that its structure, quality, and usefulness become visible. It goes beyond generating features: examine what each column represents, what values occur, how variables relate, and whether the data can represent the predictive problem. There is no fixed blueprint; descriptive statistics, tests, and plots guide the next data-cleaning or modeling decision. section 2.4 section 2.5
Begin with single columns. Identify numeric, ordinal, categorical, and date features; inspect a few rows; count distinct values; and compare variation, missing values, and suspicious sentinel values. Summaries, value counts, histograms, and box plots reveal constant features, rare categories, skew, outliers, inconsistencies, and possible typing mistakes. section 2.4.2 section 2.5
A practical exploration moves from simple inspection toward relationships:
- Inspect rows and metadata to form an initial hypothesis about column types and meanings. section 2.4.2
- Profile individual features with distinct-value counts, descriptive statistics, distributions, and category frequencies. section 2.4.2
- Compare pairs of features with correlation or association measures to find relationships and redundant predictors. section 2.4.3
- Project numeric data into fewer dimensions to spot unusual clusters or isolated observations before fitting a model. section 2.4.3
- Inspect the target alongside predictors, then make and visualize any justified filtering decision before modeling. section 7.1.6
Classifying Auto MPG columns
A first pass over Auto MPG data separates mpg, displacement, horsepower, weight, and acceleration as numeric features; cylinders and model_year as ordinal; and origin and car_name as categorical. Looking at distinct counts helps distinguish low-cardinality numeric codes from continuous measurements. Domain knowledge can then refine an ambiguous classification, such as treating cylinder count as ordinal. section 2.4.2
numeric_feats = ["mpg", "displacement", "horsepower", "weight", "acceleration"] ordinal_feats = ["cylinders", "model_year"] categorical_feats = ["origin", "car_name"]
This feature list makes the initial type decisions explicit, providing a basis for choosing summaries, encodings, and later preprocessing. section 2.4.2
Relationship checks serve data validation as well as prediction. Strong positive or negative correlations can signal redundant features worth removing. A multivariate projection such as t-SNE makes high-dimensional numeric data viewable, helping expose clusters or points that do not fit the wider dataset. These checks are preliminary validation, not an attempt to tune a model to the data. section 2.4.3
Listing 2.9 Plotting a t-SNE low-dimensional projection of a dataset
from sklearn.manifold import TSNE #1
import matplotlib.pyplot as plt #2
tsne = TSNE(n_components=2,
perplexity=30.,
init="random",
learning_rate="auto",
random_state=42) #3
X = data[numeric_feats + ordinal_feats].fillna(
data[numeric_feats + ordinal_feats].mean()) #4
projection_2D = tsne.fit_transform(X) #5
plt.figure(figsize=(15, 15))
plt.scatter(projection_2D[:, 0], projection_2D[:, 1],
edgecolor='none',
alpha=0.80,
s=10)
plt.show() #6
The listing prepares numeric and ordinal columns, fills missing values, and creates a two-dimensional t-SNE scatter plot for visual anomaly checking. section 2.4.3
Exploring an extreme target
For an Airbnb regression target, compare its minimum, mean, maximum, and tail percentiles. A maximum far above the 99th percentile indicates extreme right-skewed values. Selecting the central range, plotting its density, and checking the median can support dropping the lowest and highest 5% before training, when those examples are not useful for the modeling goal. section 7.1.6
Exploration continues into the modeling workflow. Use it to check missing data, information leakage, irrelevant or duplicated features, and the suitability of the validation strategy and data pipeline. A simple baseline model can add further evidence about underfitting, overfitting, nonlinearities, interactions, and useful hyperparameter ranges. section 7.4
Exploration also has a search-space meaning during Bayesian hyperparameter optimization: an acquisition function may try parameter combinations not tested before when the surrogate model is uncertain. That exploration is balanced against exploitation of combinations expected to improve performance. section 6.3.4
