Data exploration is the initial inspection of a dataset to understand its structure, quality, and distributions before modeling or transforming it. In financial workflows, it helps teams spot data issues and decide what to investigate or prepare next. section 5.3.2 section 9.3.1
A practical exploration can:
- Check shape, data types, missing values, unique counts, and sample rows. section 5.3.2
- Plot target-class proportions and categorical frequencies to reveal imbalance or rare labels. section 5.3.2
- Compare numeric-feature distributions across target classes to notice outliers, separation, or possible binning opportunities. section 5.3.2
- Use findings to guide data cleaning and later feature engineering, while treating plots as starting points rather than complete feature selection. section 5.3.2
Listing 5.5 Summary function
def summary(df): #1
summ = pd.DataFrame(df.dtypes, columns=['data type'])
summ['#missing'] = df.isnull().sum().values
summ['%missing'] = (df.isnull().sum().values / len(df)) * 100
summ['#unique'] = df.nunique().values
desc = df.describe(include='all').transpose() #2
if 'min' in desc.columns:
summ['min'] = desc['min'].values
else:
summ['min'] = np.nan
if 'max' in desc.columns:
summ['max'] = desc['max'].values
else:
summ['max'] = np.nan
if len(df) > 0:
summ['first value'] = df.iloc[0].values
if len(df) > 1:
summ['second value'] = df.iloc[1].values
if len(df) > 2:
summ['third value'] = df.iloc[2].values
return summ #3
print("Initial data shape:", df.shape) #4
summary_df = summary(df) #5
display(summary_df.head(10)) #6
A pandas summary function reports each column’s type, missingness, unique count, available minimum and maximum, and sample values. section 5.3.2
Credit and fraud data
A credit dataset’s target plot shows about 74% nondefaults and 26% defaults, making class imbalance visible before modeling. Initial inspection of transaction data similarly checks columns, types, non-null counts, and the fraud-label distribution before converting the table into a graph. section 5.3.2 section 9.3.1
