liveBook · Manning

Regular expression

As explained in Learn AI Data Engineering

A regular expression is a pattern language for matching, extracting, or replacing text according to specified character and sequence rules.

A regular expression, or regex, is a pattern used to find, extract, split, or replace parts of text. In data pipelines, regex turns semi-structured strings into fields that can be stored, queried, and analyzed. A pattern can match a whole value or capture selected components, such as a log level, date, and time. section 4.3 section 7.1

Regex is especially useful when source formats vary but still follow recognizable rules. Python’s re module can match phone numbers written with parentheses, dashes, dots, or spaces; capturing groups then return the area code and the two remaining number segments separately. Those matches can be loaded directly into a pandas DataFrame or converted into JSON objects. section 4.3.1 section 4.3.3

Extract phone-number components

Use a pattern such as \(?(\d{3})\)?[-.\s]?(\d{3})[-.\s]?(\d{4}) to accept common U.S. phone formats. Each parenthesized group becomes a field: area code, first three digits, and last four digits. This makes a mixed-format text block usable as a structured table. section 4.3.3

import re
import pandas as pd
 
text = """
Call me at (212) 555-7890. My backup number is 646-123-4567. You can also reach me at 917.555.1234.
"""
 
# Regex with capture groups for structured output
matches = re.findall(r'\(?(\d{3})\)?[-.\s]?(\d{3})[-.\s]?(\d{4})', text)
 
# Convert to DataFrame
df = pd.DataFrame(matches, columns=["Area Code", "First 3", "Last 4"])
display(df)

The listing pairs a grouped phone-number pattern with pandas so each regex match becomes one structured row. section 4.3.3

Common pipeline uses include:

  • Extract product name, brand, and color from a product-description string while flattening nested order JSON into a DataFrame. section 2.2.1
  • Normalize phone values to +1-XXX-XXX-XXXX and leave missing or malformed values blank. section 4.4 section 4.5
  • Remove URL protocols and trailing slashes to retain website domains. section 4.4
  • Parse messy addresses and subscription codes into structured components. section 7.5
  • Extract log type, date, and time from operational log entries for downstream analysis. section 7.1

Treat generated patterns as drafts, not as verified pipeline logic. A pattern may silently miss international values, include extensions or country codes, overmatch through greedy quantifiers, or produce plausible-looking false positives. Specify expected formats, test against real samples and edge cases, validate output, and use dedicated libraries such as phonenumbers or dateutil when the pattern becomes difficult to maintain. section 4.3.3

AI coding companions can translate a natural-language requirement into an initial pattern and help refine it, but the workflow remains human-controlled: state the expected inputs, request test cases, inspect matches, and verify results before deployment. section 4.3 section 4.3.3

How Learn AI Data Engineering covers it

How deep it goes
Demonstrates matching, capture groups, substitution, and Python re implementations, while deferring difficult cases to specialized libraries.
Examples
Uses U.S. phone numbers, brewery phone and URL fields, product descriptions, logs, addresses, and subscription codes.
What it assumes
Assumes Python and pandas pipeline work while introducing regex through concrete matching and capture-group examples.
What you can do after
Apply it via Python re and pandas to extract, normalize, and structure text while validating edge cases.
How it connects
Connects Python re, pandas, JSON, data normalization, log files, APIs, and AI coding companions.

Also explained in

Compare how 32 books explain regular expression →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Learn AI Data Engineering. The section links point to the parts of the book it draws on.