A regular expression, or regex, is a pattern used to find, extract, split, or replace parts of text. In data pipelines, regex turns semi-structured strings into fields that can be stored, queried, and analyzed. A pattern can match a whole value or capture selected components, such as a log level, date, and time. section 4.3 section 7.1
Regex is especially useful when source formats vary but still follow recognizable rules. Python’s re module can match phone numbers written with parentheses, dashes, dots, or spaces; capturing groups then return the area code and the two remaining number segments separately. Those matches can be loaded directly into a pandas DataFrame or converted into JSON objects. section 4.3.1 section 4.3.3
Extract phone-number components
Use a pattern such as \(?(\d{3})\)?[-.\s]?(\d{3})[-.\s]?(\d{4}) to accept common U.S. phone formats. Each parenthesized group becomes a field: area code, first three digits, and last four digits. This makes a mixed-format text block usable as a structured table. section 4.3.3
import re
import pandas as pd
text = """
Call me at (212) 555-7890. My backup number is 646-123-4567. You can also reach me at 917.555.1234.
"""
# Regex with capture groups for structured output
matches = re.findall(r'\(?(\d{3})\)?[-.\s]?(\d{3})[-.\s]?(\d{4})', text)
# Convert to DataFrame
df = pd.DataFrame(matches, columns=["Area Code", "First 3", "Last 4"])
display(df)
The listing pairs a grouped phone-number pattern with pandas so each regex match becomes one structured row. section 4.3.3
Common pipeline uses include:
- Extract product name, brand, and color from a product-description string while flattening nested order JSON into a DataFrame. section 2.2.1
- Normalize phone values to +1-XXX-XXX-XXXX and leave missing or malformed values blank. section 4.4 section 4.5
- Remove URL protocols and trailing slashes to retain website domains. section 4.4
- Parse messy addresses and subscription codes into structured components. section 7.5
- Extract log type, date, and time from operational log entries for downstream analysis. section 7.1
Treat generated patterns as drafts, not as verified pipeline logic. A pattern may silently miss international values, include extensions or country codes, overmatch through greedy quantifiers, or produce plausible-looking false positives. Specify expected formats, test against real samples and edge cases, validate output, and use dedicated libraries such as phonenumbers or dateutil when the pattern becomes difficult to maintain. section 4.3.3
AI coding companions can translate a natural-language requirement into an initial pattern and help refine it, but the workflow remains human-controlled: state the expected inputs, request test cases, inspect matches, and verify results before deployment. section 4.3 section 4.3.3































