Chapter 16: Regular expressions

Introduction

The introduction explains why regular expressions deserve a place in a Python book despite being an advanced feature.

• Value of regular expressions: Explain why the chapter covers regular expressions: Python is often used to parse text, and regular expressions are too useful for that work to ignore.
  Why it matters: This explains why the book gives regular expressions a chapter even though they are an advanced feature.

Key terms: regular expressions, text parsing

16.1 What is a regular expression?

Introduces regular expressions through a program that searches text-file lines for “hello” and explains compiling and searching with Python’s re module.

• Regex patterns and matches: Explain what a regular expression is, what it means for one to match text, and why metacharacters allow a pattern to match different strings.
  Why it matters: This is the basic idea behind the more flexible patterns introduced later.
• Compiling a textual regex: Describe how the example creates a compiled regex from the string "hello" and why compilation can be useful when processing large amounts of text.
  Why it matters: It explains why the example compiles its pattern before searching.
• Search results and line counts: Explain how the compiled "hello" regex tests another string and how its search result supports the example’s line count.
  Why it matters: It connects a regex search result to a practical decision about whether a line should be counted.

Key terms: regular expression, regex, match, metacharacters, re, re.compile, compiled regular expression, search, substring, None

16.2 Regular expressions with special characters

This section shows how regex special characters can match alternatives, grouped choices, and single characters from a set or range.

• Alternation and grouped choices: Explain how | and parentheses can match both "hello" and "Hello" in the example.
  Why it matters: Grouping lets you express a small variation without writing out each whole word.
• Character sets and ranges: Describe how square brackets match one character from a set, how ranges work inside them, and how to include a literal hyphen.
  Why it matters: Character sets give you a compact way to match one of several possible characters.

Key terms: special character, vertical-bar special character, parentheses special characters, special characters, ranges of characters

16.3 Regular expressions and raw strings

This section explains why Python string escaping can change a regular expression before the regex engine sees it, and introduces two ways to preserve a literal backslash.

• Two stages of escape interpretation: Explain that Python processes escapes in a normal string before passing it to a regex function, which then interprets the resulting pattern. Contrast why this makes no practical difference for \n but turns an intended literal \ten into a tab followed by en.
  Why it matters: It explains why a pattern can compile successfully yet search for the wrong text.
• Literal backslash pattern options: Describe why a normal Python string needs four typed backslashes for a regex to match one literal backslash, and state how raw strings address the underlying confusion.
  Why it matters: It gives you a way to preserve the literal backslash you meant the regex to match.

Key terms: normal Python strings, special sequences, re.compile, four backslashes, raw strings

16.3.1 Raw strings to the rescue

This section explains how raw strings preserve backslashes in Python string literals and why they are useful for regular expressions.

• Raw string notation and meaning: Describe how to write a raw string and what the leading r changes about Python’s handling of special sequences.
  Why it matters: This explains why backslashes in a raw string can reach a regular expression as written.
• Raw strings for regex patterns: Explain why the section recommends using raw strings when defining regular expressions.
  Why it matters: It gives you a reliable default for writing regex patterns containing backslashes.

Key terms: raw string, special sequences, r, raw strings, regular expressions, regex special sequences, string special sequences

16.4 Extracting matched text from strings

The section builds a regex for names and variably formatted phone numbers, then shows how to extract matched fields and handle missing optional fields.

• Name character patterns: Explain how a character class and + match names made of letters and hyphens, including why the example pattern can also match strings that are not real names.
  Why it matters: It shows how a short pattern can match a whole name without claiming to validate every possible name.
• Optional phone and middle fields: Describe how the example matches phone numbers with or without an area code and names with or without a middle name, including the roles of \d, grouping, ?, and repetition counts.
  Why it matters: Optional groups let one pattern accommodate the variations in the example records.
• Named groups for extraction: Explain how named subpatterns turn a format check into a way to extract individual names and phone numbers from a successful match.
  Why it matters: Named groups make the parsed fields available separately rather than merely confirming that a line fits the pattern.
• Failed and unmatched field values: Distinguish a failed search from a successful match whose optional middle-name field did not match, and explain why the phone field has text in a successful match.
  Why it matters: The distinction prevents treating a missing optional field as a failed record match.

Key terms: [], +, metacharacter, \d, (), ?, {}, area code, named subpatterns, ?P<name>

16.5 Substituting text with regular expressions

This section explains how a regex’s sub method replaces matching text in a string.

• Regex substitution with sub: Explain how sub uses a regex to replace every matching substring in a string and returns a new string, as in changing each "the the" to "the".
  Why it matters: It lets you make the same text correction wherever a regex finds a match.

Key terms: sub, invoking regex, matching substrings

16.5.1 Using a function with sub

Shows how to use a function with `sub` to compute a replacement string from each regular-expression match.

• Match-dependent function replacements: Explain how `sub` can use a function to replace each match based on its text, and trace the section’s example of appending `.0` to matched integers.
  Why it matters: A replacement function lets you transform each match using the text that was actually found.

Key terms: sub, match object, group, named group, int_match_to_float

16.6 Phone number normalizer

This lab specifies how to normalize differently formatted US and Canadian phone numbers and offers optional validation rules.

• Phone Number Normalization Target: Describe the phone-number formats the normalizer must accept and the single format it must produce.
  Why it matters: The target format defines what every accepted input must become.
• Optional Phone Number Validation: State the optional digit-position rules for validating a phone number and how the lab says to handle an invalid number.
  Why it matters: These optional checks distinguish an invalid number from one that merely uses a different format.

Key terms: phone number normalizer, area code, exchange code, station code, country code, normalized phone number, ValueError, invalid phone number

16.6.1 Solving the problem with AI-generated code

Warns that AI-generated regular expressions need careful testing because they may match too much or too little.

• Testing AI-generated regular expressions: Explain why an AI-generated regular expression needs careful testing, including cases that should match and cases that should not.
  Why it matters: Testing both matches and nonmatches helps catch a pattern that accepts the wrong text or misses the right text.

Key terms: regular expression, positive matches, test cases

16.6.2 Solutions and discussion

16.7 Summary