A regular grammar describes character patterns with predictable, finite-state behavior. Matching a known sequence can trigger a scripted response or extract information, making the approach useful for pattern-based dialog systems and chatbots. section 1.4.2
The underlying machine can be modeled as a finite-state machine or deterministic finite automaton. A true finite-state matcher processes input step by step, returns a match or nonmatch, and eventually halts. Python and POSIX regular expressions add lookahead and lookback features, so they are not strictly true regular grammars and may lose those guarantees. section 1.4.2 section 11.2.2
Regular grammars support several practical pattern-processing tasks:
- Recognize whether an input sequence matches a specified pattern. section 11.2.2
- Generate new symbol sequences from a formal grammar. section 11.2.2
- Find sentence boundaries, extract useful text, or trigger scripted dialog responses. section 1.4.2 section 11.2.2
Sentence segmentation
A pattern such as `re.split(r'[!.?]+[\s$]+', text)` separates ordinary sentences by punctuation followed by whitespace. More careful patterns avoid splitting decimal numbers, while spaCy provides a learned alternative whose output can be compared with the regex result. section 11.2.2
Figure 1.4 Kinds of automata
Use the automata hierarchy to place regular grammars and finite-state machines within related formal-language models. section 1.4.2
