A formal grammar is a finite set of rules specifying which symbol sequences belong to a formal language. Machines can use those rules either to recognize matching sequences or to generate new ones. Regular grammars provide a practical, predictable subset for text matching, finite-state machines, and scripted dialogue. section 1.4.2 section 11.2.2
Regular grammars are useful because they connect concise patterns with deterministic processing:
- A regular expression acts as shorthand for conditional rules that identify character patterns. section 11.2.2
- A finite-state machine processes input step by step and halts after a finite number of transitions when using a true regular grammar. section 1.4.2 section 11.2.2
- Pattern matches can trigger scripted responses or extract information for a chatbot’s knowledge base. section 1.4.2
- Python and POSIX regular expressions extend beyond true regular grammars with features such as lookahead and lookback, weakening termination guarantees. section 1.4.2
Splitting ordinary sentences
A pattern such as re.split(r'[!.?]+[\\s$]+', text) separates sentences at runs of punctuation followed by whitespace or the end of the string. It works for ordinary prose, but can consume the terminating punctuation and requires additional rules for cases such as decimal points. section 11.2.2
Figure 1.4 Kinds of automata
Compare the nested kinds of automata associated with formal languages and finite-state processing. section 1.4.2
