Grammar is the system of rules that determines how tokens combine into expressions or statements that others can understand. It applies to both natural and programming languages, although programming languages use more rigid and unambiguous rules. section 1.1
Word order is part of grammar. Rearranging a short greeting may preserve its general intent, but rearranging a longer logical statement can destroy the relationships needed for a correct interpretation. section 1.7
Grammar matters because it:
- Constrains which sequences of tokens count as valid expressions. section 1.1
- Supports formal languages such as programming languages, SQL, and regular-expression patterns. section 1.1 section 1.7
- Preserves logical relationships that bag-of-words representations discard. section 1.7
The language of locks
A combination lock accepts one meaningful sequence of symbols. Entering that sequence follows the lock’s grammar and produces a specific interpretation: releasing the hasp. Regular expressions use similarly strict patterns to recognize phrases such as a chatbot greeting and trigger an action. section 1.4.1
>>> from itertools import permutations
>>> [" ".join(combo) for combo in
... permutations("Good morning Rosa!".split(), 3)
... ]
['Good morning Rosa!',
'Good Rosa! morning',
'morning Good Rosa!',
'morning Rosa! Good',
'Rosa! Good morning',
'Rosa! morning Good']
Generate permutations of “Good morning Rosa!” to see how word order changes while some short greetings retain similar intent. section 1.7
