Punctuation can affect how text is split, counted, and loaded. For word counts, `this`, `this.`, and `this,` should often count as the same word. For data files, an unexpected comma, period, or quote can instead disrupt parsing. Decide what punctuation means for the task before removing it. section 21.2.2 section 21.4.3
For a simple Python word-normalization pass:
- Convert text to one case with `lower()`. section 6.10.2
- Create a deletion table once with `str.maketrans('', '', string.punctuation)`, then apply `translate()` to each line. Rebuilding the table inside the line loop repeats unnecessary work. section 6.10.2
- Use `split()` to separate the cleaned text on whitespace. Removing punctuation is only one normalization choice: check whether it leaves words combined or other distinctions unresolved. section 6.10.2 section 21.2.2
Clean a short line
```python import string punct = str.maketrans('', '', string.punctuation) line = 'This, this.' words = line.lower().translate(punct).split() print(words) # ['this', 'this'] ``` For a file, make `punct` before the line loop and write each resulting word followed by `\n` to put one word on each output line. section 6.10.2
moby_text = open("moby_01.txt").read() #1
moby_paragraphs = moby_text.split("\n\n")
moby = moby_paragraphs[1].lower() #2
moby = moby.replace(".", "") #3
moby = moby.replace(",", "") #4
moby_words = moby.split()
print(moby_words)
['there', 'now', 'is', 'your', 'insular', 'city', 'of', 'the',
'manhattoes,', 'belted', 'round', 'by', 'wharves', 'as', 'indian', 'isles'
'by', 'coral', 'reefs--commerce', 'surrounds', 'it', 'with', 'her',
'surf', 'right', 'and', 'left,', 'the', 'streets', 'take', 'you',
'waterward', 'its', 'extreme', 'downtown', 'is', 'the', 'battery,',
'where', 'that', 'noble', 'mole', 'is', 'washed', 'by', 'waves,', 'and',
'cooled', 'by', 'breezes,', 'which', 'a', 'few', 'hours', 'previous',
'were', 'out', 'of', 'sight', 'of', 'land', 'look', 'at', 'the', 'crowds',
'of', 'water-gazers', 'there']
#1 Reads all of the file as a single string
#2 Makes everything lowercase
#3 Removes periods
#4 Removes commas
Compare this two-character `replace()` approach with the broader translation-table approach. Removing only periods and commas leaves other punctuation, such as the dashes in `reefs--commerce`, to consider. section 21.2.2 section 6.10.2
Do not treat punctuation in structured data as disposable. Extra commas or periods can interfere with CSV and numeric fields; unmatched or unescaped quotes can confuse parsing. When cleaning fails, separate the steps and inspect a small troublesome sample. section 21.4.3
