Text is written language represented as characters, commonly stored in plain-text files. It may be unstructured, such as tweets, literary works, or news stories, or structured as rows whose fields use delimiters such as commas, tabs, or pipes. section 21.0
Reading text requires attention to encoding. ASCII covers 128 characters but cannot represent many accented letters, symbols, or writing systems. UTF-8 preserves ASCII characters while supporting the much larger Unicode character set, making it a practical default when the file’s encoding is uncertain. section 21.2.1
Process text by making its logical units and cleanup rules explicit:
- Choose whether a unit is a line, paragraph, tweet, news story, or another application-specific segment. section 21.2.2
- Split paragraphs on the file’s separator, such as two newline characters when blank lines mark paragraph boundaries. section 21.2.2
- Normalize before word-level processing by changing case, removing selected punctuation, and splitting on whitespace. section 21.2.2
- Handle undecodable bytes deliberately with policies such as ignore, replace, surrogateescape, or backslashreplace. section 21.2.1
Turning a novel into paragraphs and words
Read the file as one string with `open("moby_01.txt").read()`, then use `moby_text.split("\n\n")` to produce paragraphs. For word processing, lowercase one paragraph, remove periods and commas, and call `split()` to produce a word list. section 21.2.2
open('test.txt', errors='ignore').read()
'ABC'
open('test.txt', errors='replace').read()
'ABC ' #1
open('test.txt', errors='surrogateescape').read()
'ABC\udcff\udcc0\udcc1'
open('test.txt', errors='backslashreplace').read()
'ABC\\xff\\xc0\\xc1'
#1 Replaced error characters with chr(65533)
Compare decoding-error policies on bytes that are invalid under UTF-8: ignoring drops them, replacing marks them, and alternative policies preserve their original values or escapes. section 21.2.1
