UTF-8 is a Unicode encoding that preserves basic ASCII characters while supporting a very broad range of characters and symbols. This makes it a practical default for reading ordinary text files: ASCII-only files remain compatible, while UTF-8-encoded accented letters, punctuation, and symbols are also covered. section 21.2.1
Encoding matters because a byte outside the ASCII range is ambiguous without knowing the encoding that produced it. Python 3 strings handle Unicode by default, and open() uses strict error handling unless another errors policy is selected. section 21.2.1
Choose an error policy when file contents cannot be decoded successfully:
- strict raises an error as soon as an encoding problem is encountered. section 21.2.1
- ignore skips characters that cause encoding errors. section 21.2.1
- replace substitutes a marker character, while backslashreplace preserves an escape-style representation. section 21.2.1
- surrogateescape retains offending bytes through private Unicode code points and can translate them back when writing. section 21.2.1
Reading invalid UTF-8 bytes
A file containing the bytes 65, 66, 67, 255, 192, and 193 begins as ABC but contains invalid UTF-8 bytes. A normal read raises UnicodeDecodeError. Using open('test.txt', errors='ignore').read() returns 'ABC'; errors='replace' returns replacement markers; errors='surrogateescape' returns '\udcff\udcc0\udcc1'; and errors='backslashreplace' returns '\\xff\\xc0\\xc1'. section 21.2.1
open('test.txt', 'wb').write(bytes([65, 66, 67, 255, 192,193]))
Inspect the raw byte sequence used to demonstrate how non-ASCII values can trigger UTF-8 decoding errors. section 21.2.1
open('test.txt', errors='ignore').read()
'ABC'
open('test.txt', errors='replace').read()
'ABC ' #1
open('test.txt', errors='surrogateescape').read()
'ABC\udcff\udcc0\udcc1'
open('test.txt', errors='backslashreplace').read()
'ABC\\xff\\xc0\\xc1'
#1 Replaced error characters with chr(65533)
Compare the visible results produced by ignore, replace, surrogateescape, and backslashreplace. section 21.2.1
