A single document may need Russian, Chinese, and English together. Unicode makes this possible by giving characters from different languages and eras a shared representation. section 2.18
Unicode assigns each character a unique number called a code point. UTF-8 translates those numbers into bytes using variable-length encodings: ASCII characters use one byte, while other characters may use two, three, or more. section 2.18
Python exposes Unicode through ordinary string operations, but file handling depends on whether the data is text or binary:
- Python 3 strings contain Unicode characters and behave as sequences: they can be iterated, indexed, searched, and sorted. section 2.0 section 2.18
- Text files are read as UTF-8-formatted Unicode strings; nontext files such as PDFs and JPEGs should be opened in binary mode instead. section 5.2
- Binary mode returns bytes rather than text lines, so fixed-size reads are more appropriate than line iteration. section 5.2
Sorting characters by Unicode order
Use sorted to order the characters in a string, then join the resulting list with an empty string to produce a new string. section 2.18
with open(filename, 'rb') as f: #1
while True:
one_chunk = f.read(1000) #2
if not one_chunk:
break
print(f'This chunk contains {len(one_chunk)} bytes')
Inspect a chunked binary-reading pattern that opens a file with rb, reads fixed-size byte groups, and stops when no bytes remain. section 5.2
