When Python reads ordinary text, its strings contain Unicode characters encoded in UTF-8. UTF-8 converts Unicode code points into bytes using a variable number of bytes: ASCII characters use one byte, while many other scripts and emoji use two or more. This makes multilingual text practical, but requires care when translating between bytes and characters. section 2.0 section 2.18
Text files versus binary files
Opening a PDF or JPEG as ordinary text makes Python expect valid UTF-8 and can raise an exception. Open such files with binary mode, using a mode such as 'rb', then read fixed-size byte chunks rather than treating the data as lines. section 5.2
with open(filename, 'rb') as f: #1
while True:
one_chunk = f.read(1000) #2
if not one_chunk:
break
print(f'This chunk contains {len(one_chunk)} bytes')
A fixed-size binary-reading loop stops when read returns no bytes and reports the byte count of each chunk. section 5.2
