Unicode assigns code points to characters across writing systems and symbols. Python 3 strings use Unicode by default, so ordinary text can include international characters directly. section 6.3.2 section 21.2.1
Use Unicode through names, numbers, encodings, and file-handling options:
- Write characters with Unicode escapes such as \u00E1 or with names such as \N{LATIN SMALL LETTER A WITH ACUTE}. section 6.3.2
- Treat strings as Unicode text and bytes as integer sequences for binary data; convert between them with encode() and decode(). section 6.9
- Prefer UTF-8 for text files when a broad character set is needed; it preserves basic ASCII while supporting many other characters. section 21.2.1
- Handle undecodable input with open() errors options such as strict, ignore, replace, backslashreplace, or surrogateescape. section 21.2.1
unicode_a ='\N{LATIN SMALL LETTER A}' #1
unicode_a
'a' #2
unicode_a_with_acute = '\N{LATIN SMALL LETTER A WITH ACUTE}'
unicode_a_with_acute
'á'
"\u00E1" #3
'á'
#1 Escapes by Unicode name
#2 ASCII characters are Unicode characters.
#3 Escapes by number, using \u
Compare Unicode names and numeric escapes by evaluating both forms as the same visible character. section 6.3.2
Text versus encoded bytes
unicode_a_with_acute = '\N{LATIN SMALL LETTER A WITH ACUTE}' xb = unicode_a_with_acute.encode() xb.decode() The string contains 'á'; encoding produces bytes, and decoding returns Unicode text. Combining xb with a string raises a TypeError because the types differ. section 6.9
open('test.txt', errors='ignore').read()
'ABC'
open('test.txt', errors='replace').read()
'ABC ' #1
open('test.txt', errors='surrogateescape').read()
'ABC\udcff\udcc0\udcc1'
open('test.txt', errors='backslashreplace').read()
'ABC\\xff\\xc0\\xc1'
#1 Replaced error characters with chr(65533)
Inspect how ignore, replace, surrogateescape, and backslashreplace expose or suppress invalid UTF-8 bytes. section 21.2.1
