Why can three JavaScript characters take nine bytes? In '日本語', each character fits in one UTF-16 code unit, so the string takes six bytes in UTF-16LE. Its UTF-8 encoding takes nine. Character count and encoded byte count answer different questions. section 2.1.2
UTF-8 encodes a Unicode code point in one to four bytes. Code points from U+0000 through U+007F use one byte, preserving their ASCII byte values; higher ranges use two, three, or four. The leading byte indicates the sequence length, while additional bytes begin with 10. Unlike UTF-32's fixed four bytes per code point, UTF-8 spends fewer bytes on lower-valued code points. section 2.1.2
Table 2.2 The UTF-8 Encoding
|
Codepoint Range
|
Byte 1
|
Byte 2
|
Byte 3
|
Byte 4
|
|
|
U+0000
|
U+007F
|
0yyyzzzz
|
|
|
|
|
U+0080
|
U+07FF
|
110xxxyy
|
10yyzzzz
|
|
|
|
U+0800
|
U+FFFF
|
1110wwww
|
10xxxxyy
|
10yyzzzz
|
|
|
U+010000
|
U+10FFFF
|
11110uvv
|
10vvwwww
|
10xxxxyy
|
10yyzzzz
|
The table maps each code-point range to its UTF-8 byte pattern, showing where sequence-length prefixes and continuation bytes appear. section 2.1.2
One symbol, three bytes
In Node.js, Buffer.from('⇇') produces <Buffer e2 87 87>. The symbol is one code point, but its UTF-8 representation occupies three bytes. A Buffer exposes those bytes; a JavaScript string is instead a sequence of 16-bit values. section 2.1.2 section 2.3
Keep encoding separate from other string problems. Unicode normalization addresses visually equivalent strings built from different code-point sequences; choosing UTF-8 does not make those sequences equal. At an input boundary, also check which encoding produced the bytes: assuming UTF-8 for a filename encoded as Shift-JIS can cause a filesystem bug. section 2.1.2 section 2.3












