liveBook · Manning

UTF-8

As explained in JavaScript in Depth

A Unicode transformation format that encodes code points into a variable-length sequence of one to four bytes.

Why can three JavaScript characters take nine bytes? In '日本語', each character fits in one UTF-16 code unit, so the string takes six bytes in UTF-16LE. Its UTF-8 encoding takes nine. Character count and encoded byte count answer different questions. section 2.1.2

UTF-8 encodes a Unicode code point in one to four bytes. Code points from U+0000 through U+007F use one byte, preserving their ASCII byte values; higher ranges use two, three, or four. The leading byte indicates the sequence length, while additional bytes begin with 10. Unlike UTF-32's fixed four bytes per code point, UTF-8 spends fewer bytes on lower-valued code points. section 2.1.2

Table 2.2 The UTF-8 Encoding
Codepoint Range
Byte 1
Byte 2
Byte 3
Byte 4
U+0000
U+007F
0yyyzzzz
 
 
 
U+0080
U+07FF
110xxxyy
10yyzzzz
 
 
U+0800
U+FFFF
1110wwww
10xxxxyy
10yyzzzz
 
U+010000
U+10FFFF
11110uvv
10vvwwww
10xxxxyy
10yyzzzz

The table maps each code-point range to its UTF-8 byte pattern, showing where sequence-length prefixes and continuation bytes appear. section 2.1.2

One symbol, three bytes

In Node.js, Buffer.from('⇇') produces <Buffer e2 87 87>. The symbol is one code point, but its UTF-8 representation occupies three bytes. A Buffer exposes those bytes; a JavaScript string is instead a sequence of 16-bit values. section 2.1.2 section 2.3

Keep encoding separate from other string problems. Unicode normalization addresses visually equivalent strings built from different code-point sequences; choosing UTF-8 does not make those sequences equal. At an input boundary, also check which encoding produced the bytes: assuming UTF-8 for a filename encoded as Shift-JIS can cause a filesystem bug. section 2.1.2 section 2.3

How JavaScript in Depth covers it

How deep it goes
Works through code-point ranges, byte patterns, and Buffer output; stops short of a full decoder implementation.
Examples
Compares UTF-16LE and UTF-8 byte lengths for Japanese text and encodes ⇇ with Node.js Buffer.
What it assumes
Builds from ASCII and Unicode code points, then uses JavaScript strings and Node.js Buffer for concrete comparisons.
What you can do after
Diagnose/debug with it: distinguish a JavaScript string's character count from its encoded bytes and investigate encoding mismatches.
How it connects
Contrasts UTF-8 with ASCII, UTF-16, and UTF-32, and distinguishes encoding from Unicode normalization.

Also explained in

Compare how 13 books explain UTF-8 →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of JavaScript in Depth. The section links point to the parts of the book it draws on.