liveBook · Manning

Unicode

As explained in JavaScript in Depth

A universal character encoding standard that assigns code points to characters from many writing systems.

Two usernames can look identical yet compare unequal. One may contain a single accented code point; the other may contain a base letter followed by a combining mark. Unicode gives symbols code points, but visual similarity does not guarantee identical string contents. section 2.1.2

Unicode supports code points through U+10FFFF. Its UTF encodings turn those values into storage or transmission units: UTF-32 uses four bytes per code point; UTF-16 uses two or four bytes; UTF-8 uses one to four bytes. The compact forms avoid UTF-32’s four-byte cost for ordinary Latin text. section 2.1.2 section 2.3

One code point, two UTF-16 units

U+10437 becomes the UTF-16 surrogate pair 0xD801 0xDC37. Subtract the surrogate-range bases, combine the high part (0x400) and low part (0x37), then add 0x10000 to recover 0x10437. A JavaScript string is a sequence of 16-bit values, so its length need not count Unicode code points; those values need not even form valid UTF-16. section 2.1.2 section 2.3

Table 2.2 The UTF-8 Encoding
Codepoint Range
Byte 1
Byte 2
Byte 3
Byte 4
U+0000
U+007F
0yyyzzzz
 
 
 
U+0080
U+07FF
110xxxyy
10yyzzzz
 
 
U+0800
U+FFFF
1110wwww
10xxxxyy
10yyzzzz
 
U+010000
U+10FFFF
11110uvv
10vvwwww
10xxxxyy
10yyzzzz

The UTF-8 byte-pattern table shows why encoding a code point can take one, two, three, or four bytes. section 2.1.2

Normalize before comparing usernames

A duplicate-name check based on direct string equality can miss a visually identical username written with different combining characters. Normalize both the candidate and stored names before comparing them; normalizing only one side leaves the mismatch possible. section 2.1.2

Table 2.4 Unicode normalization forms
Form
Name
Description
Example Input
Result
Code Units
NFC
Canonical Composition
Decomposes then recombines characters
café (e + ́)
café
U+0063 U+0061 U+0066 U+00E9
NFD
Canonical Decomposition
Separates base characters from combining marks (like accents)
café
café (e + ́)
U+0063 U+0061 U+0066 U+0065 U+0301
NFKC
Compatibility Composition
Like NFC but also normalizes variants
fi (ligature, one symbol)
fi (two symbols)
U+0066 U+0069
NFKD
Compatibility Decomposition
Like NFD but also normalizes variants
½ (one symbol)
1⁄2 (three symbols)
U+0031 U+2044 U+0032

The normalization table distinguishes canonical composition and decomposition from compatibility forms, using accented text, a ligature, and a fraction. section 2.1.2

For regular expressions involving surrogate pairs or Unicode properties, use the `u` flag. Without it, a match such as `/^./` treats an emoji’s surrogate pair as separate units. Keep encoding separate from string semantics: filenames can also fail when a runtime assumes UTF-8 for bytes produced in another encoding. section 2.1.2 section 2.3

How JavaScript in Depth covers it

How deep it goes
Works through UTF encodings and a surrogate-pair calculation, then applies normalization and regex behavior without requiring formal specification knowledge.
Examples
Uses an encoded word, a calculated surrogate pair, visually identical usernames, emoji regex matches, and UTF-8 byte patterns.
What it assumes
Introduces code points and UTF encodings through ASCII and byte examples while assuming familiarity with JavaScript strings and basic code.
What you can do after
Diagnose/debug with it when byte counts, string comparisons, Unicode regex matches, or decoded filenames behave unexpectedly.
How it connects
Connects Unicode to ASCII, UTF-8, UTF-16, UTF-32, normalization, surrogate pairs, and regular expressions.

Also explained in

Compare how 9 books explain Unicode →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of JavaScript in Depth. The section links point to the parts of the book it draws on.