liveBook · Manning

UTF-16

As explained in JavaScript in Depth

A Unicode transformation format that represents code points using one or two 16-bit code units.

Why can a JavaScript string contain one visible emoji yet behave as though it contains two units? UTF-16 represents a Unicode code point with either one or two 16-bit code units. A code point above U+FFFF needs two units, called a surrogate pair. An operation that reads individual code units can therefore split what a reader sees as one symbol. section 2.1.2 section 2.3

To decode a surrogate pair, subtract 0xD800 from the high surrogate and multiply by 0x400. Add the low surrogate minus 0xDC00, then add 0x10000. For 0xD801 and 0xDC37, the pieces are 0x400 and 0x37, producing code point U+10437. This is why two 16-bit units do not simply mean two Unicode characters. section 2.1.2

A regular expression that splits an emoji

Given `const str = '😀ABC'`, `/^./.exec(str)` matches one surrogate rather than the whole emoji. `/^./u.exec(str)` uses Unicode-aware matching. The `u` flag also matters for Unicode ranges and property escapes. section 2.1.2 section 2.3

Keep the layers separate: UTF-16 is an encoding; a JavaScript string is a sequence of 16-bit unsigned integers that need not be valid UTF-16 text. UTF-16 uses two or four bytes per code point, while UTF-8 uses one to four. Neither encoding alone resolves visually identical strings built from different code points; normalize strings when that distinction matters. section 2.1.2 section 2.3

How JavaScript in Depth covers it

How deep it goes
Works through surrogate-pair decoding and distinguishes code units from code points, without treating every JavaScript string as valid UTF-16.
Examples
Decodes a surrogate pair into U+10437 and tests how the regular-expression `u` flag changes emoji matching.
What it assumes
Builds from character encodings and code points before calculating UTF-16 surrogate pairs.
What you can do after
Diagnose/debug with it when code-unit operations split surrogate pairs or regular expressions mishandle Unicode patterns.
How it connects
Connects UTF-16 to Unicode code points, UTF-8, UTF-32, JavaScript strings, regular expressions, and Unicode normalization.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of JavaScript in Depth. The section links point to the parts of the book it draws on.