liveBook · Manning

Surrogate pair

As explained in JavaScript in Depth

A two-code-unit sequence in UTF-16 used together to represent a code point outside the basic multilingual plane.

Why can a Unicode symbol occupy two positions in a JavaScript string? JavaScript strings hold 16-bit values, while a Unicode code point can exceed what one UTF-16 code unit represents. UTF-16 uses one code unit for code points through U+FFFF and a two-code-unit surrogate pair for code points from U+10000 through U+10FFFF. section 2.1.2 section 2.3

Reconstructing U+10437

The symbol 𐐷 has code point U+10437 and UTF-16 code units 0xD801 and 0xDC37. Subtract 0xD800 from the first unit and multiply by 0x400, giving 0x400. Subtract 0xDC00 from the second, giving 0x37. Add those results and 0x10000: 0x400 + 0x37 + 0x10000 = 0x10437. The pair represents one code point, not two. section 2.1.2

Do not assume every pair of 16-bit values forms valid Unicode text: a JavaScript string can contain invalid UTF-16. The distinction also matters in regular expressions. Without the `u` flag, a match such as `/^./` treats the two halves of a surrogate pair separately; with `u`, it matches the represented code point. section 2.3 section 2.1.2

Listing 2.3 Surprising behavior with Unicode and Regular Expressions
// Without the 'u' flag, regex treats surrogate pairs incorrectly
const str = '😀ABC';  #A
 
// Default behavior - sees surrogate pairs as two characters
/^./.exec(str); #B
/^.{3}/.exec(str);  #C
 
 
// With 'u' flag - Unicode-aware matching
/^./u.exec(str);  #D
/^.{3}/u.exec(str);  #E
 
 
// Character classes need 'u' flag for Unicode ranges
/[😀-😎]/.test('😀');  #F
/[😀-😎]/u.test('😀');  #G
 
 
// Unicode property escapes require 'u' flag
/\p{Emoji}/u.test("😀");  #H
/\p{Script=Greek}/u.test("α");  #I

Compare the matches with and without `u` to see when a regular expression treats a surrogate pair as one code point. section 2.1.2

How JavaScript in Depth covers it

How deep it goes
Works through decoding a concrete pair into a code point and identifies invalid UTF-16 as a possible string state.
Examples
Decodes U+10437 from 0xD801 and 0xDC37 and contrasts regular-expression matches with and without the `u` flag.
What it assumes
Introduces UTF-16 code units and code-point ranges while using hexadecimal arithmetic to explain the pair.
What you can do after
Diagnose/debug with it when JavaScript regular expressions split a supplementary code point into two matches.
How it connects
Connects surrogate pairs to Unicode code points, UTF-16 code units, JavaScript strings, and Unicode-aware regular expressions.

Also explained in

Compare how 2 books explain surrogate pair →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of JavaScript in Depth. The section links point to the parts of the book it draws on.