liveBook · Manning

Character encoding

As explained in JavaScript in Depth

A specification for representing text as bytes or code units according to a defined mapping between characters and numeric values.

A filename can exist on disk yet produce “file not found” when one system decodes its bytes with the wrong character encoding. An encoding maps numeric values to symbols; ASCII maps 97 to “a”, so “hello” becomes 104, 101, 108, 108, 111. section 2.1.1 section 2.1.4

The representation has several layers. ASCII uses one byte per symbol. ECMAScript strings instead contain 16-bit unsigned integers, although those values need not correspond to valid symbols; such uninterpreted data is often called a byte string. section 2.1.1 section 2.3

Keep these representation boundaries distinct:

  • UTF-8 uses one to four bytes per character for transmission, while UTF-16 uses two to four bytes for JavaScript’s internal representation. section 2.3
  • Surrogate pairs combine two 16-bit values for Unicode characters beyond 65,535. section 2.3
  • Shift-JIS and UTF-8 apply incompatible mappings, so decoding Shift-JIS bytes as UTF-8 changes the apparent filename. section 2.1.4

Preserving legacy filenames

A Node.js filesystem call initially returned raw directory-entry bytes, then decoded them as UTF-8 and converted them into JavaScript’s UTF-16 representation. For Shift-JIS filenames, that produced names unlike the stored bytes. Returning names as Buffer objects avoids premature interpretation, allowing the application to read files using the original byte sequences. section 2.1.4

Figure 2.1 The ASCII mapping of numbers to symbols. Source: https://en.m.wikipedia.org/wiki/File:ASCII-Table-wide.svg

Use the ASCII mapping to inspect how numeric values become visible symbols. section 2.1.1

How JavaScript in Depth covers it

How deep it goes
Traces encoding from ASCII bytes through ECMAScript string elements and filesystem failures, but does not detail encoding algorithms.
Examples
Uses “hello” numeric values, ASCII mappings, surrogate pairs, and a Shift-JIS filename failure to expose representation boundaries.
What it assumes
Assumes basic JavaScript and filesystem familiarity while introducing bytes, numeric code values, and encoding distinctions.
What you can do after
Diagnose encoding mismatches and preserve raw filenames with Node.js Buffer-based filesystem APIs.
How it connects
Connects character encoding with ASCII, Unicode, UTF-8, UTF-16, Shift-JIS, surrogate pairs, byte strings, and buffers.

Also explained in

Compare how 5 books explain character encoding →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of JavaScript in Depth. The section links point to the parts of the book it draws on.