liveBook · Manning

UTF-8

As explained in Secrets of the JavaScript Ninja, Third Edition

A variable-width Unicode encoding that represents each Unicode scalar value using one to four bytes.

UTF-8 represents each Unicode scalar value with one to four bytes. Because a character’s bytes can be split across network chunks, decoding each chunk independently with Buffer.toString("utf8") can produce gibberish. section 11.2.1

Use TextDecoder.decode(chunk, { stream: true }) to decode incrementally. TextDecoder keeps any partial character internally for the next chunk. section 11.2.1

for await (const chunk of res.body) {
  resStrings.push(
    decoder.decode(chunk, { stream: true }),
  );
  console.log(resStrings.join(""));
}

This loop decodes each response-body chunk in streaming mode before joining the resulting strings. section 11.2.1

How Secrets of the JavaScript Ninja, Third Edition covers it

How deep it goes
Explains multibyte chunk boundaries and decoder state, stopping short of byte-level encoding details.
Examples
Uses a streamed HTTP response whose body arrives in chunks and is decoded incrementally.
What you can do after
Apply TextDecoder in streaming mode to decode response chunks containing split multibyte characters.
How it connects
Connects UTF-8 to TextDecoder, Buffer.toString, HTTP response bodies, and readable streams.

Also explained in

Compare how 13 books explain UTF-8 →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Secrets of the JavaScript Ninja, Third Edition. The section links point to the parts of the book it draws on.