liveBook · Manning

TextDecoder

As explained in Secrets of the JavaScript Ninja, Third Edition

TextDecoder is a Web API interface that decodes binary data into text according to a specified character encoding while optionally preserving incomplete character sequences across calls.

When network data arrives in chunks, a character’s UTF-8 bytes can be split between chunks. Decoding each chunk independently with `Buffer.toString("utf8")` can produce gibberish. `TextDecoder` avoids this when you call `decode(chunk, { stream: true })`: it retains incomplete character sequences internally for the next chunk. section 11.2.1

Listing 11.4 Decoding a text stream one chunk at a time
import assert from "node:assert";
import { once } from "node:events";
import { createServer } from "node:http";
import { setTimeout } from "node:timers/promises";
 
const words = [
  "One ",
  "word ",
  "at ",
  "a ",
  "time.",
];
 
const server = createServer(async (req, res) => {
  for (const word of words) { #A
    res.write(word); #A
    await setTimeout(100); #A
  } #A
  res.end();
});
 
server.listen(4567);
await once(server, "listening");
 
const res = await fetch("http://localhost:4567"); #B
assert( #C
  res.body !== null, #C
  "The response must have a body", #C
); #C
 
const resStrings: string[] = []; #D
const decoder = new TextDecoder(); #D
for await (const chunk of res.body) { #D
  resStrings.push( #D
    decoder.decode(chunk, { stream: true }), #D
  ); #D
  process.stdout.write(text); #D
} #D
process.stdout.write("\n"); #D

console.assert(
  resStrings.join("") === "One word at a time.",
  "All chunks together form the complete message",
);

server.close();

See TextDecoder used as chunks arrive from an HTTP response body. section 11.2.1

How Secrets of the JavaScript Ninja, Third Edition covers it

How deep it goes
Explains the chunk-boundary problem and the decoder’s state-preserving behavior, without detailing other decoding options.
Examples
Uses a fetched HTTP response, decoded one chunk at a time with `decode(chunk, { stream: true })`.
What you can do after
Apply TextDecoder to preserve characters when decoding streamed UTF-8 chunks.
How it connects
Connects UTF-8 encoding, HTTP response bodies, and ReadableStream chunk iteration.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Secrets of the JavaScript Ninja, Third Edition. The section links point to the parts of the book it draws on.