Tokenization converts natural-language instructions, images, or other input into token sequences and numerical token IDs that an LLM can process, then translates numerical output back into human-readable text or media. Prompts therefore pass through an intermediary numerical representation rather than reaching the model as raw language. section 5.1.1
A practical tokenization pipeline connects user input to model output:
- Human input supplies the raw prompt or other media. section 5.1.1
- Tokenization splits the input and maps pieces to IDs from a predefined vocabulary or tokenizer model. section 5.1.1
- The LLM processes numerical inputs and produces numerical outputs that become text or other human-readable media. section 5.1.1
- Count both prompt and generated tokens when estimating service usage, because many LLM services bill on processed tokens. section 5.1.1
Counting prompt tokens with JavaScript
Install and use `@dqbd/tiktoken` to encode text for a selected model: `const enc = encoding_for_model("gpt-3.5-turbo"); const tokens = enc.encode(text); enc.free(); return tokens.length;`. Count the prompt and system message separately, then add their lengths to estimate the request's input-token total. section 5.1.1
Figure 5.2 The end-to-end process of how human input is transformed, processed by an LLM and converted back into human-readable output
Trace the complete transformation from human input through token IDs and LLM processing to human-readable output. section 5.1.1
