liveBook · Manning

Tokenizer configuration

As explained in Quantization and Fast Inference

A tokenizer configuration specifies vocabulary, special-token IDs, and normalization rules for converting text into model tokens.

A tokenizer configuration is required metadata in a LiteRT-LM `.litertlm` bundle. It specifies the vocabulary, special-token IDs, and normalization rules; the bundle also carries KV-cache layout and sampling defaults. section 11.2.3

How Quantization and Fast Inference covers it

How deep it goes
Names vocabulary, special-token IDs, and normalization rules, without detailing tokenization or normalization procedures.
Examples
Places tokenizer configuration beside KV-cache layout and sampling defaults in a `.litertlm` bundle.
How it connects
Connects tokenizer metadata with LiteRT-LM bundles, KV-cache layout, and sampling configuration.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Quantization and Fast Inference. The section links point to the parts of the book it draws on.