A tokenizer configuration is required metadata in a LiteRT-LM `.litertlm` bundle. It specifies the vocabulary, special-token IDs, and normalization rules; the bundle also carries KV-cache layout and sampling defaults. section 11.2.3
Tokenizer configuration
A tokenizer configuration specifies vocabulary, special-token IDs, and normalization rules for converting text into model tokens.
How Quantization and Fast Inference covers it
- How deep it goes
- Names vocabulary, special-token IDs, and normalization rules, without detailing tokenization or normalization procedures.
- Examples
- Places tokenizer configuration beside KV-cache layout and sampling defaults in a `.litertlm` bundle.
- How it connects
- Connects tokenizer metadata with LiteRT-LM bundles, KV-cache layout, and sampling configuration.
Related concepts
- chat templateconfigured by · A chat template is a formatting specification that converts conversational messages into the input representation expected by a language model.
This explainer was generated by Manning's knowledge graph from the text of Quantization and Fast Inference. The section links point to the parts of the book it draws on.
