A tokenizer is the bridge between readable text and an ML model’s numeric input. It splits a prompt into tokens, exposes their numeric IDs for inference, and decodes generated IDs back into text. In story generation, it preprocesses the prompt and postprocesses model output so the result can be returned as a coherent string. section 5.4.1 section 5.4.2
Load a tokenizer from a tokenizer.json fixture before running inference. Encode the prompt with special-token handling enabled, copy its input IDs into a mutable sequence, and pass that sequence to the model as input. Each sampled next-token ID extends the sequence. section 5.4.2 section 5.4.2
let tokenizer = Tokenizer::from_file("fixture/tokenizer.json").unwrap();
This initialization loads the tokenizer artifact used by the component before it encodes the story prompt. section 5.4.2
During generation, tokenization supports the control loop:
- Look up the <EOS> token ID to recognize a model-defined stopping marker; the fallback ID of 0 avoids an immediate lookup failure but may not be a valid stopping point. section 5.4.2
- Decode the growing ID sequence after sampling each token, making the generated text available for output and for a double-newline stopping check. section 5.4.2
- Return the final decoded sequence after trimming surrounding whitespace. section 5.4.2
Generating from “Once upon a time”
Encode “Once upon a time” into input IDs, run inference, sample a next token from the output probabilities, append its ID, and decode the sequence. Repeat for up to 100 iterations, stopping at <EOS> or when the decoded text ends with two newlines. section 5.4.2
