Tokenization converts text into discrete computational units, but the representation is an engineering choice rather than a harmless preprocessing detail. A dictionary-based representation fixes the vector size, which determines the number of input neurons; vocabulary limits and normalization therefore affect both data quality and model architecture. section 4.6.2 section 4.6.6
A practical tokenization pipeline makes its transformations explicit:
- Build a vocabulary and map words to representation positions or unique integer IDs. section 4.6.2 section 4.6.6
- Normalize text by cleaning typos, converting to lowercase, and applying other library-supported improvements instead of relying on ad hoc splitting. section 4.6.2
- Standardize sequence length with padding for short descriptions and truncation for long ones. section 4.6.6
- Check vocabulary size and sequence shape before connecting the result to an embedding or downstream classifier. section 4.6.2 section 4.6.6
Medicine descriptions
A small dictionary can encode two medicine descriptions as binary vectors: each vocabulary word receives an index, and each description sets present-word positions to 1 while leaving absent words at 0. A larger tokenizer instead maps words such as "Aspirin" to integer IDs, retains the 20,000 most frequent words, and pads or truncates every description to 180 integers. section 4.6.2 section 4.6.6
Figure 4.9 The tokenization process

Use the tokenization diagram to inspect the progression from medicine descriptions and vocabulary entries to fixed numeric vectors. section 4.6.2



































