liveBook · Manning

Tokenization

As explained in Fight Fraud with Machine Learning

Text tokenization is the conversion of text into discrete units such as words, characters, or subwords for computational processing.

Tokenization converts text into numeric representations a machine-learning model can process. In a simple word-level approach, each distinct word becomes a token and receives a one-hot vector; the vectors for a sentence form a token embedding matrix. section 5.2.1

A four-word fraud vocabulary

For the corpus “let us fight fraud,” sorting the distinct words can assign fight [1, 0, 0, 0], fraud [0, 1, 0, 0], let [0, 0, 1, 0], and us [0, 0, 0, 1]. Stacking them produces a 4×4 matrix: one dimension is vocabulary size and the other is embedding-vector size. section 5.2.1

In a transformer translation workflow, token vectors enter the encoder after positional encoding is added. Position helps preserve word order, such as “let” preceding “us,” before attention and dense layers process the sequence. section 5.2.1

Figure 5.8 Transformer model in action performing English to French language translation.

Trace how an English sentence is converted for an encoder and used to produce French output. section 5.2.1

How Fight Fraud with Machine Learning covers it

How deep it goes
Defines word-level one-hot tokenization and works through vocabulary and embedding-matrix dimensions, without covering subword methods.
Examples
Uses the corpus “let us fight fraud” to construct four one-hot word vectors and a 4×4 matrix.
What it assumes
Teaches numeric word representation from a simple corpus without assuming prior tokenization knowledge.
What you can do after
Evaluate/understand how a sentence becomes numeric encoder input before transformer processing.
How it connects
Connects tokenization with one-hot encoding, vocabulary size, embedding matrices, positional encoding, and transformer encoders.

Also explained in

Compare how 36 books explain tokenization →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Fight Fraud with Machine Learning. The section links point to the parts of the book it draws on.