To do math on text, first give each token a numerical representation. One-hot encoding assigns every vocabulary item a column and puts a 1 in that column, with 0s everywhere else. A document becomes a sequence of these vectors: one row per token, in reading order. section 2.5 section 2.5.1
Read a token matrix
An 18-token document with 15 unique words produces an 18-by-15 matrix. A 1 under “There’s” in the first row identifies the first token; a 1 under “adequate” in the fifth row identifies the fifth. The row order preserves the exact token sequence. section 2.5.1
>>> import pandas as pd
>>> onehot_vectors = np.zeros(
... (len(tokens), vocab_size), int) #1
>>> for i, tok in enumerate(tokens):
... if tok not in vocab:
... continue
... onehot_vectors[i, vocab.index(tok)] = 1 #2
>>> df_onehot = pd.DataFrame(onehot_vectors, columns=vocab)
>>> df_onehot.shape
(18, 15)
>>> df_onehot.iloc[:,:8].replace(0, '') #3
, . Survival There's adequate as fittest maybe
0 1
1
2
3
4 1
5
6
7
8 1
9 1
10 1
11
12
13
14 1
15 1
16 1
17 1
Inspect the NumPy matrix construction and its pandas display. Empty-looking cells are zeros hidden for readability; the underlying vectors remain numerical. section 2.5.1
Keep the representation consistent throughout a pipeline:
- Build a vocabulary and keep each token’s column assignment fixed. The illustrated loop skips tokens absent from that vocabulary. section 2.5.1 section 8.2.1
- Keep category-label ordering fixed too when a model predicts labels; otherwise its numerical outputs can be mapped to the wrong names. section 8.2.1
- Keep display formatting out of training data. Replacing zeros with strings may make a table readable, but the model needs numerical arrays. section 2.5.1
One-hot sequences are direct and lossless after tokenization, but sparse: each row has just one nonzero entry. Encoding characters likewise retains their order while expanding dimensionality; it does not, by itself, make documents easy to group by meaning. Denser representations such as topic vectors and word embeddings address that different goal. section 2.5.1 section 1.6
A skip-gram pipeline can define an encoder that places a 1 at a token’s vocabulary ID when preparing word pairs. The illustrated Word2Vec model instead passes token IDs to an embedding layer, so defining a one-hot vector does not mean every training batch must store one. section 6.3.3
def one_hot_encode(input_id, size):
vec = torch.zeros(size).float()
vec[input_id] = 1.0
return vec
This small PyTorch function shows the encoding operation directly: allocate a zero vector of vocabulary size, then activate the position specified by the token ID. section 6.3.3
