Categorical encoding turns a label into numbers a model can use. In one-hot encoding, each category gets its own position in a vector: that position is 1 and every other position is 0. The assignment of categories to positions is arbitrary. A wine rating assigned position 5, for example, becomes [0, 0, 0, 0, 0, 1, 0, 0, 0, 0]. section 4.3.4
Choose this representation when values are labels rather than measurements. Integer codes can imply an order and equal distances between values; grape varieties have neither. A rating may be different: if its order and intervals are meaningful, keeping it numeric may make sense. An ordinal weather level can likewise be treated as a category or as a numeric value, depending on what the model should learn from it. section 4.3.4 section 4.4.3
Add weather categories to a tensor
For four weather levels numbered 1 through 4, make a zero-filled tensor with four columns and one row per observation. Use each level, minus 1, as the column index at which to place a 1. A level of 1 becomes [1, 0, 0, 0]. Concatenate those four columns with the original data along the column dimension; the row counts must match. section 4.4.3
# In[7]:
weather_onehot.scatter_(
dim=1,
index=first_day[:,9].unsqueeze(1).long() - 1, #1
value=1.0)
# Out[7]:
tensor([[1., 0., 0., 0.],
[1., 0., 0., 0.],
...,
[0., 1., 0., 0.],
[0., 1., 0., 0.]])
The PyTorch listing shows the weather conversion: unsqueeze adds the index dimension, subtracting 1 changes levels 1–4 into column indexes 0–3, and scatter_ places the 1s in the zero-filled tensor. section 4.4.3
Text uses the same idea. At character level, make one row per character and one column per character in the chosen encoding. For English text, a 128-column ASCII example uses ord to select each column; characters outside that range are mapped to index 0. At word level, clean the text, build a dictionary from words to indexes, then set one entry in each word’s row. The example sentence has 11 words and produces a tensor with 7,261 columns. section 4.5.2 section 4.5.3
# In[8]:
word_t = torch.zeros(len(words_in_line), len(word2index_dict))
for i, word in enumerate(words_in_line):
word_index = word2index_dict[word]
word_t[i][word_index] = 1
print('{:2} {:4} {}'.format(i, word_index, word))
print(word_t.shape)
# Out[8]:
0 3394 impossible
1 4305 mr
2 813 bennet
3 3394 impossible
4 7078 when
5 3315 i
6 415 am
7 4436 not
8 239 acquainted
9 7148 with
10 3215 him
torch.Size([11, 7261])
The listing shows a word-level implementation: a zero-filled row for each word, a column for each vocabulary entry, and one assignment per word using its dictionary index. section 4.5.3
Width is the main drawback. A large vocabulary needs a column for every word, and a new word requires another column. Embeddings offer a more compact alternative. One-hot vectors also describe ideal classifier targets: airplane and bird might be [1, 0] and [0, 1]. A real classifier instead produces values between those ideals that can be interpreted as class probabilities; softmax makes its outputs sum to 1. section 4.5.4 section 7.2.3 section 7.5
