A token ID is an integer assigned to an item in a vocabulary. For text, the item might be a character or a word. Keep the IDs in their original order: a sequence is not just a collection of tokens, because an RNN uses each new item together with a summary of what came before. section 4.2.2 section 4.1.2
To turn text into an RNN input:
- Choose a vocabulary and consistently map each token to an integer ID. The code that makes this mapping is separate from the embedding layer. section 4.2.2
- Pass the ordered IDs to nn.Embedding. It looks up a vector for each ID, changing a length-T ID sequence into T feature vectors. section 4.2.2
- Feed those vectors to the RNN. For a batch, the vector input has shape (B, T, D): batch size, time steps, and features per step. section 4.2
Repeated IDs, repeated lookup
With input IDs [0, 1, 1, 0, 2] and nn.Embedding(3, 2), the ID tensor has shape [5] and the resulting vectors have shape [5, 2]. Both occurrences of ID 1 look up the same vector. Their positions still differ when the RNN processes the sequence in order. section 4.2.2 section 4.1.2
Sequence lengths need not match. Batching sequences of different lengths calls for padding and packing so the RNN processes only valid input. In sequence-to-sequence translation, source and target token sequences may also have different lengths; the encoder processes the source sequence before the decoder produces the target sequence. section 4.6 section 11.3 section 11.2
with torch.no_grad():
input_sequence = torch.tensor([0, 1, 1, 0, 2], dtype=torch.long)
embd = nn.Embedding(3, 2)
x_seq = embd(input_sequence)
print(input_sequence.shape, x_seq.shape)
print(x_seq)
torch.Size([5]) torch.Size([5, 2])
tensor([[ 0.7626, 0.1343],
[ 1.5189, 0.6567],
[ 1.5189, 0.6567],
[ 0.7626, 0.1343],
[-0.5718, 0.2879]])
Run the small PyTorch lookup to inspect the integer input, its embedding vectors, and the change in shape. section 4.2.2
