A tensor is a multidimensional numerical array that represents data and intermediate computations in a neural network. Its shape records dimensions such as tokens and embedding values, while its elements can be floating-point numbers or integer IDs. section 2.1.1 section 12.2.1
Tensor workflows connect representation, numerical type, and model input:
- Convert image data from integer pixel values ranging from 0 to 255 into PyTorch float tensors normalized between −1 and 1. section 3.2.2
- Arrange tensors into batches so training and testing data can enter a model efficiently. section 3.2.2
- Represent tokenized text as a fixed-length integer tensor, preserving separate rows for unconditional and conditional prompts. section 12.2.1
- Store embeddings and positional encodings as floating-point tensors whose dimensions reflect sequence length and embedding width. section 2.1.1
A padded text tensor
Tokenizing “panda with top hat reading a book” produces token IDs, including start and end markers. A 2 × 64 PyTorch tensor places an unconditional sequence in one row and the prompt in another, padding unused positions with the pad-token ID and optionally placing the tensor on CUDA. section 12.2.1
Listing 12.4 Formatting and padding the tokenized input
import numpy, torch
device="cuda" if torch.cuda.is_available() else "cpu"
text_tokens = numpy.ones((2, 64), dtype=numpy.int32) #1
text_tokens[0, :2] = [token_ids[0], token_ids[-1]] #2
text_tokens[1, :len(token_ids)] = token_ids #3
text_tokens = torch.tensor(text_tokens,
dtype=torch.long, device=device) #4
print(text_tokens)
Inspect the implementation that creates a two-row, 64-column tensor, fills it with token IDs and padding, converts it to a long PyTorch tensor, and selects CPU or CUDA. section 12.2.1
