Padding adds placeholder elements to variable-length sequences so a batch can be represented with one shared length. The added positions typically use a dedicated padding token such as <PAD>. section 9.1.1 section 2.2.7
Padding is useful, but the maximum sequence length creates a practical boundary:
- Adds <PAD> to tokenizer configuration when the tokenizer or model lacks an appropriate padding token. section 9.1.1
- Extends packed embeddings to the longest sequence in a batch, allowing an LSTM to process shorter variable-length inputs. section 2.2.7
- Stops short of explaining masking or padding-side choices; the treatment focuses on making batch shapes compatible. section 9.1.1 section 2.2.7
Listing 9.1 Tokenize and config
import torch
from torch import nn
from torch.nn import functional as F
import numpy as np
from numba import jit
from matplotlib import pyplot as plt
import time
from datetime import timedelta
import pandas as pd
from collections import OrderedDict
from itertools import cycle
from transformers import AutoTokenizer
from sentencepiece import SentencePieceProcessor
from datasets import load_dataset
device = "cuda:0" if torch.cuda.is_available() else "cpu"
device_cap = torch.cuda.get_device_capability()
device_type = "cuda" if "cuda" in device else "cpu"
torch.cuda.set_device(device)
torch.manual_seed(8855)
print(torch.__version__)
print(device, device_cap)
# 2.1.0+cu121
# cuda:0 (8,6)
tokenizer = AutoTokenizer.from_pretrained("./llama3/") #1
tokenizer.add_special_tokens({"pad_token": "<PAD>"})
# tokenizer.pad_token = tokenizer.eos_token #2
vocab = tokenizer.vocab
def encode(example):
return tokenizer.encode(example, return_tensors="pt")
def decode(example):
return tokenizer.batch_decode(
example,
skip_special_tokens=False,
clean_up_tokenization_spaces=True,
)[0]
print(f"Vocab Size: {len(vocab)}")
decode(
encode(
"""hello I am a specifically designed long sentence
to make sure this is working not only adequately,
but good enough for our batch functions"""
)
)
# Vocab Size: 32001
#'<s> hello I am a specifically designed long sentence to make sure this is
working not only adequately, but good enough for our batch functions'
MASTER_CONFIG = {
"vocab_size": len(vocab),
"batch_size": 16,
"context_window": 32,
"d_model": 288,
"hidden_dim": 768,
"epochs": 1000,
"log_interval": 50,
"n_heads": 6,
"n_layers": 6,
}
GLOBAL_KEEP_TRACK = []
Inspect the tokenizer configuration that adds <PAD> before vocabulary and batch-encoding setup. section 9.1.1
