A sequence model ingests tokens in order and learns dependencies across positions. Unlike set or bag-of-words models, it preserves sequence information; RNNs process tokens step by step, while Transformers can inspect multiple positions using positional information. section 14.3 section 14.5
Use sequence models when the arrangement of tokens carries useful information:
- Convert raw text into token IDs before modeling; character, word, and subword tokenization are available. section 14.6
- Pad variable-length sequences to a rectangular batch, or truncate them to a manageable maximum length. section 14.5
- Prevent padding tokens from influencing predictions, and recognize that effective sequence models generally require substantial data. section 14.5 section 14.6
IMDb sentiment classification
For raw IMDb reviews, tokenize each review, represent tokens as integer IDs, and produce fixed-length sequences of 600 positions. Shorter reviews receive [PAD] tokens; longer reviews are truncated. A sequence model can then learn sentiment-related dependencies directly from the ordered review text instead of relying on manually engineered local-order features. section 14.5
Listing 14.19 Padding IMDb reviews to a fixed sequence length
max_length = 600 max_tokens = 30_000 text_vectorization = layers.TextVectorization( max_tokens=max_tokens, split="whitespace", #1 output_mode="int", #2 output_sequence_length=max_length, #3 ) text_vectorization.adapt(train_ds_no_labels) sequence_train_ds = train_ds.map( lambda x, y: (text_vectorization(x), y), num_parallel_calls=8 ) sequence_val_ds = val_ds.map( lambda x, y: (text_vectorization(x), y), num_parallel_calls=8 ) sequence_test_ds = test_ds.map( lambda x, y: (text_vectorization(x), y), num_parallel_calls=8 )
Shows a TensorFlow TextVectorization pipeline that creates 600-token integer sequences for training, validation, and test datasets. section 14.5
