A pipeline organizes the operations that prepare data or train a model. The right workflow depends on how much control you need: Keras can handle routine work through fit(), while shared Layer and Model APIs let you add custom logic without changing frameworks. section 7.1
From image files to training batches
Suppose class-labeled JPEGs are stored in separate directories. Set image_size <- shape(180, 180) and batch_size <- 32, then call image_dataset_from_directory() for the training, validation, and test directories. Each call returns a tf.data.Dataset. Pass train_dataset to fit(), with validation_dataset as validation_data. The utility handles file indexing, reading, shuffling, decoding, resizing, and batching. section 8.2.4
Listing 8.9 Using image_dataset_from_directory() to read images from directories
image_size <- shape(180, 180)
batch_size <- 32
train_dataset <-
image_dataset_from_directory(new_base_dir / "train",
image_size = image_size,
batch_size = batch_size)
validation_dataset <-
image_dataset_from_directory(new_base_dir / "validation",
image_size = image_size,
batch_size = batch_size)
test_dataset <-
image_dataset_from_directory(new_base_dir / "test",
image_size = image_size,
batch_size = batch_size)
The complete R listing creates three image datasets with the same image size and batch size. section 8.2.4
A tf.data.Dataset gives you more control when automatic setup is not enough:
- Iterate over it to inspect elements; use dataset_batch() to group them or dataset_map() to transform them. section 8.2.4
- Use the dataset directly with Keras fit(); preprocessing can run across multiple CPU cores. section 8.2.4
A text-generation pipeline needs different stages. One example reads text shards, tokenizes them, marks document boundaries, forms fixed-length sequences, creates batches, and prefetches data for training. Interleaving shards lets CPU processors read and tokenize separate files simultaneously. The pattern stays the same: arrange the transformations so the model receives ready-to-use batches. section 16.2
Listing 16.3 Preprocessing text input for Transformer pretraining
library(tfdatasets, exclude = "shape")
library(tensorflow, exclude = c("shape", "set_random_seed"))
batch_size <- 128
sequence_length <- 256
suffix <- tf$constant(
tokenizer$token_to_id("<|endoftext|>"),
shape = shape(1L)
)
files <- dir_ls(extract_dir, glob = "*.txt")
read_file <- function(filename) {
text_line_dataset(filename) |>
dataset_map(\(x) tf$strings$regex_replace(x, "\\\\n", "\n")) |> #1
dataset_map(tokenizer, num_parallel_calls = 8) |> #2
dataset_map(\(x) tf$concat(list(x, suffix), -1L)) #3
}
ds <- tensor_slices_dataset(files) |>
dataset_interleave(read_file, cycle_length = 32,
num_parallel_calls = 32) |> #4
dataset_rebatch(sequence_length + 1, drop_remainder = TRUE) |> #5
dataset_map(\(x) list(x@r[NA:-2], x@r[2:NA]), #6
num_parallel_calls = 12) |>
dataset_batch(batch_size) |>
dataset_prefetch(buffer_size = 8)
The preprocessing listing shows how tf.data operations are composed for Transformer pretraining. section 16.2
