A `tf.data.Dataset` is an iterable input pipeline for loading and preprocessing data. It typically yields batches of inputs and labels that can be passed directly to a Keras model’s `fit()` method. The same Dataset abstraction can be used with TensorFlow, JAX, or PyTorch; its behavior with Keras does not depend on the selected backend. section 8.2.4
Listing 8.9 Using image_dataset_from_directory to read images from directories
from keras.utils import image_dataset_from_directory batch_size = 64 image_size = (180, 180) train_dataset = image_dataset_from_directory( new_base_dir / "train", image_size=image_size, batch_size=batch_size ) validation_dataset = image_dataset_from_directory( new_base_dir / "validation", image_size=image_size, batch_size=batch_size ) test_dataset = image_dataset_from_directory( new_base_dir / "test", image_size=image_size, batch_size=batch_size )
For image classification, `image_dataset_from_directory` creates training, validation, and test Datasets from class-organized directories. The resulting pipelines read, shuffle, decode, resize, and batch the images. section 8.2.4
You can also start with an array: `tf.data.Dataset.from_tensor_slices(array)` yields individual elements. Calling `.batch(32)` groups them; calling `.map(function, num_parallel_calls=8)` transforms elements with parallel calls. Inspecting a few yielded shapes is a quick way to check each step. section 8.2.4
Listing 8.13 Applying a transformation to Dataset elements using map()
>>> reshaped_dataset = dataset.map( ... lambda x: tf.reshape(x, (4, 4)), ... num_parallel_calls=8) >>> for i, element in enumerate(reshaped_dataset): ... print(element.shape) ... if i >= 2: ... break (4, 4) (4, 4) (4, 4)
This example maps 16-value elements into 4 × 4 tensors and prints the resulting shapes. section 8.2.4
Dataset can parallelize preprocessing across CPU cores and prefetch data while the model handles the previous batch. For distributed training, supply a Dataset to `fit()` and call `dataset.prefetch(tf.data.AUTOTUNE)` first if you do not know what buffer size to choose. section 8.2.4 section 18.2.2
