Data ingestion is the first operational stage of a machine-learning pipeline: it monitors a source, consumes data in batch or streaming form, and preprocesses it for model training. Its design determines whether downstream training can start reliably as datasets grow, change, or exceed the capacity of one machine. section 2.1 section 1.3.1
Start by characterizing the dataset and its operating context: where it came from, how it was collected, whether its source or size changes, and what infrastructure is needed to handle it. Distributed ingestion needs this assessment because large datasets introduce design constraints beyond a local-machine workflow. section 2.1
Choose the ingestion mode from how the dataset changes:
- Use streaming ingestion for data that grows over time; run long-lived processes that monitor source changes and continuously consume incoming data. section 2.1
- Use nonstreaming ingestion for a fixed dataset; run offline batch jobs on demand to process it. section 2.1
Scale nonstreaming pipelines with complementary patterns. Batch data into memory-sized units when a framework cannot handle the full dataset at once. Shard an extremely large dataset into smaller pieces across worker machines to accelerate training as workers are added. Cache previously fetched data when several training rounds reuse the same dataset. section 2.0 section 2.10
Fashion-MNIST ingestion workflow
A workflow downloads and preprocesses Fashion-MNIST before distributed training, then stores the dataset on a persistent volume. The ingestion script can load the dataset with tfds.load(name='fashion_mnist'); later training steps read the persisted data. The dataset has 60,000 training examples and 10,000 test examples, each a 28 × 28 grayscale image with one of 10 labels. section 9.1 section 9.4.1
Avoid repeating costly ingestion when the underlying data is still fresh. Step memoization can skip the ingestion step, while a time-based cache records the ingested dataset location and last-update timestamp. If data changes, ingest again; otherwise, route training to the cached dataset. Persisted data also matters during recovery: if training fails after in-memory data is lost, ingestion must run again before retrying training. section 7.5.2 section 6.3.2
Table 2.1 Comparison of streaming and nonstreaming data ingestion in machine learning applications (view table figure)
| Long-running processes to monitor the changes in data source |
Compare the changing-data and fixed-data cases to select a long-running monitoring process or an on-demand offline batch job. section 2.1
Figure 7.16 The workflow has been triggered. We check whether the data has been updated within the last two weeks by accessing the cache. If the data is fresh, we can skip the unnecessary data ingestion step and execute the model training step directly.
Follow a cache-based decision flow that skips ingestion when the dataset remains fresh and starts training from the stored dataset. section 7.5.2
