A dataset can be small by LLM standards and still be too large for your computer. Streaming retrieves and processes it incrementally instead of loading it all into memory. TinyStories, with about 30 million rows, is one case where this can avoid out-of-memory errors. The trade-off is some speed. section 9.1.2
Listing 9.2 Loading and preparing the data
dataset = load_dataset( #1
"text",
data_files={
"train": ["../../data/TinyStoriesv1andv2-train.txt"],
"val": ["../../data/TinyStoriesv1andv2-valid.txt"],
},
streaming=True,
)
Set streaming=True when loading the training and validation files to retrieve rows without loading the entire dataset at once. section 9.1.2
An LLM response can be streamed too, but for a different immediate reason: users see generated text as it arrives rather than waiting for the complete answer. Because response lengths vary, tokens per second is a useful measure alongside latency. Streaming adds system complexity, but it can make a slow service feel more responsive. section 6.1.5
Match the stream to the job:
- For training data, iterate over rows, clean and tokenize them, then feed batches to the model. Keep train and validation splits separate when shuffling. section 9.1.2
- For a chat application, stream at both ends: a server-side StreamingResponse must be consumed as a readable stream by the client and displayed incrementally. Users can then see where an answer is going and stop or reprompt. section 6.1.5 section 8.1.1
Do not assume every text task needs real-time data infrastructure. Streaming pipelines belong to DataOps, but some captioning and translation tasks can use pseudo–real-time processing without degrading the experience. section 3.4 section 3.4.1
