liveBook · Manning

TensorFlow I/O

As explained in Distributed Machine Learning Patterns

A TensorFlow extension library that provides additional filesystem and file-format integrations for data input.

TensorFlow I/O extends TensorFlow with filesystem and file-format integrations, allowing datasets to enter tf.Dataset pipelines without first being fully copied into memory. This avoids GraphDef size limits and reduces memory or disk pressure for larger datasets. section 2.3.2

Use TensorFlow I/O when dataset storage or scale makes direct in-memory loading impractical:

  • Load MNIST files directly from a URL through HTTP filesystem support. section 2.3.2
  • Read selected data from PostgreSQL through an IODataset SQL API. section 2.3.2
  • Feed batches incrementally to model training instead of materializing the complete dataset. section 2.3.2

Loading data from external storage

Import tensorflow_io as tfio, then create an IODataset from MNIST URLs or a PostgreSQL query. The resulting dataset exposes TensorFlow element specifications for downstream training. section 2.3.2

Listing 2.3 Loading the MNIST dataset with TensorFlow I/O
> import tensorflow_io as tfio            #1
>
> d_train = tfio.IODataset.from_mnist(    #2
    'http:/ /yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz',
    'http:/ /yann.lecun.com/exdb/mnist/train-labels-idx1-ubyte.gz')

Inspect the concrete MNIST URL-loading call and its two input files. section 2.3.2

How Distributed Machine Learning Patterns covers it

How deep it goes
Explains the memory and GraphDef problem with working URL and database APIs, while deferring detailed database setup.
Examples
Uses MNIST URLs, HTTP filesystem access, PostgreSQL queries, and sequential batch consumption during training.
What you can do after
Apply it via TensorFlow I/O APIs to construct datasets from URLs or PostgreSQL for batch-oriented model training.
How it connects
Connects filesystem support and file formats with TensorFlow datasets, HTTP storage, PostgreSQL, and incremental batching.
Cover of Distributed Machine Learning Patterns
liveBookmanning.com
Read Distributed Machine Learning Patterns on liveBook Get the book at manning.com

This explainer was generated by Manning's knowledge graph from the text of Distributed Machine Learning Patterns. The section links point to the parts of the book it draws on.