liveBook · Manning

Caching

As explained in Deep Learning with Python, Third Edition

Caching is the practice of storing reusable data or computation results so later access can avoid repeating retrieval or computation.

Caching stores reusable data or computation results so later work can avoid repeating retrieval or computation. In autoregressive generation, past tokens stay fixed, so their attention keys and values can be reused across steps. section 16.2.3

Two practical uses show what caching avoids:

  • Generation: cache past key and value vectors at each Transformer layer instead of recomputing them for the growing sequence. section 16.2.3
  • TPU training: for a small-enough dataset, call dataset.cache() to keep it in virtual-machine memory and read it from Google Cloud Storage only once. section 18.2.3

How Deep Learning with Python, Third Edition covers it

How deep it goes
Explains why past attention keys and values remain reusable and gives dataset.cache() guidance, but does not show a key/value-cache implementation.
Examples
Uses repeated mini-GPT next-token prediction and a small GCS-resident dataset cached with dataset.cache().
What it assumes
Assumes familiarity with Transformer attention for generation and tf.data datasets for the TPU data-pipeline example.
What you can do after
Apply caching with dataset.cache() to keep a small dataset in virtual-machine memory and read it from GCS once.
How it connects
Connects caching to causal attention, Transformers, recurrent-network state, TPU training, and tf.data datasets.
Cover of Deep Learning with Python, Third Edition
liveBookmanning.com
Read Deep Learning with Python, Third Edition on liveBook Get the book at manning.com

Also explained in

Compare how 38 books explain caching →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Deep Learning with Python, Third Edition. The section links point to the parts of the book it draws on.