liveBook · Manning

Caching

As explained in Software Engineering for Data Scientists

A cache is a storage mechanism that temporarily retains computed or retrieved values for faster reuse.

Caching temporarily stores a computed or retrieved result so it can be reused instead of computed again. It trades memory for computational efficiency: repeated work becomes faster, but stored results consume space. This is useful when the same function inputs occur repeatedly, such as users requesting predictions from a deployed machine-learning model. section 6.0 section 6.3 section 6.7

Memoization is caching function-call results. On a first call, the function computes a result and stores it; on a later call with the same inputs, it returns the stored result. This can reduce latency for repeated prediction lookups. section 6.3

Cache repeated model predictions

Python's functools.lru_cache can cache a prediction function by adding the @lru_cache decorator. A function can accept a tuple of artist features, format those features for a random-forest model, and return its prediction. Calling it once performs the prediction; calling it again with the same tuple retrieves the cached result. The illustrated timing falls from about 0.075 seconds to about 0.00012 seconds after the result is available. section 6.3 section 6.3

LRU means Least Recently Used. An LRU cache retains up to a chosen number of items and removes the least recently used result when a new result needs room. functools.lru_cache has no maximum size by default, so setting maxsize can limit how many results remain stored. section 6.3

Use caching when repeated computation is a recognizable bottleneck:

  • Reuse model predictions when an application receives the same input values more than once. section 6.3
  • Avoid repeating identical computations, alongside profiling, vectorization, multiprocessing, and suitable data structures for efficiency work. section 6.7
  • Practice by implementing an LRU or another cache from scratch and testing it with model predictions. section 6.6
Listing 6.20 This code is available as part of cache_ml_model_call.py in the ch6 directory. To create the cache, we just need to add the lru_cache method as a decorator to the function that we want to cache. Next, we write the function we’re caching (in this case, a function that fetches model predictions). Lastly, we time the difference between running the model to get the prediction vs. fetching it from the cache (after it’s available).
from functools import lru_cache #1

@lru_cache #2
def get_model_predictions(model, inputs): #3

    formatted_inputs = pd.DataFrame.\
        from_dict({"followers": inputs[0],
        "num_genres": inputs[1]},
        orient = "index").\
        transpose()

    return model.predict(formatted_inputs)


inputs = (2, 4) #4

start = time.time()
get_model_predictions(forest_model, inputs) #5
end = time.time()
print("Model predictions runtime (w/o) cache: ", end - start)

start = time.time()
get_model_predictions(forest_model, inputs) #6
end = time.time()
print("Model predictions runtime (w/) cache: ", end - start)

Inspect a focused Python example that decorates a model-prediction function with lru_cache and times an uncached call against a cached lookup. section 6.3

How Software Engineering for Data Scientists covers it

How deep it goes
Defines memoization, demonstrates functools.lru_cache, and explains LRU eviction and cache-size limits.
Examples
Uses repeated random-forest artist predictions and a timed Python lru_cache implementation.
What you can do after
Apply it via Python's functools.lru_cache to cache repeated model-prediction calls.
How it connects
Connects caching to scalability, profiling, vectorization, multiprocessing, and computational efficiency.

Also explained in

Compare how 49 books explain caching →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Software Engineering for Data Scientists. The section links point to the parts of the book it draws on.