Caching keeps previously computed results under keys so repeated requests avoid expensive recomputation. In inference systems, it can lower latency, reduce inference-server load, and save money even when many requests are unique. section 15.2.2
Use caching with these practical decisions:
- Choose a key-value cache and a low time to live; the exact lifetime should follow request and data analysis. section 15.5.2
- Target popular requests first: a long tail of unique inputs limits hit rates, but even modestly avoided computation can reduce cost. section 15.2.2
- Consider fuzzy matching when exact keys miss too often, using regular-expression or semantic similarity to share results across similar requests. section 15.2.2
- Use caching for runtime inference, while batch inference updates entries when the relevant key changes. section 15.5.2
PhotoStock inference serving
For PhotoStock Inc., popular queries are served from AWS ElastiCache through Redis or Memcached, reducing load on inference servers. The design starts with a simple key-value cache and low TTL, while monitoring and autoscaling handle peak real-time traffic. section 15.5.2
Listing 15.1 Example of the simplest possible in-memory cache
class InMemoryCache:
def __init__(self):
self.cache = {}
def get(self, key: str, or_else: Callable):
v = self.cache.get(key)
if v is None:
v = or_else()
self.cache[key] = v
return v
A minimal in-memory implementation shows the basic cache-aside flow: look up a key, compute on a miss, store the result, and return it. section 15.2.2
