Suppose a web service repeatedly fetches social-network posts and computes a daily sentiment summary. Fetching and analysis take time, but a later request may need a result already computed. Caching stores retrieved data or computed results in faster storage so they can be reused instead of fetched or computed again. section 7.1 section 4.7.1
A basic cache grows out of three decisions:
- Choose reusable results and a key for each one. If the possible inputs are few and requests recur, remembering results can substantially reduce repeated work. section 7.2
- Check the cache before doing the expensive work. A hit returns stored data; a miss requires fetching or computing it. Expect an initial warm-up period before many results are available. section 4.7.1 section 7.2
- Set a capacity and decide what to evict when it is full. Fast storage is limited; LRU, MRU, and LFU are replacement choices, and each can struggle with items requested only once. section 4.7.1 section 7.9
Figure 7.2 With the introduction of cache, before issuing a call to the app server, the web server checks if the results have already been computed, and in that case skips steps 3–6 in this figure. Note that cache can be added also for the single loaders, in case we allow partial results (for instance, if one of the external services is down). In theory, it can also be added to the “data cruncher” instead of the web server, delegating the interface with the data gatherer to the data cruncher, so that the web server directly calls only the data cruncher. Having cache on the web server (physically on the same machine), however, has several advantages.
The diagram places a cache check before the web server requests data and computation. On a hit, it skips those downstream steps; it also shows other possible locations for the cache. section 7.2
What fits depends on what is stored. Sentiment values for at most 100 companies may need little memory, while raw posts for those companies could take gigabytes. At larger scale, caches can run inside an application, in a separate process, or across distributed nodes. Web applications may also cache at CDN, web, application, and database layers. section 7.3 section 7.8
A cache can begin as an associative array, but capacity limits, replacement, and shared access make it a more involved component. Sharing one across threads calls for care: race conditions, inconsistent data, and deadlocks are possible. The useful trade-off is faster repeated access at the cost of storage and cache management. section 7.0 section 7.9
