liveBook · Manning

Caching

As explained in Vibe Engineering

Caching is the storage of previously computed or retrieved results so that later equivalent requests can be served with less work or cost.

Caching stores previously computed or retrieved results so later equivalent requests can avoid repeating work. It can reduce latency, but the speed comes with costs: cached data occupies memory, and loading it may delay startup or an initial request. section 7.4 section 7.4.2

For a word-existence service, an eager cache loads the dictionary before requests arrive. Lookups then avoid repeated file access, but startup takes longer and memory holds words that may never be requested. A lazy cache starts empty and loads results as requests arrive; it uses memory more selectively, but the first request for a word pays the loading cost. section 7.4 section 7.4.2

A lazy cache can retain results indefinitely when the dictionary is static and small. For larger datasets, expiration after a period without access can limit stale entries. Request logs and the distribution of intervals between requests can inform an eviction time; without traffic data, treat that choice as an assumption to validate. section 7.4

Listing 7.18 Constructing a word-exists cache
LoadingCache<String, Boolean> wordExistsCache = 
    CacheBuilder.newBuilder()
    .ticker(ticker)
    .expireAfterAccess(DEFAULT_EVICTION_TIME).recordStats() #A
    .build(
         new CacheLoader<String, Boolean>() {
           @Override
           public Boolean load(@Nullable String word) throws Exception {
       if (word == null)
             return false; #B
             }
        return checkIfWordExists(word); #C
           }
          });

This word-existence cache loads results on demand, expires entries after access, and records statistics. section 7.4

An AI-generated alternative cached the entire dictionary in a HashSet without asking about dictionary size or request distribution. It performed well for the particular workload, but that result did not eliminate the risk of higher memory use and delayed startup for larger dictionaries. Specify the expected size and distribution before accepting the design. section 7.4.1 section 7.4.2

Table 7.1 LLM-Based Improvement Results
Scenario
99th percentile
95th percentile
Max
Mean
LLM caching
17
9
36
4
Baseline
82
66
148
36
Human-Based with Caching
65
3
554
5

The load-test comparison shows why mean latency alone is insufficient: eager and lazy caches differ in tail and maximum latency. section 7.4.2

Caching also avoids repeated build work. A local cache lets an agent reuse its own build artifacts; a remote cache lets other agents download shared artifacts instead of recompiling the same dependency. The benefit depends on agents being able to reuse the artifacts already produced. section 10.3.3

How Vibe Engineering covers it

How deep it goes
Compares eager and lazy loading, eviction, memory, startup, and latency; evaluates alternatives with benchmarks and load tests.
Examples
Compares dictionary word-existence caches and illustrates agents sharing a compiled dependency through a remote cache.
What you can do after
Evaluate caching alternatives using workload assumptions, memory costs, startup time, and latency measurements.
How it connects
Connects caching to eviction, latency, memory usage, benchmarking, and build artifacts.

Also explained in

Compare how 49 books explain caching →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Vibe Engineering. The section links point to the parts of the book it draws on.