Caching stores previously computed or retrieved results so later equivalent requests can avoid repeating work. It can reduce latency, but the speed comes with costs: cached data occupies memory, and loading it may delay startup or an initial request. section 7.4 section 7.4.2
For a word-existence service, an eager cache loads the dictionary before requests arrive. Lookups then avoid repeated file access, but startup takes longer and memory holds words that may never be requested. A lazy cache starts empty and loads results as requests arrive; it uses memory more selectively, but the first request for a word pays the loading cost. section 7.4 section 7.4.2
A lazy cache can retain results indefinitely when the dictionary is static and small. For larger datasets, expiration after a period without access can limit stale entries. Request logs and the distribution of intervals between requests can inform an eviction time; without traffic data, treat that choice as an assumption to validate. section 7.4
Listing 7.18 Constructing a word-exists cache
LoadingCache<String, Boolean> wordExistsCache =
CacheBuilder.newBuilder()
.ticker(ticker)
.expireAfterAccess(DEFAULT_EVICTION_TIME).recordStats() #A
.build(
new CacheLoader<String, Boolean>() {
@Override
public Boolean load(@Nullable String word) throws Exception {
if (word == null)
return false; #B
}
return checkIfWordExists(word); #C
}
});This word-existence cache loads results on demand, expires entries after access, and records statistics. section 7.4
An AI-generated alternative cached the entire dictionary in a HashSet without asking about dictionary size or request distribution. It performed well for the particular workload, but that result did not eliminate the risk of higher memory use and delayed startup for larger dictionaries. Specify the expected size and distribution before accepting the design. section 7.4.1 section 7.4.2
Table 7.1 LLM-Based Improvement Results
|
Scenario
|
99th percentile
|
95th percentile
|
Max
|
Mean
|
|
LLM caching
|
17
|
9
|
36
|
4
|
|
Baseline
|
82
|
66
|
148
|
36
|
|
Human-Based with Caching
|
65
|
3
|
554
|
5
|
The load-test comparison shows why mean latency alone is insufficient: eager and lazy caches differ in tail and maximum latency. section 7.4.2
Caching also avoids repeated build work. A local cache lets an agent reuse its own build artifacts; a remote cache lets other agents download shared artifacts instead of recompiling the same dependency. The benefit depends on agents being able to reuse the artifacts already produced. section 10.3.3
















































