LRU

As explained in Designing Deep Learning Systems: A guide for software engineers

The least recently used algorithm evicts the item that has gone unused for the longest time when a cache reaches its capacity.

LRU evicts the model that has gone unused for the longest time when a model cache reaches memory or GPU capacity. In multitenant prediction services, it supports model swapping so new requests can load required models without exhausting resources. section 6.3.2

Use LRU as part of a model-cache resource strategy:

• Evict cached models when the process approaches memory limits or available GPU resources are exhausted. section 6.3.2
• Combine eviction with model partitioning across instances to reduce cache misses and make model swapping less disruptive. section 6.3.2
• Keep caches separated by model type when input/output formats and preprocessing logic differ substantially. section 6.3.2

Figure 6.6 A prediction service with model caching for multitenant applications

Use the architecture diagram to inspect how model caching and prediction requests fit within a multitenant prediction service. section 6.3.2

How Designing Deep Learning Systems: A guide for software engineers covers it

How deep it goes

Defines the eviction role and pairs it with model partitioning, but defers detailed cache implementation to a prediction-service example.

Examples

Uses a multitenant prediction service and an intent-classification service to illustrate model caching and swapping.

What it assumes

Assumes familiarity with model caches, prediction requests, and memory or GPU capacity constraints.

What you can do after

Operate model caches that evict inactive models and free capacity for incoming prediction requests.

How it connects

Connects model caching with model swapping, cache misses, model partitioning, multitenancy, and prediction services.

liveBookmanning.com

Read Designing Deep Learning Systems: A guide for software engineers on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Advanced Algorithms and Data StructuresAn LRU cache is a cache that evicts the entry that has been unused for the longest time when it reaches capacity.

Compare how 2 books explain LRU →

Related concepts

• cachingconfigures · Caching is the practice of storing reusable data or computation results so later access can avoid repeating retrieval or computation.
• least frequently usedcontrasts with · Least frequently used is a cache-eviction policy that removes the entry with the lowest access frequency when capacity is exhausted.
• hash mapuses · A hash map is a key-value data structure that uses a hash function to locate entries and support efficient lookup, insertion, and removal by key.
• Redisuses · Redis is an open-source, in-memory data-structure store that provides network-accessible data operations and supports persistence.
• cache replacement policykind of · A cache replacement policy is a rule or algorithm for selecting which cached item to remove when capacity is insufficient or cached data must be discarded.

This explainer was generated by Manning's knowledge graph from the text of Designing Deep Learning Systems: A guide for software engineers. The section links point to the parts of the book it draws on.