liveBook · Manning

Latency

As explained in Retrieval Augmented Generation, The Foundational Ideas

Inference time is the time required for a trained model or system to produce an output for an input.

A support assistant can retrieve a better answer and still fail its users if it responds too slowly. Latency is the time required to produce an output for an input. In a retrieval-augmented generation (RAG) pipeline, measure both retrieval latency and the response time of the combined retrieval and generation steps; improving one does not guarantee a fast answer. section 2.2 section 4.7.2 section 15.8.3

Latency arises at different points in the pipeline:

  • Similarity search: comparing a query with every document vector becomes impractical at web scale. Approximate nearest neighbor search, implemented by FAISS, makes billion-scale search feasible at millisecond-scale query times. section 2.2
  • Generation: naive RAG adds retrieval before generation, with reported minimum response times of two to three seconds. Adding stages can compound the delay. section 4.7.2 section 15.8.3
  • Hardware movement: Fusion-in-Decoder encodes passages independently, yet its decoder repeatedly reads their concatenated representations. GPU memory bandwidth, rather than arithmetic work, can dominate inference time. section 5.6 section 5.7

Compare speed with the whole deployment cost

In one reported comparison, RAG took 125 milliseconds per query, used 8 GB of memory, and cost about $65 per million queries. Pure retrieval with DPR took 45 milliseconds, while T5-11B took 890 milliseconds. The faster option is not automatically the better answerer; compare latency alongside quality, memory, and cost under the same evaluation conditions. section 4.5.2 section 3.2.1

Set a latency budget before adding quality-improving stages. Reranking adds an inference pass; multi-step retrieval adds calls; compression adds a model pass before generation. Measure whether each gain warrants its delay. In one support-assistant evaluation, retrieval accuracy rose by 8.23 points at 6.3 times the latency—a trade-off to assess, not an automatic improvement. section 15.8.3 section 1.1

How Retrieval Augmented Generation, The Foundational Ideas covers it

How deep it goes
Explains pipeline-level delays and a FiD memory-bandwidth bottleneck, but does not provide a latency-measurement procedure.
Examples
Compares DPR, RAG, REALM, and T5-11B inference times, and reports a support assistant’s accuracy gain against its latency increase.
What you can do after
Evaluate/understand it by comparing response time with quality, memory, and cost before adding pipeline stages.
How it connects
Links latency to approximate nearest neighbor search, RAG, reranking, context compression, GPU memory bandwidth, and answer quality.

Also explained in

Compare how 12 books explain latency →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Retrieval Augmented Generation, The Foundational Ideas. The section links point to the parts of the book it draws on.