A support assistant can retrieve a better answer and still fail its users if it responds too slowly. Latency is the time required to produce an output for an input. In a retrieval-augmented generation (RAG) pipeline, measure both retrieval latency and the response time of the combined retrieval and generation steps; improving one does not guarantee a fast answer. section 2.2 section 4.7.2 section 15.8.3
Latency arises at different points in the pipeline:
- Similarity search: comparing a query with every document vector becomes impractical at web scale. Approximate nearest neighbor search, implemented by FAISS, makes billion-scale search feasible at millisecond-scale query times. section 2.2
- Generation: naive RAG adds retrieval before generation, with reported minimum response times of two to three seconds. Adding stages can compound the delay. section 4.7.2 section 15.8.3
- Hardware movement: Fusion-in-Decoder encodes passages independently, yet its decoder repeatedly reads their concatenated representations. GPU memory bandwidth, rather than arithmetic work, can dominate inference time. section 5.6 section 5.7
Compare speed with the whole deployment cost
In one reported comparison, RAG took 125 milliseconds per query, used 8 GB of memory, and cost about $65 per million queries. Pure retrieval with DPR took 45 milliseconds, while T5-11B took 890 milliseconds. The faster option is not automatically the better answerer; compare latency alongside quality, memory, and cost under the same evaluation conditions. section 4.5.2 section 3.2.1
Set a latency budget before adding quality-improving stages. Reranking adds an inference pass; multi-step retrieval adds calls; compression adds a model pass before generation. Measure whether each gain warrants its delay. In one support-assistant evaluation, retrieval accuracy rose by 8.23 points at 6.3 times the latency—a trade-off to assess, not an automatic improvement. section 15.8.3 section 1.1











