liveBook · Manning

Concurrency

As explained in Quantization and Fast Inference

Concurrency is the number of requests or users being served simultaneously by a model instance.

Concurrency measures how many requests or users a model instance serves simultaneously. It affects batching, per-user speed, and the point where memory limits cause requests to queue. section 10.1 section 12.3.2

Read concurrency alongside aggregate throughput and per-user throughput:

  • On an L4, batching at concurrency 8 raises aggregate throughput above 200 tokens per second. CPU serving batches less effectively: above two concurrent users, per-user throughput falls faster than aggregate throughput rises. section 10.1
  • Concurrency also tests memory headroom. A larger KV cache can hold more in-flight sequences before the scheduler queues requests, but a small model may never use enough memory for this advantage to appear. section 12.3.2

A sweep that never reaches saturation

For a 3B model on an L4, an FP16-versus-AWQ concurrency sweep kept queue depth at zero and KV-cache occupancy below about 1.3%. The freed-memory advantage did not change observed saturation because neither configuration approached its memory limit. section 12.3.2

Figure 10.1 Effective cost per million Llama-2-7B Q4_K_M tokens versus sustained demand, on Google Cloud c3-standard-22 (CPU-only, solid) and g2-standard-4 with one NVIDIA L4 (dashed) in us-central1 at on-demand list pricing. Each curve declines hyperbolically until the instance's single-stream throughput ceiling (CPU at 20 tok/s, GPU at 50 tok/s; tick marks at successive replica boundaries), then steps up as additional replicas absorb demand above ceiling. Below saturation the ratio between curves equals the ratio of hourly prices (1.57×). At the GPU's saturation the ratio is roughly 3.9×, widening further at higher demand because the CPU instance saturates at 2.5× lower throughput. The L4 wins on cost per token at every demand point in the plot; the gap is bounded between 1.57× and roughly 5×.

Compare how sustained demand and instance throughput ceilings affect serving cost as replicas are added. section 10.1

How Quantization and Fast Inference covers it

How deep it goes
Trace batching and memory effects through throughput and queue measurements, while noting when a workload never reaches saturation.
Examples
Compare L4 batching at concurrency 8 with an FP16-versus-AWQ sweep that never filled its request queue.
What you can do after
Evaluate serving tradeoffs by comparing aggregate and per-user throughput with memory headroom and queue behavior.
How it connects
Connect concurrency to batching, throughput, KV cache, and request queues.

Also explained in

Compare how 2 books explain concurrency →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Quantization and Fast Inference. The section links point to the parts of the book it draws on.