liveBook · Manning

Replica

As explained in LLMs in Production

A replica is an additional running instance of a service or workload that provides capacity or redundancy.

A replica is an additional running instance of a service or workload. For an LLM service, load-test results help estimate how many replicas are needed for peak traffic. section 6.3.2

Estimate replica requirements by:

  • Testing the service in staging while steadily increasing requests until failure or autoscaling occurs. section 6.3.2
  • Using latency and throughput at failure to extrapolate peak capacity, while checking TTFT, TPS, CPU, GPU, and memory usage. section 6.3.2

Streaming service load test

A Locust user posts a prompt to /generate, counts streamed tokens, and records latency, time to first token, and tokens per second. Increasing users exposes the service bottleneck and informs replica sizing.

How LLMs in Production covers it

How deep it goes
Explains replicas through capacity extrapolation from failure metrics, but stops short of deployment or orchestration mechanics.
Examples
Measures a streamed /generate service with Locust while increasing users toward a visible bottleneck.
What it assumes
Assumes a staged service and basic request metrics, while defining the measurements needed to assess capacity.
What you can do after
Evaluate replica capacity by load-testing staged services and relating failure-point metrics to peak demand.
How it connects
Connects replicas with load testing, autoscaling, latency, throughput, TTFT, TPS, CPU, GPU, and memory usage.
Cover of LLMs in Production
liveBookmanning.com
Read LLMs in Production on liveBook Get the book at manning.com

Also explained in

Compare how 12 books explain replica →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLMs in Production. The section links point to the parts of the book it draws on.