liveBook · Manning

Locust

As explained in LLMs in Production

Locust is an open-source load-testing tool for defining and executing distributed user-behavior simulations.

Locust is an open-source load-testing tool for simulating user behavior against services. It distributes tests across machines, scales toward millions of users, and provides a web interface for observing runs. section 6.3.2

Use Locust when you need to:

  • Increase users gradually with custom load shapes and watch throughput for bottlenecks. section 6.3.2
  • Run tests in Docker, Kubernetes, or distributed environments while monitoring CPU, GPU, and memory usage. section 6.3.2
  • Extend standard metrics when testing streaming services, although custom metrics require additional code. section 6.3.2

Streaming LLM test

Define an HttpUser that posts prompts to a streaming endpoint, counts returned tokens, measures latency and time to first token, calculates tokens per second, and records results.

How LLMs in Production covers it

How deep it goes
Explains installation, test definition, deployment, interfaces, and custom streaming metrics, but not Locust internals.
Examples
Demonstrates a streaming LLM test, a 50-user ramp, and command-line execution with CSV output.
What it assumes
Assumes a running service and basic Python familiarity; starts with pip installation and a simple test file.
What you can do after
Operate Locust to define, distribute, observe, and extend load tests for services.
How it connects
Connects load testing with distributed systems, streaming services, autoscaling, latency, throughput, and resource utilization.
Cover of LLMs in Production
liveBookmanning.com
Read LLMs in Production on liveBook Get the book at manning.com

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLMs in Production. The section links point to the parts of the book it draws on.