liveBook · Manning

Flapping

As explained in LLMs in Production

Flapping is repeated oscillation between scaling states, such as repeatedly adding and removing service replicas.

Flapping is repeated oscillation in the number of service replicas: autoscaling starts a replica, then removes it, then starts another. In an LLM inference service, that churn is costly because a new replica must initialize its GPU and load model weights before it can serve requests. section 6.2.2

A replica that never serves

An autoscaler adds a service replica, then terminates it before it serves a single inference. Repeated additions and removals consume startup effort without adding useful capacity. section 6.2.2

Treat flapping as an HPA configuration problem. Check that the scaling metric reflects the service’s actual bottleneck: GPU utilization may be misleading if CPU or memory limits throttle the API layer first. Review the target threshold, minimum and maximum replicas, and scaling policies. Avoid scaling down so aggressively that instances are terminated before finishing assigned tasks. section 6.2.2

How LLMs in Production covers it

How deep it goes
Defines the failure mode and its startup consequences, then names HPA tuning parameters without prescribing specific settings.
Examples
Illustrates a new replica being terminated before it serves any inferences.
What you can do after
Diagnose/debug with it by recognizing repeated replica creation and termination and reviewing HPA configuration.
How it connects
Connects flapping to horizontal pod autoscaling, GPU utilization, CPU and memory bottlenecks, and scaling policies.
Cover of LLMs in Production
liveBookmanning.com
Read LLMs in Production on liveBook Get the book at manning.com

This explainer was generated by Manning's knowledge graph from the text of LLMs in Production. The section links point to the parts of the book it draws on.