liveBook · Manning

Boot time

As explained in LLMs in Production

Boot time is the elapsed time required for a machine or service instance to start and become ready for operation.

Boot time is the elapsed time a machine or service instance needs to start and become ready. For LLM services, loading a large compiled model into GPU memory can make startup painfully slow, so storage strategy becomes an operational decision. section 6.1.2

Common storage choices involve different trade-offs:

  • Object storage such as cloud buckets offers inexpensive storage, replication, fault tolerance, and high availability, but retrieving large models can be slow because assets are distributed across machines. section 6.1.2
  • Fusing mounts a bucket like an external hard drive, simplifying loading code and avoiding a separate download step, but still retrieves model objects across machines and can remain slow. section 6.1.2
Figure 6.3 Different strategies for storing LLMs and their implications at boot time. Often, we have to balance system reliability, complexity, and application load time.
figure

Compare LLM storage strategies by reliability, complexity, and application load time. section 6.1.2

How LLMs in Production covers it

How deep it goes
Explains object retrieval and fusing trade-offs, but stops before detailing additional alternatives or measurement procedures.
Examples
Uses cloud buckets, GCS or S3-style object storage, and bucket fusing as storage examples.
What it assumes
Assumes familiarity with compiled LLMs, cloud storage, and loading models into GPU memory.
What you can do after
Evaluate storage strategies by balancing reliability, complexity, retrieval performance, and application load time.
How it connects
Connects LLMs with object storage, cloud storage, artifact registries, GPU memory, replication, and high availability.
Cover of LLMs in Production
liveBookmanning.com
Read LLMs in Production on liveBook Get the book at manning.com

This explainer was generated by Manning's knowledge graph from the text of LLMs in Production. The section links point to the parts of the book it draws on.