Offloading

As explained in Domain-Specific Small Language Models

A memory-management technique that moves model tensors or state from faster, limited memory to slower or more abundant storage to fit or run a model.

Offloading moves model weights, activations, or attention KV cache from scarce GPU memory to CPU memory or disk. It makes larger-model inference feasible on cheaper hardware, but adds I/O, latency, and lower throughput. section 9.1

Operational tradeoffs include:

• Hardware cost: uses abundant, cheaper CPU RAM and disk instead of requiring many high-memory GPUs. section 9.1
• Performance: accepts slower CPUs, disks, and transfer overhead in exchange for fitting the model. section 9.1
• Optimization: FlexGen selects tensor placement and movement with linear programming, while compressing weights and cache to 4 bits. section 9.1

FlexGen on a single GPU

A 175-billion-parameter model would require roughly 420 GB in FP16, but FlexGen can distribute its state across a 16-GB GPU, CPU RAM, and disk. A policy can configure 50-percent placement across tiers and 4-bit compression for weights and cache. section 9.1

Figure 9.2 Loading a 175 billion–parameter language model with FlexGen across GPU, CPU, and disk

Shows how FlexGen loads a large language model across GPU, CPU, and disk. section 9.1

How Domain-Specific Small Language Models covers it

How deep it goes

Explains tensor placement, data movement, cost, and performance tradeoffs, then extends the technique with optimization and compression.

Examples

Uses FlexGen, a 175-billion-parameter model, a 16-GB GPU, and cross-device placement with 4-bit compression.

What it assumes

Assumes familiarity with model inference, GPU memory, CPU memory, disk storage, and tensor state.

What you can do after

Operate/configure FlexGen policies that place model state across GPU, CPU, and disk.

How it connects

Connects GPU memory, CPU memory, disk storage, tensors, attention cache, quantization, and generative inference.

liveBookmanning.com

Read Domain-Specific Small Language Models on liveBook Get the book at manning.com

Related concepts

• serverless computingaddressed by · Serverless computing is a cloud execution model in which a provider manages the underlying infrastructure and allocates or scales resources for application workloads.
• serviceuses · A service is a software or computing capability that provides defined functionality to users or other components through an interaction boundary.
• workercontrasts with · A worker is a person or computational execution unit assigned to perform a task as part of a larger system or activity.

This explainer was generated by Manning's knowledge graph from the text of Domain-Specific Small Language Models. The section links point to the parts of the book it draws on.