FlexGen

As explained in Domain-Specific Small Language Models

A high-throughput inference engine for running large language models under limited GPU memory by offloading and compressing model state across devices.

FlexGen is a high-throughput inference engine for running large language models when GPU memory is limited. It pools GPU, CPU, and disk memory and compute, then uses linear programming to place and access model tensors efficiently. section 9.1

FlexGen manages the main inference constraints as follows:

• Offloads weights, activations, and the attention KV cache across GPU, CPU, and disk according to an optimized placement policy. section 9.1
• Compresses weights and the attention cache to 4 bits, increasing throughput with negligible accuracy loss. section 9.1 section 9.5
• Trades latency and single-request speed for throughput, making it better suited to batch processing than latency-sensitive serving. section 9.1 section 9.5

Configuring an OPT model

A command such as `python -m flexgen.flex_opt --model facebook/opt-6.7b --percent 50 50 50 50 50 50` configures distributed storage percentages. A Python policy can additionally enable overlapping execution, weight and cache compression, and 4-bit compression settings.

Figure 9.2 Loading a 175 billion–parameter language model with FlexGen across GPU, CPU, and disk

Shows how FlexGen loads a very large model across GPU, CPU, and disk. section 9.1

How Domain-Specific Small Language Models covers it

How deep it goes

Explains tensor placement, linear-programming optimization, 4-bit compression, and hardware tradeoffs, but stops short of deriving the optimizer.

Examples

Uses a 175-billion-parameter model scenario, OPT command-line runs, Python policies, and reported latency and throughput measurements.

What it assumes

Assumes familiarity with earlier quantization methods and the memory requirements of model inference.

What you can do after

Operate and configure FlexGen policies for model placement, offloading percentages, execution overlap, and 4-bit weight or cache compression.

How it connects

Connects model offloading, quantization, attention cache management, linear programming, and heterogeneous GPU-CPU-disk execution.

liveBookmanning.com

Read Domain-Specific Small Language Models on liveBook Get the book at manning.com

Related concepts

• quantizationuses · Quantization maps numerical values to a finite set of discrete representations, often using reduced precision to lower model storage or computation requirements.
• GPUuses · A graphics processing unit (GPU) is a processor designed for highly parallel computation, especially graphics and data-parallel numerical workloads.
• linear programminguses · Linear programming is an optimization method that maximizes or minimizes a linear objective subject to linear equality or inequality constraints.
• CUDAuses · CUDA is NVIDIA’s parallel computing platform and programming model for performing general-purpose computations on NVIDIA GPUs.
• offloadinguses · Offloading is the transfer of computation, data, or state from one resource or context to another, often to reduce constraints on the original resource.

This explainer was generated by Manning's knowledge graph from the text of Domain-Specific Small Language Models. The section links point to the parts of the book it draws on.