How deeply does Domain-Specific Small Language Models cover PagedAttention?

Explains contiguous allocation, fragmentation, preallocation, dynamic blocks, and scheduler behavior without presenting implementation code or formal algorithm details.

PagedAttention

As explained in Domain-Specific Small Language Models

An attention algorithm that stores KV-cache blocks noncontiguously to reduce fragmentation and improve memory flexibility during LLM serving.

PagedAttention is an attention algorithm for LLM serving that stores key–value cache blocks noncontiguously. This avoids wasted reserved space and reduces internal and external memory fragmentation, making memory available for larger batches and higher throughput. section 11.1

PagedAttention addresses several serving constraints:

Replaces contiguous cache allocation with dynamically managed blocks, allowing unused capacity to be reused across requests. section 11.1
Avoids preallocating a request’s maximum sequence length when actual input or output lengths are shorter. section 11.1
Supports centralized scheduling and KV-cache management, similar to virtual memory, while batching autoregressive generation for throughput. section 11.1
Creates a foundation for more efficient memory sharing in parallel sampling and beam search. section 11.1

vLLM serving

vLLM uses PagedAttention to manage KV-cache blocks noncontiguously, combining memory-efficient serving with offline and online deployment and OpenAI-compatible API endpoints. section 11.5

How Domain-Specific Small Language Models covers it

Cover of Domain-Specific Small Language Models

liveBook manning.com

Read Domain-Specific Small Language Models on liveBook Get the book at manning.com

Related concepts

vLLMused by · vLLM is an open-source library and serving engine for high-throughput, memory-efficient inference of large language models.
virtual memoryuses · Virtual memory is a memory-management abstraction that gives processes virtual address spaces mapped to physical memory and, when needed, secondary storage.
key-value cacheuses · A key-value cache is a data structure that stores previously computed transformer attention keys and values for reuse during subsequent token generation.
cachinguses · Caching is the practice of storing reusable data or computation results so later access can avoid repeating retrieval or computation.

This explainer was generated by Manning's knowledge graph from the text of Domain-Specific Small Language Models. The section links point to the parts of the book it draws on.