How deeply does Domain-Specific Small Language Models cover grouped-query attention?

Characterizes the performance role but does not explain the internal attention computation.

Grouped-query attention

As explained in Domain-Specific Small Language Models

An attention variant that speeds inference by sharing attention computation across groups of queries.

Grouped-query attention is an inference-oriented attention variant that shares attention computation across query groups, reducing computational overhead and improving generation speed. section 7.2.4 section 7.4

Operational implications include:

Improves throughput and latency for StarCoder2 during GPU inference. section 7.2.4
Supports deployment comparisons alongside BF16 and quantized inference configurations. section 7.2.4

StarCoder2 inference

StarCoder2 3B uses grouped-query attention with a 16,384-token context window and 4,096-token sliding-window attention; GPU benchmarks report better throughput and latency than CodeGen models. section 7.2.4

How Domain-Specific Small Language Models covers it

Cover of Domain-Specific Small Language Models

liveBook manning.com

Read Domain-Specific Small Language Models on liveBook Get the book at manning.com

Related concepts

multi-head attentionequivalent to · Multi-head attention is an attention mechanism that computes multiple attention operations in parallel using distinct learned projections and combines their outputs.
key-value cacheaddresses · A key-value cache is a data structure that stores previously computed transformer attention keys and values for reuse during subsequent token generation.
keyuses · In attention, a key is a vector representation of an input element that is compared with a query to determine relevance and attention weights for its associated value.
valueuses · In attention, a value is a vector representation of input content that is combined with other values according to attention weights to produce an output.
attentionuses · Attention is a neural-network mechanism that computes input-dependent weights to combine information according to its relevance to a computation.

This explainer was generated by Manning's knowledge graph from the text of Domain-Specific Small Language Models. The section links point to the parts of the book it draws on.