Matrix-vector multiplication
Matrix-vector multiplication is the linear algebra operation of multiplying a matrix by a compatible vector to produce a vector of row-wise dot products.
CUDA for LLMs
GEMV is a general matrix-vector multiplication operation with optional scaling of the product and output vector.
- How it is used
- Uses GEMV for single-token transformer decoding, including attention against cached keys and values.
- How deep it goes
- Traces a row-wise CUDA kernel and outlines coalesced access, reductions, and vectorized loads, alongside memory-bandwidth considerations.
- Examples
- Compares single-query and 64-query attention scores and walks through a batched thread-per-row CUDA kernel.
- What you can do after
- Build a basic CUDA GEMV kernel and evaluate optimization against cuBLAS using correctness checks and profiling.
Read how CUDA for LLMs explains matrix-vector multiplication →
Math and Architectures of Deep Learning
The multiplication of a matrix by a compatible vector to produce a vector; here it is the algebraic form used to express convolution layers.
- How it is used
- Packages many weighted sums for training datasets and neural-network linear layers.
- How deep it goes
- Defines row-wise dot products, dimensions, bias addition, and layer notation, then demonstrates PyTorch without detailing multiplication internals.
- Examples
- Uses a biased weighted-sum model, a 15-by-2 PyTorch tensor, and a neural-network layer diagram.
- What you can do after
- Apply it via PyTorch to compute weighted model outputs and neural-network layer values.
Read how Math and Architectures of Deep Learning explains matrix-vector multiplication →
Also covered in
Related concepts
- inner productuses · An inner product is a scalar-valued operation on two vectors that generalizes the dot product, which for real coordinate vectors is the sum of products of corresponding components.
- memory boundkind of · A workload is memory-bound when its performance is limited primarily by memory access or data-transfer capacity rather than computational throughput.
- decode phaseused by · The decode phase is the autoregressive inference stage in which a model generates output tokens sequentially from previously processed context.
- key-value cacheproduced by · A key-value cache stores attention keys and values computed for earlier tokens so they can be reused during subsequent autoregressive generation.
- matrix-matrix multiplicationcontrasts with · Matrix multiplication is an operation on compatible matrices in which each entry of the product is the dot product of a row of the first matrix and a column of the second.
Questions about matrix-vector multiplication
What is matrix-vector multiplication?
Matrix-vector multiplication is the linear algebra operation of multiplying a matrix by a compatible vector to produce a vector of row-wise dot products.
Which Manning books explain matrix-vector multiplication?
CUDA for LLMs and Math and Architectures of Deep Learning.
What is the difference between matrix-vector multiplication and matrix-matrix multiplication?
Matrix-vector multiplication is the linear algebra operation of multiplying a matrix by a compatible vector to produce a vector of row-wise dot products. Matrix multiplication is an operation on compatible matrices in which each entry of the product is the dot product of a row of the first matrix and a column of the second.
What is matrix-vector multiplication a kind of?
Matrix-vector multiplication is a kind of memory bound. A workload is memory-bound when its performance is limited primarily by memory access or data-transfer capacity rather than computational throughput.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.

