Matrix-vector multiplication

As explained in Math and Architectures of Deep Learning

The multiplication of a matrix by a compatible vector to produce a vector; here it is the algebraic form used to express convolution layers.

Matrix-vector multiplication applies a matrix to a compatible column vector and returns a vector. Each output element is the dot product of one matrix row with the input vector. This packages many weighted sums into one operation, making it useful for processing an entire training dataset or neural-network layer at once. section 2.5.3

For a data matrix X and weight vector w, a biased weighted-sum model can be written as y = Xw + b. If X has m rows and n columns, w has n elements and y has m elements; the bias is then added to the resulting vector. section 2.5.3 section 2.7.3

In a neural network, the same operation expresses a complete linear layer:

• Each row of W contains the weights feeding one neuron in the next layer. section 8.3.1
• Multiplying W by the previous layer's activation vector produces all neuron weighted sums together. section 8.3.1
• Adding a bias vector and applying an activation function completes the layer's forward computation. section 8.3.1

A PyTorch computation

With X shaped as 15 × 2 and w shaped as 2 × 1, torch.matmul(X, w) produces a 15 × 1 result. Adding b = 5.0 shifts every output: y = torch.matmul(X, w) + b. section 2.7.3

Figure 8.6 Linear layer outputting layer l from layer (l − 1). The weights belonging to row 1 of the weight matrix (coming from all the input neurons, layer (l − 1), which sum together to form output neuron 1) are shown in bold.

Trace how rows of a weight matrix collect incoming neuron activations to produce the outputs of the next layer. section 8.3.1

How Math and Architectures of Deep Learning covers it

How deep it goes

Defines row-wise dot products, dimensions, bias addition, and layer notation, then demonstrates PyTorch without detailing multiplication internals.

Examples

Uses a biased weighted-sum model, a 15-by-2 PyTorch tensor, and a neural-network layer diagram.

What it assumes

Builds from vectors and dot products while assuming familiarity with matrix dimensions and neural-network layers.

What you can do after

Apply it via PyTorch to compute weighted model outputs and neural-network layer values.

How it connects

Connects vectors, matrices, dot products, bias, activation functions, and linear layers.

liveBookmanning.com

Read Math and Architectures of Deep Learning on liveBook Get the book at manning.com

Related concepts

• embeddinguses · A numerical vector representation of an item or input that encodes features or relationships useful for computational tasks.
• matrix-matrix multiplicationuses · Matrix multiplication combines compatible matrices to form a result whose entries are sums of products between rows of the first matrix and columns of the second.
• arrayuses · An array is an indexed, ordered collection of elements arranged in one or more dimensions, with elements accessed by position.
• vectorproduces · A vector is an ordered collection of components representing an element of a vector space.
• inner productequivalent to · An inner product is a scalar-valued operation on two vectors that combines their corresponding components, summing their products and, for complex vectors, using conjugation.

This explainer was generated by Manning's knowledge graph from the text of Math and Architectures of Deep Learning. The section links point to the parts of the book it draws on.