Tensor operations are the small numerical transformations from which neural-network layers are built. A dense layer can be read as a short pipeline: multiply an input tensor by weights, add a bias, then apply ReLU. section 2.3
A dense layer as tensor operations
def dense(inputs, W, b): return tf.nn.relu(tf.matmul(inputs, W) + b) This combines tensor-product multiplication, addition, and rectification. ReLU replaces negative values with zero, using max(x, 0). section 2.3 section 3.3.1
Common operations include:
- Addition combines tensors or adds a bias vector to a matrix result. section 2.3
- Tensor products multiply compatible tensors; element-wise multiplication applies multiplication position by position. section 2.6
- TensorFlow also provides square, square root, matrix multiplication, and concatenation operations. section 3.3.1
- Differentiable operations let gradient computation connect the result back to model weights during training. section 2.6 section 3.3.1
output = relu(matmul(input, W) + b)
Use the compact expression to inspect the three operations inside a dense layer. section 2.3
