A linear layer turns a vector of input features into a vector of outputs using learned weights and biases. In Keras, a `Dense` layer performs this mapping. Dense layers are also called fully connected: each output unit can use every input feature. They are a natural building block for vector data, whose features are not assumed to have a spatial structure. section 2.5.1 section 20.1.6
From 784 features to 32 outputs
Give a dense layer 32 units and an input tensor shaped `(2, 784)`: its output has shape `(2, 32)`. The first dimension still counts samples; 32 is the number of output features. The layer multiplies each input by a weight matrix shaped `(784, 32)`, then adds a 32-element bias. If an activation such as ReLU is specified, it runs after that transformation. section 3.6.2 section 4.1.3
import keras #1 from keras import ops class NaiveDense: def __init__(self, input_size, output_size, activation=None): self.activation = activation self.W = keras.Variable( shape=(input_size, output_size), initializer="uniform" #2 ) #2 self.b = keras.Variable(shape=(output_size,), initializer="zeros") #3 def __call__(self, inputs): #4 x = ops.matmul(inputs, self.W) x = x + self.b if self.activation is not None: x = self.activation(x) return x @property def weights(self): #5 return [self.W, self.b]
This small implementation makes the operations visible: create weights and biases, multiply the inputs by the weights, add the biases, and optionally apply an activation. section 2.5.1
The activation is an important distinction. `Dense(1)` with no activation produces an unconstrained scalar output—a typical final layer for scalar regression. `Dense(1, activation="sigmoid")` instead produces a value between 0 and 1, suitable as a probability for binary classification. A Dense layer with ReLU is therefore not just the underlying linear mapping: it adds a further operation. section 4.3.3 section 4.1.3
Practical choices:
- Stack Dense layers to build a model for vector data; choose both the number of layers and the number of units in each. section 4.1.3 section 20.1.6
- Use more units when a more complex representation is needed, but remember that they increase computation and can encourage unwanted patterns. section 4.1.3
- For images, consider convolutional layers when local patterns matter: Dense layers learn global patterns across their input features rather than local image patches. section 8.1.1
