A neural network maps inputs to predictions by chaining data transformations. Each transformation is a layer: a module that takes one or more tensors and produces one or more tensors. Stacking layers lets a model learn successive representations; the number of layers contributing to those representations is its depth. section 3.6.2 section 1.5
Two layers for digit classification
A Keras Sequential model can chain Dense(512, activation="relu") with Dense(10, activation="softmax"). Digit images enter as float32 tensors of shape (60000, 784) during training. Each Dense layer performs tensor operations using its weight tensors; together, the layers turn image data into predictions. section 2.5
Weights are the layer’s learned state, though some layers are stateless. A loss function compares the model’s predictions with targets, and an optimizer uses that feedback to update weights during training. The knowledge retained by a trained model resides in those weights. section 3.6.2 section 1.6
Choose a layer that fits the shape of the data:
- For vectors in 2D tensors shaped (samples, features), use a densely connected layer such as Keras Dense. section 3.6.2
- For sequences in 3D tensors shaped (samples, timesteps, features), consider an LSTM or Conv1D layer. section 3.6.2
- For images in rank-4 tensors, use 2D convolution layers such as Conv2D. section 3.6.2
To make a Keras layer, subclass Layer: define weights in build() and the forward computation in call(). A simple dense layer holds a weight matrix and bias, then computes a matrix product, adds the bias, and optionally applies an activation. Calling the layer can build it from the input shape before running call(). section 3.6.2
Listing 3.33 A simple dense layer from scratch in Keras
import keras class SimpleDense(keras.Layer): #1 def __init__(self, units, activation=None): super().__init__() self.units = units self.activation = activation def build(self, input_shape): #2 batch_dim, input_dim = input_shape self.W = self.add_weight( #3 shape=(input_dim, self.units), initializer="random_normal" #3 ) #3 self.b = self.add_weight(shape=(self.units,), initializer="zeros") def call(self, inputs): #4 y = keras.ops.matmul(inputs, self.W) + self.b if self.activation is not None: y = self.activation(y) return y
The custom dense-layer listing puts weight creation in build() and the input transformation in call(), making the separation between learned state and computation concrete. section 3.6.2
Layers can also be grouped into reusable blocks. ConvNet architectures often repeat groups of layers and build feature hierarchies. More depth is not an unlimited improvement: vanishing gradients constrain how far layers can simply be stacked. For a small regression dataset, a small model can also help limit overfitting; a single-unit linear output leaves predictions unconstrained by an activation. section 9.1 section 4.3.3
