liveBook · Manning

Weight initialization

As explained in Build AI Drug Discovery Pipelines

Weight initialization is the procedure for assigning initial numerical values to a model's learnable weights before training.

Weight initialization assigns starting values to a model’s learnable weights before training. The scale matters: weights that are too small can lead to vanishing gradients, leaving early layers with tiny learning signals. Weights that are too large can produce exploding gradients and unstable training. section 10.2.3

Kaiming (He) initialization is designed for networks using the ReLU activation, which sets negative values to zero. In a molecular SMILES autoencoder, it initializes linear-layer weights before the model learns to compress and reconstruct strings. section 10.2.3

Choosing a range for a linear layer

With Kaiming uniform initialization in fan_out mode and ReLU gain, weights are sampled from −√(6 / fan_out) to +√(6 / fan_out). For a layer with 256 outputs, the range is approximately −0.153 to +0.153. The bound changes with the fan mode and activation gain. section 10.2.3

Listing 10.6 Kaiming Initialization
   def _initialize_weights(self):
        for module in self.modules():
            if isinstance(module, nn.Linear):    #A
                nn.init.kaiming_uniform_(
                    module.weight, mode='fan_out', nonlinearity='relu'
                )
                if module.bias is not None:    #B
                    nn.init.zeros_(module.bias)
            elif isinstance(module, nn.Embedding):
                nn.init.normal_(module.weight, mean=0, std=0.1)
                if module.padding_idx is not None:
                    module.weight.data[module.padding_idx].zero_()

The initializer applies Kaiming uniform values to linear weights and zeros their biases. It initializes embedding weights separately with a normal distribution and zeros the padding row when one exists. section 10.2.3

Initialization is a starting point, not evidence that the autoencoder will meet its design goal. The model can reconstruct SMILES strings, but its learned map of chemical space is flawed for generating new molecules. section 10.2.3

How Build AI Drug Discovery Pipelines covers it

How deep it goes
Explains gradient-scale risks and Kaiming uniform bounds, then implements initialization for linear and embedding layers.
Examples
Calculates the sampling range for a 256-output layer and initializes a molecular SMILES autoencoder.
What you can do after
Apply it via PyTorch to initialize linear and embedding layers with different strategies.
How it connects
Connects initialization to ReLU, backpropagation, vanishing gradients, exploding gradients, and autoencoders.

Also explained in

Compare how 3 books explain weight initialization →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build AI Drug Discovery Pipelines. The section links point to the parts of the book it draws on.