Weight initialization

As explained in Deep Learning with JAX

The process of assigning initial values to neural-network weights before training begins.

Before training begins, a neural network needs initial values for its weights and biases. Weight initialization supplies those values, using randomness as part of the model-building process. section 9.0 section 2.4

In JAX, initialization receives a PRNGKey from outside the model. This keeps random-number generation functionally pure: keys represent generator state, while parameters remain explicit data passed to the stateless forward pass. section 9.0 section 2.4

Figure 2.4 The process of initializing and applying a neural network in JAX

Follow the initialization-to-application flow for a JAX neural network, including externally supplied randomness and explicit model parameters. section 2.4

How Deep Learning with JAX covers it

How deep it goes

Explains the role of initialization and external PRNG keys, but stops short of specifying initialization distributions or algorithms.

Examples

Uses a fully connected network with 784 inputs, 512 hidden neurons, and 10 output neurons to illustrate initialization and application.

What it assumes

Assumes familiarity with feed-forward layers, weights, biases, activations, and forward passes while contrasting JAX with PyTorch and TensorFlow.

What you can do after

Build it from scratch by initializing layer parameters with externally supplied random state and passing them into a pure forward pass.

How it connects

Connects weight initialization with PRNG keys, functional programming, model parameters, forward passes, and neural-network layers.

liveBookmanning.com

Read Deep Learning with JAX on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Sutskever's ListWeight initialization is the procedure for assigning initial values to a neural network's trainable weights before training.

Compare how 2 books explain weight initialization →

Related concepts

• parameterconfigures · A parameter is a value or variable that specifies or controls a characteristic, computation, or behavior of a function, model, operation, or request.
• random restartused by · Random restart is a search or optimization technique that repeatedly begins from newly selected initial states to escape local optima or other poor regions.
• autoencoderused by · An autoencoder is a neural network that encodes input data into an internal representation and decodes it to reconstruct the input or an approximation.
• exploding gradientaddresses · An exploding gradient is a training instability in which gradient magnitudes grow excessively, causing unstable parameter updates or numerical and convergence failures.
• vanishing gradient problemaddresses · The vanishing-gradient problem is a neural-network training difficulty in which gradients shrink toward zero as they propagate backward, making updates to earlier layers or distant sequence steps ineffective.

This explainer was generated by Manning's knowledge graph from the text of Deep Learning with JAX. The section links point to the parts of the book it draws on.