Shared weight

As explained in Natural Language Processing in Action, Second Edition

Weight sharing is the reuse of the same learned parameters across multiple inputs, positions, or computation steps.

Shared weight means reusing the same learned weight matrices across multiple positions or computation steps. In an RNN, one set of matrices is applied to every token, while hidden states and outputs pass from one step to the next. section 8.3

This reuse reduces the model’s distinct parameters but makes training dependent across time. Backpropagation through time must propagate each token’s error backward through every earlier use of the shared matrices, serially. section 8.3

Shared matrices in an unrolled RNN

For a text sequence, the RNN applies the same weight matrices to token 1, token 2, and later tokens. After comparing the final output with the correct label, training works backward through those repeated applications and adjusts the shared weights using each token’s error. section 8.3

Figure 8.10 Data fed into a recurrent network

Trace how token inputs, hidden states, and recurrent computations are arranged across time. section 8.3

How Natural Language Processing in Action, Second Edition covers it

How deep it goes

Explains repeated matrix use and serial gradient propagation, but does not provide implementation code or parameter-count analysis.

Examples

Uses an unrolled RNN processing text tokens and propagating errors backward through repeated weight-matrix multiplications.

What it assumes

Assumes familiarity with tokens, vectors, weight matrices, outputs, labels, and gradients while relating gradient calculation to the chain rule.

What you can do after

Understand how recurrent models reuse parameters and why their training must process sequence errors backward one token at a time.

How it connects

Connects shared weight to recurrent neural networks, hidden states, backpropagation through time, gradients, and the chain rule.

liveBookmanning.com

Read Natural Language Processing in Action, Second Edition on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Inside Deep Learning: Math, Algorithms, ModelsA convolutional-network technique that reuses the same parameters at multiple input positions instead of learning separate parameters for each position.

Compare how 2 books explain shared weight →

Related concepts

• recurrent neural networkused by · A recurrent neural network is a neural network that processes sequences by updating an internal state across successive inputs, allowing earlier inputs to influence later computations.
• convolutional layerused by · A convolutional layer is a neural-network layer that applies shared learned filters to local regions of structured input to produce feature representations.
• convolution kernelconfigures · A convolution kernel is a small array or tensor of weights applied across local input regions to compute weighted responses and detect or transform patterns.
• autoencodercontrasts with · An autoencoder is a neural network that encodes input data into an internal representation and decodes it to reconstruct the input or an approximation.
• layer normalizationused by · Layer normalization standardizes neural-network activations across specified feature dimensions for each example independently, often applying learned scale and bias parameters.
• LSTMuses · An LSTM is a recurrent neural network architecture with gated memory mechanisms that control information flow and help retain or update information across sequences.

This explainer was generated by Manning's knowledge graph from the text of Natural Language Processing in Action, Second Edition. The section links point to the parts of the book it draws on.