liveBook · Manning

Parameter

As explained in Deep Learning with Python, Third Edition

A model parameter that is updated by backpropagation during training.

A parameter is a learned value that controls how a neural network transforms its inputs. In a layer, parameters are usually called weights: learning adjusts them so predictions move closer to their targets and the loss decreases. section 1.6 section 20.1.1

Training treats parameters as adjustable state rather than fixed data. The loss measures prediction quality, backpropagation supplies the feedback needed for adjustment, and an optimizer updates the values. section 1.6

A parameter in PyTorch

PyTorch can store trainable state in an ordinary tensor, but torch.nn.parameter.Parameter makes its meaning explicit. A Parameter is a tensor subclass that PyTorch can automatically track and retrieve when assigned to a model, much like a Keras Variable. section 3.4.1

Keep these practical distinctions in mind:

  • A tensor can represent any numerical data; a Parameter specifically signals trainable model state. section 3.4.1
  • A model’s learned knowledge is stored in its weights, which may number from millions to hundreds of billions. section 1.8 section 20.1.1
  • Pretrained weights are usable only with a matching model architecture and tokenizer. section 15.4.2
Listing 3.24 Creating a PyTorch parameter
>>> x = torch.zeros(size=(2, 1))
>>> p = torch.nn.parameter.Parameter(data=x)     #1

Inspect the minimal PyTorch construction of a Parameter from an existing zero-filled tensor. section 3.4.1

>>> backbone.summary()
Model: "roberta_backbone"
┏━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┓
┃ Layer (type)          ┃ Output Shape      ┃     Param # ┃ Connected to       ┃
┡━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━┩
│ token_ids             │ (None, None)      │           0 │ -                  │
│ (InputLayer)          │                   │             │                    │
├───────────────────────┼───────────────────┼─────────────┼────────────────────┤
│ embeddings            │ (None, None, 768) │  38,996,736 │ token_ids[0][0]    │
│ (TokenAndPositionEmb… │                   │             │                    │
├───────────────────────┼───────────────────┼─────────────┼────────────────────┤
│ embeddings_layer_norm │ (None, None, 768) │       1,536 │ embeddings[0][0]   │
│ (LayerNormalization)  │                   │             │                    │
├───────────────────────┼───────────────────┼─────────────┼────────────────────┤
│ embeddings_dropout    │ (None, None, 768) │           0 │ embeddings_layer_… │
│ (Dropout)             │                   │             │                    │
├───────────────────────┼───────────────────┼─────────────┼────────────────────┤
│ padding_mask          │ (None, None)      │           0 │ -                  │
│ (InputLayer)          │                   │             │                    │
├───────────────────────┼───────────────────┼─────────────┼────────────────────┤
│ transformer_layer_0   │ (None, None, 768) │   7,087,872 │ embeddings_dropou… │
│ (TransformerEncoder)  │                   │             │ padding_mask[0][0] │
├───────────────────────┼───────────────────┼─────────────┼────────────────────┤
│ transformer_layer_1   │ (None, None, 768) │   7,087,872 │ transformer_layer… │
│ (TransformerEncoder)  │                   │             │ padding_mask[0][0] │
├───────────────────────┼───────────────────┼─────────────┼────────────────────┤
│ ...                   │ ...               │ ...         │ ...                │
├───────────────────────┼───────────────────┼─────────────┼────────────────────┤
│ transformer_layer_11  │ (None, None, 768) │   7,087,872 │ transformer_layer… │
│ (TransformerEncoder)  │                   │             │ padding_mask[0][0] │
└───────────────────────┴───────────────────┴─────────────┴────────────────────┘
 Total params: 124,052,736 (473.22 MB)
 Trainable params: 124,052,736 (473.22 MB)
 Non-trainable params: 0 (0.00 B)

Inspect how a pretrained Transformer exposes parameter counts for layers such as embeddings and layer normalization. section 15.4.2

How Deep Learning with Python, Third Edition covers it

How deep it goes
Defines parameters, relates them to loss and backpropagation, and demonstrates PyTorch tracking without deriving gradient updates internally.
Examples
Uses PyTorch Parameter construction, linear regression, pretrained RoBERTa weights, and parameter-count summaries.
What it assumes
Assumes basic tensors and neural-network layers while introducing parameters through concrete PyTorch operations.
What you can do after
Evaluate trainable state, create a PyTorch Parameter, and understand how optimization changes model weights.
How it connects
Connects parameters with weights, layers, loss functions, backpropagation, gradient descent, optimizers, pretrained models, and tokenizers.
Cover of Deep Learning with Python, Third Edition
liveBookmanning.com
Read Deep Learning with Python, Third Edition on liveBook Get the book at manning.com

Also explained in

Compare how 31 books explain parameter →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Deep Learning with Python, Third Edition. The section links point to the parts of the book it draws on.