Linear layer

As explained in Transformers in Action

nn.Linear is a PyTorch module that applies an affine linear transformation to its input.

A linear layer is a fully connected transformation that maps a fixed-length input vector to an output vector. Every output unit connects to every input unit, allowing the layer to learn combinations of the input features. section 2.2.4

A position-wise feed-forward network uses one linear layer to expand the vector into a higher-dimensional space, applies a nonlinear activation, and uses another linear layer to restore the original dimension. The same transformation operates independently at each sequence position. section 2.2.4

Two linear layers in a position-wise FFN

class PositionwiseFeedforward(nn.Module): def __init__(self, input_dim, hidden_dim, dropout=0.1): super().__init__() self.dropout = nn.Dropout(dropout) self.fc1 = nn.Linear(input_dim, hidden_dim) self.fc2 = nn.Linear(hidden_dim, input_dim) def forward(self, x): x = self.dropout(torch.relu(self.fc1(x))) return self.fc2(x) Here, fc1 increases representational capacity, while fc2 returns the result to the model's original width. section 2.2.4

Figure 6.4 Linear projection W maps patch vectors from an input space ℝ^dto an embedding space ℝ^k. Dashed arrows indicate the mapping of example patches through W.

Visualizes a linear projection mapping flattened image-patch vectors from the input space into the transformer's embedding space. section 6.3.1

How Transformers in Action covers it

How deep it goes

Explains full connectivity, dimensional expansion and contraction, nonlinear placement, and independent reuse across sequence positions.

Examples

Uses a position-wise FFN, a vision-transformer patch projection, and a masked-token prediction head.

What it assumes

Assumes fixed-length vectors and basic neural-network layers while explaining connectivity and dimensional reshaping through examples.

What you can do after

Build it from scratch as a PyTorch module inside an FFN or as a projection from patches or hidden states.

How it connects

Connects linear layers to fully connected networks, nonlinear activation, transformer hidden size, patch embeddings, and masked language modeling.

liveBookmanning.com

Read Transformers in Action on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Inside Deep Learning: Math, Algorithms, ModelsA PyTorch module that applies an affine linear transformation to its input.
• 

  liveBookmanning.com
  Deep Learning with Python, Third Edition
• 

  liveBookmanning.com
  Deep Learning with PyTorch, Second EditionA PyTorch linear layer module that applies an affine transformation to map features to outputs.
• 

  liveBookmanning.com
  Deep Learning with R, Third EditionA Dense layer is a neural-network layer in which every output unit is connected to every input unit through learned weights and biases.
• 

  liveBookmanning.com
  Math and Architectures of Deep Learning
• 

  liveBookmanning.com
  Grokking Machine LearningA neural-network architecture in which every node in a layer connects to every non-bias node in the next layer, with no connections between nonconsecutive layers.
• 

  liveBookmanning.com
  Natural Language Processing in Action, Second EditionA fully connected linear layer is a neural-network layer that computes an affine transformation from every input activation to its output units.
• 

  liveBookmanning.com
  Learn Generative AI with PyTorchA PyTorch layer that applies a learned linear transformation to its input.
• 

  liveBookmanning.com
  Sutskever's List
• 

  liveBookmanning.com
  Evolutionary Deep Learning: Genetic algorithms and neural networksA dense layer is a neural-network layer in which each unit receives inputs from all units in the preceding layer.
• 

  liveBookmanning.com
  Build a Text-to-Image Generator (from Scratch)A fully connected layer is a neural-network layer in which every output unit receives input from every unit in the preceding layer.
• 

  liveBookmanning.com
  Grokking AI Algorithms, Second EditionA linear layer is a neural-network layer that computes an affine or linear transformation of its input using learned weights and usually a bias.

Compare how 13 books explain linear layer →

Related concepts

• affine transformimplements · An affine transformation is a mapping that combines a linear transformation with an additive translation or offset.
• feedforward networkpart of · A feedforward network is a neural network or subnetwork in which information flows from inputs toward outputs through successive computations without recurrent connections or cycles.
• logitproduces · A logit is a real-valued quantity representing an outcome’s log-odds or, in machine learning, an unnormalized model score used to derive probabilities.
• matrix-matrix multiplicationuses · Matrix multiplication combines compatible matrices to form a result whose entries are sums of products between rows of the first matrix and columns of the second.
• softmaxuses · Softmax is a function that exponentiates a vector of scores and normalizes the results into nonnegative values that sum to one.
• neural network layerkind of · A neural network layer is a computational stage or component that transforms input representations into outputs within a neural network.

This explainer was generated by Manning's knowledge graph from the text of Transformers in Action. The section links point to the parts of the book it draws on.