Linear layer

As explained in Sutskever's List

A linear layer is a neural-network layer that maps input features to outputs by a learned affine transformation, typically connecting each output to every input feature.

A fully connected, or dense, layer links every unit to every unit in the preceding layer. In AlexNet, these layers follow convolutional feature extraction: they flatten feature maps and combine image-wide information, giving up spatial structure to produce classification predictions. section 2.5.1

AlexNet’s first two fully connected layers contain most of its roughly 60 million parameters, increasing representational capacity while raising the risk of overfitting. The final layer produces 1,000 scores for ImageNet categories; softmax converts them into predicted probabilities. section 2.5.1

AlexNet and dropout

AlexNet’s FC6 and FC7 each have 4,096 neurons; during training, dropout disables 50% of the neurons in those layers to help limit overfitting. section 2.5.1 section 2.5.2

Figure 2.3 The AlexNet architecture (cropped in the original paper). The numbers indicate spatial dimensions and channel depth at each stage. The input image (224 × 224 × 3 RGB) passes through a series of convolutional and pooling layers that progressively reduce spatial resolution (224→55→27→13) while increasing channel depth. Early downsampling is driven by the first convolutional layer (11 × 11 filters with a stride of 4), followed by max pooling layers (3 × 3 windows with a stride of 2) that further reduce spatial dimensions. The stride determines how far a filter moves across the input at each step, controlling how densely the image is sampled and how quickly spatial resolution is reduced. Filter sizes (11, 5, 3) are depicted as cubes and define the local receptive field of each convolution. The two horizontal streams reflect the model’s original two-GPU training setup, where feature maps were split across devices (e.g., 48 + 48 = 96 channels). The fully connected layers (4096 neurons, implemented as 2048 + 2048 across GPUs) feed into a 1000-way output for ImageNet classification. Used with permission granted by the lead author (Alex Krizhevsky).

Trace AlexNet’s path from convolutional feature maps through fully connected layers to its 1,000-way output. section 2.5.1

How Sutskever's List covers it

How deep it goes

Explains dense connectivity, flattening, parameter concentration, and output scoring, but gives no weight-level calculations.

Examples

Uses AlexNet’s three fully connected layers, 4,096-unit hidden layers, 1,000-way output, and 50% dropout in FC6 and FC7.

What you can do after

Understand how dense layers aggregate image features into classification scores and probabilities.

How it connects

Connects convolutional layers, feature maps, dropout, overfitting, and softmax classification.

liveBookmanning.com

Read Sutskever's List on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Inside Deep Learning: Math, Algorithms, ModelsA PyTorch module that applies an affine linear transformation to its input.
• 

  liveBookmanning.com
  Deep Learning with Python, Third Edition
• 

  liveBookmanning.com
  Deep Learning with PyTorch, Second EditionA PyTorch linear layer module that applies an affine transformation to map features to outputs.
• 

  liveBookmanning.com
  Deep Learning with R, Third EditionA Dense layer is a neural-network layer in which every output unit is connected to every input unit through learned weights and biases.
• 

  liveBookmanning.com
  Math and Architectures of Deep Learning
• 

  liveBookmanning.com
  Grokking Machine LearningA neural-network architecture in which every node in a layer connects to every non-bias node in the next layer, with no connections between nonconsecutive layers.
• 

  liveBookmanning.com
  Natural Language Processing in Action, Second EditionA fully connected linear layer is a neural-network layer that computes an affine transformation from every input activation to its output units.
• 

  liveBookmanning.com
  Learn Generative AI with PyTorchA PyTorch layer that applies a learned linear transformation to its input.
• 

  liveBookmanning.com
  Evolutionary Deep Learning: Genetic algorithms and neural networksA dense layer is a neural-network layer in which each unit receives inputs from all units in the preceding layer.
• 

  liveBookmanning.com
  Transformers in Actionnn.Linear is a PyTorch module that applies an affine linear transformation to its input.
• 

  liveBookmanning.com
  Build a Text-to-Image Generator (from Scratch)A fully connected layer is a neural-network layer in which every output unit receives input from every unit in the preceding layer.
• 

  liveBookmanning.com
  Grokking AI Algorithms, Second EditionA linear layer is a neural-network layer that computes an affine or linear transformation of its input using learned weights and usually a bias.

Compare how 13 books explain linear layer →

Related concepts

• affine transformimplements · An affine transformation is a mapping that combines a linear transformation with an additive translation or offset.
• feedforward networkpart of · A feedforward network is a neural network or subnetwork in which information flows from inputs toward outputs through successive computations without recurrent connections or cycles.
• logitproduces · A logit is a real-valued quantity representing an outcome’s log-odds or, in machine learning, an unnormalized model score used to derive probabilities.
• matrix-matrix multiplicationuses · Matrix multiplication combines compatible matrices to form a result whose entries are sums of products between rows of the first matrix and columns of the second.
• softmaxuses · Softmax is a function that exponentiates a vector of scores and normalizes the results into nonnegative values that sum to one.
• neural network layerkind of · A neural network layer is a computational stage or component that transforms input representations into outputs within a neural network.

This explainer was generated by Manning's knowledge graph from the text of Sutskever's List. The section links point to the parts of the book it draws on.