A linear layer is a trainable, fully connected mapping that combines every value in an input encoding to produce the output units needed for a prediction. It is especially useful after convolution and pooling have compressed a text representation into a fixed-size vector. section 7.5.3 section 7.5.1
Use a linear layer to:
- Combine information from a high-dimensional encoding into an answer such as a binary newsworthiness prediction. section 7.5.3
- Size the input to match the number of values produced by preceding convolution and pooling layers. section 7.5.1
- Reduce a latent representation to the dimensions required by regression or classification targets. section 7.5.3
- Compute predictions from recurrent hidden states, including inside hand-built RNNs and LSTMs. section 8.2.2 section 8.4.3
From pooled text features to a classifier
A CNN applies five convolution-and-pooling paths, each producing one pooled value per channel. With five channels per path, concatenation creates a 5 × 5 tensor, or 25 values. A linear output layer consumes those 25 values and produces the dimensions required by the classifier. Reducing each convolution from five channels to three changes the concatenated representation from 25 to 15 values, limiting overfitting while potentially slowing convergence. section 7.5.1
Listing 7.29 Computing the tensor size for the output of a 1D convolution
>>> out_pool_total = 0 #1 >>> for kernel_len, stride in zip(kernel_lengths, strides): >>> out_conv = ( ... (in_seq_len - dilation * (kernel_len - 1) - 1) ... // stride ... ) + 1 >>> out_pool_total += ( ... (out_conv - dilation * (kernel_len - 1) - 1) ... // stride ... ) + 1
Inspect the calculation that totals the pooled convolution outputs before sizing the fully connected layer. section 7.5.3
