Prior knowledge

As explained in Deep Learning with R, Third Edition

Prior knowledge is information or structural assumptions available to an agent before it learns from or encounters the current problem.

Prior knowledge is what an agent brings to a problem before learning from the current data. In a neural network, it can take the form of assumptions built into the architecture: the chosen layers and their connections determine which functions gradient descent can search for. A useful assumption makes that search easier; a poor one can limit the model even when more training data is available. section 9.0

An assumption hidden in a layer choice

A two-class classifier made from one Dense layer with no activation assumes the classes are linearly separable. The layer is not merely an implementation choice: it determines what kind of solution the model can learn. section 3.6.3

Image-model choices offer two more examples:

• Convolutional layers encode the assumption that relevant image patterns can appear at different locations. section 9.0
• Depthwise separable convolution assumes spatial locations in intermediate activations are highly correlated while channels are relatively independent. When that assumption fits, it can use training data more efficiently and produce smaller models. section 9.4

Prior knowledge also matters when judging generalization. To test how efficiently an agent adapts, hold its past experience and prior knowledge fixed, then assess it on sufficiently different tasks it was not trained or programmed to handle. This asks what the agent can do with the information it already has, rather than whether it has practiced a particular task. section 19.3.4

Figure 19.9 An ARC-AGI task: the nature of the task is demonstrated by a couple of input–output pair examples. Provided with a new input, you must construct the corresponding output.

The grid task makes on-the-fly adaptation concrete: a few input–output examples establish a new task, and a fresh input calls for a corresponding output. section 19.3.4

Not all priors are architectural. Innate priors are among the resources associated with human reasoning and generalization in unfamiliar situations, alongside abstraction, symbolic models, logic, and common sense. section 5.1.2

How Deep Learning with R, Third Edition covers it

How deep it goes

Explains priors through layer choices and a novel-task evaluation setup, without giving a general procedure for discovering the right priors.

Examples

Uses a linear classifier, image convolutions, and an ARC-AGI input–output grid task.

What it assumes

Builds on basic ideas of layers, classification, and training while explaining prior knowledge as assumptions embodied in model architecture.

What you can do after

Evaluate/understand it by identifying assumptions in an architecture and distinguishing trained task skill from adaptation to unfamiliar tasks.

How it connects

Connects prior knowledge to hypothesis spaces, gradient descent, convolution, and generalization.

liveBookmanning.com

Read Deep Learning with R, Third Edition on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Bayesian Optimization in ActionPrior knowledge is information or assumptions about an objective function that are available before observing all relevant data.

Compare how 2 books explain prior knowledge →

Related concepts

• covariance functionuses · A covariance function, also called a kernel, specifies the covariance between function values at pairs of inputs in a Gaussian process.
• mean functionuses · A mean function maps an input to the expected value of a random function before conditioning on observed data.
• problem solvingused by · Problem solving is the process of analyzing a problem and developing, selecting, or applying a solution to address it.

This explainer was generated by Manning's knowledge graph from the text of Deep Learning with R, Third Edition. The section links point to the parts of the book it draws on.