Chapter 10: From foundational algorithms to modern AI: David J.C. MacKay and the unifying logic of learning
The introduction presents MacKay’s synthesis of seemingly separate algorithms as ways of learning from incomplete evidence under practical constraints, then connects that synthesis to modern AI.
After this chapter you will be able to
- Shared logic of learning. Explain why MacKay placed methods with different histories and machinery in a common framework: observed data bears on unknown possibilities, probabilistic models weigh those possibilities, and practical limits constrain inference.
- Shared structure of learning methods. Explain how the section extends MacKay’s view of learning to Bayesian and non-Bayesian methods: each begins with something unresolved, uses available information, transforms it into an answer, and faces constraints on what is feasible.
- Shared Structure Beneath Distinct Methods. Explain that the earlier contributions used different vocabulary, mathematics, and objectives, yet MacKay’s lens brings a deeper shared intellectual structure into focus without erasing those differences.
- Prior evidence and posterior probability. Explain how disease prevalence, test sensitivity, and the false-positive rate combine to revise the probability of disease after a positive test.
- Informative but inconclusive test result. Explain why a positive result raises the probability of disease in this example without establishing a diagnosis, and what would help narrow the remaining uncertainty.
- Likelihood-based coin estimate. Explain why maximum likelihood estimation selects p = 0.7 after seven heads in 10 tosses: that value makes the observed result most likely among the candidate probabilities of heads.
- P Value Decision Threshold. Describe what the p-value measures and how comparing it with a predetermined significance level leads to a decision with controlled Type I error risk.
- Entropy as message uncertainty. Explain what Shannon’s entropy measures using the example of eight equally likely messages, and describe how unequal probabilities change it.
- Compression and communication limits. Distinguish how compression and error-correcting codes use redundancy, and explain the limit imposed by a noisy channel’s capacity.
- Recursive Global Optimization. Describe how Bellman’s recurrence combines a choice’s immediate cost with the best outcome from the following state, and how retaining subproblem results helps produce a global optimum.
- Exactness and Practical Limits. Distinguish what dynamic programming resolves in a correctly specified deterministic problem from the conditions that limit its guarantee or computational benefit.
- From probability to action. Explain why a posterior probability alone does not determine whether the physician should treat, withhold treatment, or seek more information. Describe how expected utility compares actions using their probabilities and possible consequences.
- Hard versus soft margins. Explain why a noisy or mislabeled point can distort a hard-margin SVM boundary and why a soft-margin SVM may allow violations instead.
- Slack and penalty tradeoff. Describe what slack variables measure and how the penalty parameter C changes the balance between violations and margin width.
- Optimization and generalization limits. Distinguish finding the best boundary for the specified training objective from knowing which boundary will perform best on future observations.
- Random forest prediction stability. Explain why a single decision tree can be unstable and how bootstrap samples, random feature selection, and aggregation make random-forest predictions more stable.
- Cross-domain migration of ideas. Explain that a probability calculation, computational pattern, or mathematical quantity can reappear in another field under different terminology and for a different purpose. Give one of the section’s examples without explaining its mechanics.
- Decoding and model comparison correspondence. Explain how Bayesian model comparison mirrors decoding, including what corresponds across the two settings and why model comparison can involve additional uncertainty.
- Shared local inference pattern. Trace how belief propagation updates messages in a graphical model and explain the correspondence with code decoding.
- Cross-entropy as training loss. Describe how the same cross-entropy quantity becomes a classifier's training loss, including how the probability assigned to the correct class affects loss and its relationship to likelihood.
- Portability of underlying methods. Explain how a method developed for one problem can help solve another when their underlying structures correspond, even if the algorithm is not transferred intact. Describe the practical value of adapting such methods rather than starting from scratch.
- Foundations as operating machinery. Explain how modern AI made earlier ideas about inference, information, optimization, representation, and generalization part of working learning systems rather than replacing them.
- Training and learned representations. Trace the training cycle from examples to predictions, measured error, gradients, and parameter updates. Describe how successive layers also learn features rather than relying entirely on features specified in advance.
- Foundations within network training. Explain how the section connects Fisher’s likelihood, Shannon’s cross-entropy, Bayes’s probabilistic language, and Vapnik’s concern with generalization to neural network training.
- Sequential and relational processing. Distinguish recurrent processing from transformer self-attention, and explain why the transformer can scale more efficiently.
- Earlier ideas within transformers. Explain why the transformer is presented as a new arrangement of earlier learning ideas rather than a break from them, while noting the limits the section identifies.
- Predictable gains from scaling. Explain that experiments found relatively smooth power-law declines in prediction error as model parameters, training data, and computation increased, so improvement did not require a new learning algorithm each time.
- Expanded but persistent limits. Explain how more usable data, computing infrastructure, and efficient parallel architecture push the boundaries of feasible learning outward without eliminating limits such as data quality, memory, energy, cost, and diminishing returns.
What it builds on
- Bayesian inference in real-world scenarios (chapter 2) · Bayesian inference applied to real-world scenarios.
- Maximum likelihood estimation (chapter 3) · The earlier introduction to maximum likelihood estimation.
- From prior to posterior: how Bayes’ Theorem works (chapter 2) · The earlier account of Bayesian updating from prior to posterior.
- Step 1: state the hypotheses (chapter 4) · Stating null and alternative hypotheses
- Step 5: establish the decision rule (chapter 4) · Using a predefined decision rule in hypothesis testing
- Step 7: draw a conclusion (chapter 4) · Drawing a conclusion from a hypothesis test
- Entropy as the measure of uncertainty (chapter 5) · Introduces entropy as a measure of uncertainty.
- Channel capacity (chapter 5) · Introduces channel capacity as a communication limit.
Section by section
Introduction
The introduction presents MacKay’s synthesis of seemingly separate algorithms as ways of learning from incomplete evidence under practical constraints, then connects that synthesis to modern AI.
- Shared logic of learning: Explain why MacKay placed methods with different histories and machinery in a common framework: observed data bears on unknown possibilities, probabilistic models weigh those possibilities, and practical limits constrain inference.
Why it matters: This connection helps you see why methods taught separately belong in the same story of learning. - Cumulative foundations of modern AI: Describe how MacKay’s perspective recasts earlier methods as advances that reinforce one another, and how modern AI draws on many of their principles together at a much larger scale.
Why it matters: It connects the book’s earlier algorithms to the larger learning systems discussed next.
Key terms: probabilistic inference, uncertainty, constraint, incomplete evidence, complementary building blocks, modern AI, representations, computational resources
10.1 Synthesis rather than invention
The section uses MacKay’s synthesis to frame diverse algorithms as ways of reaching justified conclusions from incomplete information under constraints.
- Shared structure of learning methods: Explain how the section extends MacKay’s view of learning to Bayesian and non-Bayesian methods: each begins with something unresolved, uses available information, transforms it into an answer, and faces constraints on what is feasible.
Why it matters: This lens lets you compare methods without treating their different technical forms as wholly unrelated. - Approximation and preserved uncertainty: Explain why probabilistic inference may be approximated when exact inference is too computationally costly, and what tradeoff that approximation introduces.
Why it matters: A feasible answer may represent less uncertainty than the exact inference would. - Portability of the shared lens: Justify why recognizing the shared problem structure makes insights portable across domains and helps interpret both new and familiar methods.
Why it matters: The perspective helps you recognize when an idea might apply outside the problem that first motivated it.
Key terms: Bayesian and non-Bayesian, uncertainty, information, transformation, constraints, exact inference, approximation, computational resources, portability, shared structure
10.2 Looking back: isolated contributions, shared foundations
This introduction frames MacKay’s perspective as a way to see shared foundations beneath earlier contributors’ distinct methods.
- Shared Structure Beneath Distinct Methods: Explain that the earlier contributions used different vocabulary, mathematics, and objectives, yet MacKay’s lens brings a deeper shared intellectual structure into focus without erasing those differences.
Why it matters: This perspective connects the earlier chapters without treating their contributions as identical.
Key terms: MacKay’s lens, distinct formulations, deeper intellectual structure
10.2.1 Bayes: revising belief with new evidence
A medical-test example shows how Bayes’ Theorem revises a prior belief with evidence while leaving uncertainty about the patient’s condition.
- Prior evidence and posterior probability: Explain how disease prevalence, test sensitivity, and the false-positive rate combine to revise the probability of disease after a positive test.
Why it matters: A test result is evidence to weigh against the starting probability, not a probability of disease by itself. - Informative but inconclusive test result: Explain why a positive result raises the probability of disease in this example without establishing a diagnosis, and what would help narrow the remaining uncertainty.
Why it matters: A substantial increase in probability can still fall far short of a definitive diagnosis.
Key terms: Bayes’ Theorem, prior probability, sensitivity, false-positive rate, posterior probability, base rate, uncertainty, independent evidence
10.2.2 Fisher: estimating unknown parameters through maximum likelihood
This section uses a coin-toss example to show how maximum likelihood produces a parameter estimate, how it differs from Bayesian updating, and why the estimate remains uncertain.
- Likelihood-based coin estimate: Explain why maximum likelihood estimation selects p = 0.7 after seven heads in 10 tosses: that value makes the observed result most likely among the candidate probabilities of heads.
Why it matters: This connects Fisher’s estimation method to a concrete choice of parameter value. - Fixed unknown parameter: Distinguish maximum likelihood from Bayesian updating in this example: maximum likelihood treats the coin’s probability of heads as fixed but unknown and assigns no prior or posterior probabilities to its possible values.
Why it matters: The distinction prevents a best-fitting estimate from being mistaken for a posterior belief about the parameter. - Estimate assumptions and limits: Explain why p = 0.7 is a defensible estimate rather than a precise statement of the coin’s true behavior: it rests on a small sample and assumes independent tosses with a constant probability, so more observations would help.
Why it matters: A best-fitting value should not be reported as certainty about the coin.
Key terms: maximum likelihood estimation, likelihood function, point estimate, fixed but unknown, Bayesian updating, prior, posterior, small sample size, independent, constant probability
10.2.3 Neyman and Pearson: making decisions while controlling errors
The section shows how Neyman–Pearson hypothesis testing turns sample evidence into a decision about a defect-rate improvement while controlling, but not eliminating, error risk.
- Sampling Variation and Hypotheses: Explain why an observed defect-rate improvement does not establish that the production process improved, and describe the null and alternative hypotheses used to test it.
Why it matters: The hypotheses give the sample result a defined claim to test rather than treating an apparent improvement as proof. - P Value Decision Threshold: Describe what the p-value measures and how comparing it with a predetermined significance level leads to a decision with controlled Type I error risk.
Why it matters: This distinction lets you interpret a test decision without mistaking the p-value for the probability that the null hypothesis is true. - Limits of Test Conclusions: Explain why the test decision remains uncertain and state which factors the section says affect Type II error and statistical power.
Why it matters: A defensible test conclusion acknowledges both the information gained and the uncertainty that remains.
Key terms: sampling variation, null hypothesis, alternative, p-value, significance level, Type I error, Type II error, statistical power, sample size, accepted error rate
10.2.4 Shannon: quantifying information and the limits of communication
This section explains how Shannon measured information as uncertainty reduction and identified limits on reliable communication.
- Entropy as message uncertainty: Explain what Shannon’s entropy measures using the example of eight equally likely messages, and describe how unequal probabilities change it.
Why it matters: It distinguishes information gained from the size of the object carrying the message. - Compression and communication limits: Distinguish how compression and error-correcting codes use redundancy, and explain the limit imposed by a noisy channel’s capacity.
Why it matters: It shows why communicating efficiently and protecting against errors are different tasks, both subject to a channel limit. - Decoding and resolved uncertainty: Describe how a received signal helps resolve uncertainty about a transmitted message and what decoding produces.
Why it matters: It connects Shannon’s measure of uncertainty to the practical task of inferring a message from a received signal.
Key terms: entropy, bits, compression, predictable redundancy, noisy channel, channel capacity, error-correcting codes, structured redundancy, received signal, decoding
10.2.5 Bellman: making global optimization tractable through decomposition
The section explains how Bellman’s dynamic programming finds a globally optimal route by reusing solutions to connected subproblems, and when that approach is effective.
- Why Local Choices Fail: Explain why choosing the cheapest next step or comparing every complete path is inadequate for a large multistage routing problem.
Why it matters: It explains why the problem needs more than a greedy choice or an exhaustive search. - Recursive Global Optimization: Describe how Bellman’s recurrence combines a choice’s immediate cost with the best outcome from the following state, and how retaining subproblem results helps produce a global optimum.
Why it matters: The recurrence accounts for downstream consequences without repeatedly solving the same subproblems. - Exactness and Practical Limits: Distinguish what dynamic programming resolves in a correctly specified deterministic problem from the conditions that limit its guarantee or computational benefit.
Why it matters: It keeps an exact solution guarantee distinct from whether the method is applicable or practical.
Key terms: multistage network, globally optimal route, dynamic programming, recurrence, tabulation, memoization, optimal substructure, deterministic problem, state representation, overlapping subproblems
10.2.6 Raiffa and Schalifer: From Bayesian inference to rational action
This section shows how Bayesian decision analysis turns a probability of disease into a choice among actions despite remaining uncertainty.
- From probability to action: Explain why a posterior probability alone does not determine whether the physician should treat, withhold treatment, or seek more information. Describe how expected utility compares actions using their probabilities and possible consequences.
Why it matters: A probability informs a medical decision, but the consequences of each choice determine what to do with it. - Decision trees and information value: Describe how decision trees arrange choices, uncertain events, and consequences for evaluation through rollback, and how the value of information addresses whether an additional test is worth its cost.
Why it matters: A further test is an option to evaluate, not an automatic next step.
Key terms: posterior probabilities, Bayesian decision analysis, utilities or losses, expected utility, decision trees, rollback, value of information
10.2.7 Vapnik: generalizing through soft margins
This subsection explains how soft-margin SVMs trade perfect training-set separation for a boundary that may generalize better, while acknowledging the limits of that optimization.
- Hard versus soft margins: Explain why a noisy or mislabeled point can distort a hard-margin SVM boundary and why a soft-margin SVM may allow violations instead.
Why it matters: A boundary that fits every training point is not necessarily the best boundary for new data. - Slack and penalty tradeoff: Describe what slack variables measure and how the penalty parameter C changes the balance between violations and margin width.
Why it matters: The penalty setting makes the compromise between fitting training data and preserving margin width explicit. - Optimization and generalization limits: Distinguish finding the best boundary for the specified training objective from knowing which boundary will perform best on future observations.
Why it matters: Solving the training problem exactly does not guarantee the best predictions on data that has not yet been seen.
Key terms: hard-margin SVM, soft-margin SVM, margin, slack variables, penalty parameter C, margin width, training objective, global solution, model capacity, feature representation
10.2.8 Breiman: stabilizing predictions through randomness and aggregation
The section explains how random forests stabilize predictions and uses them to illustrate a shared pattern across foundational algorithms.
- Random forest prediction stability: Explain why a single decision tree can be unstable and how bootstrap samples, random feature selection, and aggregation make random-forest predictions more stable.
Why it matters: It shows why introducing variation among trees can make their combined prediction more reliable than one tree’s prediction. - Limits of forest predictions: Describe what a random forest can and cannot resolve when predicting an unseen observation.
Why it matters: A steadier prediction should not be mistaken for certainty about a new case. - Shared foundational algorithm pattern: Explain the common pattern the section identifies across otherwise different foundational contributions.
Why it matters: The pattern connects the forest example to the chapter’s broader synthesis without treating different algorithms as identical.
Key terms: decision tree, random forest, bootstrap samples, random feature selection, aggregation, variance, unseen observation, training sample, strength and correlation, uncertainty
10.3 When ideas cross boundaries
The section introduces how foundational ideas migrate across domains, acquiring new names and purposes while retaining a shared underlying structure.
- Cross-domain migration of ideas: Explain that a probability calculation, computational pattern, or mathematical quantity can reappear in another field under different terminology and for a different purpose. Give one of the section’s examples without explaining its mechanics.
Why it matters: Recognizing a familiar idea in a new setting helps connect methods that might otherwise seem unrelated.
Key terms: migration and adaptation, Bayesian model comparison, message passing, cross-entropy
10.3.1 Decoding as Bayesian model comparison
The section shows that recovering a message from a noisy signal and comparing models against data use the same Bayesian inference pattern.
- Bayesian decoding of noisy signals: Explain how a Bayesian decoder uses prior probabilities and a channel likelihood to assess candidate messages when a received signal is noisy.
Why it matters: Decoding provides the concrete example that makes the later comparison with learning from data clear. - Decoding and model comparison correspondence: Explain how Bayesian model comparison mirrors decoding, including what corresponds across the two settings and why model comparison can involve additional uncertainty.
Why it matters: It reveals why a method for recovering messages can also describe learning from data.
Key terms: prior probabilities, channel model, posterior probability, Bayesian decoder, Bayesian model comparison, marginal likelihood, evidence, posterior model probabilities, channel likelihood
10.3.2 Message passing across codes and graphical models
The section shows how decoding error-correcting codes and inference in graphical models use the same local message-passing pattern, and where that pattern has limits.
- Graph representation of decoding: Explain why noisy codewords are represented as a graph and what its nodes and messages represent.
Why it matters: The graph turns a costly search over whole bit sequences into calculations organized around connected bits and constraints. - Shared local inference pattern: Trace how belief propagation updates messages in a graphical model and explain the correspondence with code decoding.
Why it matters: The correspondence reveals why decoding and probabilistic inference can use the same computational pattern despite their different purposes. - Limits imposed by loops: Distinguish what message passing can produce on graphs without loops from what may happen on graphs with loops.
Why it matters: Graph structure affects whether the resulting beliefs can be treated as exact and whether the updates settle.
Key terms: error-correcting code, variable nodes, constraint nodes, messages, probabilistic graphical models, factor nodes, belief propagation, local belief updates, loops, exact marginal probabilities
10.3.3 Cross-entropy: from coding efficiency to model training
The section shows how cross-entropy moves from measuring coding inefficiency to serving as a loss for training classifiers.
- Coding with mismatched probabilities: Explain what cross-entropy measures in coding and why an inaccurate probability model increases average code length.
Why it matters: The coding interpretation makes clear what the later training loss is measuring. - Cross-entropy as training loss: Describe how the same cross-entropy quantity becomes a classifier's training loss, including how the probability assigned to the correct class affects loss and its relationship to likelihood.
Why it matters: It connects an information measure from coding directly to how predictive models are trained.
Key terms: entropy, cross-entropy, average code length, probability model, training loss, classifier, likelihood
10.4 Why the connections matter
The section explains how recognizing shared structures makes foundational methods portable across fields and keeps them useful in modern learning systems.
- Portability of underlying methods: Explain how a method developed for one problem can help solve another when their underlying structures correspond, even if the algorithm is not transferred intact. Describe the practical value of adapting such methods rather than starting from scratch.
Why it matters: A connection between fields can give you a tested starting point for a new problem. - Continuing value of foundations: Describe why earlier foundational contributions remain useful rather than merely historical, and relate that continuing use to MacKay’s synthesis.
Why it matters: It explains why older ideas can remain useful even when modern systems look very different.
Key terms: portability of ideas, underlying method, shared structure, unification, Bayesian updating, likelihood, entropy, recursive decomposition, aggregation
10.5 Looking forward: AI operationalizes the synthesis
The section explains how modern AI turned MacKay’s synthesis of information, inference, and learning into systems that learn from data.
- Foundations as operating machinery: Explain how modern AI made earlier ideas about inference, information, optimization, representation, and generalization part of working learning systems rather than replacing them.
Why it matters: It connects the book’s foundational ideas to modern AI without treating modern systems as a break from those ideas.
Key terms: intellectual synthesis, operational one, foundations
10.5.1 Neural networks: the synthesis becomes trainable
The section shows how neural network training makes earlier ideas about probability, information, inference, and generalization work together in a practical learning system.
- Training and learned representations: Trace the training cycle from examples to predictions, measured error, gradients, and parameter updates. Describe how successive layers also learn features rather than relying entirely on features specified in advance.
Why it matters: This is how the section turns a collection of foundational ideas into a repeatable learning process. - Foundations within network training: Explain how the section connects Fisher’s likelihood, Shannon’s cross-entropy, Bayes’s probabilistic language, and Vapnik’s concern with generalization to neural network training.
Why it matters: It explains why deep learning is a synthesis of earlier ideas, not a replacement for them. - MacKay’s conceptual synthesis: Explain why the section calls MacKay’s treatment prescient without claiming that he predicted deep learning’s architectures or dominance.
Why it matters: It distinguishes a lasting explanation of learning from a prediction of particular technologies.
Key terms: forward pass, loss function, backpropagation, gradients, optimization, representation, likelihood, cross-entropy, Bayes, generalization
10.5.2 Transformers: the synthesis gains a scalable architecture
The section explains how transformers reorganize neural-network learning around attention and parallel computation while retaining earlier ideas about likelihood, information, and representation.
- Sequential and relational processing: Distinguish recurrent processing from transformer self-attention, and explain why the transformer can scale more efficiently.
Why it matters: This contrast explains what the transformer changed and why that change helped it scale. - Attention weighting steps: Trace how self-attention turns token relationships into context-sensitive representations at the level described here.
Why it matters: These steps make the section’s account of attention more specific than simply saying that tokens relate to one another. - Earlier ideas within transformers: Explain why the transformer is presented as a new arrangement of earlier learning ideas rather than a break from them, while noting the limits the section identifies.
Why it matters: It places the transformer within the chapter’s synthesis without claiming that earlier thinkers invented its architecture.
Key terms: recurrent neural networks, hidden state, self-attention, parallel, query–key comparisons, softmax, normalized weights, values, context-sensitive representations, likelihood
10.5.3 Scaling laws: expanding the boundaries of what is feasible
This section explains how scaling an effective AI framework can predictably reduce prediction error and expand what systems can do, while leaving practical limits in place.
- Predictable gains from scaling: Explain that experiments found relatively smooth power-law declines in prediction error as model parameters, training data, and computation increased, so improvement did not require a new learning algorithm each time.
Why it matters: This distinguishes progress from scaling an existing approach from progress that requires inventing a new algorithm. - Learning as uncertainty reduction: Describe how the section interprets modern AI through MacKay’s logic: data helps reduce uncertainty, while architectures and training methods turn that information into learned parameters and predictions.
Why it matters: It connects scaling to the chapter’s broader account of how AI learns from information. - Expanded but persistent limits: Explain how more usable data, computing infrastructure, and efficient parallel architecture push the boundaries of feasible learning outward without eliminating limits such as data quality, memory, energy, cost, and diminishing returns.
Why it matters: It prevents predictable scaling gains from being mistaken for unlimited progress.
Key terms: scaling laws, prediction error, power-law relationships, model parameters, training data, computation, MacKay’s unifying logic, uncertainty, learned parameters, predictions
10.6 Summary
Where this is used later
- Intellectual positioning (chapter 12) · MacKay’s role in synthesizing earlier learning ideas.
- Foundations carried forward (chapter 12) · MacKay’s role in synthesizing existing learning ideas
- Bayes, MacKay, and uncertainty (chapter 11) · The discussion of Bayes and uncertainty draws on the distinction between updating an assessment and eliminating uncertainty.
- Cross-entropy as likelihood and information (chapter 11) · The later discussion connects model-training loss to likelihood.
- Overfitting and regularization (chapter 11) · Provides an earlier example of balancing training fit against model capacity when discussing overfitting and regularization.
- From the 2015 synthesis to modern AI (chapter 11) · The unifying view of learning under constraints.
- From learned representations to relational architecture (chapter 13) · The significance of connecting foundational learning ideas
- The complete learning cycle (chapter 11) · Develops the complete neural network learning cycle introduced here.
Concepts explained in this chapter
- uncertaintyUncertainty is the lack of complete information about the state of a system, a quantity, or the outcome of an event.
- cross entropy lossCross-entropy loss measures the discrepancy between target outcomes and predicted probabilities.
- approximate inferenceApproximate inference estimates distributions or other inferential quantities when exact computation is impractical.
- constraintA condition that restricts the feasible choices or solutions in an optimization problem.
- scaling lawA scaling law describes how a system’s properties change as its scale or resource levels vary.
- probabilistic reasoningProbabilistic reasoning is inference that represents and updates beliefs using probabilities under uncertainty.
- message passingMessage passing is a procedure that propagates information between connected nodes to compute or approximate quantities in a network.
- backpropagationA backward pass propagates learning signals from a model’s output back through its components to determine their contributions to the loss.
- deep learningDeep learning is a machine-learning paradigm that learns representations through multiple parameterized computational layers.
- forward passThe computation that propagates input data through a model to produce its output.
- generalizationGeneralization is the ability of a learned model to perform effectively on observations not used during training.
- neural networkA neural network is a parameterized computational model composed of interconnected layers that transform inputs through learned functions.
- random forestA random forest is an ensemble model that combines predictions from multiple decision trees trained with randomized sampling or feature selection.
- self-attentionSelf-attention forms representations of sequence elements by relating each element to others in the same sequence.
- transformer architectureA transformer is a neural-network architecture that uses attention to model relationships among sequence elements.
- dynamic programmingDynamic programming is an algorithmic technique that solves problems by decomposing them into related subproblems and combining their solutions, often reusing computed results.
- hypothesis testingHypothesis testing, also called statistical testing, formally evaluates evidence against a null hypothesis under specified assumptions.
- level of significanceA significance level is the maximum probability of rejecting a null hypothesis when it is true, specified for a statistical test.
- null hypothesisA null hypothesis is a baseline statistical claim, commonly asserting that an effect or difference is absent.
- p-valueA p-value is the probability, under a null hypothesis, of observing a result at least as extreme as the one obtained.
- decision analysisDecision analysis formally evaluates choices by representing uncertain states, possible actions, consequences, and preferences.
- expected utilityExpected utility is the probability-weighted average desirability of the possible outcomes of an action.
- value of informationThe expected improvement in a decision outcome attributable to acquiring additional information before acting.
- decision boundaryA decision boundary is a geometric surface separating regions assigned to different predicted classes.
- marginA boosting margin measures the combined classifier's support for the correct class relative to a competing class.
- optimizationAn optimization method or algorithm is a procedure for searching for solutions that improve an objective function.
- regularization parameterRegularization strength is a setting that controls how strongly a regularization penalty or constraint affects model fitting.
- slack variableA nonnegative optimization variable measuring how much an observation violates a constraint, such as a classification margin.
- soft-margin extensionSoft-margin optimization finds a separating boundary by balancing margin width against penalties for margin violations.
- maximum likelihood estimationA maximum likelihood estimate is a parameter value that maximizes a statistical model’s likelihood function.
- Bayes' theoremBayes' theorem calculates a conditional probability from the reverse conditional probability and the relevant marginal probabilities.
- posterior probabilityA posterior is a probability or probability distribution updated after incorporating observed evidence.
- channel capacityThe maximum rate at which information can be transmitted reliably through a communication channel under specified conditions.
- entropyEntropy is a quantitative measure of uncertainty across the possible outcomes of an information source and their probabilities.
- informationInformation is a quantifiable reduction in uncertainty about the state or outcome of a variable or system.
- decoding strategyA procedure for selecting a generated token from a model’s next-token probability distribution.
- bootstrap samplingA resampling technique that draws observations with replacement from an original dataset.
- data aggregationAggregation is the combination of multiple observations, models, or outputs into a consolidated result.
- random feature selectionRandom feature selection restricts the features considered at a decision-tree split to a randomly chosen subset.
- varianceVariance is the model's sensitivity to changes in the training sample, reflected in how its behavior varies across samples.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
