What you will learn in Timeless Algorithms, The Foundational Ideas
Chapter by chapter
Chapter 1: Seeing inside the black box
The introduction explains why producing model results is not the same as judging them and presents foundational ideas as the layers beneath modern learning systems.
What you learn in chapter 1 →Chapter 2: From effect to cause: Thomas Bayes and the first algorithm of learning
The introduction presents Bayesian reasoning as a way to infer unseen causes from observations and revise beliefs as evidence accumulates, grounded in the rules of probability.
What you learn in chapter 2 →Chapter 3: The algorithm of estimation: Ronald Fisher and maximum likelihood
The introduction contrasts Fisher’s data-first approach with Bayesian inference and presents maximum likelihood as a way to estimate unknown parameters.
What you learn in chapter 3 →Chapter 4: From evidence to decisions: Neyman, Pearson, and the principles of hypothesis testing
The introduction explains why learning from data is not enough to make decisions and how the Neyman–Pearson framework adds explicit, advance rules for controlling errors.
What you learn in chapter 4 →Chapter 5: Entropy and the bit: Claude Shannon and the quantification of information
The introduction contrasts earlier ways of reducing uncertainty with Shannon’s way of measuring it, then shows why that shift mattered for communication.
What you learn in chapter 5 →Chapter 6: Dynamic programming: Richard Bellman and recursive optimization
The introduction frames dynamic programming as Bellman’s way to optimize interconnected decisions by reusing solutions to smaller problems.
What you learn in chapter 6 →Chapter 7: Beyond Bayesian inference: Raiffa, Schlaifer, and rational decision-making
The introduction explains how Raiffa and Schlaifer extended Bayesian inference into a framework for choosing actions and deciding whether more information is worth obtaining.
What you learn in chapter 7 →Chapter 8: Learning to generalize: Vladimir Vapnik and support vector machines
The introduction explains why SVMs were designed to generalize beyond training data, using wide margins and, when necessary, tolerated violations.
What you learn in chapter 8 →Chapter 9: From single trees to random forests: Leo Breiman and the logic of ensemble learning
This introduction explains why unstable decision trees can benefit from an ensemble and contrasts Breiman’s approach to generalization with Vapnik’s.
What you learn in chapter 9 →Chapter 10: From foundational algorithms to modern AI: David J.C. MacKay and the unifying logic of learning
The introduction presents MacKay’s synthesis of seemingly separate algorithms as ways of learning from incomplete evidence under practical constraints, then connects that synthesis to modern AI.
What you learn in chapter 10 →Chapter 11: Learning representations: LeCun, Bengio, Hinton, and the rise of deep learning
The introduction explains why deep learning became influential, centering on learned representations and the convergence of earlier ideas and practical advances.
What you learn in chapter 11 →Chapter 12: Beyond recurrence: Google Brain and the transformer architecture
The introduction explains why recurrent sequence models became a bottleneck and how the transformer’s attention-based architecture changed sequence representation and scaling.
What you learn in chapter 12 →Chapter 13: Expanding the boundaries of learning: OpenAI and the scaling laws of neural language models
This introduction connects earlier learning ideas and transformer architecture to evidence that scaling model parameters, data, and compute yields predictable gains while leaving important constraints in place.
What you learn in chapter 13 →Concepts explained in this book
- maximum likelihood estimationA maximum likelihood estimate is a parameter value that maximizes a statistical model’s likelihood function.
- self-attentionSelf-attention forms representations of sequence elements by relating each element to others in the same sequence.
- generalizationGeneralization is the ability of a learned model to perform effectively on observations not used during training.
- likelihoodA likelihood function measures how compatible observed data are with different parameter values of a statistical model.
- random forestA random forest is an ensemble model that combines predictions from multiple decision trees trained with randomized sampling or feature selection.
- transformer architectureA transformer is a neural-network architecture that uses attention to model relationships among sequence elements.
- entropyEntropy is a quantitative measure of uncertainty across the possible outcomes of an information source and their probabilities.
- decision ruleA decision rule is a specified mapping from observed evidence or a test statistic to an action or hypothesis-testing decision.
- decision treeA decision tree is a hierarchical predictive model that recursively partitions data using feature-based rules.
- cross entropy lossCross-entropy loss measures the discrepancy between target outcomes and predicted probabilities.
- hypothesis testingHypothesis testing, also called statistical testing, formally evaluates evidence against a null hypothesis under specified assumptions.
- neural networkA neural network is a parameterized computational model composed of interconnected layers that transform inputs through learned functions.
- uncertaintyUncertainty is the lack of complete information about the state of a system, a quantity, or the outcome of an event.
- parameterA parameter is a quantity that determines aspects of a model’s behavior.
- level of significanceA significance level is the maximum probability of rejecting a null hypothesis when it is true, specified for a statistical test.
- dynamic programmingDynamic programming is an algorithmic technique that solves problems by decomposing them into related subproblems and combining their solutions, often reusing computed results.
- expressive powerThe range and complexity of relationships that a model can represent.
- Bayesian inferenceBayesian inference updates prior probabilities or distributions about hypotheses or parameters with observed data.
- logitA logit is an unnormalized real-valued score that can be transformed into a probability.
- null hypothesisA null hypothesis is a baseline statistical claim, commonly asserting that an effect or difference is absent.
- marginA boosting margin measures the combined classifier's support for the correct class relative to a competing class.
- baggingBagging, or bootstrap aggregation, is an ensemble technique that trains models on bootstrap samples and combines their predictions to reduce variability.
- data representationA representation encodes informative properties of data in a form suitable for processing or interpretation.
- random feature selectionRandom feature selection restricts the features considered at a decision-tree split to a randomly chosen subset.
- scaling lawA scaling law describes how a system’s properties change as its scale or resource levels vary.
- inner productA dot product is an operation that maps two equal-length numerical vectors to the sum of their pairwise coordinate products.
- support vector machineA support vector machine is a supervised learning model that finds a class-separating boundary, typically by maximizing the margin.
- data aggregationAggregation is the combination of multiple observations, models, or outputs into a consolidated result.
- valueA projected representation of information that is combined with other values according to attention weights.
- backpropagationA backward pass propagates learning signals from a model’s output back through its components to determine their contributions to the loss.
- Bayes' theoremBayes' theorem calculates a conditional probability from the reverse conditional probability and the relevant marginal probabilities.
- soft-margin extensionSoft-margin optimization finds a separating boundary by balancing margin width against penalties for margin violations.
- tokenA token is a discrete unit of text represented in a model's vocabulary, such as a word, word fragment, or punctuation mark.
- attention weightAn attention weight is a normalized value indicating how strongly one element contributes to an attention-based representation.
- informationInformation is a quantifiable reduction in uncertainty about the state or outcome of a variable or system.
- feature importance scoreVariable importance is a quantitative assessment of how much a predictor contributes to a model's predictions or reduction of uncertainty.
- autoregressive generationAutoregressive generation is the process of producing a sequence by predicting one token at a time, conditioning each prediction on tokens already available.
- posterior probabilityA posterior is a probability or probability distribution updated after incorporating observed evidence.
- keyIn an attention mechanism, a projected representation of an element that is compared with queries to determine relevance.
- queryIn an attention mechanism, a projected representation of an element that is compared with keys to determine which information to attend to.
- embeddingAn embedding is a numerical vector representation of an item, such as a word or token, in a learned space.
- alternative hypothesisA statistical claim of an effect, difference, or relationship that is contrasted with the null hypothesis.
- sigmoidA sigmoid function is a smooth S-shaped function that maps real-valued inputs to values between zero and one.
- multiplication ruleThe multiplication rule is a probability rule for calculating the probability that multiple events occur together from joint or conditional probabilities.
- representation learningRepresentation learning transforms raw data into representations that retain information useful for downstream tasks.
- tokenizationThe process of dividing text into discrete units called tokens.
- logistic regressionLogistic regression is a statistical model that estimates the probability of an outcome from a weighted combination of input variables.
- kernel functionA function that computes an inner product in a feature space, allowing some learning methods to operate implicitly in that space.
- varianceVariance is the model's sensitivity to changes in the training sample, reflected in how its behavior varies across samples.
- critical regionA rejection region is the set of test-statistic outcomes for which a hypothesis-testing procedure rejects the null hypothesis.
- Neyman–Pearson frameworkA statistical framework for making hypothesis-testing decisions by controlling error rates under specified hypotheses.
- optimal substructureThe principle of optimality is the property that an optimal strategy can be formed from optimal decisions for the remaining stages after any state.
- model architectureAn architecture is the structural organization of components and operations in a computational model or system.
- slack variableA nonnegative optimization variable measuring how much an observation violates a constraint, such as a classification margin.
- activation functionAn activation function transforms a neural network unit’s combined inputs before the result is passed onward.
- deep learningDeep learning is a machine-learning paradigm that learns representations through multiple parameterized computational layers.
- p-valueA p-value is the probability, under a null hypothesis, of observing a result at least as extreme as the one obtained.
- hard-margin support vector machineA solution whose decision boundary maximizes the separation margin between classes.
- addition ruleThe addition rule calculates the probability that at least one of multiple mutually exclusive events occurs by summing their probabilities.
- correlationCorrelation measures the strength and direction of association between quantities.
- test statisticA test statistic is a numerical quantity calculated from data and used to assess a statistical hypothesis.
- attentionAn attention mechanism weights relationships among elements to form context-dependent representations.
- power-law relationshipA power-law relationship is one in which a quantity varies as a power of another quantity.
- prior probabilityA prior distribution represents uncertainty about an unknown quantity before incorporating observed data.
- expected utilityExpected utility is the probability-weighted average desirability of the possible outcomes of an action.
- conjugacyA prior distribution that yields a posterior in the same distribution family when updated with data under a particular likelihood.
- linear predictorA linear predictor is a weighted additive combination of input variables and coefficients that produces an intermediate model score.
- overlapping subproblemAn overlapping subproblem is a repeated smaller instance generated by the decomposition of a larger computational problem.
- interceptA bias term is an additive constant that shifts a model's output independently of its input features.
- bootstrap samplingA resampling technique that draws observations with replacement from an original dataset.
- optimizationAn optimization method or algorithm is a procedure for searching for solutions that improve an objective function.
- value of informationThe expected improvement in a decision outcome attributable to acquiring additional information before acting.
- tabulationTabulation is a dynamic-programming implementation technique that iteratively evaluates subproblems in a predetermined dependency order and stores their results.
- support vectorA support vector is a training example that lies on the edge of the separating margin and helps determine the decision boundary.
- computeA computational budget is the fixed amount of computation available for a task.
- thresholdA decision threshold is a cutoff used to convert a score or estimated probability into a decision or predicted class.
- objective functionA training objective is a quantity that a learning procedure seeks to optimize while fitting a model.
- parallelismParallel computation executes multiple computations concurrently across processing resources to reduce execution time or increase throughput.
- conceptual stackA conceptual layered architecture for explaining how data-driven systems represent data, structure relationships, learn, and interpret results.
- expected value of sample informationThe expected value of sample information is the expected improvement in decision outcome from imperfect information before its acquisition cost is deducted.
- false positiveA positive test result or classification when the tested condition is absent.
- neural network layerA layer is a stage in a neural network that transforms values received from a preceding stage.
- decoding strategyA procedure for selecting a generated token from a model’s next-token probability distribution.
- evidenceEvidence is information or observations considered in support of a claim or conclusion.
- regularization parameterRegularization strength is a setting that controls how strongly a regularization penalty or constraint affects model fitting.
- one-sided testA one-tailed test is a hypothesis test whose rejection region lies in one direction of the test-statistic distribution.
- base caseA base case is a directly known computational state that requires no further recursive decomposition.
- independenceAn independent event is an event whose probability is unaffected by the occurrence or outcome of another event.
- information gainInformation gain is the reduction in uncertainty produced by a split or observation, commonly measured using entropy.
- constraintA condition that restricts the feasible choices or solutions in an optimization problem.
- oddsOdds are the ratio of the number or probability of favorable outcomes to the number or probability of unfavorable outcomes.
- probability distributionA probability distribution assigns nonnegative probabilities to possible outcomes so that the probabilities over the defined outcome space sum to one.
- attention headA component of an attention architecture that evaluates relationships among sequence elements, potentially focusing on a different type of relationship from other heads.
- validation setHeld-out data used to assess model performance and guide model selection during development.
- cross-validationCross-validation is a model evaluation technique that repeatedly withholds portions of data from fitting and assesses performance on those held-out observations.
- gradient descentGradient descent is an iterative optimization technique that updates model parameters in the direction that reduces an objective function.
- critical valueA critical value is a boundary value of a test statistic that separates the rejection region from the non-rejection region.
- masked attentionCausal masking is a restriction on attention that prevents a sequence position from accessing subsequent positions.
- permutation feature importancePermutation importance measures a feature's contribution to a model by assessing the effect of permuting its values on predictive performance.
- stabilityStability is the persistence of patterns and relationships in data under comparable conditions or over time.
- redundancyRedundancy is the deliberate inclusion of repeated or structured information to enable reliable recovery of a message despite errors or noise.
- leaf nodeA terminal node is a node marking the end of a decision-tree path and recording its resulting outcome.
- stateA representation of a problem configuration that captures information needed to determine subsequent choices and outcomes.
- diminishing returnsThe principle that successive increases in an input produce progressively smaller incremental gains in an outcome.
- composite hypothesisA hypothesis that permits multiple parameter values or probability models.
- simple hypothesisA simple hypothesis specifies one exact value for an unknown parameter or one fully specified probability model.
- out-of-bag evaluationOut-of-bag evaluation estimates a model’s predictive performance using observations omitted from the samples used to train the models making those predictions.
- binomial distributionThe binomial distribution gives the probabilities of numbers of successes in a fixed number of independent trials with a constant success probability.
- evaluation datasetA test set is data kept separate from model fitting and selection for estimating the performance of a completed model.
- log-likelihoodA log-likelihood is the logarithm of a likelihood function, converting products of probabilities into sums for analysis and optimization.
- forward passThe computation that propagates input data through a model to produce its output.
- overfittingOverfitting occurs when a model fits its training observations too closely without capturing relationships that remain useful beyond them.
- bitA bit is a binary unit of information that can take one of two possible values.
- decision boundaryA decision boundary is a geometric surface separating regions assigned to different predicted classes.
- meanExpected value is the probability-weighted average of the possible numerical outcomes of a random variable.
- empirical riskEmpirical risk is the average loss a model incurs on the observations in a given sample.
- recurrent neural networkA neural network that processes sequential inputs using a state carried across sequence positions.
- BICThe Bayesian information criterion is a likelihood-based model-selection score that penalizes model complexity, with a penalty that increases with sample size.
- t-testA t-test is a statistical test for comparing means using a t-distribution under specified assumptions.
- two-tailed testA two-tailed test is a hypothesis test whose rejection region includes extreme values in both directions of the test-statistic distribution.
- tradeoffA trade-off is a decision situation in which improving one outcome requires accepting a reduction in another.
- recurrenceA rule that defines a quantity in terms of its values for other instances.
- hyperplaneA hyperplane is a flat subspace of one dimension less than its ambient space.
- feedforward networkA neural network component that applies learned transformations independently at each sequence position.
- token idAn integer index that identifies a token in a model’s vocabulary.
- dependent eventA dependent event is an event whose probability is affected by the occurrence or outcome of another event.
- impurity-based importanceImpurity-based importance is a measure that aggregates the impurity reductions attributed to a feature across a model's tree splits.
- independent variableA feature is an input attribute whose values are considered by a predictive model when making predictions.
- naive bayes classifierA naive Bayes classifier is a probabilistic classification model that applies Bayes' theorem while assuming features are conditionally independent given the class.
- binary random variableA Bernoulli random variable has two possible values, conventionally 0 and 1.
- hyperparameterA hyperparameter is a setting that controls a model or learning procedure and is selected outside ordinary parameter fitting.
- kernel trickThe kernel trick computes inner products in an implicitly transformed feature space without explicitly constructing the transformed feature vectors.
- net value of sample informationThe net value of sample information is the expected value of sample information after subtracting the cost of acquiring the information.
- statistical learning theoryStatistical learning theory is a mathematical framework for analyzing how data, model capacity, and learning procedures affect predictive performance.
- validation lossHeld-out loss is the value of a loss function evaluated on observations excluded from model training.
- complement ruleThe complement rule states that the probability of an event not occurring is one minus the probability of that event occurring.
- Bayesian A/B testingBayesian A/B testing compares alternatives by updating probability distributions about their performance using observed data.
- Bayesian networkA Bayesian network is a probabilistic graphical model that represents variables and their conditional dependencies using a directed acyclic graph.
- Bayesian optimizationBayesian optimization is a method for optimizing an expensive or black-box objective by using a probabilistic surrogate model to guide evaluations.
- Thompson samplingThompson sampling is a Bayesian sequential decision algorithm that samples from posterior distributions over alternatives and selects the alternative with the highest sampled value.
- actionA choice available to a decision-maker in a decision problem.
- causationCausality, or causation, is a relationship in which a change in one factor brings about a change in another.
- gradientA gradient is a vector of partial derivatives that describes the direction and rate of change of a function with respect to its inputs.
- knapsack problemAn optimization problem that selects items to maximize total value subject to a capacity constraint.
- pruningPruning is a technique that reduces a decision tree by removing branches to limit its size or improve its predictive behavior.
- model weightA learned numerical coefficient that controls an input’s contribution in a neural network.
- clusteringClustering is an unsupervised modeling technique that groups observations according to similarity without predefined labels.
- channel capacityThe maximum rate at which information can be transmitted reliably through a communication channel under specified conditions.
- self-supervised learningSelf-supervised learning trains a model using supervisory signals derived from the data itself rather than manually supplied labels.
- binomial testA binomial test is an exact statistical test for assessing whether the observed number of successes in fixed independent binary trials is consistent with a specified success probability.
- VC dimensionThe VC dimension of a hypothesis class is the largest number of points it can shatter, measuring its capacity to realize distinct labelings.
- multi-head attentionAn attention mechanism that computes multiple attention patterns in parallel using distinct learned projections and combines their outputs.
- feature engineeringFeature engineering is the process of transforming raw data into input features suitable for statistical or machine-learning models.
- efficiencyEfficiency describes how small an estimator's sampling variance is relative to competing estimators or an information-based bound.
- decision frameworkAn organized structure of methods and assumptions for evaluating choices and selecting actions.
- sufficient statisticA sufficient statistic is a statistic that contains all information in a sample relevant to a specified parameter under a specified model.
- statistical significanceStatistical significance is an assessment of whether an observed result is sufficiently unlikely under a specified null hypothesis.
- out-of-bag observationAn out-of-bag observation is a training observation omitted from a particular bootstrap sample and evaluated using trees that did not include it.
- squared errorSquared-error loss measures prediction error by squaring the difference between a predicted and observed value.
- rollbackA procedure for evaluating a decision tree in reverse, combining probabilities and utilities at chance nodes and selecting the best branch at decision nodes.
- ordinary least squaresOrdinary least squares estimates regression parameters by minimizing the sum of squared differences between observed and predicted responses.
- estimator consistencyConsistency is the property that an estimator converges to the true parameter value as the amount of data increases under specified conditions.
- polynomial kernelA polynomial kernel is a kernel function whose value is based on a polynomial of the input observations’ inner product.
- radial basis function kernelA radial basis function kernel is a kernel function whose similarity decreases with distance between observations according to a radial basis function.
- context windowThe finite amount of input context a model can consider when producing an output.
- semantic associationA semantic association is a learned relationship between concepts that makes one concept more likely to be recalled or predicted in the presence of another.
- treatment thresholdA treatment threshold is the probability of a condition at which treating and not treating have equal expected utility.
- data driftA distribution shift is a change in the input data distribution between the conditions under which a model was trained and those under which it is evaluated or applied.
- z-scoreA z-score is a standardized value expressing a measurement's distance from a mean in units of standard deviation.
- z-testA z-test is a statistical hypothesis test that uses a standardized statistic with a standard normal reference distribution under specified assumptions.
- forest marginA forest margin is the difference between a forest's voting support for an observation's correct class and its strongest competing class.
- strengthTree strength measures the predictive ability of individual trees in a forest.
- feature selectionFeature selection identifies a subset of predictors that contributes useful information while excluding predictors that add little value.
- probabilityProbability is a numerical measure representing the likelihood of an event or outcome conditional on information and assumptions.
- bootstrap sampleA dataset formed by drawing observations with replacement from an original dataset.
- expected value of perfect informationExpected value of perfect information is the expected improvement in utility from knowing the true state before choosing an action.
- transitionA transition is a rule that maps a current state and a decision to a resulting state in a multi-stage computation.
- recallThe true positive rate is the proportion of actually positive cases correctly identified.
- decision theoryThe formal study of how decision-makers choose among actions under uncertainty and in light of preferences or objectives.
- data augmentationData augmentation creates additional training observations by transforming existing observations while preserving their correct outcomes.
- dropoutDropout is a neural-network regularization technique that randomly sets selected unit outputs to zero during training.
- early stoppingEarly stopping is a regularization technique that halts training when further learning is likely to harm performance on held-out data.
- weight decayWeight decay is a regularization technique that penalizes large weights during model training.
- bayesian neural networkA Bayesian neural network is a neural network whose weights and biases are represented by probability distributions, allowing predictions to account for parameter uncertainty.
- cluster entropyCluster entropy measures the uncertainty in category membership among the observations within a cluster.
- normalized mutual informationNormalized mutual information is a normalized measure of the association between cluster assignments and known categories.
- purityPurity measures the extent to which each cluster is dominated by a single known category.
- combinationA combination is a selection or grouping in which order does not matter.
- depthNetwork depth is the number of successive computational layers in a neural network.
- widthNetwork width is the number of computational units in a neural network layer.
- data qualityData quality is the degree to which data are accurate, relevant, consistent, and sufficiently complete for their intended use.
- support vector formThe support-vector form is a formulation of a support vector machine's decision function as a sum of contributions from support vectors and a bias term.
- contextual embeddingA contextual representation is a numerical encoding of a token or sequence that incorporates information from its surrounding context.
- model calibrationProbability calibration adjusts model scores so that predicted probabilities correspond to observed outcome frequencies.
- standard deviationThe square root of variance, measuring dispersion in the same units as the quantity described.
- chance nodeA node in a decision tree representing an uncertain event that the decision-maker cannot control.
- decision nodeA node in a decision tree representing a point at which the decision-maker chooses among available actions.
- pre-activationA pre-activation value is the numerical result computed at a neural-network node before an activation function transforms it.
- probabilistic reasoningProbabilistic reasoning is inference that represents and updates beliefs using probabilities under uncertainty.
- decision analysisDecision analysis formally evaluates choices by representing uncertain states, possible actions, consequences, and preferences.
- regularizationRegularization is a method that constrains or penalizes a model to limit complexity and improve estimation, especially when data is limited.
- foundation modelA foundation model is a model trained on broad data at large scale and adaptable to multiple downstream tasks.
- multimodal trainingMultimodal learning models relationships among information represented in different modalities, such as images, text, audio, and video.
- decision makingDecision-making is the process of selecting an action or outcome under available information, constraints, and uncertainty.
- error rateAn error rate is the proportion of predictions or classifications that are incorrect.
- margin distributionA margin distribution is the distribution of classification-margin values across labeled observations.
- false-positive rateThe false positive rate is the proportion of actually negative cases incorrectly classified as positive.
- problem definitionProblem formulation is the process of specifying a problem's states, decisions, constraints, objectives, and relationships in a form suitable for analysis or computation.
- beta distributionA continuous probability distribution on values from zero to one, parameterized by two positive shape parameters.
- model complexityModel complexity is the degree of structural or parametric elaboration in a model that affects its flexibility and risk of fitting noise.
- model selectionModel selection is the process of choosing among candidate models based on their assumptions, representations, and expected behavior.
- ReLUA rectified linear unit (ReLU) is an activation function that outputs the greater of zero and its input.
- decoderA communication-system component that reconstructs a transmitted message from its encoded form.
- encoderA communication-system component that converts a message into a form suitable for transmission.
- ensembleEnsemble learning combines multiple predictive models to improve predictions.
- independent and identically distributedIndependent and identically distributed describes random variables that are mutually independent and share the same probability distribution.
- approximate inferenceApproximate inference estimates distributions or other inferential quantities when exact computation is impractical.
- mutual informationMutual information is a measure of the reduction in uncertainty about one variable provided by observing another variable.
- change-making problemThe coin change problem is the computational problem of determining ways to form a target amount from a specified set of coin denominations, typically with a counting or optimization objective.
- model diversityEnsemble diversity is the extent to which constituent models differ in their structures, predictions, or errors.
- boostingBoosting is an ensemble technique that builds models sequentially, with later models focusing on cases earlier models handled poorly.
- experimentationAn experiment is an information-gathering procedure that can reveal something about uncertain states before a decision.
- hidden stateA hidden state is an internal representation that a recurrent neural network carries forward to summarize information from earlier sequence elements.
- next-word predictionA language-modeling objective in which a model learns to predict the next token from preceding tokens in a sequence.
- decision functionA decision function is a mathematical function that assigns a score to an input for determining its predicted class.
- hypothesis spaceA hypothesis class is the set of candidate functions or decision boundaries a learning method is permitted to select from.
- information theoryInformation theory is the mathematical study of information, uncertainty, and communication, including measures of entropy and error.
- reinforcement learningReinforcement learning is a machine-learning approach in which an agent learns behavior through interaction with an environment and feedback about its actions.
- training datasetTraining data is the dataset used to fit a model.
- base rate fallacyThe base rate fallacy is the error of neglecting an event's underlying prevalence when interpreting evidence about it.
- law of total probabilityThe law of total probability calculates an event's overall probability by summing its probabilities across mutually exclusive and collectively exhaustive cases.
- assumptionA mathematical assumption is a premise about mathematical properties or conditions required for a result to hold.
- one-class svmA one-class support vector machine estimates a region containing most training observations and identifies observations outside it as unusual.
- Gini impurityGini impurity measures the probability of misclassifying an observation when its class is assigned randomly according to the class distribution in a node.
- internal nodeAn internal node is a decision-tree node that tests a feature and directs observations along branches.
- statistical powerStatistical power is the probability that a test rejects the null hypothesis when a specified alternative condition is true.
- reasoning under uncertaintyReasoning under uncertainty is inference and decision-making when available information does not determine a single certain conclusion.
- sampling without replacementSampling without replacement is a sampling procedure in which each selected item is not returned to the sample space before the next trial.
- nonlinear mappingNonlinearity is the property of a transformation that does not preserve linear combinations, allowing neural networks to represent relationships beyond repeated linear combinations.
- parameter countModel size is the scale of a model's structure or parameterization.
- convex optimizationConvex optimization is optimization in which the objective and feasible region have convex structure, allowing local optima to be global optima under standard conditions.
- memoizationMemoization is a dynamic-programming implementation technique that recursively evaluates only needed subproblems and caches their computed values.
- linear kernelA linear kernel computes similarity using the ordinary dot product of observations in the supplied feature space.
- splitA decision-tree split partitions observations at a node according to a feature-based rule.
- batchingA mini-batch is a small subset of training examples processed together during an iteration of model training.
- support vector regressionSupport vector regression is a regression model that fits a function while controlling prediction errors through a tolerance margin.
- tanhThe hyperbolic tangent function maps real-valued inputs to values between −1 and 1 and is centered at zero.
- reward signalA reward is a numerical consequence associated with an action or state transition that represents its desirability in a decision process.
- transition probabilityThe probability of a particular next state given a current state and action.
- prosecutor's fallacyThe prosecutor's fallacy is the error of confusing the probability of evidence given a hypothesis with the probability of the hypothesis given the evidence.
- estimatorAn estimator is a rule or statistic that maps observed data to an estimate of an unknown parameter.
- linear regressionLinear regression is a statistical model that represents a continuous response as a linear function of predictors and model parameters.
- learning rateA learning rate controls the size of parameter updates during iterative model training.
- state of natureA possible condition outside a decision-maker’s control that affects the outcome of an act.
- p-hackingP-hacking is the practice of selectively changing analyses or reporting choices to obtain apparently statistically significant results.
- Bellman equationThe Bellman equation is a recursive equation that expresses the value of a decision state in terms of immediate reward and the value of subsequent states.
- false negativeA negative test result or classification when the tested condition is present.
- linearly separableLinear separation is the property that classes can be divided by a linear decision boundary in a feature space.
- empirical risk minimizationEmpirical risk minimization selects a model from a permitted class by minimizing its average loss on observed data.
- structural risk minimizationStructural risk minimization is a statistical learning framework that balances training accuracy against model capacity to promote generalization.
- independent repetitionIndependent repetition is the repeated performance of an experiment under identical conditions with independent outcomes across trials.
- softmaxThe softmax function transforms a vector of real-valued scores into a probability distribution.
- backtrackingBacktracking is a technique that reconstructs a solution by tracing stored decisions backward from a final state.
- opportunity lossOpportunity loss is the loss incurred by choosing an action instead of the best action that would have been selected with the relevant information.
- practical importancePractical significance is the real-world importance or usefulness of an effect, independent of whether it meets a statistical significance threshold.
- maximax methodMaximax is a decision heuristic that selects the option with the highest possible outcome.
- maximin methodMaximin is a decision heuristic that selects the option with the best worst-case outcome.
- minimax regret methodMinimax regret is a decision heuristic that selects the option minimizing the worst possible regret relative to the best alternative in hindsight.
- neuronA neuron is a computational unit in a neural network that combines inputs and produces an output.
- specificitySpecificity is the probability that a test gives a negative result when the condition being tested for is absent.
- feature spaceA feature space is the geometric space whose dimensions correspond to the features used to describe observations.
- kernel principal component analysisKernel principal component analysis applies principal component analysis in a feature space induced by a kernel to capture nonlinear structure.
- loss surfaceA loss surface is a geometric representation of how a model's loss varies across possible parameter values.
- Neyman–Pearson lemmaThe Neyman–Pearson lemma states that a likelihood-ratio rejection rule is most powerful for testing a simple null hypothesis against a simple alternative at a specified significance level.
- hierarchical clusteringHierarchical clustering constructs nested groups by successively merging or splitting clusters.
- K-means clusteringK-means clustering partitions observations into a specified number of groups by assigning them to the nearest centroid and iteratively updating the centroids.
- weighted sumA linear combination is a sum of quantities multiplied by scalar coefficients.
- riskRisk is the possibility of an undesirable outcome together with uncertainty about its occurrence or impact.
- exhaustive event setCollective exhaustiveness is the property of a set of outcomes that together they cover every possibility in a defined outcome space.
- mutually exclusive eventMutually exclusive events are events that cannot occur simultaneously.
- margin-based reasoningMargin-based learning emphasizes the strength of separation between classes when selecting a classifier.
- Markov chainA Markov chain is a stochastic model of transitions among states in which the next state depends only on the current state.
- preposterior analysisPreposterior analysis is decision analysis conducted before information is obtained, evaluating possible results and their consequences for later choices.
- terminal analysisTerminal analysis is decision analysis conducted after information is observed, using the resulting beliefs to choose an act.
- epochAn epoch is one complete pass through a training dataset during model training.
- utility functionUtility is a numerical representation of the desirability or preference associated with an outcome.
- beta-binomial distributionThe beta-binomial model combines a Beta prior for an unknown success probability with a binomial likelihood for observed trial outcomes.
- frequentist paradigmFrequentist statistics interprets probability through long-run frequencies of repeated trials under fixed conditions.
- Cramer-Rao lower boundThe Cramér–Rao lower bound is a theoretical lower limit on the variance of unbiased estimators under specified regularity conditions and a probability model.
- out-of-bag errorOut-of-bag error estimates prediction error using observations omitted from the bootstrap samples used to train the relevant trees.
- interpretabilityInterpretability is the degree to which a model’s behavior or predictions can be understood by people.
- state spaceA state space is the collection of distinct configurations that can occur in a problem or dynamical process.
- convolutional neural networkA convolutional neural network is a neural network architecture that uses convolutional operations to learn patterns in structured data such as images.
- permutationA permutation is an arrangement of objects or outcomes in which order matters.
- coding theoryCoding is the systematic representation of messages as symbols or codewords for efficient transmission, storage, or error protection.
- message passingMessage passing is a procedure that propagates information between connected nodes to compute or approximate quantities in a network.
- method of momentsThe method of moments estimates parameters by equating theoretical distribution moments with corresponding sample statistics.
- end-to-end learningEnd-to-end learning is an approach in which a single objective guides the joint learning of parameters throughout a model.
- combinatoricsCombinatorics is the branch of mathematics concerned with counting, arrangement, and selection of discrete objects.
- marginal probabilityA marginal probability is the probability assigned to a value of one variable after accounting for the possible values of other variables.
- Markov decision processA Markov decision process is a model of sequential decisions in which actions influence transitions among states and rewards, with future dynamics depending on the current state and action.
- chain ruleThe chain rule is a calculus rule for differentiating a composite function by multiplying the derivatives of its component functions.
- cumulative rewardExpected cumulative reward is the probability-weighted expected total of rewards accumulated over a decision process.
- binomial coefficientA binomial coefficient counts the number of ways to choose a specified number of items from a larger set, without regard to order.
- capacity controlCapacity control limits the complexity of a learning model to help manage its generalization behavior.
- stochastic gradient descentStochastic gradient descent is an optimization method that updates model parameters using gradients computed from individual observations or small batches.
- biasBias is a systematic deviation that can consistently distort data, estimates, or predictions.
- node impurityImpurity is a measure of how mixed the outcomes or classes are within a decision-tree node.
- conditional probabilityConditional probability is the probability of an event given that another event or condition is known to hold.
- degree of beliefA belief is a degree of credence assigned to a proposition or state of the world.
- compressionEncoding information using fewer bits or symbols while preserving what is needed for its recovery or use.
- sequential decision-makingSequential decision-making is the selection of actions over multiple stages where each action affects later states, options, or outcomes.
- Fisher information matrixFisher information is a quantity measuring how much information a probability model and its data provide about an unknown parameter.
- markov chain monte carloMarkov chain Monte Carlo is a family of algorithms that uses Markov chains to generate samples from probability distributions.
- hidden layerA hidden layer contains neural network values that are neither supplied as inputs nor observed as outputs.
- hidden Markov modelA hidden Markov model is a probabilistic model of sequences in which hidden states evolve according to transition probabilities and generate observable outputs.
- kernel methodA kernel method is a computational technique that uses a kernel function to represent relationships between data points without explicitly constructing their feature representations.
- AICThe Akaike information criterion is a model selection criterion that estimates relative information loss while balancing fit against model complexity.
- policyA policy is a rule that specifies how decisions are selected as a process moves through stages or states.
- likelihood ratio testA likelihood ratio test compares nested statistical models using their maximized likelihoods to assess whether the more complex model provides a significant improvement.
- learning systemA learning system is a computational system that derives or adjusts a model from data or experience.
- negative log-likelihoodA negative log-likelihood is the negative logarithm of a likelihood, commonly minimized as an objective equivalent to maximizing likelihood.
- tokenA next token is the linguistic unit that follows the current context in a sequence being modeled.
- approximationApproximation is a technique for obtaining a tractable solution by accepting some deviation from an exact result.
- trainingTraining is the process of fitting a model by adjusting its parameters using data.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
