liveBookmanning.com

What you will learn in Machine Learning System Design

With end-to-end examples

by Valerii Babushkin and Arseny Kravchenko

Read it on liveBook Get the book at manning.com

Chapter by chapter

Concepts explained in this book

• specificationA structured document that records a system's goals, priorities, requirements, and design decisions.
• pipelineAn ordered set of computational steps and dependencies that accomplishes a defined process.
• driftModel drift is a change over time in a machine learning model’s behavior or performance caused by changes in its inputs, relationships, or operating environment.
• performanceA metric is a defined quantitative measure used to evaluate performance, behavior, or outcomes.
• baselineA feature baseline is a baseline that makes predictions from a selected set of input features using a comparatively simple modeling approach.
• class imbalanceA condition in which the frequencies of classes in a classification dataset or population differ substantially.
• similarity searchThe field concerned with finding and ordering relevant information items in response to queries.
• conversionThe proportion of entities that complete a specified transition or desired action among those eligible to do so.
• proxy metricA measurable quantity that serves as an indirect approximation of a desired outcome that is difficult to measure directly.
• machine learning systemA software system that incorporates machine learning models together with the data, infrastructure, and operational components required to produce behavior in an application or production context.
• seasonality effectA recurring pattern of variation in data or demand associated with a regular calendar or time cycle.
• importance scoreA quantitative attribution describing how much an input feature contributes to a model's output or predictive performance.
• inputAn input variable or derived input representation provided to a machine-learning model.
• monitoringThe practice of recording timestamped events, inputs, outputs, and contextual identifiers generated by a software system.
• data driftA discrepancy between the statistical characteristics of data used for model development and those of data encountered in the target operating environment.
• concept driftA change over time in the relationship between input features and the target outcome, making previously learned patterns invalid.
• objective functionA loss is a numerical value representing the discrepancy between a machine learning model's output and a target value for an instance or collection of instances.
• recallThe proportion of actual positive cases that a model correctly predicts as positive.
• learning curveA curve showing how a model's loss changes across training progress or another ordered evaluation dimension.
• machine learning system designThe practice of architecting, building, and evaluating software systems whose behavior depends on machine learning models and their surrounding data and operational components.
• datasetA dataset is an organized collection of data records or examples used for analysis, training, evaluation, or other computational purposes.
• labelReference labels or outcomes regarded as the actual values against which model predictions are evaluated.
• consistencyA quantitative measure of how consistently a model or machine learning system produces similar outputs under specified changes to inputs, data, or training conditions.
• feature selectionFeature selection is the process of choosing a subset of available input features for a machine learning model.
• data qualityData quality is the degree to which data is accurate, complete, valid, consistent, timely, and suitable for its intended purpose.
• scalabilityScalability is the ability of a system to maintain acceptable operation as workload, data volume, or users increase.
• decompositionDecomposition is a release strategy that allows different system components to be deployed and managed independently.
• metadataData that describes the content, structure, context, provenance, or other attributes of data.
• data annotationThe process of assigning annotations or target values to data examples for machine learning.
• thresholdA threshold is a cutoff value used to convert a score or probability into a discrete decision or category.
• metric hierarchyAn organized structure that decomposes a high-level objective into progressively more specific measurable metrics.
• offline metricAn offline metric is a quantitative measure evaluated on fixed historical, validation, or test data rather than through live system interaction.
• residual errorThe difference between a model prediction and its corresponding true value, especially in regression and more broadly as a sample-wise prediction error.
• linear regressionA linear model predicts an output as a linear combination of input features, optionally followed by a task-specific link or threshold function.
• relevancy scoreA numerical score expressing how relevant an item is to a particular user query.
• classifierA machine-learning task that assigns an input to one or more discrete categories.
• constant baselineA constant baseline is a baseline model that produces a fixed or stepwise fixed output rather than learning a complex mapping from inputs to predictions.
• demand forecastDemand forecasting is the process of estimating future demand for a product or service over a specified time period.
• foundational modelA machine learning model whose parameters were learned previously on a dataset or task before being applied or adapted to another task.
• language modelA large language model is a machine learning model trained on large text corpora to generate or transform language and perform related language tasks.
• migrationA data integration process that extracts data from sources, transforms it, and loads the result into a destination.
• validationA procedure for partitioning or otherwise structuring data and evaluation conditions to estimate a machine learning system's performance under expected operating conditions.
• retrainingRetraining is the process of training a machine learning model again, typically with changed data, initialization, or training conditions.
• contractA defined interface through which software components request functionality or exchange data.
• feature engineeringFeature engineering is the process of constructing, transforming, or selecting input features for a machine learning model.
• key performance indicatorA key performance indicator that quantitatively measures progress toward an organization's business objective.
• experimentationAn A/B test is a controlled experiment that compares outcomes for users or units exposed to two alternative variants.
• quantile loss functionA quantile loss function is an asymmetric loss function whose minimizer is a specified quantile of the predictive distribution.
• versioningModel versioning is the practice of assigning and managing distinct identifiers for machine learning model revisions.
• accountabilityAccountability is the assignment of responsibility for specific decisions, tasks, system components, or outcomes.
• metricAn evaluation measure that can be calculated only after a system is deployed and its output is used in a real business context.
• evaluation datasetA designated partition of data reserved for evaluating or tuning a machine learning model during development.
• redundancyThe condition in which multiple input features provide overlapping or duplicative information.
• dataset splitA procedure or method for partitioning data into separate subsets for training, validation, testing, or comparative evaluation.
• human-in-the-loopJudgments or preference signals supplied by people about the quality or desirability of system outputs.
• mean reciprocal rankA ranking metric equal to the average reciprocal rank of the first relevant result across queries.
• normalized discounted cumulative gainA ranking metric that discounts relevance gains by position and normalizes them by the ideal discounted cumulative gain.
• meanThe arithmetic average of a set of numerical values.
• interpretabilityThe property of a model or automated decision-making system whose behavior and rationale are visible or understandable to relevant people.
• mean absolute errorThe arithmetic mean of the absolute differences between predictions and actual values.
• mean squared errorThe arithmetic mean of squared differences between predictions and actual values.
• false-positive rateThe proportion of actual negative cases that a model incorrectly predicts as positive.
• precisionThe proportion of predicted positive cases that are actually positive.
• specificityThe proportion of actual negative cases that a classifier correctly identifies.
• build-versus-buy decisionA decision process for choosing between developing a capability internally and obtaining it from an external provider.
• robustnessA software quality property of continuing to operate and produce some result despite errors or abnormal conditions, even when the result may be inaccurate.
• minimum detectable effectThe smallest effect size that an experiment is designed to detect with specified statistical power and significance criteria.
• two-stage systemA machine-learning architecture that processes candidates through two sequential stages, typically retrieval followed by ranking.
• on-call scheduleAn on-call schedule is a timetable specifying which people or teams are responsible for incident response during particular shifts.
• processing cascadeA cascade is a multi-stage retrieval or decision process in which successive stages progressively filter, retrieve, and rank candidates.
• fallbackA fallback is an alternative behavior or implementation used when the primary behavior or system cannot operate normally.
• lagA feature or time-series value obtained from an earlier time point relative to the prediction time.
• maintenanceThe ongoing work of keeping existing system functionality working as its environment changes.
• data sourceData origin is the source, generating process, or provenance from which data is obtained.
• biasA systematic error caused by a model's limited capacity to capture useful signals, leading it toward simplified assumptions about the data.
• feature storeAn infrastructure system for storing, managing, and serving machine-learning features.
• point estimateA single numerical estimate of a population quantity or model-performance measure without an associated uncertainty range.
• quantile metricA performance measure that evaluates forecasts for a specified quantile of a predictive distribution.
• overrideAn explicitly configured result or behavior that supersedes an automated system decision for specified inputs.
• RACI matrixA RACI matrix is a responsibility-assignment framework that identifies who is responsible, accountable, consulted, and informed for each task or component.
• time windowThe number of observations included in a time-series training or testing window.
• complexityThe degree of structural or parametric richness in a machine learning model that determines the range of functions it can represent.
• click-through rateThe proportion of users who click a displayed search result or other presented link.
• backward compatibilityThe ability of new software or an interface version to support existing code or clients without changes.
• probability distributionA statistical pattern describing the frequencies or probabilities of values in a population or dataset.
• preprocessingPreprocessing is the transformation and preparation of data before it is supplied to a machine learning model for training or inference.
• cold start problemA situation in which a system lacks sufficient initial data or interaction history to provide effective predictions, recommendations, or other outputs.
• evaluationModel evaluation is the process of measuring a machine learning model's performance using metrics on data separate from fitting data.
• error analysisA systematic examination of prediction errors to identify their distributions, causes, patterns, and implications for model improvement.
• securitySecurity is the protection of a system, its models, data, and interfaces from unauthorized access, misuse, or compromise.
• reportThe process of presenting measurements, analyses, or system outcomes to inform evaluation and decision-making.
• rule-based baselineA rule-based baseline is a baseline model that produces predictions through explicit conditional rules or category-specific statistics rather than learned general-purpose parameters.
• keyword search indexA search index is a data structure that organizes searchable representations so that relevant items can be retrieved efficiently.
• data quality monitoringA check that evaluates whether data conforms to expected formats, ranges, completeness, or consistency constraints.
• predictabilityPredictability is the property that a system's behavior and outcomes can be reliably anticipated from its inputs and conditions.
• simplicitySimplicity is the property of having relatively few concepts, dependencies, or implementation details that must be understood or managed.
• costCompute cost is the computational resource expenditure required to execute a model or system operation.
• adaptive request batchingDynamic batching is an inference-serving technique that combines requests arriving within a configured time window into a batch for execution.
• cachingA storage mechanism that keeps copies of data or results so repeated requests can be served without recomputation or retrieval from the original source.
• elasticity plotA curve representing how demand or another response variable changes as an influencing variable changes.
• batch inferenceThe process of applying a machine learning model to a collected batch of inputs rather than processing each input continuously.
• determinismDeterministic behavior is behavior in which the same relevant inputs and state produce the same output.
• forward compatibilityThe ability of older software or an interface version to work with new code or clients without changes.
• leaky abstractionA leaky abstraction is an abstraction that exposes underlying implementation details that its interface is intended to hide.
• nondeterminismNondeterminism is the property that identical apparent inputs may produce different outcomes because of uncontrolled state, data, timing, or randomness.
• underfittingA condition in which a model is too simple to capture important relationships between features and the target variable, resulting in poor performance on both training and new data.
• weighted mean absolute percentage errorA forecast-error metric calculated as the sum of absolute errors divided by the sum of actual values, expressed as a percentage.
• postmortemA postmortem is a structured analysis of an incident and its causes, effects, and follow-up actions.
• proportional responseProportional response is an incident-management principle in which postincident actions are scaled according to the failure’s impact and likelihood of recurrence.
• runbookA runbook is an operational document containing procedures for diagnosing and resolving recurring system problems.
• A/A testAn A/A test is an experiment that assigns participants to two equivalent variants to check experimental instrumentation and statistical behavior.
• cross entropy lossA classification loss that penalizes incorrect probabilistic predictions using the negative logarithm of the probability assigned to the true class.
• bus factorBus factor is the number of key contributors whose loss could prevent a project or system from continuing effectively.
• uptimeThe property that required data and related systems can be obtained or accessed reliably by a system and its engineers.
• privacyThe protection of information about individuals or entities from unauthorized collection, use, disclosure, or inference.
• antigoalAn explicitly identified outcome or condition that a system design should avoid.
• provenanceData lineage is the record of data origins, transformations, dependencies, and movement through processing systems.
• model exportModel conversion is the transformation of a trained model into a representation or format supported by another execution framework or deployment environment.
• documentationDocumentation is a maintained collection of information that describes a system, its implementation, operation, and associated procedures.
• stratified samplingStratification is a sampling technique that divides data into groups and controls the amount selected from each group.
• goalA desired outcome that a system or design is intended to achieve.
• canary deploymentCanary deployment is a release strategy that sends a limited portion of users or traffic to a new version before fully switching over.
• percentileA value that divides a probability distribution so that a specified proportion of its mass lies at or below the value.
• tradeoffA design compromise in which improving one property requires accepting a cost or reduction in another.
• recursive feature eliminationAn iterative feature-selection algorithm that repeatedly removes the least important features according to a fitted model.
• proportional z-testA statistical test for comparing proportions between groups using a standardized normal-test statistic.
• Spearman's rank correlation coefficientA correlation coefficient that measures the monotonic association between ranked variables.
• data scientistA professional role involving analysis of data, development of data-driven models or insights, and communication of findings to support decisions.
• machine learning engineerA software-oriented professional who develops, deploys, operates, and improves machine-learning systems.
• control metricA supplementary metric tracked alongside a primary experimental outcome to reveal effects or failures that the primary outcome may not immediately show.
• overfittingA condition in which a model fits its training data too closely and performs poorly on unseen data because it captures insignificant or non-generalizable patterns.
• false negativeA classification or alert failing to indicate a condition that is actually present.
• image segmentationSegmentation is a computer-vision labeling formulation that identifies the pixels belonging to an object or region.
• regressionA software defect in which a change causes previously working behavior to fail or deteriorate.
• search spaceThe set of questions, goals, constraints, stakeholders, and contextual facts that define why a system is needed and what it should accomplish.
• api modelA model API is a programmatic interface that exposes machine-learning model operations such as generating embeddings and scoring candidates.
• handlerA software component that receives and processes requests made through an API.
• Condorcet's jury theoremCondorcet's jury theorem states that independent voters who are individually more likely than not to be correct become increasingly likely to reach a correct majority decision as their number grows.
• inter-rater reliabilityInterrater reliability is the degree to which multiple raters produce consistent or mutually agreeing judgments on the same items.
• labeling platformA labeling platform is a software service or environment for managing data annotation tasks, workers, interfaces, and resulting labels.
• labeling toolA labeling tool is software that presents data and annotation controls for creating or reviewing machine-learning labels.
• modeMajority vote is an aggregation method that selects the label or decision supported by more than half of the votes.
• preference dataPairwise comparison is an evaluation technique that asks a rater to compare two alternatives rather than assign an absolute score to one alternative.
• third-party crowdsourcing serviceA third-party crowdsourcing service provides externally managed distributed workers for completing annotation tasks.
• weakly supervised learningWeakly supervised learning is a machine-learning approach that learns from labels that are incomplete, noisy, imprecise, or otherwise weaker than fully accurate annotations.
• stakeholderA stakeholder is a person or group with an interest in, influence over, responsibility for, or authority regarding a project or system.
• downsamplingThe process of reducing the number of data examples or observations retained for analysis or training.
• prediction driftA significant change over time in the distribution or statistical characteristics of a model’s predictions.
• abstractionAbstraction is a technique for managing complexity by exposing higher-level operations while encapsulating lower-level implementation details.
• feature interactionA dependency or combined effect between features that influences their computation or a model's behavior.
• area under the curveThe area under the receiver operating characteristic curve, summarizing ranking performance across classification thresholds.
• dataset sizeThe quantity of records or examples contained in a dataset or used for model training.
• residual analysisResidual analysis is the examination of prediction errors to diagnose model behavior, assumptions, and performance differences across data.
• candidate filteringThe process of quickly reducing a large set of possible items to a smaller candidate set for downstream processing.
• end-to-end modelA model architecture that maps inputs directly to final outputs without explicit intermediate representations or manually designed processing stages.
• hallucinationA generative-model behavior in which false or unsupported information is presented as if it were true.
• retrieval-augmented generationA generation approach that retrieves relevant external information and supplies it to a generative model as input.
• batchingA model-training approach that learns from a collected batch of data rather than updating continuously per incoming example.
• ONNXONNX is an open format and interoperability standard for representing machine learning models across frameworks and runtimes.
• flexibilityFlexibility is the capacity of a system to accommodate changes in models, processing steps, features, interfaces, or implementation technologies.
• model qualityThe degree to which a machine learning model produces accurate, reliable, and appropriate outputs for its intended task.
• throughputThroughput is the amount of work or number of results processed by a system per unit of time.
• focal lossA classification loss that down-weights well-classified high-confidence examples so training emphasizes difficult examples.
• continuous improvementA feedback loop is a pattern in which a model or system affects the data-generating process that later supplies its inputs or labels.
• software release lifecycleA release cycle is the recurring process and timing by which changes to a software system are prepared, verified, and deployed.
• categorical encodingOne-hot encoding is a representation technique that converts a categorical value into a vector with one active indicator for its category.
• SHAPA feature-attribution technique based on Shapley values that assigns contributions of input features to a model prediction.
• rolling cross-validationA time-ordered cross-validation procedure that repeatedly advances training and validation windows through a series.
• inverted pyramid schemeA top-down problem-analysis technique that begins with a broad high-level statement and progressively investigates more specific details and nuances.
• problem definitionA concise written description of a problem, including its relevant goals, context, and important characteristics.
• inference engineAn inference engine is software that loads a machine-learning model and executes it to produce predictions from inputs.
• data validation pipelineA data validation pipeline is an automated sequence that checks data against defined quality constraints and reports the resulting status.
• feature statisticsFeature statistics are numerical summaries describing the values, relationships, or distribution of one or more model input features.
• data governanceA data governance framework is a coordinated set of roles, responsibilities, policies, and processes for managing data quality and use.
• missing valueMissing data consists of absent values, records, or fields that are expected in a dataset or input.
• schema validationData schema validation is the process of checking that input fields, types, columns, and categories conform to an expected schema.
• reproducibilityReproducibility is the property of being able to recreate a dataset or computational result from its original source and documented procedure.
• expressive powerModel capacity is the flexibility of a model class to represent functions or fit data, often related to the number of parameters.
• data representationA learned or constructed encoding of data that captures information in a form suitable for input to a machine learning model.
• varianceAn error caused by a model's sensitivity to small fluctuations in its training data, producing substantially different behavior on new data.
• liftLift is the measured change in an outcome for a treatment variant relative to a control or baseline.
• hyperparameter tuningHyperparameter optimization is the process of searching for configuration values that improve a model's performance.
• accuracyAccuracy is the proportion of evaluated predictions that match the corresponding correct labels.
• mean target encodingTarget encoding is a categorical-variable encoding technique that replaces each category with a statistic of the target values associated with that category.
• active learningActive learning is a machine-learning technique that selectively chooses informative or uncertain examples for labeling or inclusion in a dataset.
• sequenceData consisting of observations indexed or ordered by time.
• bootstrapped lower confidence bound of recall at a given precisionA one-sided bootstrap-based lower confidence estimate of recall conditional on achieving a specified precision.
• bootstrapped lower confidence bound of recall at a given specificityA one-sided bootstrap-based lower confidence estimate of recall conditional on achieving a specified specificity.
• binomial distributionA probability distribution for the number of successes in a fixed number of independent Bernoulli trials with a common success probability.
• confidence boundA statistical limit that bounds an estimated quantity with a stated level of confidence or coverage.
• fixed ratioA split-update procedure that preserves the ratio of training data to validation data as new labeled data arrives.
• fixed setA split-update procedure that evaluates models on an unchanged reference dataset across modeling iterations.
• fixed shiftA split-update procedure that replaces the validation window with a fixed-length set of the most recent data.
• decision boundaryA decision boundary is the boundary in an input space that separates regions assigned different model decisions or predicted classes.
• marginA measure of the separation between observations and the classifier boundary that determines their predicted classes.
• multinomial distributionA probability distribution for counts across multiple categories in a fixed number of independent trials.
• noise additionThe deliberate perturbation of data values with random noise to assess or improve the stability of model decisions.
• Poisson bootstrapA bootstrap approximation that independently assigns Poisson-distributed multiplicities to observations for each resampling iteration.
• Poisson distributionA probability distribution for the count of events occurring independently at a constant average rate in a fixed interval or exposure.
• kissKISS is a design principle that favors keeping systems and solutions as simple as practical.
• YAGNI principleYAGNI is a software design principle that advises implementing functionality only when it is actually needed.
• false positiveA classification or alert indicating that a condition is present when it is not.
• count dataData consisting of nonnegative integer-valued observations, often recorded over time as counts of events or units.
• intermittent demandDemand that occurs irregularly, with periods of zero demand interspersed among nonzero observations.
• mean absolute percentage errorThe average absolute prediction error expressed as a percentage of the corresponding actual values.
• predictive distributionA probability distribution representing the possible future values of a target conditional on available information.
• quantile forecastA forecast specifying a selected quantile of a predictive distribution.
• scaled mean squared errorA mean squared error whose squared errors are scaled by the squared average actual over the forecast horizon.
• Tweedie distributionA family of exponential-dispersion probability distributions that includes several continuous, discrete, and compound Poisson-gamma cases.
• Tweedie loss functionA loss function derived from a Tweedie probability model for targets with distributional characteristics such as mass at zero and nonnegative continuous values.
• deploymentDeployment is the process of making a software system or model available to operate in a target production environment.
• aleatoric uncertaintyAleatoric uncertainty is irreducible uncertainty arising from randomness, noise, or ambiguity inherent in the data-generating process.
• stock management systemA system that manages inventory availability, replenishment, and related stock decisions.
• synthetic dataArtificially generated data designed to approximate the characteristics of data from a real-world domain.
• experiment trackingThe process of organizing, tracking, preserving, and overseeing experiments and their results.
• fairnessThe property that a model’s behavior or performance does not produce unjustified disparities across relevant demographic or other groups.
• asymmetric cost functionA cost function that assigns different penalties to different types or directions of error.
• gradient boostingA gradient-boosting model is an ensemble model built by sequentially fitting weak predictive models to reduce the errors of the current ensemble.
• convergence analysisAn analysis of whether an iterative training procedure approaches stable or sufficiently low loss as iterations increase.
• granularityThe level of detail at which data is represented, measured, or modeled.
• isotonic regressionA regression technique that fits a monotonic function to data while preserving the specified ordering relationship.
• uncovered demandCustomer demand that is not fulfilled because insufficient inventory is available.
• reliabilityReliability is the ability of a system to continue performing its required functions consistently despite faults, failures, or changing load.
• z-scoreA z-score is the number of standard deviations by which a value differs from the mean of a reference distribution.
• revenueIncome generated from selling goods or services before subtracting expenses.
• error costThe adverse value assigned to an incorrect prediction, which may differ according to the error's direction and magnitude.
• optimizationOptimization is the improvement of a system's performance or resource efficiency through changes to its models, serving process, code, or hardware.
• unstructured dataData that lacks a fixed machine-readable schema for its content, such as free text or images.
• complexity biasA tendency to give undue credit to or prefer complex ideas and systems over simpler alternatives.
• minimal productThe smallest product version that delivers enough functionality to test or serve a core use case.
• gross merchandise valueThe total monetary value of merchandise transactions over a specified period before deductions such as returns or expenses.
• hyperparameterA configuration value selected outside model parameter learning that controls training or model behavior.
• embedded methodA feature-selection technique that derives feature choices from the parameters or importance values of a model trained as part of the selection process.
• filter methodA statistical feature-selection method that evaluates each feature independently using a single-feature property or score.
• L1 regularizationA regularization penalty based on the sum of absolute parameter values that promotes sparse model coefficients.
• wrapper methodA feature-selection technique that evaluates feature subsets by repeatedly training or assessing a predictive model.
• fine-tuningThe process of continuing training on an existing model to adapt its parameters to new data or a related task.
• frameworkSoftware that provides reusable infrastructure for configuring and executing machine-learning model training.
• inference frameworkSoftware that supports model serving and inference through an engine plus associated conversion, optimization, and operational components.
• model optimizationModel optimization is the improvement of a trained model's execution efficiency or resource requirements while preserving acceptable predictive behavior.
• ONNX RuntimeONNX Runtime is a cross-platform runtime for executing ONNX machine-learning models with hardware-specific optimizations.
• Amazon Elastic Compute CloudA cloud computing service that provides resizable virtual compute instances.
• Amazon ElastiCacheAWS ElastiCache is a managed in-memory caching service compatible with engines including Redis and Memcached.
• artifact packagingArtifact packaging is the process of bundling a trained model and its supporting files into a deployable format.
• AWS Application Load BalancerAWS Application Load Balancer is a managed service that distributes application traffic across multiple targets using application-level routing.
• directed acyclic graphA directed acyclic graph is a directed graph whose edges have no cycle, so its nodes can be arranged in an evaluation order.
• NVIDIA Triton Inference ServiceNVIDIA Triton Inference Server is open-source serving software for deploying and executing machine learning models through standardized inference interfaces.
• similarity metricVector similarity is a measure of how closely two numerical vectors correspond in direction, distance, or another defined geometric relationship.
• trainingModel training is the process of fitting a machine learning model to data by adjusting its learned parameters.
• Five WhysA questioning technique that repeatedly asks why to trace a problem toward its underlying origin.
• negative feedback loopA self-reinforcing cycle in which system outputs create data or conditions that cause progressively worse future outputs.
• risk landscapeA structured view of potential failure outcomes, their consequences, and their relevant likelihoods for a system.
• risk ratingA numerical value representing the estimated risk associated with a borrower or other subject.
• joinData joining is the technique of combining records or collections from multiple data sources into a connected dataset.
• multimodalA property of a system or model that represents or processes information from multiple data modalities, such as text and images.
• feature bundleA set of related features consolidated into one computational and operational unit because they share keys, lineage, or transformations.
• feature hierarchyA relationship in which a feature is computed from other features or data sources, forming a hierarchy of feature calculations.
• preaggregationThe technique of aggregating lower-level data before applying further aggregations or feature calculations.
• precalculationPrecomputation performs a calculation before its result is needed so that later processing can reuse the stored result.
• read-write skewAn imbalance or mismatch between the read and write workloads, volumes, or latency requirements of a feature management system.
• feature scalingFeature scaling is the transformation of numerical input features to comparable numerical ranges or scales.
• data availabilityThe condition in which required data can be accessed reliably during model training and system runtime.
• control groupThe group in an experiment that receives the existing condition or status quo and provides a comparison for an intervention group.
• treatment groupThe group in an experiment that receives the new condition or intervention being evaluated.
• smoke testA preliminary test that checks whether a system or component operates sufficiently to warrant further testing.
• aggregate featureA feature computed by summarizing multiple observations over a specified set or time window.
• bag-of-words modelA bag-of-words representation describes text using counts or indicators of its words while disregarding word order.
• BERTBERT is a pretrained bidirectional transformer language model that produces contextual representations of text.
• quantile quantizationA transformation that converts continuous values into discrete buckets defined by value quantiles.
• term frequency-inverse document frequencyA text-weighting representation that scores terms by their frequency in a document relative to their frequency across documents.
• hypothesis testingA statistical procedure for evaluating whether observed data provide evidence against a specified hypothesis or reference distribution.
• diversityA measure of how varied the items or categories represented in a set of results are.
• entropyEntropy is a measure of uncertainty or dispersion in a probability distribution.
• LIMELIME is a model-agnostic explanation technique that approximates a complex model locally around a particular input with a simpler interpretable model.
• permutation feature importancePermutation feature importance measures a feature's importance by randomly shuffling its values and measuring the resulting change in model performance.
• data aggregationThe process of combining records into summarized groups according to selected dimensions and time periods.
• user-item interaction matrixA matrix whose rows represent users, columns represent items, and entries represent interactions or preferences.
• embeddingAn embedding is a numerical vector representation of an object that places semantically or structurally similar objects near one another in a representation space.
• configurabilityConfigurability is the degree to which a software system exposes changeable behavior through explicit settings rather than fixed implementation choices.
• knowledge baseA knowledge base is an organized repository of structured facts, entities, relationships, or domain-specific information.
• labeling functionA labeling function is a heuristic program or rule that assigns a possibly noisy label to an unlabeled data example.
• multisource weak supervisionMultisource weak supervision is a training-data construction technique that combines multiple imperfect labeling sources to produce noisy labels for unlabeled examples.
• named entity recognitionNamed entity recognition is a natural-language-processing technique that identifies and classifies mentions of entities in text.
• rule-based systemA rule-based system produces outputs by applying explicitly specified logical or procedural rules rather than learned model parameters.
• training-serving skewA condition in which a machine learning model performs poorly because its training data does not represent the data encountered during application or serving.
• SQLA request containing terms or criteria submitted to a search engine to retrieve relevant information.
• FlyteFlyte is an open-source workflow orchestration platform for defining, executing, and monitoring data and machine-learning workflows.
• ParquetApache Parquet is a columnar storage file format designed for efficient storage and retrieval of structured data.
• Welch's t-testA statistical test for comparing the means of two groups without assuming that their variances are equal.
• epistemic uncertaintyEpistemic uncertainty is uncertainty caused by limited knowledge or insufficient coverage of the input space in the training data.
• samplingSampling is the process of selecting a subset of observations from a larger dataset for analysis or model training.
• memory boundA bottleneck is a system component or operation that limits overall performance because it constrains throughput or increases processing time.
• profilingProfiling is the measurement and analysis of a program's runtime behavior and resource consumption to identify performance bottlenecks.
• nonfunctional requirementA requirement that specifies qualities or operational constraints of a system, such as performance, security, or portability.
• contextual banditA sequential decision method that selects an action using the observed context and learns its outcome or reward.
• ecpmA revenue metric normalized to a quantity of 1,000 interaction opportunities, calculated in this section as revenue per click multiplied by 1,000.
• environmentA computational representation of an online environment that generates or replays interactions for offline algorithm evaluation.
• replay evaluationAn offline recommendation-evaluation procedure that replays historical interactions and retains cases where a candidate recommendation matches the historically selected item.
• simulatorA computational system that reproduces relevant behavior of a real or online environment for experimentation.
• delayed shift validationA validation procedure that shifts the validation period at relatively infrequent intervals while keeping it fixed between shifts.
• correlation matrixA correlation matrix is a table whose entries quantify pairwise statistical association between variables.
• Evidently AIEvidently AI is an open-source Python library for evaluating, testing, and monitoring machine learning models and data.
• population stability indexThe population stability index is a divergence measure that quantifies how a population's distribution differs from a reference distribution.
• schema complianceSchema compliance is adherence of data to an expected structure, including required fields, data types, and permitted columns.
• time-series validationA validation procedure that separates training and evaluation data according to their temporal order.
• Wasserstein distanceWasserstein distance is a metric between probability distributions defined by the minimum cost of transporting mass from one distribution to another.
• saliency mapA saliency map is an explanation representation that indicates how strongly different input elements influence a model's output.
• assumption checkAn assumption check is a procedure for assessing whether the conditions required by a statistical or machine learning model are supported by data or diagnostics.
• attention matrixAn attention matrix is a two-dimensional array of attention weights relating positions in one sequence to positions in another sequence.
• attention plotAn attention plot is a visualization of attention activations between elements of an input sequence and elements of an output sequence.
• diagnostic plotA diagnostic plot is a statistical visualization designed to reveal violations of model assumptions, influential observations, or other systematic patterns in model results.
• guided attentionGuided attention is a training technique that adds a penalty encouraging an attention matrix to follow a desired alignment pattern.
• heteroscedasticityHeteroscedasticity is the condition in which regression-error variance changes across predictor or fitted-value levels.
• homoscedasticityHomoscedasticity is the condition in which the variance of a regression error is constant across predictor or fitted-value levels.
• leverageLeverage is a measure of how much an observation's predictor values can influence fitted regression coefficients.
• deployment pipelineA set of continuous integration and continuous delivery or deployment practices that automate software integration, testing, and release.
• exploding gradientAn exploding gradient is a training condition in which gradient magnitudes become excessively large and destabilize optimization.
• gradient clippingGradient clipping is a training technique that bounds gradient values or their norm before parameter updates.
• nanNaN is a floating-point value representing an undefined or unrepresentable numerical result.
• regularizationRegularization is a technique that constrains model fitting to discourage overly complex solutions and improve generalization.
• vertical scalingA scaling technique that increases the capacity of a single machine by upgrading or replacing its hardware.
• soft targetA label representation that assigns weighted or probabilistic membership rather than a single categorical class.
• invariantThe property of remaining unchanged under specified transformations or perturbations.
• functional requirementA requirement that specifies the functionality, value, or user flow a system or feature must provide.
• model opacityThe property of a machine-learning model whose internal reasoning or decision process is difficult for observers to understand.
• risk modelA structured description of the threats, failure scenarios, and undesirable outcomes that a system should prevent or mitigate.
• auxiliary modelA secondary machine learning model is an additional learned model that provides an alternative or corrective output when a primary model is unsuitable.
• bias-variance tradeoffThe tradeoff between error from a model's restrictive assumptions and error from its sensitivity to variation in training data as model complexity changes.
• integration testA test that evaluates interactions among multiple software components or between a component and an external system.
• shadow modeAn operating mode in which a candidate system runs alongside the active system and produces outputs without affecting users or decisions.
• wrapperA software component that encapsulates another component behind a controlled interface while coordinating surrounding infrastructure behavior.
• overconfigurationOverconfiguration is a design condition in which a system exposes excessive configurable choices and abstraction layers, increasing unnecessary complexity.
• underconfigurationUnderconfiguration is a design condition in which a system exposes too few configurable choices and hardcodes behavior that may need to change.
• distanceA function that assigns a numerical measure of separation or dissimilarity between mathematical objects or representations.
• innovationThe extent to which a proposed solution introduces novel methods, components, or capabilities beyond existing approaches.
• mlopsA set of practices for operationalizing machine-learning systems through reproducibility, testing, automation, deployment, and pipeline management.
• fuzzy cacheA fuzzy cache is a cache that matches requests using relaxed similarity conditions rather than exact key equality.
• routingRouting between models is a pattern that selects among models according to the characteristics or difficulty of each input.
• normalizationThe transformation of numeric data so that its values conform to a specified scale or distribution.
• hypothesisA hypothesis is a testable proposition about an expected relationship or effect in an experiment.
• statistical significanceA statistical procedure that evaluates whether observed data provide sufficient evidence against a null hypothesis according to a chosen significance criterion.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.