What you will learn in AI Model Evaluation
Chapter by chapter
Chapter 1: Setting the stage for offline evaluations
The chapter introduction frames model evaluation as important across varied AI products, especially before models are exposed to users.
What you learn in chapter 1 →Chapter 2: Anatomy of an offline evaluation
The introduction explains why offline evaluations provide an early check of model behavior before a new version reaches users.
What you learn in chapter 2 →Chapter 3: Using offline evaluations as diagnostics
This introduction explains why diagnostic offline evaluations matter: they help reveal the reasons behind model behavior that performance metrics alone may miss.
What you learn in chapter 3 →Chapter 4: Engineering system performance evaluations
This introduction explains why evaluating an AI feature requires assessing the production system around the model, not just the model’s offline performance.
What you learn in chapter 4 →Chapter 5: Counterfactual evaluations
Introduces counterfactual evaluations as estimates of how an AI system might have performed had it made different decisions.
What you learn in chapter 5 →Chapter 6: Evaluating models in an A/B test
The chapter introduction explains why offline evaluations are necessary but A/B tests are needed to judge a model’s effect on real users and product outcomes.
What you learn in chapter 6 →Chapter 7: From offline evaluation to live experiment
Positions A/B testing as one part of a broader model evaluation system that connects offline evaluation with online experimentation.
What you learn in chapter 7 →Chapter 8: Pitfalls of online metrics
The chapter introduction argues that trustworthy AI model evaluation requires more than online metrics alone, while previewing pitfalls discussed later.
What you learn in chapter 8 →Chapter 9: LLM-as-a-judge fundamentals
Introduces LLM-as-a-judge as a way to evaluate model outputs that require semantic judgment, while stressing clear criteria and validation rather than treating the judge as objective truth.
What you learn in chapter 9 →Chapter 10: LLM-as-a-judge in production
The chapter introduction frames production LLM judging as a trust problem and explains why teams should expect quiet evaluation failures.
What you learn in chapter 10 →Concepts explained in this book
- offline evaluationA diagnostic evaluation performed on non-live data to reveal model failures or unintended optimization behavior.
- LLM-as-a-judgeAn evaluation method in which a large language model assesses another model's outputs or behaviors and assigns judgments or scores.
- diagnostic evaluationA diagnostic evaluation examines model behavior or outputs to reveal specific causes, patterns, strengths, weaknesses, biases, or failure modes beyond aggregate performance.
- counterfactual reasoningCounterfactual evaluation estimates outcomes under decisions or policies other than those that generated the observed data.
- latencyThe total elapsed time from receiving a user request to displaying the complete response, including all intermediate processing steps.
- A/B testA controlled online experiment that compares outcomes between two variants, typically a control and a treatment, using assigned users or traffic.
- offline metricAn offline metric is a quantitative measure of model or algorithm behavior computed from historical or otherwise non-live evaluation data.
- evaluationA performance evaluation measures how well a model or AI system achieves its intended objective against predefined metrics.
- frontend load time metricThe elapsed time required for a web page to load, often summarized across request-time percentiles.
- inverse propensity weightingInverse propensity scoring estimates counterfactual outcomes by weighting observed outcomes inversely by the probabilities that generated their actions.
- policyA decision rule that maps a context to an action or a probability distribution over actions.
- data driftFeature drift detection is the identification of changes over time in the distributions or behavior of model input features.
- biasA bias is a learnable additive parameter that shifts a model computation independently of its inputs.
- metricA quantitative measure of model performance derived from behavior or outcomes observed during live operation.
- latency degradation experimentA controlled evaluation that deliberately increases or varies response latency to measure effects on system, user, product, or business outcomes.
- language modelA neural language model trained on large text corpora to understand, generate, or transform natural-language content.
- continuous improvementA dynamic process in which a model's predictions influence user behavior and that behavior changes the data observed by the model later.
- rubricA defined set of criteria and scoring guidance for assessing model outputs.
- logLogging is the practice of recording detailed information about system behavior, user interactions, and outcomes.
- historical dataHistorical interaction data records past user actions and engagement events in a product.
- recommendation modelA model that selects or ranks items for presentation to users based on predicted interest or utility.
- off-policy evaluationOff-policy evaluation estimates the performance of a target policy using data collected under a different behavior policy.
- contextA contextual feature is an input variable describing the circumstances surrounding a user's current interaction or request.
- trustThe degree to which a model or system merits confidence because its behavior is reliable, responsible, and interpretable.
- online evaluationEvaluation of an AI system in a live or production environment using real-time system behavior or user interactions.
- position biasPosition bias is the tendency for a response's displayed position in an evaluation prompt to affect the judge's verdict independently of the response content.
- shadow testingA technique that sends copies of live production requests to a non-serving candidate system and records its outputs without affecting user-visible behavior.
- samplingThe process of selecting a subset of observations or traffic from a larger population for analysis or processing.
- relevanceThe degree to which retrieved, recommended, or generated results address the intended query, context, task, or user need.
- recallA classification metric measuring the fraction of relevant instances that a system successfully retrieves or identifies.
- online experimentationA live controlled evaluation of a model or product change using real users or production traffic to measure outcomes.
- costCost measures the computational, financial, or resource expenditure required to operate a model or system.
- rerankingRe-ranking is the process of reordering an initially retrieved or recommended list using additional criteria or signals.
- effective sample sizeEffective sample size is the amount of statistically usable evidence remaining after observations are weighted or otherwise adjusted.
- evaluation designAn evaluation design is a deliberate specification of the data, metrics, judgments, comparisons, scenarios, and decision rules used to assess a model or AI system.
- model calibrationCalibration is the agreement between a model's predicted confidence and the observed frequency of its outcomes.
- checklistAn evaluation procedure that decomposes an assessment into explicit item-level or atomic checks rather than assigning one holistic score.
- model selection playbookA model selection playbook is a structured process for comparing model variants and documenting the evidence, trade-offs, and readiness checks behind a selection decision.
- proxy metricA proxy metric is an indirect measure whose value is intended to indicate a target quality or outcome that is difficult to measure directly.
- versioningVersioning is the practice of recording distinct revisions of data, models, code, prompts, metrics, and configuration so that changes can be identified and reproduced.
- policy overlapThe condition that a candidate policy's relevant actions have positive probability under the policy that generated the logged data.
- promptA prompt is text or other input provided to a generative model to specify a task or elicit a response.
- resource utilizationHardware utilization is the proportion of available processor capacity consumed by a running system.
- evaluation datasetData kept separate from training and tuning for a relatively independent assessment of a finalized or mostly finalized model.
- counterfactual loggingCounterfactual logging records observed interactions together with information needed to estimate what would have occurred under alternative experiment variants or policies.
- varianceVariance is a measure of how much a set of observed outcomes fluctuates around its expected or average value.
- reproducibilityReproducibility is the ability to repeat an evaluation and obtain comparable results from the same documented inputs, code, configuration, and procedures.
- guardrail metricA guardrail metric is a metric monitored to ensure that an experiment does not cause unacceptable harm while optimizing its primary objective.
- diversityA diversity index is a quantitative measure summarizing the variety of items, categories, or outcomes in a set.
- robustnessRobustness is the ability of a model to maintain acceptable behavior across disturbances, unusual conditions, or changes in inputs.
- input-output frameworkAn input-output framework organizes an evaluation around its input data, evaluation process, and resulting outputs.
- heat mapA heatmap is a table-based visualization that represents values with color intensity to reveal patterns and variations across cells.
- ablation studyAn ablation study evaluates the effect of removing or varying selected inputs, features, or components while holding other conditions constant.
- exposure metricAn exposure metric measures the amount or type of content presented to users or user segments by a recommendation system.
- validationA procedure for determining whether an evaluator's outputs are sufficiently reliable for a specified decision by comparing them with trusted references or human judgments.
- evaluation contractAn evaluation contract is a formal specification of what is judged, which criteria apply, what context is available, and what output the evaluator must produce.
- probability theoryProbability is a numerical measure from zero to one representing the likelihood of an event or choice.
- simulation-based approachA simulation-based evaluation assesses how a model or system might behave in hypothetical, rare, unobserved, or risky conditions by constructing representative scenarios.
- variantA test variant, or treatment, is an experimental condition that receives the change being evaluated.
- quantizationA model optimization technique that represents model values with lower-precision numerical formats to reduce computation, memory use, or latency.
- metric driftMetric drift is a change over time in the relationship between an evaluation metric and the real-world quality or outcome it is intended to represent.
- cohortA stable cohort is a subgroup of users sharing defined characteristics that remain sufficiently consistent for comparison over an evaluation period.
- stability checkStability testing evaluates whether an evaluator produces sufficiently consistent judgments across comparable conditions or repeated evaluations.
- policy valuePolicy value is the expected performance of a decision-making policy if deployed under the evaluation conditions.
- watchpointA live monitoring check that tracks specified online behavior, model weaknesses, or risks and triggers investigation or intervention when needed.
- judge promptA judge prompt is an instruction given to an evaluating language model that defines how it should assess another model's output.
- importance weightAn importance weight is a factor that reweights an observation to account for a difference between the distribution that generated it and a target distribution.
- user experienceUser experience is the overall experience a user has when interacting with a product, including its usefulness, quality, and responsiveness.
- internal beta testingStructured evaluation of a model or product change by employees before exposure to external users.
- thresholdA threshold is a predefined cutoff value used to determine whether a metric or evaluation result satisfies a requirement.
- two-stage architectureA recommendation architecture that first generates candidate items and then ranks a smaller set using additional signals.
- recommender systemAn AI system that selects, predicts, or ranks items for users based on preferences, behavior, or item characteristics.
- verbosity biasVerbosity bias is a judgment tendency to favor longer model outputs because they appear more complete, thoughtful, or polished, regardless of their actual quality or usefulness.
- failure modeA failure mode is a recurring way in which a model or system fails under particular conditions.
- hypothesisA hypothesis is a testable statement about a population or experimental effect evaluated using statistical evidence.
- performanceA performance metric is a quantitative measure of how well a model or system achieves its intended task or objective.
- diagnostic metricA diagnostic metric is a quantitative measure designed to reveal specific strengths, weaknesses, or failure patterns in a model's behavior.
- noiseNoise is random or irrelevant variation in an observed evaluation result that does not represent the product effect of interest.
- model maturityModel maturity is the stage of development and operational experience of an AI model, from early experimentation to established production use.
- output formatStructured output behavior is the tendency of a model to produce responses that conform consistently to a specified structure or schema.
- percentileA latency percentile is a value below which a specified proportion of observed system response times falls.
- ensembleAn evaluation setup that combines judgments from multiple evaluators, models, runs, or prompts for a single assessment.
- validation setA validation set is a dataset reserved for assessing or comparing model behavior during development rather than fitting model parameters.
- fallbackAn alternative system behavior or response used when a preferred processing path or primary component is unavailable or fails.
- covariate adjustmentCovariate adjustment is a statistical technique that incorporates pre-treatment variables associated with an outcome to improve the precision of an estimated treatment effect.
- tier 2An evaluation tier is a level in a multi-stage evaluation system with a defined evaluator cost, capability, and escalation position.
- time to first tokenTime to first token is the elapsed time from a request until the first generated token becomes available to the user.
- deterministic metricA metric computed by fixed rules or calculations that produce the same result for the same inputs without interpretive judgment.
- context windowA context window is the bounded amount of input and generated text that a language model can process in one invocation.
- non-inferiority marginA predefined maximum acceptable difference from a reference outcome that must not be exceeded to conclude that performance is not meaningfully worse.
- regressionA regression is an unintended deterioration in a model or system behavior after a change.
- gold exampleA gold example is an evaluation case whose expected judgment or score has been established by human experts or another trusted reference process.
- completion rateTask completion rate is the proportion of attempted tasks that an automated system completes successfully.
- Precision@kA ranking metric measuring the proportion of relevant items among the top K retrieved or recommended items.
- canary setA canary set is a fixed evaluation dataset used to detect changes in evaluator behavior or other evaluation-pipeline components.
- expected rewardExpected reward is the predicted average benefit of a policy over its possible actions and outcomes.
- reliabilityThe ability of a system to provide correct and available service consistently under expected operating conditions and failures.
- evaluation criterionA primary metric, or success metric, is the main outcome measure selected to evaluate whether an experiment achieves its intended objective.
- fairness evaluationA fairness evaluation assesses whether a model's outcomes or error rates differ unfairly across protected or otherwise relevant groups.
- signalA signal is the portion of an observed evaluation result that reflects the real product effect or outcome of interest.
- long-term holdbackAn experimental design that persistently reserves a user or traffic cohort from a change for comparison against a stable reference condition.
- treatment variantA model variant is an alternative model configuration or experimental treatment evaluated relative to a control or other variants.
- AI agentAn agentic system is an AI system that performs tasks through multi-step actions, often by deciding when and how to invoke tools.
- retrieval rankingOrdering multiple outputs from best to worst according to their relative quality or utility.
- alignmentAlignment is the degree to which a model's behavior, reasoning, and outputs match specified evaluation objectives and human judgment.
- discoveryA measure of whether recommendations introduce items that are new or unexpected while remaining relevant to the user.
- batchingThe grouping of multiple requests or data items into batches for joint processing.
- interpretabilityExplainability is the degree to which the basis for a model's output or judgment can be understood and inspected by people.
- observabilityThe extent to which a system's internal state can be inferred from its outputs, logs, and other external signals.
- metric hierarchyA metric hierarchy is an ordered structure of metrics that establishes their relative priority when evaluating trade-offs.
- variance reductionVariance reduction is the use of statistical methods to decrease outcome variability and improve the precision or power of an estimate.
- blind audit sampleA blind audit sample is a fixed random sample of lower-tier judgments routed independently to a stronger evaluator or human reviewer.
- confidenceA numerical value produced by a model that represents its estimated certainty in an output or prediction.
- confident-error rateConfident-error rate is the proportion of high-confidence lower-tier judgments that disagree with an independent stronger evaluation.
- content-based routingStatic content-based routing sends examples to an evaluation tier according to properties of the input rather than the lower-tier evaluator's judgment.
- evaluation escalationThe routing of an evaluation example from a cheaper evaluator to a stronger evaluator or human reviewer when specified conditions are met.
- beta testingPre-release testing in which selected users evaluate a product or model to identify issues before wider release.
- self-enhancement biasSelf-preference bias is an evaluator's tendency to favor outputs that resemble its own characteristic style, structure, tone, or reasoning patterns.
- personalizationPersonalization is the adaptation of recommendations or other outputs to an individual user's preferences, history, or behavior.
- edge caseAn edge case is an uncommon or boundary-condition input or situation that can expose unusual model behavior.
- population stability indexThe Population Stability Index measures the degree of distribution shift between two datasets or populations.
- combined evaluationA combined evaluation integrates performance, diagnostic, and simulation-based assessments to provide a broader view of model behavior.
- correlationCorrelation is a statistical association between variables that does not by itself establish that one variable causes another.
- baselineA baseline is an existing model, method, heuristic, or reference result against which a new approach is compared.
- out-of-distribution performanceOut-of-distribution performance measures how well a model performs on data whose distribution differs from the data used for training or evaluation.
- metric libraryA metric library is a reusable software collection that implements quantitative measures for evaluating models or systems.
- swapped-order evaluationA swapped-order rerun is a repeated pairwise evaluation in which the same two responses are placed in opposite prompt positions to test whether the judgment changes with ordering.
- online-offline correlationOffline-to-online correlation is the analysis of how changes in offline evaluation metrics relate to outcomes observed in live experiments.
- rolling datasetA rolling dataset is a continuously maintained collection of records that incorporates newly available evaluation and operational results over time.
- monitoringThe continuous observation of model behavior, system signals, and user outcomes during live operation.
- reward hackingA failure mode in which a model exploits weaknesses in an objective or reward specification to obtain high measured reward without achieving the intended goal.
- driftA change over time in a model's data distribution, behavior, or performance relative to its earlier state.
- click-through rateClick-through rate is the proportion of users or impressions that result in a click on a presented item or recommendation.
- versionA model version is a distinct released or identified state of a model with a specific implementation, parameters, or training artifacts.
- postmortemA postmortem is a blameless retrospective analysis of a production or evaluation incident that records what happened, why detection or validation failed, and what changes will follow.
- throughputTokens per second is the rate at which a generative model produces output tokens after generation begins.
- debate-style judgingAn evaluation pattern in which competing cases are constructed for alternatives before an evaluator weighs them and renders a judgment.
- multi-call debateA debate-style evaluation technique that uses separate model calls for the competing advocates and a final judging call.
- advocate-strength effectAn apparent evaluation difference caused by one assigned advocate arguing more effectively than another rather than by a quality difference between the evaluated responses.
- single-judge debateA debate-style evaluation technique in which one model invocation argues for both alternatives and then judges the arguments.
- propensity scoreA propensity score is an estimated probability that a logging policy selects a particular action for a given context.
- segmented evaluationSegment analysis compares evaluation results across defined subsets of users or data to reveal differences hidden by aggregate results.
- memory usageMemory footprint is the amount of memory occupied by a model or software process during operation.
- cold-start behaviorThe behavior of a recommendation system when users or items have little or no interaction history.
- candidate setA candidate set is the collection of items or actions considered for selection by a decision-making system.
- metric suiteA metric suite is a coordinated collection of evaluation measures that jointly characterize model performance, reliability, and product fit.
- unit of evaluationA unit of evaluation is the individual entity at which a model or system is assessed, such as a query, session, response, or completed task.
- rank-aware metricA ranking metric measures the quality of an ordered list of items relative to desired relevance or preference.
- labelA target annotation or outcome associated with a data example for supervised learning or evaluation.
- exposure loggingThe recording of whether an assigned user or unit actually received or experienced the changed model output.
- randomizationThe assignment of experiment units to variants by chance according to a specified allocation procedure.
- control groupA control variant is the reference condition in an experiment against which treatment variants are compared.
- distillationA model compression technique in which a smaller student model learns to reproduce the behavior of a larger teacher model.
- pruningA model optimization technique that removes parameters, connections, or structures judged unnecessary for the target task or inference.
- disagreement reviewDisagreement analysis is the investigation of differing evaluation judgments to identify the specific output behavior or criterion responsible for the disagreement.
- coherenceLogical coherence measures whether the claims and reasoning in an output fit together consistently and intelligibly.
- unit of randomizationA unit of randomization is the entity, such as a user, session, or account, that is independently assigned to an experiment variant.
- demographic featureDemographic information describes attributes of people or groups such as age, gender, race, or ethnicity.
- capacity planningCapacity planning is the process of determining the computational resources required to operate a system under expected and peak workloads.
- test suiteA collection of evaluations that are run regularly to assess a model across defined criteria or scenarios.
- CUPEDCUPED is a variance-reduction technique for randomized experiments that adjusts outcomes using pre-treatment covariates correlated with those outcomes.
- variance inflationAn increase in the variance of an estimate or comparison caused by additional sources of variation among the observations or groups.
- backtestingAn evaluation technique that compares model predictions with outcomes that actually occurred in historical data.
- benchmarkA fixed reference dataset or evaluation standard used to compare the performance of different models or approaches.
- synthetic dataArtificially generated data designed to represent properties or scenarios of real-world data.
- training datasetThe portion of a dataset from which a machine learning model learns patterns and relationships.
- correlation matrixA correlation matrix is a tabular array whose cells contain pairwise correlation values for variables represented by its rows and columns.
- prompt injectionPrompt injection is an attack in which untrusted input contains instructions intended to manipulate a model's behavior or override its intended instructions.
- demographic parityDemographic parity is a fairness criterion requiring a model's positive prediction or outcome rate to be comparable across demographic groups.
- equality of opportunityEquality of opportunity is a fairness criterion requiring comparable true-positive or qualified-positive outcome rates across demographic groups.
- scoring mechanismAn evaluation technique that assigns numerical values to one or more dimensions of a model output.
- pairwise comparisonA pairwise comparison is an evaluation procedure that compares two alternatives against shared criteria to determine which is preferable or whether they are tied.
- accuracyA measure of how often a model's outputs match desired or correct outcomes.
- behaviorThe observable outputs and actions produced by a model in response to inputs and conditions.
- error rateError rate is the proportion of predictions or decisions that are incorrect for a defined task.
- error rate by subgroupError rate by subgroup is the proportion of incorrect model outputs calculated separately for a defined user, item, or scenario subgroup.
- irrelevant recommendation rateIrrelevant recommendation rate is the proportion of recommended items that are not relevant to the user.
- miss rateMiss rate is the proportion of relevant items that a recommendation model fails to recommend or surface.
- dry-run tool adapterA dry-run tool adapter is an integration layer that simulates a tool invocation without performing its externally visible side effects.
- network throttlingThrottling is the deliberate limitation of request or traffic processing rates to control resource use.
- sink componentA sink component is a system component that captures outbound actions without delivering them to their external destination.
- statefulnessA stateful operation is an operation whose effects depend on or modify persistent application or external system state.
- labeled dataLabeled data consists of examples paired with human- or system-provided annotations representing target outputs, judgments, or categories.
- AI/ML model development workflowThe model development life cycle is the sequence of activities involved in developing, deploying, monitoring, and iterating an AI model.
- tiered judgingTiered judging is an evaluation technique that applies a lower-cost judge broadly and routes selected examples to a stronger judge for higher-fidelity adjudication.
- abstentionAbstention is an evaluation behavior in which a judge declines to issue a verdict when the available information is insufficient for a meaningful judgment.
- noveltyNovelty measures how unfamiliar or non-obvious recommended items are to users or relative to prior recommendations.
- dashboardA dashboard is an interactive interface that presents selected data, metrics, and trends for monitoring or analysis.
- fairnessFairness is the extent to which a model treats relevant user or content groups equitably and without unjustified disparities.
- action spaceAn action space is the set of actions available to a policy or decision-making system in a given context.
- checklist itemA checklist item is a single, grounded, independently traceable criterion in a checklist that can be resolved as pass or fail.
- metric frameworkAn organized set of evaluation metrics and rules that connects model objectives with product goals, constraints, and downstream outcomes.
- sample sizeSample size is the number of observations or participants included in a statistical experiment.
- infrastructureInfrastructure is the computing, storage, networking, and runtime support required to operate a software system.
- causal inferenceCausal analysis estimates the effects of actions or interventions on outcomes while accounting for how observations were generated.
- first contentful paintFirst Contentful Paint measures when the first content is rendered on a screen.
- fully loaded timeFully loaded time measures the elapsed time until a page and its content have completed loading.
- time to first byteTime to first byte measures the elapsed time between a client's request and receipt of the first byte of response data.
- agent step countThe number of execution steps performed while an agent system completes a task.
- loop/retry rateThe frequency with which an agent system repeats an execution loop during task completion.
- time to final tokenTime to final token is the elapsed time from a request until the complete generated response is available.
- heuristicA heuristic is a straightforward rule or procedure that produces an outcome without requiring a machine-learning model.
- covariateA covariate is a measured variable associated with an outcome that can be included in statistical adjustment of an effect estimate.
- cachingThe storage of previously computed or retrieved data so subsequent requests can be served more quickly.
- progressive renderingProgressive rendering is the incremental display of interface content as portions become available rather than waiting for the entire result.
- chain-of-thoughtThe process of deriving conclusions or solutions through logical or evidential steps expressed in a model output.
- action-outcome pairAn action-outcome pair records a system action together with the resulting observed outcome.
- consistencyConsistency is the degree to which an evaluator produces stable judgments across comparable inputs or repeated evaluations.
- incremental action valueThe estimated change in outcome attributable to taking a particular alternative action instead of the observed or reference action.
- impactProduct impact is the effect that a model or product change has on product outcomes such as retention, engagement, or satisfaction.
- noise sensitivityNoise sensitivity measures how much a model's outputs change when realistic noise or perturbations are introduced into its inputs.
- out-of-distributionOut-of-distribution data consists of examples drawn from a distribution that differs meaningfully from the data used to train or evaluate a model.
- fidelityGroundedness is a metric measuring whether an output is supported by the information it is expected to rely on.
- content discovery rateContent discovery rate measures how many new titles or genres users try within a defined period.
- minimum detectable effectThe minimum detectable effect is the smallest true change in an outcome that an experiment is designed to detect with specified statistical power and significance.
- power analysisPower analysis is a statistical procedure for determining the sample size needed to detect an effect under specified assumptions.
- replay evaluationReplay evaluation is an evaluation technique that reuses previously logged events or interactions to simulate and assess system behavior under an alternative strategy.
- decision frameworkA decision framework is a structured set of criteria and reasoning rules for choosing among actions based on evidence.
- error analysisError analysis is the systematic examination of model mistakes to identify recurring patterns, causes, and failure conditions.
- sandboxingAn isolated execution environment that enables experimentation without affecting real users, systems, or production traffic.
- fairness metricA fairness metric is a quantitative measure of disparities in model outcomes or errors across relevant groups.
- tool-use correctnessTool-call accuracy measures whether an agent selects and invokes the appropriate tool correctly.
- customer satisfactionViewing satisfaction is the degree to which users find their consumption of recommended content fulfilling and worthwhile.
- top-k evaluationTop-k evaluation assesses only the first K results in a ranked output list.
- human-in-the-loopHuman review is the assessment of model outputs or decisions by human evaluators.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
