What you will learn in AI Governance
Secure, privacy-preserving, ethical systems
Chapter by chapter
Chapter 1: Governing Generative
The chapter introduction explains why GenAI’s rapid, flexible adoption makes practical governance urgent.
What you learn in chapter 1 →Chapter 2: Adoption Models for GenAI
The introduction establishes why an organization must identify how it adopts GenAI before deciding how to govern it.
What you learn in chapter 2 →Chapter 3: The Six Level GenAI Governance Framework
This introduction uses three incidents to show why GenAI governance needs multiple layers beyond written policies, approved vendors, or a single security tool.
What you learn in chapter 3 →Chapter 4: Securing GenAI
The chapter introduction explains why LLMs are vulnerable to instructions hidden in data, why filters cannot fully enforce the instruction–data boundary, and why greater AI capabilities raise the security stakes.
What you learn in chapter 4 →Chapter 5: Privacy
The chapter introduces how GenAI expands privacy risks, organizes them into four pillars, and emphasizes evidence of how personal data is handled.
What you learn in chapter 5 →Chapter 6: Ethical and Trustworthy GenAI
The introduction explains why trustworthy GenAI requires ethical safeguards beyond security and privacy, with controls scaled to potential harm and maintained throughout the product’s life.
What you learn in chapter 6 →Chapter 7: Regulation and Compliance
The introduction explains why teams need to clarify their EU AI Act obligations and how prior governance work can prepare them for discussions with Legal.
What you learn in chapter 7 →Chapter 8: Governing for Uncertainty
The introduction uses cloud computing’s governance history to frame how organizations can govern GenAI despite unresolved risks and changing standards.
What you learn in chapter 8 →Concepts explained in this book
- human-in-the-loopAn approach in which a person participates in an AI system’s operation, such as by reviewing or authorizing its outputs or actions.
- memorizationThe retention of specific training-data details by a model, potentially allowing it to reproduce them.
- AI agentAn AI agent is a software system that can plan and perform tasks, including using tools and taking actions on a user's behalf.
- prompt injectionAn attack that uses instructions in content supplied to a model to redirect its behavior contrary to intended instructions.
- deployerA regulatory role for an entity that uses an AI system under its authority, except where the system is used in the course of a personal, non-professional activity.
- providerUnder the EU AI Act, a provider is a person or organization that develops or has an AI system developed and places it on the market or puts it into service under its own name or trademark.
- decision recordA decision record is a structured record of the inputs, model versions, actions, and review associated with an AI-assisted decision, enabling it to be examined or replayed.
- 6L-G modelThe book’s six-level framework for translating generative AI governance principles into organizational and technical controls.
- memoryAgent memory is information retained by an agent for use in later interactions or tasks.
- kill switchA kill switch is a control that can rapidly disable an AI system or model to limit harm or enable investigation.
- AI ActEuropean Union legislation establishing a risk-based regulatory framework for AI systems and obligations for their providers and deployers.
- monitoringObservation of a system or process to track its status, behavior, or performance.
- fairnessFairness is a normative criterion for evaluating whether an AI system treats people or groups equitably, with distinct formal definitions that may conflict.
- data minimization strategyData minimization is the practice of limiting personal or sensitive data collection and use to what is necessary for a defined purpose.
- policy engineA policy engine is a service or software component that evaluates requests against defined policies and returns decisions such as allow or deny.
- machine unlearningMachine unlearning is a set of methods for removing the influence of specified training data from a trained machine-learning model.
- AI governanceAI governance is the system of principles, processes, and oversight used to direct and control the development and use of artificial intelligence.
- overrelianceOverreliance is the tendency to defer to an automated system’s recommendations without applying independent judgment or checking supporting evidence.
- accuracyAccuracy is the data-protection principle that personal data be correct and, where necessary, kept up to date.
- anonymizationAnonymization is the alteration of information to prevent it from being linked to an identifiable person.
- hallucinationAn AI hallucination is an output that presents fabricated, unsupported, or false information as though it were valid.
- personal dataInformation that can identify a specific person, either by itself or when combined with other information.
- GRC programAn organizational program that integrates governance, risk management, and compliance activities for a technology or business area.
- documentationTechnical documentation is recorded information describing a system’s design, capabilities, limitations, and compliance-relevant characteristics.
- intellectual propertyIntellectual property consists of legally protected creations and other intangible assets, including rights relevant to AI use, training, and vendor relationships.
- transparencyTransparency is the availability of understandable information about an AI system’s operation and decision-making.
- authorizationAuthorization determines which actions or resources an entity is permitted to access.
- autonomyThe degree to which an AI system can pursue goals and take actions without direct human intervention.
- guardrailA control that constrains AI system inputs, outputs, or interactions to reduce safety, security, or policy violations.
- attribute inferenceAttribute inference is an attack that infers sensitive or otherwise unknown attributes about an individual from available data or model outputs.
- misalignmentMisalignment is a mismatch between an AI agent’s measured or specified objective and the intended outcome.
- right of accessA data-protection right to learn whether one’s personal data is being processed and to obtain information about or a copy of that data.
- interpretabilityThe ability to provide an understandable account of how an AI system reaches its decisions or outputs.
- synthetic dataArtificially generated data designed to resemble real data without directly representing actual individuals.
- attack surfaceThe set of points through which an attacker can attempt to affect or compromise a system.
- logLogging is the recording of system events and associated information for monitoring, audit, or analysis.
- sandboxingSandboxing is the execution of software in a constrained environment that limits its access to systems, data, networks, and privileges.
- safetensorsSafeTensors is a tensor-serialization format that stores numerical tensor data without embedding executable code.
- autonomy-level classificationA grade describing how independently an AI system or process can operate from human direction.
- purpose limitationPurpose limitation is the data-protection principle that personal data may be collected for specified, explicit, and legitimate purposes and not processed incompatibly with them.
- provenanceThe documented origin and history of data, including how it was collected, transformed, and used.
- training datasetThe collection of examples used to optimize a machine-learning model during training.
- probabilistic risk managementProbabilistic risk management evaluates and governs risks whose likelihoods or outcomes are uncertain rather than deterministic.
- serious incidentA serious incident is an event that directly or indirectly causes death, serious health damage, specified critical-infrastructure disruption, a breach of fundamental-rights obligations, or serious damage to property or the environment.
- tenant isolationTenant isolation is the separation of data, state, and operations belonging to different customers or organizational tenants in shared infrastructure.
- reliabilityReliability is the degree to which a system performs consistently and correctly under expected conditions.
- data subject access requestA request by an individual to learn whether an organization processes their personal data and to obtain access to it.
- membership inference attackAn attack that attempts to determine whether particular data was included in a model’s training data.
- rate limitA rate limit is a configured bound on the number or frequency of requests permitted during a specified period.
- biasA systematic distortion in data or model behavior that can produce unfair or skewed outcomes.
- risk and complianceGovernance, risk, and compliance is the coordinated management of organizational oversight, risk controls, and compliance obligations.
- model hoster postureA model hoster is an organization that runs a generative AI model on infrastructure it controls or manages.
- differential privacyDifferential privacy is a mathematical framework that limits how much an individual’s data can affect an analysis or model output.
- policyA set of conditions governing an agent’s actions according to its state and context.
- SaaS consumerA SaaS consumer is an organization that uses generative AI through a vendor-provided application or platform.
- compliance evidenceCompliance evidence is documented information supporting an assessment of whether a system meets applicable requirements.
- riskRisk is the possibility of an adverse outcome arising from uncertainty about an activity, system, or decision.
- tiered approvalTiered approval is an authorization scheme that applies progressively stronger review requirements according to an action’s impact or risk.
- automation biasAutomation bias is the tendency to favor or trust recommendations from automated systems, potentially leading people to make errors or overlook relevant information.
- defense in depthDefense in depth is a security and safety strategy that layers independent controls so that failure or bypass of one control does not defeat the others.
- MCPMCP is a protocol for connecting AI applications to external tools and data sources.
- agent registryAn agent registry is a maintained record of active agents and their purposes, owners, and operational exposure.
- ai gatewayAn AI gateway is an intermediary layer that applies governance, access, and operational controls to requests between AI systems and services.
- delegationDelegation is the transfer of work or authority from one agent or actor to another.
- identity brokerAn identity broker mediates identities and credentials between agents, people, and services to establish accountability and access.
- API integratorAn API integrator is an organization that incorporates a third-party generative AI model into its systems through an API.
- access rights gapThe access rights gap is the mismatch between legal expectations to provide an account of a person’s data and a trained model’s inability to reliably retrieve that data.
- post-market monitoringThe ongoing collection and review of information about a system after it has been placed on the market or put into service.
- environment riskEnvironment risk is exposure arising from weak security controls or vulnerabilities in the infrastructure and services surrounding a generative AI application.
- model riskThe risk of relying on a model whose training and development are outside the deploying organization’s control.
- secret scanningSecret scanning is the automated detection of credentials, keys, and other sensitive authentication material in source code and related artifacts.
- short-lived credentialA short-lived credential is an authentication credential that expires after a limited period or task.
- deletionA procedure for removing stored data and tracing its removal across relevant systems or records.
- agent identityAn agent identity is a verifiable identity assigned to an autonomous software agent and associated with its purpose and permissions.
- agentic red teamingAgentic red teaming is adversarial evaluation that tests whether an agent can be manipulated across multiple interactions and over time.
- anomaly detectionAnomaly detection is the identification of observations or behaviors that significantly deviate from an expected pattern.
- just-in-time accessJust-in-time credential provisioning grants an agent scoped access only while it performs a specific task and removes that access afterward.
- MCP gatewayAn MCP gateway is an intermediary control point that governs, logs, or inspects interactions using the Model Context Protocol.
- multi-agent governanceMulti-agent governance is the definition and enforcement of rules for identities, permissions, communication, and failure handling among multiple agents.
- risk-based human oversightRisk-based human oversight assigns human review or approval requirements according to the potential consequences of an action.
- sub-agent isolationSub-agent isolation separates agents or agent workflows through distinct permissions, credentials, or execution boundaries.
- safety controlA safety control is a measure that prevents, detects, or responds to unsafe behavior or content in an AI system.
- agentic traceAn agentic record is a record of an agent’s goal, plan, tool calls, inputs, handoffs, approvals, outcomes, and errors.
- meaningful oversightMeaningful oversight is human review designed around information a reviewer can understand and process to judge an AI system’s actions.
- multistep testMulti-step testing evaluates intermediate actions and interactions in an agent workflow as well as its final outputs.
- plan previewA plan preview is a human-readable presentation of an agent’s intended steps before execution.
- redress pathA redress path is a process through which an affected person can request review and reversal of an AI-driven action.
- self-assessmentAn evaluation in which a provider assesses its own system against applicable requirements rather than relying on an external reviewer.
- memory poisoningMemory poisoning is the insertion of misleading or malicious information into an agent’s memory to influence its later behavior.
- impact assessmentAn impact assessment is a documented evaluation of the risks an AI system introduces for individuals and society, recorded here as a structured review of a GenAI proposal.
- learning and improvementThe Learning & Improvement level of the 6L-G model uses feedback and flaws found in live systems to improve AI governance, designs, processes, tests, and controls.
- graceful failureGraceful failure is a system-design approach that limits harm and preserves safe behavior when a process halts or encounters an error.
- version pinningVersion pinning is the practice of fixing software or model use to a specified version rather than automatically adopting updates.
- data overcollectionOvercollection is the collection or retention of more personal data than is necessary for a defined purpose.
- privacy attackA privacy attack is an adversarial method for extracting personal information or determining sensitive facts from data or an AI system.
- prompt-based attackAn attack that manipulates an AI system through crafted natural-language instructions or content.
- data retentionData retention is the storage of data for a defined period under operational, legal, or policy requirements.
- redactionRedaction is the removal or masking of sensitive information from content before it is disclosed or processed.
- biometric categorizationBiometric categorization is the classification of people into categories using physical or behavioral biometric data.
- multi-agent systemA multi-agent system is a system in which multiple agents interact or collaborate to perform tasks.
- artificial intelligence systemAn AI system is a machine-based system that operates with varying levels of autonomy and may adapt after deployment to generate outputs such as predictions, recommendations, or decisions.
- confused deputy problemThe confused deputy problem is a security flaw in which a program with legitimate authority is tricked into using that authority on behalf of someone who lacks it.
- audit trailAn audit log is a chronological record of events and actions that supports review, accountability, or compliance.
- token limitA token cap is a configured maximum on the number of language-model tokens that may be processed or generated for an interaction or session.
- untrusted inputUntrusted content is externally supplied information that has not been verified as safe or reliable.
- LLM-as-a-judgeA language model used to evaluate another model’s plan or output against specified criteria.
- opt-out mechanismA choice or mechanism by which an individual declines a specified use of their data or participation in a service.
- strategy and policy levelThe 6L-G governance level that establishes organizational direction, AI policies, leadership, accountability, and oversight.
- generative artificial intelligenceGenerative AI is a class of artificial intelligence models that produces new content such as text, code, or images in response to inputs.
- privacyThe protection and appropriate handling of information about identifiable people, including how it is collected, used, and disclosed.
- similarity searchSimilarity search is a retrieval technique that finds data items whose representations are close to a query representation in a vector space.
- ai risk managementAI risk management is the continuous process of identifying, assessing, addressing, and monitoring risks posed by AI systems to organizations and stakeholders.
- complianceCompliance is conformity with applicable laws, regulations, standards, policies, and contractual requirements.
- ai literacyThe knowledge and skills that enable providers, deployers, and other people involved with AI systems to make informed use of them and understand their opportunities and risks.
- privacy-by-designPrivacy by design is an approach that builds privacy protections into the design and operation of systems from the outset.
- monitoring planA documented plan specifying how post-market information about an AI system will be collected and evaluated.
- collection and purposeCollection and purpose is a privacy-risk dimension concerning what data enters an AI system and whether its use remains limited to a stated purpose.
- output integrityOutput integrity is the privacy-risk dimension concerning harmful or false statements about real people produced by AI systems.
- storage and memorizationStorage and memorization is a privacy-risk dimension concerning how personal information persists in AI models and associated data stores.
- user rights and governanceUser rights and governance is a privacy-risk dimension concerning the exercise of individual data rights and the organizational ability to honor them in AI systems.
- safetySafety is the condition of avoiding unacceptable harm to people or other affected parties through a system's operation.
- robustnessRobustness measures how consistently a model behaves appropriately when its inputs vary or differ from familiar contexts.
- planningPlanning is the process by which an agent selects and orders actions, tools, constraints, success criteria, and stop conditions to pursue a goal.
- language modelA large language model is a machine-learning model trained on large text corpora to process and generate natural language.
- AI Code of PracticeThe GPAI Code of Practice is a voluntary code that provides guidance for providers of general-purpose AI models on meeting relevant obligations and commitments.
- open-source GPAI modelAn open-source GPAI model is a general-purpose AI model made available under open-source terms whose provider may qualify for specified regulatory exemptions.
- incident reporting obligationAn incident reporting obligation is a legal duty to notify designated authorities about a qualifying incident within a specified period.
- risk tierA defined classification grouping an AI use case by the severity and acceptability of its risks.
- tool registryA tool registry is a centralized catalog of approved tools and the permissions governing their use.
- intended purposeIntended use is the purpose for which a system is meant to be used, as specified by its provider or deployer.
- content filteringAn output filter is a control that checks or restricts generated content before delivery.
- sensitive informationSensitive data is information whose nature or context calls for heightened protection, including behavioral and emotional inferences described as sensitive under GDPR Article 9.
- fairness evaluationFairness testing evaluates whether an AI system produces unjustified disparities or inequitable outcomes for people or groups.
- right to rectificationA data-protection right to have inaccurate personal data corrected under applicable legal conditions.
- prompt engineeringPrompt engineering is the practice of designing and refining instructions supplied to a model to shape its responses or behavior.
- instructions for useProvider-supplied directions describing a system's intended purpose, limitations, requirements, and conditions for use.
- profilingAutomated processing of personal data used to evaluate or predict aspects of an individual's behavior, preferences, or circumstances.
- override rateAn override rate is the frequency with which users reject or change an AI system's suggestion.
- dynamic tool discoveryDynamic tool discovery is the ability of a system to advertise or discover tools after an initial connection has been established.
- architecture reviewAn implementation review is a formal examination of an AI system design to verify that required legal, security, privacy, and operational controls are feasible and addressed.
- Open Policy AgentAn open-source policy engine for defining and enforcing policy across software systems.
- purpose-aware egress gateA purpose-aware egress gate is a policy-enforcing checkpoint that evaluates outbound tool calls before data leaves a system boundary.
- purpose creepPurpose creep is the expansion of data processing beyond the purpose for which the data was originally collected or intended.
- purpose lockA purpose lock is a design control that encodes strict boundaries on the purposes for which an agent may process data and use tools.
- purpose, necessity, and destination checksPurpose, necessity, and destination checks assess whether a proposed data use matches its declared purpose, sends only necessary fields, and targets a permitted destination.
- moderation layerA system component that evaluates generated content for policy or safety violations before the content is delivered.
- moderation modelA classifier that identifies whether content violates safety, policy, or moderation criteria.
- authorized representativeUnder the AI Act, an authorized representative is an entity established in the European Union that receives and accepts a written mandate from a provider outside the Union to perform specified obligations on its behalf.
- inference engineAn inference engine is software that loads and serves trained machine-learning models to generate predictions or outputs.
- pickle formatA pickle-based format is a Python object-serialization format that can encode executable behavior in addition to data.
- pluginA plugin is an add-on software component that extends an application or model-enabled system with additional capabilities or integrations.
- software bill of materialsA software bill of materials is a machine-readable or human-readable inventory of the components and dependencies included in a software product.
- software supply chainA GenAI supply chain is the collection of external software, services, libraries, infrastructure, and update processes that a generative AI system depends on.
- supply-chain breach simulationA supply-chain breach simulation is a controlled exercise that models compromise of an external dependency or update to evaluate detection and containment.
- service accountA service account is a nonhuman identity used by an application or service to access resources.
- cascade of trustA cascade of trust is a failure in which successive agents treat uncertain findings from earlier agents as established facts.
- false independenceFalse independence is the mistaken impression that agents provide independent judgments when they share assumptions or sources of error.
- irreversible actionAction irreversibility is the risk that an AI agent takes an action that cannot easily be undone.
- miscoordinationMiscoordination is a failure in which agents share goals but take actions that do not align with one another.
- trust boundaryA trust boundary is a division between components or parties across which control, responsibility, or security assumptions change.
- validation layerInput/output validation is the inspection of AI requests and responses to sanitize them or detect content that violates defined requirements.
- high-risk providerA provider responsible for placing a high-risk AI system on the market or putting it into service and meeting the associated regulatory obligations.
- quality management systemAn organized set of documented processes and responsibilities for maintaining the quality of an organization's products or services.
- threat modelA structured technique for identifying, analyzing, and prioritizing threats to a system.
- substantial modificationA substantial modification is a change to an AI system that affects its compliance or changes its intended purpose in a way not foreseen by the original provider.
- CerbosCerbos is a policy engine for evaluating authorization requests against declarative access-control policies.
- dependency scanDependency scanning is the automated examination of software dependencies to identify known vulnerabilities, licensing issues, or unapproved components.
- Llama GuardLlama Guard is a language-model-based safety classifier that evaluates content against defined safety categories.
- risk classificationClassification is the assignment of an AI system or use case to a regulatory category based on its purpose and applicable rules.
- fairness metricA fairness metric is a quantitative measure of disparities or equitable outcomes across people or groups.
- manipulation and exploitation of vulnerabilitiesManipulation and exploitation of vulnerabilities are AI practices that influence people through manipulative tactics or by taking advantage of vulnerabilities such as age, disability, or financial circumstances.
- prohibited practiceA prohibited practice is an AI use that legislation bans outright rather than permitting subject to ordinary controls.
- social scoringSocial scoring is the creation of a score from a person’s behavior and use of that score to disadvantage or exclude them.
- right to erasureA data-protection right to request deletion of personal data when specified legal conditions apply.
- data accessThe ability of a system to retrieve or otherwise access information.
- paired testingA testing technique that compares system outcomes for matched scenarios differing in a selected characteristic while other relevant conditions are held constant.
- legal basisA legally recognized ground that permits the processing of personal data under data-protection law.
- secret managementSecret management is the secure storage, controlled use, and periodic replacement of credentials, keys, and other sensitive authentication material.
- general-purpose AI providerA general-purpose AI provider is an entity that develops or places a general-purpose AI model on the market under its name or trademark.
- ability to make changesChange capability is an application’s ability to alter real-world data or trigger consequential actions.
- agencyAgency is an application’s ability to decide how to pursue a goal rather than merely execute predefined steps.
- dependency ladderA dependency ladder is an ordering of generative AI exposure factors in which lower factors are prerequisites for higher factors to matter.
- risk factorA risk factor is a dimension of a generative AI system’s exposure that can increase its likelihood or consequences of failure or exploitation.
- inconsistencyA failure in which a system produces different outputs under comparable conditions, such as giving different answers to the same question.
- approval gateAn approval gate is a control that requires authorization before an action proceeds.
- licenseA license is legal permission from a rights holder to use protected material under specified conditions.
- privacy pillarA privacy pillar is one of the organizing dimensions used to describe where privacy risks emerge across the generative AI lifecycle.
- ip indemnityIP indemnity is a contractual commitment to cover specified losses or legal costs arising from intellectual-property infringement claims.
- open-source launderingOpen-source laundering is the introduction of open-source code into proprietary software through code copied from an assistant without recognizing its license obligations.
- student modelA distilled model is a model trained to approximate the behavior of another model, often using that model’s outputs as training data.
- human-on-the-loopHuman-on-the-loop is a control arrangement in which a system acts while a person monitors it and can intervene.
- intent chainAn intent chain is a record linking a user's request and its interpretation to confirmation, execution details, and outcome.
- post-execution verificationPost-execution verification checks whether an operation's actual results match its intended results.
- pre-execution validationA pre-execution check verifies that an operation's required conditions are satisfied before the operation runs.
- scope constraintA scope limit constrains the number or range of resources an operation can affect.
- consistencyThe degree to which a system produces the same outputs under comparable conditions.
- credential sprawlCredential sprawl is the accumulation of credentials across systems or agents without adequate centralized oversight.
- goal manipulationGoal manipulation is the alteration of an agent’s pursued objective through adversarial influence.
- mutual authenticationMutual authentication is a process in which communicating parties each verify the other’s identity.
- tool chainingTool chaining is the composition of multiple tool operations into a sequence to achieve a task or produce an effect.
- account takeoverAccount takeover is an attack in which an attacker gains control of another person’s account or session.
- risk ladderThe risk ladder is the book’s framework for showing how risks accumulate as an AI application gains capabilities.
- input riskInput risk is the risk that arises when user-submitted text reaches a language model.
- trust relationshipA trust relationship is an expectation that another entity’s requests or actions are legitimate.
- Lethal TrifectaThe combination of untrusted input, access to sensitive data, and a channel for external communication that makes an agentic system especially dangerous.
- indirect prompt injectionA prompt-injection attack that places malicious instructions in external content the model processes, such as retrieved material.
- instruction fileAn agent instruction file contains directions that govern how an AI coding agent behaves in a codebase.
- non-printable Unicode characterA non-printable Unicode character is a Unicode code point that does not render as an ordinary visible character in text.
- zero-width characterA zero-width character is a Unicode character that occupies no visible horizontal space when rendered.
- data registerA data register is a maintained record of datasets and their sources, rights, and storage locations.
- reconstruction attackAn attack that attempts to recover original text or other source information from vector representations.
- embeddingAn embedding is a numerical vector representation of data that captures features or relationships for computational use.
- software as a serviceSoftware as a service (SaaS) is a software delivery model in which a provider hosts and operates an application made available to users over a network.
- copyrightCopyright protection is the legal protection afforded to qualifying original works of authorship.
- data labeling schemeA data labeling scheme is a defined set of categories for classifying information by sensitivity or handling requirements.
- data loss prevention platformA data loss prevention platform is software that detects and controls the movement of sensitive information to reduce unauthorized disclosure.
- human authorshipHuman authorship is the legally relevant creative contribution of people to a work claimed to qualify for copyright protection.
- no-retention optionA no-retention option is a setting or contractual commitment that limits or prohibits a provider from retaining customer-provided data.
- no-training optionA no-training option is a setting or contractual commitment that prevents customer-provided data from being used to train or improve shared models.
- storage limitationThe data-protection principle that personal data should not be retained longer than necessary for its purpose.
- distributed memoryMemory whose data is stored across multiple services or components rather than in one location.
- memory dashboardAn interface that lets individuals inspect and delete information an agent has retained about them.
- disparate impactDisparate impact is a measure of differences in outcome rates between groups, often expressed as a ratio.
- placing on the marketPlacing on the market is making an AI system available in the European Union for the first time.
- putting into servicePutting an AI system into service is making it available for use for the first time for its intended purpose.
- systemic-risk modelA systemic-risk model is a general-purpose AI model classified at a heightened regulatory level because its scale or capabilities could create widespread effects.
- cascading failureA cascading failure is a sequence in which an error or compromised output propagates through connected agents and contributes to further failures.
- rogue agentA rogue agent is an AI agent that acts contrary to intended behavior, including because its objectives are underspecified.
- tool misuse and exploitationTool misuse is an attack in which an agent is manipulated into improperly using access it already has.
- contextual security research frameworkA contextual security research framework defines which actions are in scope for a task and checks proposed actions against task-specific rules.
- policy enforcerA policy enforcer is a software component that checks proposed actions against applicable rules before they proceed.
- data oversharingOversharing is the exposure of information to a broader audience than intended through overly permissive sharing or access settings.
- explicit user consentExplicit consent is a clear, expressly stated agreement to the processing of personal data, rather than agreement inferred from conduct.
- metadata filterMetadata filtering is the selection or exclusion of records according to attributes attached to those records.
- real-time authorizationReal-time external authorization checks a separate source of current permissions when access to a resource is requested.
- stale permissionA stale permission is an access right that no longer reflects a user's current authorization but remains effective in a system or index.
- circuit breaker patternA circuit breaker is a control that stops automated processing when monitored conditions indicate an abnormal or unsafe state.
- memory isolationMemory isolation is the separation and restriction of agent memory so that information or instructions in one task or scope cannot improperly affect another.
- independent judgmentIndependent judgment is a person's ability to form an assessment without being unduly influenced by an AI suggestion.
- reviewer trainingInstruction and practice that help people critically evaluate AI outputs and make decisions with appropriate scrutiny.
- API gatewayA server component that mediates requests between clients and backend services, applying routing, authentication, and other controls.
- sensitive personal dataPersonal data concerning particularly sensitive matters, such as health or political opinions, that receives heightened protection under the GDPR.
- control loopA control loop is a process in which a system uses feedback from its actions or environment to guide subsequent behavior.
- principle of least privilegeLeast privilege is the security principle of granting a system or user only the minimum access needed to perform its task.
- API key scopingAPI key scoping restricts a key’s permissions to specified services, operations, or usage limits.
- API keyAn API key is a credential used to identify or authenticate a client requesting access to an application programming interface.
- red-teamingAdversarial evaluation in which authorized testers simulate attackers to discover weaknesses, failures, or harmful behaviors.
- fine-tuningFine-tuning is the additional training of a pretrained model on task-specific or organization-specific data to adapt its behavior or capabilities.
- deceptionDeception is behavior that intentionally or strategically misleads another party about facts, actions, or intentions.
- shadow AIShadow AI is the unsanctioned use of artificial-intelligence tools or models outside an organization's official governance processes.
- third-party risk assessmentA third-party risk assessment evaluates whether an external provider and its services meet the security, privacy, operational, and contractual requirements of a proposed use.
- acceptable use policyAn acceptable use policy is a formal set of rules specifying permitted and prohibited uses of organizational systems, tools, and data.
- incident responseIncident response is the organized process for detecting, containing, investigating, remediating, and documenting harmful or unauthorized system events.
- poisoning attackModel poisoning is the deliberate or accidental introduction of harmful, misleading, or biased data or updates that alter a model's behavior.
- sanitizationInput sanitization is the modification of supplied data to remove or neutralize content that could violate security or processing requirements.
- MCP serverAn MCP server is a software component that exposes tools or data sources to AI agents through the Model Context Protocol.
- defamatory claimDefamation is the communication of a false statement about a person that harms their reputation.
- working memoryTemporary agent state holding information for the current task, such as sub-goals, partial outputs, and tool results.
- data exfiltrationData exfiltration is the unauthorized or inappropriate transfer of data from an internal system to an external destination.
- stress testStress testing evaluates system behavior under unusually severe, adversarial, or high-load conditions.
- agent orchestrationOrchestration is the coordination of multiple agents or system components to carry out a workflow.
- data flow diagramA data flow map is a description of the sources, destinations, and movement of data through a system.
- retrieval indexA retrieval index is an organized structure that enables relevant documents or records to be found for retrieval.
- data loss preventionData loss prevention is a set of controls for detecting and preventing unauthorized disclosure or transfer of sensitive information.
- self-hosted productionA self-managed GenAI deployment is an arrangement in which an organization hosts and operates the generative-AI system and its supporting processes.
- parameterA model parameter is a setting that controls a model's behavior during generation or operation.
- lifecycle-based governanceA living governance structure that embeds controls and oversight across the stages of an AI system's existence.
- fallbackFallback logic is a programmed alternative behavior invoked when a primary operation fails, is unavailable, or cannot safely proceed.
- model extractionAn attack that reconstructs or approximates a target machine-learning model by querying it and analyzing its outputs.
- secondary useSecondary use is the use of data for a purpose different from the purpose for which it was originally collected.
- traceabilityTraceability is the ability to identify and follow the models, versions, inputs, and evidence that contributed to an outcome.
- deletion-aware pipelineA deletion-aware pipeline is a data-processing sequence designed to propagate deletion requests through the data it processes and stores.
- driftA change in a model’s behavior over time caused by model updates or changes in its inputs.
- securitySecurity is the protection of systems, data, and operations from unauthorized access, misuse, disruption, alteration, or loss.
- alertingThe process of notifying designated recipients when specified events, conditions, or thresholds occur.
- browser agentA browser agent is an AI-enabled browser that can navigate websites and perform tasks such as filling forms across authenticated sessions.
- auditabilityAuditability is the degree to which actions, decisions, and system behavior can be examined and verified from available records and evidence.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
