When a GenAI system produces a harmful response, an output and timestamp may not be enough to identify the failed safeguard. A log records events and associated details so operators can monitor behavior, investigate incidents, and produce audit evidence. section 3.6.3 section 3.6.1
Record request metadata before a vendor call crosses the organization’s trust boundary, including the originating user, timestamp, and key payload indicators. Operational logs can also capture usage, token consumption, and errors for budget forecasting and compliance. section 2.3.1
For a traceable GenAI request, record more than the final input and output:
- Capture timestamps, user and session identity, model version, token counts, tools invoked, and whether actions were allowed or blocked. section 4.3.1
- Record prompt versions, retrieval sources, guardrail decisions, and human review steps; link events with a shared request identifier. section 4.3.1
- For agentic systems, include the tool called, data it could reach, requested action, and any human approval. section 3.6.1
Logs can themselves expose sensitive information. Redact or tokenize personal information before storage, and decide retention and export arrangements before launch. For high-risk systems requiring six months of operating logs, a provider’s 30-day API retention alone is insufficient. section 4.3.1 section 7.4.3
An insufficient incident trail
A medical pilot gave a borderline case unapproved reassurance without clinician review. Its logs showed the output, timestamp, and model name, but not the prompt, provider model version, or reason review was missed. The team preserved the logs, opened an incident, and required clinician review of every output while assigning owners to the gaps. section 3.6.3
Table 3.14 Operations and monitoring artifacts
|
Activity
|
Common Artifacts
|
Main Owner
|
Systems commonly used
|
|
System configuration record
|
Model, prompts, data sources, connected services, safety
settings, and effective dates
|
Engineering
|
Source control systems such as GitHub. MLFlow or a cloud
model registry for model versions.
|
|
Request Record
|
Technical record of each request: user input information
retrieved, safeguards applied, human escalation, model responses, tool calls.
|
Engineering
|
Application logs or an AI tracing system such as Langfuse,
Arize AX, Datadog, Braintrust. OpenTelemetry can send these records to an
existing logging platform.
|
|
Multi Stakeholder Monitoring
|
What Engineering, Security, Product, Support, and
specialist reviewers must check; how often; when to raise an alert; and who
must respond. Also includes complaints, appeals, and human overrides
|
System owner coordinates, each team owns its checks.
|
Arize AX, Fiddler, WhyLabs, Arthur, or Datadog for system
and model behaviour; Splunk or Microsoft Sentinel for security alerts;
Zendesk, Salesforce, or Jira Service Management for user reports
|
|
Change Review
|
For each proposed change to the model, prompts, knowledge
sources, connected services, or safety settings: the reason, test results,
approval, and date it went live.
|
Product, Engineering
|
ServiceNow or Jira Service Management for approval;
GitHub, GitLab, or Azure DevOps for the change and the date it went live
|
|
Incident Response and Recovery
|
Incident ticket, timeline, preserved evidence, who was
affected, action, retesting, and approval to resume
|
Incident Response, Security, Product, Legal
|
PagerDuty, incident.io, Rootly, FireHydrant, ServiceNow,
or Jira Service Management
|
|
Secure Decommissioning
|
Records the shutdown, closed access, treatment of data,
and unresolved cases.
|
System Owner
|
ServiceNow or Jira for the shutdown work; the AI inventory
for retired status; vendor or cloud accounts for access closure
|
Use the operations artifact table to identify request records, their owners, and possible logging or tracing systems. section 3.6.1




















