A generative AI application should not assume that a model’s raw response is ready for users. Content filtering screens material for harmful or inappropriate content before it is shared. It can identify and restrict hate speech, profanity, or violent content. This matters because model responses can vary, and even a carefully written prompt may produce an unsuitable result. section 10.6 section 13.5 section 6.7
In an application architecture, response filtering sits in the output-processing path. Classifiers and NLP tools can screen generated output for appropriateness, bias, and accuracy before presentation. A safety system can also monitor inputs as well as outputs, helping prevent attacks and catch model mistakes. Filtering is one safeguard within a broader responsible-AI approach, not a substitute for application design, testing, or user review. section 10.6 section 13.3.3
A practical rollout has three parts:
- Select a filtering model or service, then configure it for the application’s needs. Google Perspective API and Azure Content Filtering are named options; their capabilities and setup are not specified here. section 13.5
- Decide how users encounter an intervention. For potentially harmful queries, a predetermined response can direct users to relevant policies; review-and-edit controls let users check generated output. section 13.3.3
- Test and refine continuously. Check both missed harmful content and legitimate content blocked by mistake: overfiltering can restrict open dialogue, while filtering cannot catch every harmful result. section 13.5
Recognizing a filtered completion
When inspecting a chat completion API response, check `finish_reason`. A value of `content_filter` means some of the completion was filtered because of harmful content. Treat that as a diagnostic signal when investigating why a response differs from what the application expected; it does not describe the full moderation decision. section 3.4.2
Table 13.2 Application-level RAI mitigations
|
Mitigation
|
Description
|
|---|---|
|
Review and edit
|
Encourage users to critically assess AI-generated outputs, supporting efficient correction and highlighting potential inaccuracies.
|
|
User responsibility
|
Remind users of their accountability for the final content, especially when reviewing suggestions such as code.
|
|
Citations
|
If AI content is reference based, cite sources to clarify the origin of the information.
|
|
Predetermined responses
|
For potentially harmful queries, provide thoughtful, precrafted responses to maintain decorum and direct users to appropriate policies.
|
|
Input/output limitation
|
Restrict input and output lengths to minimize the production of undesirable content and prevent misuse.
|
|
AI role disclosure
|
Inform users they are interacting with an AI, not a human, and disclose if the content is AI generated, which may be legally required.
|
|
Bot detection
|
Implement mechanisms to prevent the creation of APIs over your product, ensuring controlled use.
|
|
Anthropomorphism prevention
|
Anthropomorphism prevention means ensuring that AI systems don’t seem human. It’s about clear communication that AI doesn’t think or feel to avoid confusion and ensure people use AI properly, without expecting it to act like a human. Implement safeguards against AI outputs that suggest human-like qualities or capabilities, reducing misinterpretation risks.
|
|
Structured inputs/outputs
|
Use prompt engineering to structure inputs and limit outputs to specific formats, avoiding open-ended responses.
|
The application-level mitigation table gives concrete interventions to pair with filtering, including review and edit, predetermined responses, input/output limits, and AI-role disclosure. section 13.3.3
For enterprise deployment, place filtering alongside output validation, quality assurance, continuous monitoring, and responsible-AI controls. The aim is to reduce harmful results while keeping useful, legitimate responses available—not to claim that every unsafe or misleading output has been eliminated. section 10.6 section 10.2.3 section 13.5
