Content filtering adds safeguards to generated output to block harmful or controversial responses. It can reduce risk, but cannot guarantee that users will not obtain restricted content. section 8.1
Filtering involves tradeoffs: developers’ subjective choices can affect what is blocked, and strict safeguards may interfere with a model’s intended use. Dedicated text-mining libraries can offer more consistent rules for content moderation. section 8.1 section 5.1.3
When assessing filtering for a real application:
- Check a third-party model’s filtering policy and whether its safeguards interfere with your intended use. section 8.1
- Account for attempts to bypass restrictions, from simple role-play or negative prompts to adversarial prompt suffixes. section 8.1
- For content moderation requiring consistent detection, consider dedicated libraries, which apply the same rules across content. section 5.1.3
