A classifier is a model that assigns an input to one or more predefined categories or labels. In language-model systems, it can identify whether text is safe, toxic, positive, negative, neutral, or related to self-harm. section 3.7 section 2.9
Classifiers can support safety before or after generation. A safety classifier may scan large pretraining datasets so toxic material can be filtered or labeled as unsafe for conditional pretraining. It may also detect risky model responses after generation or help evaluate outputs during safety post-training. section 3.7 section 3.8
Their usefulness depends on both the task and the cost of making reliable judgments:
- Apply them to ordinary language tasks, such as rating product reviews as positive, negative, or neutral. section 2.9
- Use them to identify toxic or unsafe content across vast and varied datasets. section 3.7
- Treat their decisions cautiously: frontier-scale datasets make classification extremely expensive, so efficient classifiers may be less accurate than the language models they support. section 3.7
Self-harm safety
A classifier trained on labeled conversations can detect whether a generated response contains self-harm instructions or encouragement. Such judgments can guide post-processing or safety training, although producing dependable labels requires substantial work on sensitive material. section 3.8
