Toxicity
Toxicity is the presence or degree of abusive, hateful, offensive, threatening, or otherwise harmful content in language or other material.
Natural Language Processing in Action, Second Edition
Toxicity is the presence of abusive, hateful, threatening, or otherwise harmful language in generated or user-provided text.
- How it is used
- Supports moderation of online comments and filtering of harmful chatbot behavior.
- How deep it goes
- Defines toxic comments and works through labeled datasets, feature representations, classifier metrics, BERT fine-tuning, and guardrail design.
- Examples
- Uses labeled Wikipedia comments, an LDiA classifier comparison, and a six-label social-media dataset for BERT fine-tuning.
- What you can do after
- Apply it via scikit-learn or a pretrained BERT classifier to identify harmful comments, and evaluate missed cases with recall.
Read how Natural Language Processing in Action, Second Edition explains toxicity →
Transformers in Action
Toxicity is the presence of harmful, abusive, hateful, or otherwise damaging content in model outputs or data.
- How it is used
- Detects harmful model behavior and supports prompt filtering, output review, bias analysis, and safer production deployment.
- How deep it goes
- Explains classifier scoring and scanner integration procedurally, but defers detailed classifier architecture and threshold calibration.
- Examples
- Uses a Llama 2 completion, RoBERTa-Toxicity scoring code, and LLM Guard tests containing violent language and ordinary prompts.
- What you can do after
- Diagnose toxicity with a fine-tuned classifier and operate configurable scanners that block unsafe prompts before model inference.
Related concepts
- guardrailaddressed by · A guardrail is a constraint, safeguard, or control that limits actions or behavior to reduce risk and keep a system or process within acceptable bounds.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.