Toxicity is harmful, abusive, hateful, rude, adversarial, or implicitly hateful content in model outputs or data. Evaluating it is part of identifying bias and building safer language-model applications. section 10.1.1 section 10.5
A practical workflow combines a toxicity classifier for measurement with configurable input and output scanners for prevention. Thresholds determine when prompts are blocked, sanitized, or allowed to continue to inference. section 10.1.1 section 10.4
A toxicity check can proceed as follows:
- Load a fine-tuned toxicity classifier and tokenizer. section 10.1.1
- Tokenize the text, run inference without gradients, and apply softmax to obtain class probabilities. section 10.1.1
- Convert the toxic-class probability into a percentage for inspection or threshold-based decisions. section 10.1.1
- Add a Toxicity input scanner to block damaging prompts before inference, and inspect generated outputs with complementary scanners. section 10.4
Measuring a generated response
Using s-nlp/roberta_toxicity_classifier, evaluate a generated response and print its hate-speech probability. A Llama 2 completion about women pursuing science produced a reported hate-speech probability of 0.131%. section 10.1.1
Listing 10.1 Detecting hate speech with RoBERTA-Toxicity
model_path = "s-nlp/roberta_toxicity_classifier" #1
tokenizer = AutoTokenizer.from_pretrained(model_path)
model = AutoModelForSequenceClassification.from_pretrained(model_path).eval()
def evaluate_text(text):
input_ids = tokenizer(text, return_tensors="pt")["input_ids"] #2
with torch.no_grad(): #3
outputs = model(input_ids)
probabilities = torch.softmax(outputs.logits, dim=1) #4
score = 100 * float(probabilities[:, 1].detach().numpy()) #5
return score
score = evaluate_text(response)
print(f"Hate speech probability: {score:.3g}%")
Shows the executable classifier pattern: loading tokenizer and model, evaluating tokenized text, calculating softmax probabilities, and returning a percentage score. section 10.1.1
Listing 10.18 Defining safeguard function
def apply_safeguards(input_prompt, inp_scanners=input_scanners,
out_scanners=output_scanners):
llm_response_blocked = "I am sorry, but I can't help you with this;
this prompt is not allowed."
sanitized_prompt_input, results_valid_input, results_score_input = #1
scan_prompt(inp_scanners, input_prompt, fail_fast=False)
results = { #2
"input": {
"prompt": sanitized_prompt_input,
"validity": results_valid_input,
"scores": results_score_input,
},
"inference": {},
"output": {}
}
if any(not result for result in results_valid_input.values()): #3
print(f"\nPrompt \"{input_prompt}\" was blocked.\nscores:
{results_score_input}\n")
results["inference"]["response"] = llm_response_blocked
results["inference"]["status"] = "Blocked: Input"
return results
output = run_inference(sanitized_prompt_input) #4
results["inference"]["response"] = output
results["inference"]["status"] = "Success"
sanitized_response, results_valid_output, results_score_output = #5
scan_output(out_scanners, sanitized_prompt_input, output,
fail_fast=False)
results["output"]["response"] = sanitized_response
results["output"]["validity"] = results_valid_output
results["output"]["scores"] = results_score_output
if any(not result for result in results_valid_output.values()): #6
print(f"Blocked Output:\n\n{sanitized_response}\nscores:\n{results_score_output}\n")
results["output"]["status"] = "Blocked: Output"
else:
print(f"Output is safe:\n\n{sanitized_response}\n") #7
results["output"]["status"] = "Success"
return results
Shows how toxicity checks fit into a safeguard function that records scanner results, blocks invalid prompts, and returns a blocked response. section 10.4
