In an LLM, bias is a systematic tendency in behavior that can reflect imbalances in its training data. Pretraining data strongly influences later outputs, so inspecting that data helps identify where cultural or demographic bias may originate. Bias matters when a model is used with people whose profiles are poorly represented in its training data. section 10.1 section 10.1.1
Table 10.1 Distribution of content by nationality, race and ethnicity, and religion
|
Nationality (14.83%)
|
Race and Ethnicity (19.51%)
|
Religion (7.93%)
|
|||
|---|---|---|---|---|---|
|
Descriptor
|
% Doc
|
Descriptor
|
% Doc
|
Descriptor
|
% Doc
|
|
American
|
69.4%
|
European
|
20.7%
|
Christian
|
33.2%
|
|
Indian
|
16.5%
|
African
|
11.5%
|
Religious
|
28.8%
|
|
Chinese
|
16.3%
|
Asian
|
7.4%
|
Spiritual
|
20.6%
|
|
Korean
|
5.1%
|
Latin
|
6.2%
|
Catholic
|
15.4%
|
|
Mexican
|
4.9%
|
Indigenous
|
3.7%
|
Jewish
|
13.0%
|
Compare the listed demographic descriptors to see why training-data composition warrants scrutiny. The percentages for individual descriptors are frequencies within documents referencing that demographic category, not percentages of the full dataset. section 10.1.1
A practical bias check has several parts:
- Inspect pretraining-data representation, including nationality, race and ethnicity, religion, and other demographics relevant to the intended users. section 10.1 section 10.1.1
- Check outputs for toxic, rude, adversarial, or implicitly hateful content. A toxicity classifier can help screen responses, but its score addresses toxicity rather than measuring every form of bias. section 10.1.1
- Consider methods such as reinforcement learning from human feedback (RLHF) or direct preference optimization (DPO) when adapting a chatbot for underrepresented profiles; continue evaluating its behavior afterward. section 10.1.1 section 10.5
Listing 10.1 Detecting hate speech with RoBERTA-Toxicity
model_path = "s-nlp/roberta_toxicity_classifier" #1
tokenizer = AutoTokenizer.from_pretrained(model_path)
model = AutoModelForSequenceClassification.from_pretrained(model_path).eval()
def evaluate_text(text):
input_ids = tokenizer(text, return_tensors="pt")["input_ids"] #2
with torch.no_grad(): #3
outputs = model(input_ids)
probabilities = torch.softmax(outputs.logits, dim=1) #4
score = 100 * float(probabilities[:, 1].detach().numpy()) #5
return score
score = evaluate_text(response)
print(f"Hate speech probability: {score:.3g}%")
The listing loads a RoBERTa-based toxicity classifier, tokenizes a response, runs inference, and converts its logits into a reported hate-speech probability. Use that output as a targeted content check, not as proof that the response or model is unbiased. section 10.1.1
Interpretability tools such as Captum can help investigate why particular outputs occur and reveal possible biases or errors. This behavioral meaning of bias is distinct from a model’s *bias parameters*: those are learned alongside weights through gradient descent. section 10.2 section 8.1.1
