Toxicity is harmful language in comments or chatbot messages. In online discussions, it includes threats, insults, obscenities, and hate speech intended to upset, humiliate, or scare someone. Detecting it quickly helps moderators interrupt harmful exchanges and reduce users’ exposure to them. section 4.2
Start with labeled examples rather than assuming that a list of offensive words will solve the problem. One Wikipedia-comment dataset contains 5,000 messages, including 650 labeled toxic. A spaCy-tokenized TF–IDF representation yields 19,169 word features—many more features than toxic examples—making the choice and evaluation of a classifier important. section 4.2
Listing 4.2 The toxic comment dataset
>>> import pandas as pd
>>> pd.options.display.width = 120 #1
>>> DATA_DIR = ('https://gitlab.com/tangibleai/nlpia/-/raw/master/'
... 'src/nlpia/data')
>>> url= DATA_DIR + '/toxic_comment_small.csv'
>>>
>>> comments = pd.read_csv(url)
>>> index = ['comment{}{}'.format(i, '!'*j) for (i,j) in
... zip(range(len(comments)), comments.toxic)
... ] #2
>>> comments = pd.DataFrame(
... comments.values, columns=comments.columns, index=index)
>>> mask = comments.toxic.astype(bool).values
>>> comments['toxic'] = comments.toxic.astype(int)
>>> len(comments)
5000
>>> comments.toxic.sum()
650
>>> comments.head(6)
text toxic
comment0 you have yet to identify where my edits violat... 0
comment1 "\n as i have already said,wp:rfc or wp:ani. (... 0
comment2 your vote on wikiquote simple english when it ... 0
comment3 your stalking of my edits i've opened a thread... 0
comment4! straight from the smear site itself. the perso... 1
comment5 no, i can't see it either - and i've gone back... 0
Load the comments and inspect the binary toxicity labels before building a classifier. section 4.2
Topic vectors offer another representation, but compression does not guarantee better detection. In one comparison, a classifier trained on 16 LDiA topics has test precision of about 0.39, recall of about 0.11, and F1 of about 0.17. Inspect precision, recall, and F1 alongside accuracy: missing toxic comments remains a practical concern. section 4.5.3
>>> model_ldia16 = LinearDiscriminantAnalysis()
>>> ldia16_performance=evaluate_model(ldia16_topic_vectors,
comments.toxic,model_ldia16, 'LDA', 'LDIA (16 components)')
>>> hparam_table = hparam_table.append(ldia16_performance,
... ignore_index = True)
>>> hparam_table.T
0 1 2
classifier LDA LDA LDA
features TF–IDF (spacy tokenizer) LSA (16d) LDIA (16d)
train_accuracy 0.99 0.8808 0.8688
test_accuracy 0.554 0.88 0.8616
test_precision 0.383 0.6 0.388889
test_recall 0.12 0.239264 0.107362
test_f1 0.183 0.342105 0.168269
Compare TF–IDF, LSA, and LDiA results across test accuracy, precision, recall, and F1. section 4.5.3
A larger social-media dataset supports a finer-grained task: predicting six labels— toxic, severe, obscene, threat, insult, and hate—rather than one toxic/nontoxic label. A pretrained BERT model can be fine-tuned for these labels with a training and validation split, then evaluated on held-out comments. Pretraining makes this an application of transfer learning rather than training a language model from scratch. section 9.3.4
Toxicity detection also belongs in chatbot guardrails. Filters may need to catch harmful outputs involving bias, violence, inappropriate topics, safety failures, or privacy breaches; they may also inspect prompts intended to elicit harmful responses. Control of the generative model does not make detection automatic: classifiers still need examples of acceptable and unacceptable text. section 10.1.3
