A message can express more than one intent, just as a comment can be toxic in more than one way. Multilabel classification assigns zero or more labels to each input rather than forcing a single choice. It is also called multilabel tagging; “tagger” is often clearer than “classifier” when several labels can apply at once. section 8.2.4 section 9.3.4
The distinction from multiclass classification matters. A multiclass classifier chooses one of several mutually exclusive categories. A tagger can assign every applicable category: a surname may have multiple appropriate nationalities, and a user message may contain several intents. One implementation uses an independent binary classifier for each possible tag. When adapting a multiclass neural classifier, use an element-wise sigmoid rather than a softmax output. section 8.2.4 section 12.2.2
Tagging toxic comments
A comment dataset has six label columns: toxic, severe, obscene, threat, insult, and hate. Each row pairs comment text with a multi-hot vector: [0, 0, 0, 0, 0, 0] has none of those labels, while multiple 1s can mark overlapping kinds of toxicity. To fine-tune a pretrained BERT model, split the data into training and evaluation sets, convert each row to [text, labels], and pass those rows to simpletransformers’ MultiLabelClassificationModel. Evaluate on held-out comments rather than assuming that a plausible tag is correct. section 9.3.4
Listing 9.33 Creating datasets for your model
>>> def get_dataset(X, y):
... data = [[X.iloc[i][0], y.iloc[i].values.tolist()]
for i in range(X.shape[0])]
... return pd.DataFrame(data, columns=['text', 'labels'])
...
>>> train_df = get_dataset(X_train, y_train)
>>> eval_df = get_dataset(X_test, y_test)
>>> train_df.shape, eval_df.shape
((127656, 2), (31915, 2))
>>> train_df.head() #1
text labels
0 Grandma Terri Should Burn in Trash \nGrandma T... [1, 0, 0, 0, 0, 0]
1 , 9 May 2009 (UTC)\nIt would be easiest if you... [0, 0, 0, 0, 0, 0]
2 "\n\nThe Objectivity of this Discussion is dou... [0, 0, 0, 0, 0, 0]
3 Shelly Shock\nShelly Shock is. . .( ) [0, 0, 0, 0, 0, 0]
4 I do not care. Refer to Ong Teng Cheong talk p... [0, 0, 0, 0, 0, 0]
The dataset conversion makes the training format concrete: one text column and one list of label values per row. section 9.3.4
Intent tagging can also guide a chatbot’s response. A math tutor might detect an answer or numerical expression while disregarding an apparent insult. Scores are not a resolution of every ambiguity, though: “mean” could refer to an average or to unkindness, and the intent recognizer may need more context. section 12.2.2
>>> pip install mathtext
>>> from mathtext.predict_intent import predict_intents_list
>>> predict_intents_list('you are mean forty 2') #1
[('answer', 0.7568096903748207),
('numerical_expression', 0.25867391608346874),
('support', 0.23595954822965573),
('exit', 0.2259397417552966),
('negative_ux', 0.22204299015973564),
('next_lesson', 0.19376506261045864),
...]
>>> predict_intents_list('you are jerk infinity')
[('negative_ux', 0.7653253605153358),
('harder', 0.39947314108352083),
('main_menu', 0.3104721326013856),
('next_lesson', 0.2752392592506451),
('hint', 0.2745974777104082),
('answer', 0.22114933410265608),
('faq', 0.20082098769547646),
...]
The mathtext predictions show several scored intents for each ambiguous message, rather than one compulsory class. section 12.2.2
In production, a tagger can distinguish kinds of potentially toxic chatbot output for filtering and review. Collect examples of failures, retrain the filter, and track its scores to prioritize messages for human inspection. Such guardrails provide measurable signals; they do not make a chatbot perfectly safe. section 10.1.3
