A training dataset is the collection of examples a model learns from. For toxic-comment classification, each example pairs a comment with a label: toxic or nontoxic for a binary classifier, or a set of labels such as toxic, threat, and insult for a multilabel classifier. The examples used to train a model should not also be its only evidence of success. section 4.2.1 section 9.3.4
A practical workflow keeps the examples useful and the evaluation honest:
- Inspect the comments and labels before training. Check what the task asks a person to judge, how the text is formatted, and whether the labels appear accurate. section 9.3.4
- Separate training examples from held-out examples. Use the training portion to fit the model and the held-out portion to assess predictions on comments it has not seen during fitting. section 4.2.1 section 9.3.4
- Put the examples in the form the model expects: TF–IDF vectors and toxicity labels for a linear discriminant classifier, or text and label lists for BERT fine-tuning. section 4.2.1 section 9.3.4
Why a training score can mislead
A linear discriminant classifier fitted to TF–IDF comment vectors scored 0.999 when evaluated on those same examples. After a train/test split, its training score was still 0.999, but its held-out score was 0.554. The first score describes performance on familiar comments; it does not establish how well the classifier will handle new ones. section 4.2.1
Listing 4.6 LDA model performance with train/test split
>>> from sklearn.model_selection import train_test_split >>> X_train, X_test, y_train, y_test = train_test_split(tfidf_docs,\ ... comments.toxic.values, test_size=0.5, random_state=271828) >>> lda_tfidf = LinearDiscriminantAnalysis(n_components=1) >>> lda = lda_tfidf.fit(X_train, y_train) #1 >>> round(float(lda.score(X_train, y_train)), 3) 0.999 >>> round(float(lda.score(X_test, y_test)), 3) 0.554
The split, fit, and two score calls make the difference between training performance and held-out performance visible in a short Python experiment. section 4.2.1
For BERT fine-tuning, the toxic-comment CSV provides comment text and six labels. A split of its 159,571 rows produces 127,656 training examples and 31,915 held-out examples, reformatted as `text` and `labels` for `MultiLabelClassificationModel`. The pretrained model is fine-tuned on the training portion and evaluated separately; a later run uses evaluation during training and early stopping. Reviewing the comments and labels remains important: a convincing metric cannot replace understanding the task or the data. section 9.3.4
