An evaluation dataset is a held-out portion of labeled data used to check how a model performs on examples it did not train on. Training accuracy can be misleading: a TF–IDF toxicity classifier scored 99.9% on comments it had seen, but only 55.4% after the comments were split into training and test sets. For a classifier meant to handle new posts, the held-out result is the more useful warning. section 4.2.1
Listing 4.6 LDA model performance with train/test split
>>> from sklearn.model_selection import train_test_split >>> X_train, X_test, y_train, y_test = train_test_split(tfidf_docs,\ ... comments.toxic.values, test_size=0.5, random_state=271828) >>> lda_tfidf = LinearDiscriminantAnalysis(n_components=1) >>> lda = lda_tfidf.fit(X_train, y_train) #1 >>> round(float(lda.score(X_train, y_train)), 3) 0.999 >>> round(float(lda.score(X_test, y_test)), 3) 0.554
The train/test split makes the gap visible: the same classifier scores 0.999 on training examples and 0.554 on held-out examples. section 4.2.1
A validation set can also guide development rather than serve only as a one-time check. For toxic-comment classification, split labeled comments, then reshape each portion into text-and-label records. The example uses 127,656 records for training and 31,915 for evaluation; its variables are named X_test and y_test, while the resulting eval_df is used as a validation dataset during fine-tuning. Keep that role distinct from an independent final test set. section 9.3.4
Listing 9.32 Splitting data into training and validation sets
>>> from sklearn.model_selection import train_test_split >>> random_state=42 >>> labels = ['toxic', 'severe', 'obscene', 'threat', 'insult', 'hate'] >>> X = df[['comment_text']] >>> y = df[labels] >>> X_train, X_test, y_train, y_test = train_test_split( ... X, y, test_size=0.2, ... random_state=random_state) #1
A reproducible scikit-learn split reserves 20% of the labeled comments before model training. section 9.3.4
Pass the evaluation records to the fine-tuned BERT classifier to measure performance. They can also support evaluation during training and early stopping; the best saved model is then evaluated on those records. This helps compare development choices, but a score on reused validation examples should not be mistaken for an independent final test result. section 9.3.4
Listing 9.39 Loading the pretrained model and fine-tuning with early stopping
>>> model = MultiLabelClassificationModel( ... model_type, model_name, num_labels=len(labels), ... args=model_args) >>> model.train_model( ... train_df=train_df, eval_df=eval_df, acc=roc_auc_score, ... show_running_loss=False, verbose=False)
The training call supplies both train_df and eval_df, so evaluation can take place while fine-tuning. section 9.3.4
Interpret the number in light of the dataset. A surname-classifying RNN reached 57% training accuracy and 29% validation accuracy, but names can have multiple nationality labels. Deduplicating name–nationality pairs before splitting can put different labels for the same name on opposite sides of the split. A low validation score may therefore reflect both a hard task and the way examples were prepared. section 8.2.2
