A classifier assigns an input to a predefined category. For text, it first needs a numerical representation of the words or their meaning. The output can be a binary decision, such as toxic versus nontoxic, or a choice among several categories. That simple decision can help prioritize comment review, analyze sentiment, or route a query to the right service. section 4.2 section 10.1.3 section 2.6.2
Start with labeled examples. A Wikipedia-discussion dataset contains 5,000 comments, 650 labeled toxic. A spaCy-tokenized TF–IDF representation turns them into vectors with 19,169 word features—far more features than toxic examples. That imbalance matters when choosing a model: there is limited labeled evidence for which words signal toxicity. section 4.2
Listing 4.3 Creating TF–IDF vectors for the toxic comments dataset
>>> from sklearn.feature_extraction.text import TfidfVectorizer
>>> import spacy
>>> nlp = spacy.load("en_core_web_sm")
>>>
>>> def spacy_tokenize(sentence):
... return [token.text for token in nlp(sentence.lower())]
>>>
>>> tfidf_model = TfidfVectorizer(tokenizer=spacy_tokenize)
>>> tfidf_docs = tfidf_model.fit_transform(\
... raw_documents=comments.text).toarray()
>>> tfidf_docs.shape
(5000, 19169)
Inspect the tokenization and TF–IDF step, especially the resulting 5,000-by-19,169 feature matrix. section 4.2
For two classes, linear discriminant analysis (LDA) looks for an axis on which their vectors separate. A useful approximation subtracts the nontoxic centroid from the toxic centroid, then projects each comment vector onto that direction with a dot product. Higher projections point toward the toxic examples. Scale the scores and apply a cutoff to produce labels; a scaled score alone should not be mistaken for a calibrated probability. section 4.2.1
>>> mask = comments.toxic.astype(bool).values #1 >>> toxic_centroid = tfidf_docs[mask].mean(axis=0) #2 >>> nontoxic_centroid = tfidf_docs[~mask].mean(axis=0) #3 >>> centroid_axis = toxic_centroid - nontoxic_centroid >>> toxicity_score = tfidf_docs.dot(centroid_axis) #4 >>> toxicity_score.round(3) array([-0.008, -0.022, -0.014, ..., -0.025, -0.001, -0.022])
Follow the centroid calculation and vectorized dot product that give each comment a toxicity score. section 4.2.1
Check performance on comments the classifier has not seen
A scikit-learn LDA classifier fitted to TF–IDF vectors scores 0.999 on its training half but only 0.554 on its test half. Its test confusion matrix records 913 nontoxic comments predicted toxic and 201 toxic comments predicted nontoxic. The striking training score is therefore not evidence that the classifier is ready to moderate new comments; inspect held-out results and the kinds of mistakes it makes. section 4.2.1
Listing 4.6 LDA model performance with train/test split
>>> from sklearn.model_selection import train_test_split >>> X_train, X_test, y_train, y_test = train_test_split(tfidf_docs,\ ... comments.toxic.values, test_size=0.5, random_state=271828) >>> lda_tfidf = LinearDiscriminantAnalysis(n_components=1) >>> lda = lda_tfidf.fit(X_train, y_train) #1 >>> round(float(lda.score(X_train, y_train)), 3) 0.999 >>> round(float(lda.score(X_test, y_test)), 3) 0.554
Compare the training and test scores in the runnable train/test split. section 4.2.1
The label design is as important as the model. A multiclass classifier chooses one category per object. A multi-label tagger can assign several when categories overlap—for example, all appropriate nationalities associated with an ambiguous surname. Search for multi-output classification when looking for scikit-learn models for that task. section 8.2.4
Classifiers also make small operational decisions: a query classifier can distinguish questions for a chatbot’s knowledge base from other requests, while NLP filters can flag toxic or erroneous chatbot content. Keep evaluating them in their intended setting. A bag-of-words sentiment classifier that reaches about 93% accuracy on its movie-review examples reaches only about 56% when applied to product reviews. section 10.1.3 section 2.6.2
