Two comments can mean nearly the same thing while sharing few words. A TF–IDF search may miss that connection because it compares terms rather than meanings. Latent semantic analysis (LSA), also called latent semantic indexing, compresses word-usage patterns into topic vectors that represent words and documents in fewer dimensions. section 4.4 section 1.6
Start with a bag-of-words or TF–IDF matrix of documents and terms. Words that occur across similar documents contribute to shared topics. LSA reduces the many vocabulary dimensions to a smaller set of topic dimensions while retaining much of the information useful for comparing documents. TF–IDF generally gives slightly better results than raw bag-of-words counts. section 4.1.4 section 4.3
Reducing a comment vocabulary
A collection of 5,000 Wikipedia discussion comments produces TF–IDF vectors with 19,169 dimensions; only 650 comments carry a toxic label. That leaves many more word features than labeled toxic examples, inviting a classifier to overfit particular words. Reducing the vectors to LSA topics offers a more manageable representation for classification. In one comparison, a classifier using 16-dimensional LSA features reached 0.88 test accuracy, but that dataset result is not a guarantee for moderation elsewhere. section 4.2 section 4.4 section 4.5.4
Listing 4.3 Creating TF–IDF vectors for the toxic comments dataset
>>> from sklearn.feature_extraction.text import TfidfVectorizer
>>> import spacy
>>> nlp = spacy.load("en_core_web_sm")
>>>
>>> def spacy_tokenize(sentence):
... return [token.text for token in nlp(sentence.lower())]
>>>
>>> tfidf_model = TfidfVectorizer(tokenizer=spacy_tokenize)
>>> tfidf_docs = tfidf_model.fit_transform(\
... raw_documents=comments.text).toarray()
>>> tfidf_docs.shape
(5000, 19169)
The Python listing tokenizes the comments with spaCy and builds their TF–IDF matrix with scikit-learn. Inspect its 5,000-by-19,169 shape to see the high-dimensional input that LSA is meant to compress. section 4.2
Topic vectors can support semantic search, document comparison, clustering, and classification. LSA yields both word-topic and document-topic vectors. It is related to PCA and singular value decomposition (SVD); scikit-learn exposes an LSA-style interface through TruncatedSVD. Topic vectors are useful approximations of meaning, not proof that a toxicity decision is correct. section 4.0 section 6.4.3 section 4.5.4
