A topic vector represents a document—or a word—by its scores on a chosen set of topics rather than by a separate score for every word in a vocabulary. This smaller space lets documents with related subject matter be compared even when they use different words. A document has one document-topic vector; words have word-topic vectors that can be combined to represent new documents. section 4.1.2 section 4.0
Picture a document about pets in New York City. You might score it on three dimensions: pets, animals, and cities. Give “cat” and “dog” positive pet weights and “NYC” a city weight, then combine those weights with the document’s TF–IDF values. The result is a three-component vector. This hand-picked recipe illustrates the transformation, not an algorithm that discovers topics. section 4.1.3
Listing 4.1 Sample weights for your topics
>>> import numpy as np
>>> topic = {}
>>> tfidf = dict(list(zip('cat dog apple lion NYC love'.split(),
... np.random.rand(6)))) #1
>>> topic['petness'] = (.3 * tfidf['cat'] +\
... .3 * tfidf['dog'] +\
... 0 * tfidf['apple'] +\
... 0 * tfidf['lion'] -\
... .2 * tfidf['NYC'] +\
... .2 * tfidf['love']) #2
>>> topic['animalness'] = (.1 * tfidf['cat'] +\
... .1 * tfidf['dog'] -\
... .1 * tfidf['apple'] +\
... .5 * tfidf['lion'] +\
... .1 * tfidf['NYC'] -\
... .1 * tfidf['love'])
>>> topic['cityness'] = ( 0 * tfidf['cat'] -\
... .1 * tfidf['dog'] +\
... .2 * tfidf['apple'] -\
... .1 * tfidf['lion'] +\
... .5 * tfidf['NYC'] +\
... .1 * tfidf['love'])
The Python listing shows individual TF–IDF values contributing different amounts—including negative weights—to three topic scores. Its weights are illustrative, not learned. section 4.1.3
In practice, topics are learned from word usage across documents:
- Latent semantic analysis (LSA) reduces a TF–IDF or bag-of-words matrix to fewer dimensions, using co-occurrence patterns. High-weight words can help a person name a topic, but naming is not required to compare vectors. section 4.1.4 section 4.4
- Latent Dirichlet allocation (LDiA) takes a different, nonlinear statistical approach. It works with raw bag-of-words counts here and can produce document-topic vectors with many near-zero values. Its results can change between runs unless the random seed is fixed. section 4.1.4 section 4.5.2
>>> ldia16_topic_vectors = ldia.transform(bow_docs)
>>> ldia16_topic_vectors = pd.DataFrame(ldia16_topic_vectors,\
... index=index, columns=columns)
>>> ldia16_topic_vectors.round(2).head()
topic0 topic1 topic2 ... topic13 topic14 topic15
comment0 0.0 0.0 0.00 ... 0.00 0.0 0.0
comment1 0.0 0.0 0.28 ... 0.00 0.0 0.0
comment2 0.0 0.0 0.00 ... 0.00 0.0 0.0
comment3 0.0 0.0 0.00 ... 0.95 0.0 0.0
comment4! 0.0 0.0 0.07 ... 0.00 0.0 0.0
The `ldia.transform(bow_docs)` output makes the data structure tangible: each comment is a row, and each of 16 topic columns holds that comment’s score. section 4.5.2
From comment counts to a toxicity classifier
In a comment dataset, 19,169 terms were mapped to 16 LDiA topics, and the resulting vectors were used as classifier features. That reduction did not automatically improve prediction: the 16-topic LDiA classifier had lower test F1 than the 16-topic LSA classifier in the reported comparison. Evaluate the downstream task, not just how tidy the topics look. section 4.5.2 section 4.5.3
For semantic search, compare a query’s and documents’ topic vectors by similarity or distance rather than relying only on exact word matches. Check whether nearby vectors really represent similar subjects: LSA can preserve large distances without preserving every close relationship. Dense, continuous topic vectors also cannot be indexed as easily with a traditional inverted index as sparse word-based vectors. section 4.6 section 4.8.1
