Keyword counts can miss a document’s subject when related words do not match exactly. Latent Dirichlet allocation (LDiA) turns bag-of-words counts into topic vectors: compact representations based on which words tend to occur together. These vectors can support semantic search, topic analysis, and downstream classifiers. section 4.0 section 4.11
Think of a document as a mixture of topics and each topic as a distribution over words. LDiA starts with a chosen number of topics and uses Dirichlet probability distributions as priors for topic weights and word assignments. An imagined document generator would choose a length, mix topics, and draw words appropriate to them; the model works backward from observed word counts to estimate the mixtures. It models bags of words, not word order. section 4.5 section 4.5.1
Inspect topics in comments
For a toxic-comments dataset, make raw count vectors with `CountVectorizer(tokenizer=spacy_tokenize)`, then fit `LatentDirichletAllocation(n_components=16, learning_method='batch')`. Inspect `components_` to see how terms are allocated across topics; compare words such as “hate,” “stupid,” and “idiot.” Call `transform(bow_docs)` to obtain one topic vector per comment. The model allocates 19,169 terms across 16 topics in this example, but individual allocations can change between runs because training is stochastic; fix a random seed when reproducibility matters. section 4.5.2
>>> from sklearn.decomposition import LatentDirichletAllocation as LDiA >>> ldia = LDiA(n_components=16, learning_method='batch') >>> ldia = ldia.fit(bow_docs) #1 >>> ldia.components_.shape (16, 19169)
The fitting snippet shows the scikit-learn estimator, chosen topic count, raw-count input, and shape of its topic-by-term output. section 4.5.2
More interpretable topics do not guarantee a better classifier. A linear discriminant analysis classifier trained on 16-topic LDiA vectors performs worse on the toxicity task than one trained on 16-topic latent semantic analysis (LSA) vectors. Raising LDiA to 32 topics improves precision, recall, and F1 in this dataset, but still does not match LSA. Here, LDA names the classifier; LDiA names latent Dirichlet allocation. section 4.5.3 section 4.5.4
Try LSA first for most topic modeling, semantic search, recommendation, classification, or regression work: its linear approach is faster and often more practical. Choose LDiA when explaining the words in each topic matters, including topic analysis and some summarization tasks. Check downstream performance rather than assuming human-readable topics will predict better. section 4.1.4 section 4.5 section 4.5.4
