Latent Dirichlet allocation (LDA) discovers topics shared across a collection of documents. It represents each document as a finite mixture of topics and each topic as a distribution over words. The aim is to infer both the shared topics and each document’s topic proportions. section 9.1
Start with a vocabulary of V words and D documents. Under LDA’s bag-of-words assumption, word order does not matter: each document becomes a vector of word counts. Stack those vectors into a V × D term-document matrix A. With K topics, the representation also has a nonnegative-matrix-factorization view: A ≈ WH, where W is a V × K matrix of topics and H is a K × D matrix of document topic proportions. Term frequency–inverse document frequency (tf-idf) can adjust the input weights to reduce the influence of terms appearing across many documents. section 9.1
The probabilistic view names what must be inferred:
- Each word xᵢ,ᵈ has a latent topic label zᵢ,ᵈ among K topics. section 9.1
- Each document has topic proportions θᵈ; each topic βₖ is shared across documents. The hyperparameters α and η express prior knowledge about proportions and topics. section 9.1
Figure 9.1 Latent Dirichlet allocation graphical model
The graphical model helps distinguish document-level topic proportions, shared topics, and word-level topic labels. section 9.1
For an observed document, the central inference problem is its posterior over topic proportions and word-topic labels, p(θ, z | x, α, β). Variational Bayes approximates this posterior with a fully factored distribution q. Its parameters are optimized by maximizing the evidence lower bound (ELBO), expressed as an energy term minus an entropy term. Coordinate-ascent updates provide the iterative fitting procedure; an increasing ELBO is one way to inspect its progress. section 9.1.1
For a large or streaming corpus, online variational Bayes processes a document’s word-count vector at a time. It first finds locally optimal γ and ϕ while holding the shared parameter λ fixed, then moves λ toward an estimate from that document using a learning-rate weight ρ. This avoids requiring a full corpus pass at every iteration. section 9.1.1
Topics in newsgroup documents
A from-scratch variational-Bayes implementation applies LDA to the 20 Newsgroups dataset. The ELBO is plotted against mean-field iterations, and word clouds display inferred topics. The highest-ranked words in those topics match the dataset’s categories. As a further check on the model’s structure, explain how its term-document representation relates to nonnegative matrix factorization. section 9.1.1 section 9.7
