Latent semantic analysis (LSA) retrieves documents by finding low-dimensional structure in document-term data. It applies principal component analysis to bag-of-words document vectors, then projects documents onto topic axes. This compression suppresses smaller variations and exposes indirect similarities between documents that share few or no terms directly. section 4.7 section 4.1
A compact LSA workflow connects linear algebra with retrieval:
- Represent each document as a term-frequency vector in a document-term matrix. section 4.7
- Compute the matrix singular value decomposition, interpret the dominant right-singular directions as topic axes, and retain a lower rank. section 4.6.3 section 4.7
- Project documents into topic space and compare their projected vectors, often with cosine similarity. section 4.6.3 section 4.7
Toy document matrix
A PyTorch example uses the terms “violence,” “gun,” “america,” and “roses.” Keeping only the dominant topic assigns nearly equal weights to “violence” and “gun.” Two documents have cosine similarity 0.0 in the original term space but 1.0 after projection, showing how LSA can reveal a shared latent topic despite no direct term overlap. section 4.6.3
Listing 4.8 Computing LSA
terms = ["violence", "gun", "america", "roses"] #1
X = torch.tensor([[0, 0, 0, 2],
[1, 1, 1, 0],
[2, 2, 0, 0],
[3, 3, 0, 0],
[5, 5, 0, 0],
[0, 1, 0, 0],
[1, 0, 0, 0]]).float() #2
U, S, V_t = torch.linalg.svd(X) #3
V = V_t.T
rank = 1
U = U[:, :rank]
V = V[:, :rank] #4
topic0_term_weights = list(zip(terms, V[:, 0])) #5
def cosine_similarity(vec_1, vec_2):
vec_1_norm = torch.linalg.norm(vec_1)
vec_2_norm = torch.linalg.norm(vec_2)
return torch.dot(vec_1, vec_2) / (vec_1_norm * vec_2_norm)
d5_d6_cosine_similarity = cosine_similarity(X[5], X[6]) #6
doc_topic_projection = torch.dot(X, V)
d5_d6_lsa_similarity = cosine_similarity(doc_topic_projection[5],
doc_topic_projection[6])
Inspect the executable SVD, rank-reduction, topic-weight, projection, and cosine-similarity steps. section 4.6.3
