A document-term matrix turns a text collection into numbers: its rows represent vocabulary terms, its columns represent documents, and each entry records a term–document association. TF–IDF supplies one common weighting scheme, making the structure useful for search and topic modeling. section 4.4 section 3.6.3
A toxic-comment corpus
A comment corpus produces a TF–IDF representation with shape (5000, 19169): 5,000 messages and 19,169 dimensions in the displayed dataset. The vocabulary is large relative to the available messages, making the representation cumbersome and increasing overfitting risk. Similar meanings such as “leave this page” and “go away” can still receive low similarity because their words differ. section 4.4
The matrix supports several practical choices:
- Use it as a traditional information-retrieval representation, then compare TF–IDF weighting with BM25 or BM25F when ranking relevance. section 3.6.3
- Apply dimensionality reduction such as PCA, SVD, or LSA to make document relationships and latent topics easier to analyze. section 4.4
>>> tfidf_docs.shape (5000, 19169)
Inspecting the matrix shape exposes the number of documents and feature dimensions before modeling. section 4.4
