A term frequency vector turns a document into a row of numbers: each vocabulary slot records how often a word appears, or whether it appears. This makes text usable for computation without preserving exact word order. section 1.1.1 section 2.5.3
Practical properties shape how these vectors are built and used:
- Use a vectorizer, including scikit-learn’s CountVectorizer, to convert text into token-count or binary word vectors. section 1.1.1 section 2.5.3
- Store high-dimensional vectors sparsely because most documents contain only a small fraction of a potentially very large vocabulary. section 2.5.3
- Inspect individual vectors with a pandas Series or DataFrame, while using vector operations for retrieval and search. section 2.5.3
- Support keyword search, full-text search, and toxic social-media-message detection, but represent occurrence rather than word semantics. section 1.1.1 section 2.5.3
Figure 1.1 Encoding natural language
Trace how raw natural-language input becomes a numerical vector in an NLP pipeline. section 1.1.1
