In natural language processing, a vectorizer (or encoder) transforms text into numerical vectors that computers can process. Vectorized operations also replace explicit Python loops with arithmetic on arrays or data frames. section 1.1.1 section 3.2.2
Text representations support different kinds of NLP work:
- Token-count and term-frequency vectors represent word occurrence and can support keyword search, full-text search, and toxic-message detection. section 1.1.1
- Embeddings represent semantics, supporting search based on what a query means rather than only which words occur. section 1.1.1
For numerical work, NumPy and pandas can perform array and data-frame arithmetic without a Python loop. This can greatly reduce runtime even when the algorithm’s big-O complexity is unchanged. section 3.2.2
Listing 3.5 Increasing the vectorized math in your code
>>> v1 = np.arange(5) #1 >>> v2 = pd.Series(reversed(range(5))) >>> slow_answer = sum([4.2 * (x1 * x2) for x1, x2 in zip(v1, v2)]) >>> slow_answer 42.0 >>> faster_answer = sum(4.2 * v1 * v2) #2 >>> faster_answer 42.0 >>> fastest_answer = 4.2 * v1.dot(v2) #3 >>> fastest_answer 42.0
Compare a loop, array arithmetic, and a dot product: each computes the same result, with increasingly vectorized code. section 3.2.2
