Imagine searching for a document by finding the word-count vector closest to a query. Give every distinct token in the corpus its own position, and the vector has as many dimensions as the vocabulary has tokens. A short text can require 246 positions; real-world word-count vectors may require millions. Trigram vocabularies can also reach millions of dimensions. section 1.5.3 section 3.2.3
From two coordinates to a vocabulary
Start with two word-frequency axes: the vectors [1, 0] and [2, 1] fit on a flat plot. Adding a separate axis for every other vocabulary term leaves the vector math intact, but the result can no longer be inspected on a 2D screen. section 3.2.3
That scale creates the curse of dimensionality: as dimensions increase, vectors grow farther apart in Euclidean distance, and finding nearby vectors becomes difficult without checking them one by one. Sorting many vectors by distance from a query can become impractical; search engines use approximate nearest-neighbor search to make that task feasible. The term-frequency matrices considered here have an important qualification: most entries are zero, and their sparsity protects them from this particular curse. section 3.2.3
Reducing the number of dimensions offers another route. Separate token dimensions give no immediate reason to treat “Good morning” and “Guten Morgen” as related; a smaller, continuous-valued representation can instead describe qualities such as subject matter or sentiment. Cosine similarity supplies a practical way to compare high-dimensional vector directions, while reduced vectors can be plotted to look for patterns and clusters. section 1.6 section 3.3
Listing 3.8 2D term frequency vector examples
>>> from matplotlib import pyplot as plt
>>> import seaborn as sns
>>> palette = sns.color_palette("muted") #1
>>> sns.set_theme() #2
>>> vecs = pd.DataFrame([[1, 0], [2, 1]], columns=['x', 'y'])
>>> vecs['color'] = palette[:2]
>>> vecs['label'] = [f'vec{i}' for i in range(1, len(vecs)+1)]
>>> fig, ax = plt.subplots()
>>> for i, row in vecs.iterrows():
... ax.quiver(0, 0, row['x'], row['y'], color=row['color'],
... angles='xy', scale_units='xy', scale=1)
... ax.annotate(row['label'], (row['x'], row['y']),
... color=row['color'], verticalalignment='top'
... )
>>> plt.xlim(-1, 3)
>>> plt.ylim(-1, 2)
>>> plt.xlabel('X (e.g. frequency of word "vector")')
>>> plt.ylabel('Y (e.g. frequency of word "space")')
>>> plt.show()
Plot two term-frequency vectors as a manageable starting point before considering what changes when each vocabulary term needs its own axis. section 3.2.3
