How can a chatbot find the right policy document when someone asks a question in different words? An embedding helps turn text into a numerical representation that can be compared with others. A word embedding places words in a space where words used similarly tend to be close together. Document embeddings can serve a similar purpose for longer text. section 1.12 section 8.13
Nearby words
“Cucumber” and “pickle” may appear close together in a word-embedding space, while “cucumber” and “philosophy” appear farther apart. Unlike assigning each word an arbitrary numerical point, an embedding captures patterns related to meaning. section 1.12
Figure 1.4 Representation of word embeddings in the vector space
The figure illustrates the idea of representing words as positions in a space. section 1.12
Embeddings are also used in retrieval-augmented generation (RAG). Documents are converted into vectors and may be stored in a vector database. A question is converted into a vector too; nearby documents are retrieved and placed in the language model’s context. This can help when the relevant information cannot all fit into that context, but it depends on the embeddings bringing back useful documents. section 8.13
Finding a travel policy
An employee asks whether a premium-economy ticket for a five-hour sales trip is reimbursable. A chatbot could retrieve the most similar travel-policy documents and use their wording to answer, rather than relying only on information already in the model’s context. section 8.13
A meaningful numerical representation is not necessarily a neutral one. Word embeddings trained on news text have exhibited gender stereotypes about occupations and descriptions. Embeddings can also complicate privacy rights: after personal data has been transformed into internal representations, identifying or removing one person’s contribution may be technically infeasible. How existing data-protection rights apply to models remains contested. section 1.12 section 3.15
