liveBook · Manning

One-hot encoding

As explained in Machines that Think

One-hot encoding represents each categorical item as a vector with exactly one nonzero entry indicating its index.

One-hot encoding turns a token’s numerical ID into a vector. For a vocabulary of N tokens, each vector has N entries: one entry is 1 at the token’s index, and all others are 0. section 2.5.1

A token’s vector

In a 10,000-token vocabulary, “dog” at index 42 has a vector of length 10,000 with a 1 at position 42 and zeros elsewhere. section 2.5.1

The representation makes tokens usable in mathematical operations, but it does not encode similarity: “dog” is no closer to “cat” than to “spoon.” Its vocabulary-sized vectors are also sparse, using space inefficiently. Dense vectors, including embeddings, offer a later alternative that can represent similarity. section 2.5.1

How Machines that Think covers it

How deep it goes
Defines vocabulary-length vectors, illustrates a token index, and identifies sparsity and similarity limits.
Examples
Uses dog at index 42 in a 10,000-token vocabulary and contrasts dog, cat, and spoon.
What it assumes
Assumes tokens have numerical IDs and explains how to represent them as vectors.
What you can do after
Evaluate one-hot vectors as token representations by recognizing their form and limitations.
How it connects
Connects tokenization and numerical token IDs to dense vectors and embeddings as subsequent representations.

Also explained in

Compare how 10 books explain one-hot encoding →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Machines that Think. The section links point to the parts of the book it draws on.