liveBook · Manning

Embedding layer

As explained in Build AI Drug Discovery Pipelines

A token embedding is a learned vector representation assigned to a discrete input token.

An embedding layer converts discrete token IDs into dense, learned vectors. For tokenized molecular SMILES, it replaces raw integers with representations that can capture chemical similarity, structural relationships, and context before encoding. section 10.2.3

The layer is a learnable lookup table: each vocabulary entry selects one row from an embedding matrix. The embedding dimension trades richer relationships against additional parameters and computation, while the resulting sequence can be flattened for an encoder. section 10.2.3

In molecular and protein models, embeddings support several practical workflows:

  • Represent SMILES tokens more efficiently than sparse one-hot vectors, reducing the parameter burden of the first neural-network layer. section 10.2.3
  • Pair byte-pair-encoded molecular fragments with a larger vocabulary to shorten sequences, speed training, and provide a stronger chemical prior. section 10.4.4
  • Combine protein token embeddings with positional encodings and self-attention so masked-language models learn residue patterns and long-range dependencies. section 12.3.3

A molecular embedding pipeline

Tokenize a SMILES sequence, map tokens such as C to vocabulary IDs, and use an embedding matrix—illustrated as 30 tokens by 128 dimensions—to produce dense vectors for the encoder. A PyTorch autoencoder can define this operation with nn.Embedding and initialize its embedding weights separately from linear layers.

Figure 10.3 Three-step process of converting molecular SMILES strings into dense vector representations for neural network processing. (1) Character-level tokenization breaks the input sequence into individual tokens, including special tokens like <sos> (start of sequence) and <eos> (end of sequence). (2) Vocabulary mapping converts each token to its corresponding integer ID using a learnable lookup table. (3) The embedding matrix (30 tokens × 128 dimensions) transforms these integer IDs into dense, learnable vector representations. The bottom flow diagram shows how this embedding layer fits into the overall autoencoder pipeline, taking a molecular input and converting the tokenized text to numerical vectors before passing to the encoder for compression. This approach enables the model to learn semantic relationships between chemically similar tokens during training, providing a more efficient and meaningful representation than sparse one-hot encoding.

Trace the representation from character-level SMILES tokenization through vocabulary IDs and dense vectors before encoder compression. section 10.2.3

How Build AI Drug Discovery Pipelines covers it

How deep it goes
Explains lookup behavior, dimensional tradeoffs, encoding integration, initialization, and alternatives, but stops short of embedding optimization theory.
Examples
Uses tokenized SMILES, a 30-by-128 embedding matrix, chemical fragments, and masked amino-acid prediction.
What it assumes
Assumes token IDs and sequences, while defining how vocabulary mapping and dense representations replace raw indices.
What you can do after
Build it from scratch with PyTorch's nn.Embedding inside an autoencoder or protein language model.
How it connects
Connects tokenization, SMILES, one-hot encoding, autoencoders, byte-pair encoding, positional encoding, and self-attention.

Also explained in

Compare how 5 books explain embedding layer →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build AI Drug Discovery Pipelines. The section links point to the parts of the book it draws on.