liveBook · Manning

Rotary Position Embedding

As explained in Build AI Drug Discovery Pipelines

A rotary position embedding encodes relative positional information by rotating query and key representations within attention computations.

Rotary position embedding (RoPE) encodes relative position in attention by rotating query and key representations. In protein models, this helps represent where amino acids occur relative to one another in a sequence. section 12.3.5

ESM-2 uses RoPE instead of sinusoidal position encodings. RoPE allows the model to generalize to sequences longer than those seen during training, an important consideration because protein lengths vary widely. section 12.3.5

Listing 12.5 Loading ESM-2 and Computing Protein Embeddings
from transformers import AutoTokenizer, AutoModel
 
model_name = "facebook/esm2_t33_650M_UR50D"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModel.from_pretrained(model_name)    #A
model.eval()
 
sequence = "MKTAYIAKQRQISFVKSHFSRQLEERLGLIEVQ"
inputs = tokenizer(sequence, return_tensors="pt")    #B
with torch.no_grad():
    outputs = model(**inputs)    #C
embeddings = outputs.last_hidden_state    #D

Load a pretrained ESM-2 model and compute embeddings for a protein sequence. The model uses RoPE internally; the listing does not implement it separately. section 12.3.5

How Build AI Drug Discovery Pipelines covers it

How deep it goes
Defines the attention-level purpose and contrasts sinusoidal encodings, without detailing the rotation implementation.
Examples
Illustrates use through a pretrained ESM-2 model computing embeddings for a protein sequence.
What you can do after
Apply it via Transformers by loading a pretrained ESM-2 model to compute protein embeddings, rather than implementing RoPE directly.
How it connects
Connects rotary embeddings with Transformer attention, sinusoidal position encodings, and masked language modeling.

Also explained in

Compare how 5 books explain Rotary Position Embedding →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build AI Drug Discovery Pipelines. The section links point to the parts of the book it draws on.