Rotary position embedding (RoPE) encodes relative position in attention by rotating query and key representations. In protein models, this helps represent where amino acids occur relative to one another in a sequence. section 12.3.5
ESM-2 uses RoPE instead of sinusoidal position encodings. RoPE allows the model to generalize to sequences longer than those seen during training, an important consideration because protein lengths vary widely. section 12.3.5
Listing 12.5 Loading ESM-2 and Computing Protein Embeddings
from transformers import AutoTokenizer, AutoModel
model_name = "facebook/esm2_t33_650M_UR50D"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModel.from_pretrained(model_name) #A
model.eval()
sequence = "MKTAYIAKQRQISFVKSHFSRQLEERLGLIEVQ"
inputs = tokenizer(sequence, return_tensors="pt") #B
with torch.no_grad():
outputs = model(**inputs) #C
embeddings = outputs.last_hidden_state #DLoad a pretrained ESM-2 model and compute embeddings for a protein sequence. The model uses RoPE internally; the listing does not implement it separately. section 12.3.5




