An encoder-only transformer processes an entire input sequence without a separate output decoder. Its bidirectional self-attention lets each token use context from both sides, producing contextual embeddings suited to understanding text rather than generating it token by token. section 3.3 section 3.6
Unlike a decoder-only model, it does not feed each output back in for autoregressive generation. It processes the input in one forward pass, needs no KV cache, and is easier to batch for high-throughput inference. That makes it a practical choice when the required output is a classification, an extracted answer, or a representation for retrieval. section 3.3
Listing 3.4 Transformer encoder used in encoder-only models (Fairseq abstracted)
class TransformerEncoderBase(FairseqEncoder):
def __init__(self, cfg, dictionary, embed_tokens, return_fc=False):
super().__init__(dictionary) #1
self.cfg = cfg
self.embed_tokens = embed_tokens
self.return_fc = return_fc
self.embed_positions = ( #2
PositionalEmbedding(
cfg.max_source_positions,
embed_tokens.embedding_dim,
embed_tokens.padding_idx,
learned=cfg.encoder_learned_pos
) if not cfg.no_token_positional_embeddings else None
)
self.layernorm_embedding = ( #3
LayerNorm(embed_tokens.embedding_dim)
if cfg.layernorm_embedding else None
)
self.layers = nn.ModuleList([ #4
self.build_encoder_layer(cfg) for _ in range(cfg.encoder.layers)
])
self.layer_norm = ( #5
LayerNorm(embed_tokens.embedding_dim)
if cfg.encoder.normalize_before else None
)
def forward(self, src_tokens, src_lengths=None, **kwargs):
x = self.embed_tokens(src_tokens) #6
if self.embed_positions is not None:
x += self.embed_positions(src_tokens)
if self.layernorm_embedding is not None:
x = self.layernorm_embedding(x)
x = F.dropout(x,
p=self.cfg.dropout,
training=self.training) #7
x = x.transpose(0, 1) # (batch, seq_len, dim) -> (seq_len, batch, dim)
for layer in self.layers: #8
x = layer(x)
if self.layer_norm is not None: #9
x = self.layer_norm(x)
return { #10
'encoder_out': [x], # shape: (seq_len, batch, dim)
'encoder_padding_mask': [src_tokens.eq(self.embed_tokens.padding_idx)]
}
The abstracted Fairseq encoder shows the implementation skeleton: token embeddings, optional positional embeddings and embedding layer normalization, followed by encoder layers. section 3.3
A common pretraining objective is masked language modeling. Roughly 15% of input tokens are replaced with a mask token; the model predicts their originals from surrounding context and computes loss only at masked positions. The masked-token prediction head is used for pretraining and removed for fine-tuning. BERT uses static masking, while RoBERTa re-randomizes masked positions each epoch. section 3.3.1
Predicting a masked token
Given “The capital of France is [MASK],” the masked-language-modeling task predicts “Paris.” The prediction can draw on tokens on either side of a mask when they are present. section 3.3.1
Listing 3.5 Generalized masked token projection during encoder-only training
class MaskedLMHead(nn.Module):
def forward(self, hidden_states, masked_tokens=None):
if masked_tokens is not None: #1
hidden_states = hidden_states[masked_tokens, :]
x = self.dense(hidden_states) #2
x = self.activation_fn(x) #3
x = self.layer_norm(x) #4
return self.vocab_projection(x) #5
The generalized masked-language-model head selects masked hidden states before applying its projection layers, so token predictions need not be computed for every input position. section 3.3.1
BERT, RoBERTa, and ModernBERT illustrate the family. Typical uses include sentiment analysis, named entity recognition, semantic search, and retrieval of context for retrieval-augmented generation. An encoder-based classifier can also act as a fast guardrail around a larger generative model, labeling prompts without generating a response. section 3.3 section 7.6
