ModernBERT is an encoder-only transformer for efficient classification and retrieval. It modernizes the BERT-style design with rotary positional embeddings, alternating local and global attention, and context lengths up to 8,192 tokens. section 7.3.3
Its practical advantages combine architectural improvements with compact-model deployment:
- Processes longer contexts than traditional compact encoder setups, while remaining memory- and inference-efficient on standard GPUs. section 7.3.3
- Supports high-volume, latency-sensitive classification tasks such as sentiment analysis, compliance checks, and intent detection. section 7.7
- Offers an efficiency–accuracy trade-off and achieves state-of-the-art performance across classification and retrieval benchmarks. section 7.3.3 section 7.7
Financial PhraseBank classification
A three-class sentiment classifier loads answerdotai/ModernBERT-base, maps negative, neutral, and positive labels, tokenizes examples to a maximum length of 128, and uses dynamic padding. Hugging Face Trainer evaluates each epoch, selects the best checkpoint by macro F1, and reports accuracy, precision, recall, F1, and a confusion matrix. Undersampling caps neutral training examples at 500 while retaining all positive and negative examples. section 7.3.3
Listing 7.6 Loading ModernBERT and tokenizer
model_name = "answerdotai/ModernBERT-base" #1
tokenizer = AutoTokenizer.from_pretrained(model_name, use_fast=True)
id2label = {0: "negative", 1: "neutral", 2: "positive"}
label2id = {v: k for k, v in id2label.items()}
model = AutoModelForSequenceClassification.from_pretrained( #2
model_name,
num_labels=3,
id2label=id2label,
label2id=label2id,
device_map="auto",
trust_remote_code=True,
)
Inspect the executable model and tokenizer setup, including label mappings, automatic device placement, and sequence-classification configuration. section 7.3.3
Listing 7.7 Tokenization function and collator
max_length = 128
def preprocess(batch): #1
toks = tokenizer( #1
batch["text"], #1
truncation=True, #1
max_length=max_length #1
) #1
toks["labels"] = batch["label"] #1
return toks #1
encoded = dataset.map(preprocess, batched=True,
remove_columns=dataset["train"].column_names)
data_collator = DataCollatorWithPadding(tokenizer=tokenizer) #2
Follow the preprocessing implementation for truncation, maximum sequence length, label attachment, and dynamic batch padding. section 7.3.3
Listing 7.9 Training configuration
training_args = TrainingArguments( #1
output_dir="./results_modernbert_cls",
overwrite_output_dir=True,
num_train_epochs=3,
per_device_train_batch_size=16,
per_device_eval_batch_size=16,
eval_strategy="epoch",
save_strategy="epoch",
save_total_limit=2,
learning_rate=3e-5,
weight_decay=0.01,
logging_dir="./logs",
logging_steps=100,
load_best_model_at_end=True,
metric_for_best_model="f1",
greater_is_better=True,
)
Review a compact training configuration with three epochs, learning rate 3e-5, epoch evaluation, checkpoint limits, and macro-F1 model selection. section 7.3.3
Listing 7.10 Instantiating the Trainer
trainer = Trainer( #1
model=model,
args=training_args,
train_dataset=encoded["train"],
eval_dataset=encoded["validation"],
compute_metrics=compute_metrics,
data_collator=data_collator,
tokenizer=tokenizer,
)
trainer.train() #2
See how the model, encoded datasets, metrics, collator, and tokenizer are assembled into a Trainer and trained. section 7.3.3
