Unlabeled text is raw language without annotations or desired outputs. It includes research papers, technical documentation, customer interactions, financial news, and clinical notes. section 3.6
Use it for unsupervised fine-tuning, also called continual pre-training, to expose a model to domain vocabulary, writing patterns, and contextual nuances before task-specific supervised fine-tuning. section 3.6
Unlabeled data is valuable when it offers:
- Abundant domain language where labeled examples are expensive, scarce, or time-consuming to curate. section 3.6
- A preparatory adaptation stage that complements rather than replaces supervised fine-tuning. section 3.6
Domain corpus
A corpus of 50,000 financial news articles or clinical notes can adapt a model’s language priors before task-specific fine-tuning, without labels. section 3.6
Listing 3.34 Domain-adaptive pretraining (DAPT) with masked language modeling
import torch
from transformers import (
AutoTokenizer,
AutoModelForMaskedLM,
DataCollatorForLanguageModeling,
Trainer,
TrainingArguments
)
from datasets import load_dataset
model_checkpoint = "distilbert-base-uncased" #A
domain_text = """ #B
The patient exhibited severe symptoms of acute myocardial infarction.
troponin levels were elevated, indicating significant cardiac stress.
The electrocardiogram showed ST-segment elevation in leads V1-V4.
Treatment began immediately with aspirin and nitroglycerin.
"""
with open("domain_data.txt", "w") as f:
f.write(domain_text)
dataset = load_dataset("text", data_files={"train": "domain_data.txt"}) #C
tokenizer = AutoTokenizer.from_pretrained(model_checkpoint) #D
def tokenize_function(examples): #E
return tokenizer(examples["text"], truncation=True, max_length=128)
tokenized_datasets = dataset.map(tokenize_function, batched=True, remove_columns=["text"]) #F
data_collator = DataCollatorForLanguageModeling( #G
tokenizer=tokenizer,
mlm=True,
mlm_probability=0.15
)
model = AutoModelForMaskedLM.from_pretrained(model_checkpoint) #H
training_args = TrainingArguments( #I
output_dir="./domain_adapted_model",
overwrite_output_dir=True,
num_train_epochs=3,
per_device_train_batch_size=4,
save_steps=10_000,
save_total_limit=2,
learning_rate=2e-5,
weight_decay=0.01,
)
trainer = Trainer( #J
model=model,
args=training_args,
train_dataset=tokenized_datasets["train"],
data_collator=data_collator,
)
print("Starting Unsupervised Fine-Tuning (Domain Adaptation)...")
trainer.train() #K
trainer.save_model("./final_adapted_model") #L
print("Adaptation complete. The model is now tuned to your domain's vocabulary.")
Demonstrates domain-adaptive pretraining with masked language modeling, including text loading, tokenization, and training components. section 3.6





