liveBook · Manning

Unlabeled data

As explained in Building Specialized AI Systems

Unlabeled text is raw textual data without annotations or desired output labels.

Unlabeled text is raw language without annotations or desired outputs. It includes research papers, technical documentation, customer interactions, financial news, and clinical notes. section 3.6

Use it for unsupervised fine-tuning, also called continual pre-training, to expose a model to domain vocabulary, writing patterns, and contextual nuances before task-specific supervised fine-tuning. section 3.6

Unlabeled data is valuable when it offers:

  • Abundant domain language where labeled examples are expensive, scarce, or time-consuming to curate. section 3.6
  • A preparatory adaptation stage that complements rather than replaces supervised fine-tuning. section 3.6

Domain corpus

A corpus of 50,000 financial news articles or clinical notes can adapt a model’s language priors before task-specific fine-tuning, without labels. section 3.6

Listing 3.34 Domain-adaptive pretraining (DAPT) with masked language modeling
import torch
from transformers import (
    AutoTokenizer, 
    AutoModelForMaskedLM, 
    DataCollatorForLanguageModeling, 
    Trainer, 
    TrainingArguments
)
from datasets import load_dataset
 
model_checkpoint = "distilbert-base-uncased"  #A
 
domain_text = """  #B
The patient exhibited severe symptoms of acute myocardial infarction.
troponin levels were elevated, indicating significant cardiac stress.
The electrocardiogram showed ST-segment elevation in leads V1-V4.
Treatment began immediately with aspirin and nitroglycerin.
"""
with open("domain_data.txt", "w") as f:
    f.write(domain_text)
 
dataset = load_dataset("text", data_files={"train": "domain_data.txt"})  #C
 
tokenizer = AutoTokenizer.from_pretrained(model_checkpoint)  #D
 
def tokenize_function(examples):  #E
    return tokenizer(examples["text"], truncation=True, max_length=128)
 
tokenized_datasets = dataset.map(tokenize_function, batched=True, remove_columns=["text"])  #F
 
data_collator = DataCollatorForLanguageModeling(  #G
    tokenizer=tokenizer, 
    mlm=True, 
    mlm_probability=0.15
)
 
model = AutoModelForMaskedLM.from_pretrained(model_checkpoint)  #H
 
training_args = TrainingArguments(  #I
    output_dir="./domain_adapted_model",
    overwrite_output_dir=True,
    num_train_epochs=3,
    per_device_train_batch_size=4,
    save_steps=10_000,
    save_total_limit=2,
    learning_rate=2e-5,
    weight_decay=0.01,
)
 
trainer = Trainer(  #J
    model=model,
    args=training_args,
    train_dataset=tokenized_datasets["train"],
    data_collator=data_collator,
)
 
print("Starting Unsupervised Fine-Tuning (Domain Adaptation)...")
trainer.train()  #K
 
trainer.save_model("./final_adapted_model")  #L
print("Adaptation complete. The model is now tuned to your domain's vocabulary.")
 

Demonstrates domain-adaptive pretraining with masked language modeling, including text loading, tokenization, and training components. section 3.6

How Building Specialized AI Systems covers it

How deep it goes
Explains the adaptation rationale and gives a masked-language-modeling implementation, stopping short of detailed training mechanics.
Examples
Uses financial news, clinical notes, research papers, technical documentation, and customer interactions as unlabeled corpora.
What it assumes
Distinguishes unlabeled adaptation from labeled input-output training and assumes later task-specific fine-tuning.
What you can do after
Apply it via domain-adaptive pretraining to align language priors with specialized vocabulary and writing patterns.
How it connects
Connects domain adaptation with continual pre-training, supervised fine-tuning, and masked language modeling.

Also explained in

Compare how 6 books explain unlabeled data →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Building Specialized AI Systems. The section links point to the parts of the book it draws on.