liveBook · Manning

Data collator

As explained in Natural Language Processing in Action, Second Edition

A data collator is a component that dynamically assembles samples into batches and may apply batch-level preprocessing such as padding.

A data collator dynamically assembles individual samples into training batches and can apply batch-level preprocessing, such as padding. In a language-model fine-tuning pipeline, it connects prepared datasets with the training loop while keeping batch construction consistent. section 10.2.2

GPT-2 fine-tuning

Create a DataCollatorForLanguageModeling with the tokenizer and mlm=False, then pass it to Hugging Face Trainer alongside the model, training arguments, and training and evaluation datasets. The batch size should reflect available GPU RAM; single-digit values provide a practical starting point when monitoring out-of-memory errors.

How Natural Language Processing in Action, Second Edition covers it

How deep it goes
Defines dynamic batching and padding, but stops short of implementation internals.
Examples
Uses DataCollatorForLanguageModeling with GPT-2, tokenizer, training arguments, and prepared training and evaluation datasets.
What it assumes
Assumes tokenized datasets and familiarity with training arguments, while simplifying batch preparation.
What you can do after
Apply it via Hugging Face Trainer to batch samples for language-model fine-tuning.
How it connects
Connects tokenizers, PyTorch datasets, padding, language modeling, and the Hugging Face Trainer.
Cover of Natural Language Processing in Action, Second Edition
liveBookmanning.com
Read Natural Language Processing in Action, Second Edition on liveBook Get the book at manning.com

This explainer was generated by Manning's knowledge graph from the text of Natural Language Processing in Action, Second Edition. The section links point to the parts of the book it draws on.