Dataset

A dataset is a structured collection of data, often used in machine learning and data science to train and evaluate models. In the context of large language models (LLMs) and deep learning, datasets are crucial for providing the necessary scale and diversity of information that models require to learn effectively.

Large Datasets in Language Models

Large datasets are fundamental in training language models like GPT-3 and BERT. These datasets consist of billions of words and cover a wide range of topics and languages, which is essential for models to perform well across various tasks. The vastness and variety of these datasets enable models to understand and generate human-like text by learning language syntax, semantics, and context. For example, the dataset used for GPT-3 consists of 300 billion tokens, providing a comprehensive text corpus for training.

Efficient data loading is a critical aspect of handling such large datasets. This involves organizing the data into structures that facilitate quick access and processing. For instance, inputs can be collected in a tensor, where each row represents one input context, and a second tensor contains the corresponding prediction targets. This method is illustrated in the following figure:

Figure 2.13 To implement efficient data loaders, we collect the inputs in a tensor, x, where each row represents one input context. A second tensor, y, contains the corresponding prediction targets (next words), which are created by shifting the input by one position.

Preparing Datasets for Fine-Tuning

Preparing datasets for parameter-efficient fine-tuning, such as with Low-Rank Adaptation (LoRA), involves several steps. This process includes downloading and preparing the dataset, balancing it, and splitting it into training, validation, and test sets. Such preparation is essential for ensuring that the model is trained effectively and can generalize well to new data. Here is an example code snippet demonstrating this process:

from pathlib import Path
import pandas as pd
from ch06 import (
    download_and_unzip_spam_data,
    create_balanced_dataset,
    random_split
)

url = "https://archive.ics.uci.edu/static/public/228/sms+spam+collection.zip"
zip_path = "sms_spam_collection.zip"
extracted_path = "sms_spam_collection"
data_file_path = Path(extracted_path) / "SMSSpamCollection.tsv"

download_and_unzip_spam_data(url, zip_path, extracted_path, data_file_path)

df = pd.read_csv(
    data_file_path, sep="\t", header=None, names=["Label", "Text"]
)
balanced_df = create_balanced_dataset(df)
balanced_df["Label"] = balanced_df["Label"].map({"ham": 0, "spam": 1})

train_df, validation_df, test_df = random_split(balanced_df, 0.7, 0.1)
train_df.to_csv("train.csv", index=None)
validation_df.to_csv("validation.csv", index=None)
test_df.to_csv("test.csv", index=None)

Datasets in PyTorch

In PyTorch, a dataset is an object designed to provide uniform access to data, primarily through the implementation of the __len__ and __getitem__ methods. This design allows datasets to be used seamlessly for loading and preprocessing data in machine learning tasks. The dataset abstraction is versatile, capable of returning a sample and label for image classification tasks, or any data point with or without corresponding labels.

CIFAR-10 Dataset

The CIFAR-10 dataset is a widely used dataset in the field of image recognition. It consists of 60,000 32x32 color images divided into 10 distinct classes, with each class containing 6,000 images. This dataset is particularly useful for training machine learning models to perform image classification tasks. In PyTorch, the CIFAR-10 dataset can be easily accessed through the torchvision module. Here is a code snippet to download and instantiate the dataset:

from torchvision import datasets
data_path = '../data-unversioned/p1ch7/'
cifar10 = datasets.CIFAR10(data_path, train=True, download=True)    #1
cifar10_val = datasets.CIFAR10(data_path, train=False, download=True)    #2
#1 Instantiates a dataset for the training data; TorchVision downloads the data if it is not present.
#2 With train=False, this gets us a dataset for the validation data, again downloading as necessary.

Normalizing Data

Normalization is a crucial preprocessing step in machine learning, especially for image data. It involves adjusting the dataset so that each channel of the images has a zero mean and a unitary standard deviation. In PyTorch, this can be achieved using the transforms.Normalize function, which requires pre-computed mean and standard deviation values for each channel. Normalizing the data helps in stabilizing and speeding up the training process of machine learning models.

In summary, datasets are a foundational component in the development and training of machine learning models, providing the necessary data for models to learn and generalize effectively.

| Book Title | Usage of dataset | Technical Depth | Connections to Other Concepts | Examples Used | Practical Application |
|------------|------------------|-----------------|-------------------------------|---------------|-----------------------|
| [Build a Large Language Model (From Scratch)](https://www.manning.com/books/build-a-large-language-model-from-scratch) | Discusses the role of large datasets in training language models, emphasizing scale and diversity. | Covers efficient data loading and preparation for fine-tuning, including Low-Rank Adaptation (LoRA). | Connects datasets to model training tasks like understanding syntax, semantics, and context. | Examples include GPT-3 and BERT datasets, and code snippets for dataset preparation. | Practical steps for downloading, balancing, and splitting datasets for training. [more](https://livebook.manning.com/book/build-a-large-language-model-from-scratch/chapter-1/56) |
| [Deep Learning with PyTorch, Second Edition](https://www.manning.com/books/deep-learning-with-pytorch-second-edition) | Describes datasets as objects for uniform data access in PyTorch, using methods like `__len__` and `__getitem__`. | Details on dataset instantiation and preprocessing, including normalization. | Links datasets to image classification tasks and data preprocessing techniques. | Uses CIFAR-10 dataset as a primary example, with code for downloading and normalizing data. | Focuses on practical implementation in PyTorch, including data loading and preprocessing. [more](https://livebook.manning.com/book/deep-learning-with-pytorch-second-edition/chapter-7/8) |