A dataset gives a model a consistent way to access examples. In PyTorch, a Dataset provides __len__ for the number of items and __getitem__ for retrieving one item by index. An item might be an image and its class label, but labels are not required, and the object need not keep all its data in memory. section 7.1.2
Start with an image dataset
torchvision.datasets.CIFAR10 can download separate training and validation sets. Indexing an instance returns an RGB image and an integer label; in one example, label 1 identifies an automobile. Images can then be transformed into tensors and normalized before reaching a model. section 7.1.1 section 7.1.2 section 7.5
The examples should match the job. For a bird-versus-airplane classifier, keep just those two CIFAR-10 classes and remap their labels to 0 and 1. A list works as a quick dataset because it supports indexing and len; a custom Dataset is useful when that shortcut becomes limiting. section 7.2.1
Listing 7.2 Filtering CIFAR-10 to two classes (birds and airplanes)
# In[5]:
label_map = {0: 0, 2: 1}
class_names = ['airplane', 'bird']
cifar2 = [(img, label_map[label])
for img, label in cifar10
if label in [0, 2]]
cifar2_val = [(img, label_map[label])
for img, label in cifar10_val
if label in [0, 2]]
See the filtering code that constructs separate two-class training and validation collections. section 7.2.1
For data that needs custom loading, subclass Dataset and implement the same two methods. If __len__ returns N, __getitem__ must work for indices 0 through N−1. A CT dataset, for example, turns annotated candidate regions from scans into tensors and labels, while also returning identifiers and coordinates. DataLoader can fetch dataset items in the background and assemble batches for a training loop. section 12.5 section 1.5
def __len__(self):
return len(self.candidateInfo_list)
def __getitem__(self, ndx):
# ... line 200
return (
candidate_t, #1
pos_t, #1
candidateInfo_tup.series_uid, #1
torch.tensor(center_irc), #1
)
Inspect the custom methods and the tuple returned for a CT candidate. section 12.5
Keep training and validation examples separate, representative, and free of unfair hints. A consistent ordering can support a repeatable split, but some tasks require splitting by patient or scan before extracting individual examples to prevent leakage. Check the data itself, too: a quick look at candidate sizes can expose mistaken assumptions before training. section 12.5.3 section 12.8
The dataset's output shape can change even when its source data does not. CT scans and annotations used for 3D examples can instead be prepared as 2D examples when a segmentation model expects them. section 15.6
