liveBook · Manning

DataLoader

As explained in Build AI Drug Discovery Pipelines

PyTorch DataLoader is a utility for batching, optionally shuffling, and iterating over samples from a dataset.

A PyTorch DataLoader turns a dataset into an iterable of batches. It can also shuffle samples. Batching makes training more memory-efficient and can speed it up. section 8.1.2

A compatible custom Dataset supplies its size through __len__ and individual samples through __getitem__. For molecule activity prediction, each sample can be a molecular fingerprint paired with its activity value. section 8.1.2

Listing 8.3 MoleculeDataset Data Loader
def smiles_to_fp(smiles, fp_size=2048):
    mol = Chem.MolFromSmiles(smiles)
    if mol is None:
        return None
    mfpgen = rdFingerprintGenerator.GetMorganGenerator(
        radius=3, fpSize=fp_size
    )
    fp = mfpgen.GetFingerprint(mol)
    arr = np.zeros((fp_size,), dtype=np.int8)
    DataStructs.ConvertToNumpyArray(fp, arr)
    return arr
 
class MoleculeDataset(Dataset):
    def __init__(self, smiles, activities):
        self.smiles = smiles
        self.activities = activities
        self.fingerprints = [smiles_to_fp(s) for s in smiles]
        
    def __len__(self):
        return len(self.smiles)
    
    def __getitem__(self, idx):
        fp = torch.tensor(self.fingerprints[idx], dtype=torch.float32)
        activity = torch.tensor([self.activities[idx]], dtype=torch.float32)
        return fp, activity
 

The MoleculeDataset listing converts SMILES strings into fingerprints and returns a fingerprint and activity tensor for each index. section 8.1.2

Create separate loaders for training, validation, and test datasets. With batch_size=32, each batch contains up to 32 samples. The training loader shuffles samples; the validation and test loaders do not. section 8.1.2

train_loader = DataLoader(train_dataset, batch_size=32, shuffle=True)
val_loader = DataLoader(val_dataset, batch_size=32, shuffle=False)
test_loader = DataLoader(test_dataset, batch_size=32, shuffle=False)
 

These loader definitions set the batch size and shuffle behavior for each split. section 8.1.2

An epoch is one pass through the training examples. The final batch can be smaller than the requested batch size; drop_last=True discards that incomplete batch. section 8.1.2

During training, iterate over the loader to receive inputs and targets one batch at a time. Move them to the chosen device, compute predictions and loss, then update the model. A validation loader supplies batches for checking performance separately. section 8.2.3

Listing 8.7 Train the Model
def train_model(
    model, train_loader, val_loader, criterion, optimizer, 
    scheduler, device, num_epochs=10
):
    model.train()    #A
    train_losses = []    #B
    val_losses = []
    
    for epoch in range(num_epochs):    #C
        running_loss = 0.0    #D
        for inputs, targets in train_loader:
            inputs, targets = inputs.to(device), targets.to(device)
            
            optimizer.zero_grad()    #E
            outputs = model(inputs)    #F
            loss = criterion(outputs, targets)    #G
            loss.backward()    #H
            optimizer.step()    #I
            
            running_loss += loss.item()
        
        train_loss = running_loss / len(train_loader)
        train_losses.append(train_loss)
        
        model.eval()    #J
        val_loss = 0.0
        with torch.no_grad():    #K
            for inputs, targets in val_loader:
                inputs, targets = inputs.to(device), targets.to(device)
                outputs = model(inputs)
                loss = criterion(outputs, targets)
                val_loss += loss.item()
        
        val_loss /= len(val_loader)
        val_losses.append(val_loss)
        
        print(f'Epoch {epoch+1}/{num_epochs}', 
              f'Train Loss: {train_loss:.4f}',
              f'Val Loss: {val_loss:.4f}')    #L
        
        scheduler.step(val_loss)    #M
        model.train()    #N

The training loop iterates over batches, moves them to the device, and uses them to update the model. section 8.2.3

How Build AI Drug Discovery Pipelines covers it

How deep it goes
Explains Dataset methods, batch size, shuffling, incomplete batches, and iteration in training and evaluation loops.
Examples
Uses a molecule activity dataset with separate 32-sample training, validation, and test loaders.
What it assumes
Introduces PyTorch tensors and a custom Dataset before constructing loaders.
What you can do after
Apply it via PyTorch to configure loaders and iterate over input-target batches in a training loop.
How it connects
Connects DataLoader to PyTorch Dataset, molecular fingerprints, neural network training, and validation.

Also explained in

Compare how 6 books explain DataLoader →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build AI Drug Discovery Pipelines. The section links point to the parts of the book it draws on.