Testing is an iterative loop for evaluating code or model behavior against test inputs during training, between training steps, or before deployment. It turns validation into an operational checkpoint rather than a one-time notebook task. section 9.3.2 section 2.1.5
Use testing at several points in an ML delivery workflow:
- Run unit tests in CI after repository commits, before building images or triggering deployment. section 3.4.1
- Evaluate model behavior during training with a separate test dataloader; run every iteration when feedback is needed, or periodically to reduce training time. section 9.3.2
- Test the optimized model in its final deployment environment, because GPU measurements may differ from embedded-device performance after quantization or accelerator-specific processing. section 2.1.5
- Use staging as an evaluation gate against production models, then promote the evaluated version automatically and retain version lineage for rollback. section 2.1.5
A CI testing stage
A GitLab CI pipeline can define test, build, and update stages. Its test_code job uses Python 3.10, installs requirements, and runs pytest; only afterward does the pipeline build and push a Docker image and update the Helm chart. section 3.4.1
Listing 9.11 Model training code
def train_model(
model_embedding_factors=20,
model_learning_rate=1e-3,
model_hidden_dims=256,
model_dropout_rate=0.2,
optimizer_step_size=10,
optimizer_gamma=0.1,
training_epochs=30,
train_batch_size=64,
test_batch_size=64,
shuffle_training_data=True,
shuffle_testing_data=True):
import torch
from torch.autograd import Variable
from torch.utils.data import DataLoader
class MatrixFactorization(torch.nn.Module): #1
def __init__(
self,
n_users,
n_items,
n_factors,
hidden_dim,
dropout_rate):
super().__init__()
self.n_items = n_items
self.user_factors = torch.nn.Embedding(
n_users + 1,
n_factors,
sparse=False)
self.item_factors = torch.nn.Embedding(
n_items + 1,
n_factors,
sparse=False)
self.linear = torch.nn.Linear(
in_features=n_factors,
out_features=hidden_dim
)
self.linear2 = torch.nn.Linear(
in_features=hidden_dim,
out_features=1
)
self.dropout = torch.nn.Dropout(p=dropout_rate)
self.relu = torch.nn.ReLU()
def forward(self, user, item):
user_embedding = self.user_factors(user)
item_embedding = self.item_factors(item)
embeddding_vector = torch.mul(user_embedding, item_embedding)
x = self.relu(self.linear(embeddding_vector))
x = self.dropout(x)
rating = self.linear2(x)
return rating
dataset_map = get_datasets_local(split=['train', 'test'])
Inspect the model-training listing for the matrix-factorization training loop, separate train and test dataloaders, and configurable testing frequency. section 9.3.2
