Data quality measures whether data is accurate, complete, consistent, timely, valid, and fit for its intended purpose. In a production ML pipeline, quality checks run before training so defective inputs fail early rather than corrupting experiments or models. section 9.3 section 9.3.1
A practical first QA layer checks:
- Schema requirements, such as the presence of user_id, item_id, and rating columns. section 9.3.1
- Data types, catching malformed values before model training. section 9.3.1
- New defects discovered during operation, extending the repository of checks iteratively. section 9.3.1
- The access path: local copies reduce network dependence but increase storage and startup costs, while direct object-store reads reverse that trade-off. section 9.3.1
Listing 9.10 Data quality check
def check_dataset(bucket:str = 'datasets', dataset:str = 'ml-25m'): from pyarrow import fs, parquet print("Running QA") minio = fs.S3FileSystem( endpoint_override='http://minio-service.kubeflow:9000', access_key='minio', secret_key='minio123', scheme='http') train_parquet = minio.open_input_file( f'{bucket}/{dataset}/train.parquet.gzip') df = parquet.read_table(train_parquet).to_pandas() is_subset = set(['user_id', 'item_id', 'rating']).issubset(df.columns) assert is_subset, f'Required columns not found. Found {df.columns}' print('QA passed!')
The runnable check reads a Parquet training file from MinIO, verifies required columns, and stops the pipeline when the schema is incomplete. section 9.3.1
