A pipeline script orchestrates a repeatable machine-learning workflow: it reads configuration, prepares training inputs, launches a managed training job, and optionally deploys the resulting model. Unlike a local notebook workflow, it uses containerized Python code and Cloud Storage locations that can be shared across pipeline components. section 11.3.1 section 11.5
A practical pipeline script typically performs these steps:
- Loads project, region, dataset, staging, training, deployment, and split settings from a pipeline configuration file. section 11.3.6
- Creates a Vertex AI CustomTrainingJob with the training script, prebuilt container, staging bucket, requirements, and serving image. section 11.3.6
- Builds a TabularDataset reference, runs training with train-validation-test proportions and a machine type, then receives the trained model. section 11.3.6
- Creates or selects an endpoint and deploys the model when deployment is enabled; successful execution reports completion, while failures provide log links. section 11.3.6
End-to-end control flow
The main function loads pipeline_config.yml, passes the Cloud Storage configuration path to the training script, creates the job and managed dataset, runs training, and conditionally deploys the model. Run it with python pipeline_script.py after configuring project, region, dataset, staging, and storage paths. section 11.3.6
Listing 11.5 Creating a CustomTrainingJob object
def create_job(config):
model_display_name = '{}-{}'.format(config['ENDPOINT_NAME'], TIMESTAMP)
job = aiplatform.CustomTrainingJob(
display_name='train-{}'.format(model_display_name),
script_path = config['script_path'],
container_uri=config['train_image'], #1
staging_bucket = config['staging_path'],
requirements=['gcsfs'], #2
model_serving_container_image_uri= \
config['deploy_image'] #3
)
return job
Inspect the CustomTrainingJob definition to see how training code, containers, staging storage, dependencies, and serving configuration become one managed job. section 11.3.6
