liveBook · Manning

GitLab Pipelines

As explained in Machine Learning Platform Engineering

GitLab CI is a continuous integration and continuous delivery service for automating software build, test, and deployment workflows.

GitLab Pipelines automate the path from a repository commit to a tested, packaged, and deployment-ready application. A pipeline can run tests, build a Docker image, push it to Docker Hub, and update a Helm chart for deployment. section 3.4 section 3.4.1

A GitLab CI configuration organizes work into stages and jobs:

  • The test stage runs Python dependencies and unit tests. section 3.4.1
  • The build stage builds a Docker image, tags it with part of CI_COMMIT_SHA, and pushes it to Docker Hub. section 3.4.1
  • The update stage changes the Helm chart to reference the newly built image tag. section 3.4.1
  • Jobs declare scripts, execution images, services, and predefined variables such as CI_PROJECT_DIR and CI_COMMIT_SHA. section 3.4.1

FastAPI deployment pipeline

A FastAPI application responding with random jokes is stored in GitLab. Each pipeline tests the Python code, builds and publishes its Docker image, and updates the Helm chart used by the deployment workflow. section 3.4.1 section 3.4

Listing 3.13 GitLab CI example
image: docker:20.10.16
variables:
  DOCKER_TLS_CERTDIR: "/certs"
stages:    #1
  - test
  - build
  - update
services:    #2
  - docker:20.10.16-dind    #3
test_code:    #4
    stage: test
    image:
      name: python:3.10
    script:
      - echo "testing code"
      - pip install -r $CI_PROJECT_DIR/requirements.txt
      - pytest $CI_PROJECT_DIR/

build_image:    #5
    stage: build
    script:
      - echo "building docker image"
      - cd $CI_PROJECT_DIR
      - echo ${CI_COMMIT_SHA:0:8}
      - docker build . -t varunmallya/hello-joker:${CI_COMMIT_SHA:0:8}
      - docker login -u ${DOCKER_USERNAME} -p ${DOCKER_PASSWORD}
      - docker push varunmallya/hello-joker:${CI_COMMIT_SHA:0:8}
 update_helm_chart:    #6
    stage: update
    image:
      name: python:3.10
    script:
      - apt-get update && apt-get install git
      - git clone 
https://${GITLAB_USER_NAME}:${GITLAB_ACCESS_TOKEN}@\
gitlab.com/learn-mlops/gitlab-ci-example.git  #7
      - cd $CI_PROJECT_DIR
      - wget 
https://github.com/mikefarah/yq/releases/download/v4.2.0/\
yq_linux_amd64 -O /usr/bin/yq && chmod +x /usr/bin/yq  #8
      - git config --global user.email "${GITLAB_USER_EMAIL}"
      - git config --global user.name "${GITLAB_USER_NAME}"
      - yq e -i ".image.tag |= "${CI_COMMIT_SHA:0:8}"" \
        hello-joker/values.yaml  #9
      - git add hello-joker/values.yaml
      - git commit -m "[skip ci]Update helm chart"    #10
      - git push https://${GITLAB_USER_NAME}:\
        ${GITLAB_ACCESS_TOKEN}@gitlab.com/learn-mlops/\
        gitlab-ci-example.git HEAD:main  #11

Inspect the complete .gitlab-ci.yaml configuration, including stages, Docker-in-Docker service, test execution, image publishing, and Helm updates. section 3.4.1

How Machine Learning Platform Engineering covers it

How deep it goes
Builds a runnable three-stage configuration with jobs, scripts, images, services, and predefined variables.
Examples
Uses a FastAPI random-jokes service, pytest tests, a Docker image, and a Helm chart.
What it assumes
Assumes a GitLab repository, Docker runtime, Docker Hub, and credentials configured as CI variables.
What you can do after
Operate/configure a GitLab CI pipeline that tests code, publishes an image, and updates deployment metadata.
How it connects
Connects continuous integration with Docker, Docker Hub, Helm, Kubernetes deployment, and Argo CD.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Machine Learning Platform Engineering. The section links point to the parts of the book it draws on.