GitHub Actions turns a push into a repeatable delivery gate for a RAG application. A workflow in .github/workflows defines the runner, runtime, dependency installation, tests, secrets, and—when the checks pass—the path to a deployed container. This replaces manual rebuilding, docker pushes, and restarts with a pipeline that can catch broken changes before users receive them. section 5.3 section 5.5
A minimal evaluation workflow runs whenever code is pushed to main. It uses an Ubuntu runner, checks out the repository, sets up Python 3.11, installs pinned requirements, exposes OPENAI_API_KEY from repository secrets, and runs pytest. The tests send evaluation questions to the RAG application, compare generated answers against expected answers, and require a PASS result. section 2.4.3
Build a production-oriented workflow in stages:
- Store dependencies in requirements.txt and pin the OpenAI package version so automated runs use a stable dependency set. section 2.4.3
- Place workflow YAML under .github/workflows and trigger the test job on pushes to main. section 2.4.3
- Use checkout, Python setup, dependency installation, and pytest steps to make the runner reproduce the test environment. section 2.4.3
- Keep credentials in GitHub Secrets rather than source code; inject them as environment variables only for the step that needs them. section 2.4.3 section 5.2.4
- Bind a deployment workflow to a GitHub environment so tests can access the live credentials needed for Cosmos DB, embeddings, and search-index checks. section 5.2.4
- Run tests inside the container image intended for production, making a passing result evidence that the image includes required libraries as well as working logic. section 5.2.4
Automatic RAG evaluation on push
A test loops over questions such as “What is Underwhelming Spatula?”, obtains each RAG answer, and asks an LLM evaluator whether it matches the expected information. The GitHub Actions job runs pytest after installing openai, dotenv, and pytest; a missing PASS fails the pipeline. As requirements grow, the evaluator prompt can demand exact dates or numbers and explanations for failures. section 2.4.3
For deployment, GitHub Actions can rebuild the container and rerun tests after each commit. Only a passing run swaps the new image into the Azure Web App, creating a short feedback loop from code change to tested release. Logs and later load testing complement this gate when diagnosing behavior under real traffic. section 5.3 section 5.5
Figure 2.4 The automated eval system for RAG development. Evals compare chatbot answers to expected outputs and automate testing via GitHub Actions, ensuring high-quality performance.
The diagram links expected-answer evaluations, RAG responses, automated tests, and the GitHub Actions gate in one delivery flow. section 2.4.3










