6 A scientific approach for validating LLM-based solutions
This chapter covers
- Encapsulation of the LLM API to not tightly couple with the specific provider
- Solving a mistake of the models’ silent accuracy drop with a Scientific approach
- Trade-off between context size and model accuracy
- Trade-off between context size and the cost of using the model
Let’s say we are given the task of implementing text-to-SQL functionality. Users will input human text, such as “generate a SQL for counting all the employees in our marketing department,” and the service will output the corresponding SQL query that can be submitted to any SQL database.
We aim to harness the power of generative AI to enhance query generation. However, while doing it, there are a couple of trade-offs and mistakes that we need to avoid.
Firstly, we want to avoid being tightly coupled with a specific LLM API provider. Such a mistake can result in significant maintenance overhead in the future if we choose to change the model provider company.
To achieve this flexibility, we will implement a custom service that hides the model provider behind a clean API for end-users, starting by encapsulating the OpenAI API for programmatic access. We will create a service that leverages Vibe Engineering techniques (generating SQL from human input) to deliver value while keeping the underlying infrastructure decoupled