liveBook · Manning

Chapter 6: A scientific approach for validating LLM-based solutions

What you will learn in chapter 6 of Vibe Engineering

The chapter introduces a text-to-SQL service and explains why provider independence and measured accuracy matter when building an LLM-based solution.

After this chapter you will be able to

Section by section

Introduction

The chapter introduces a text-to-SQL service and explains why provider independence and measured accuracy matter when building an LLM-based solution.

Key terms: text-to-SQL, LLM API provider, clean API, managed AI gateways, vendor lock-in, integration tests, model accuracy, validation framework, metric, bird-bench

6.1 Creating a text-to-SQL service

Introduces the Spring Boot text-to-SQL service and explains why it is used to expose the service over HTTP.

Key terms: Spring Boot, embedded HTTP service, REST APIs, text-to-SQL

6.1.1 Creating the service skeleton

Identifies the Spring application entry point and configuration files that form the text-to-SQL service skeleton.

Key terms: Java main function, application.yaml, Spring settings, application-test.yaml, integration tests

6.1.2 Defining the API of the service

This section defines the text-to-SQL service’s REST request and response, explains how table metadata guides SQL generation, and separates the controller from the LLM provider.

Key terms: SqlGeneratorQueryRequest, user_prompt, list of tables, Table, tableName, SQL, fields, types, SqlGeneratorController, SqlGeneratorService

6.2 Integration with the OpenAI API

The section explains how a text-to-SQL service configures an LLM provider, builds a SQL-generation prompt, controls requests, and turns the model response into executable SQL.

Key terms: OpenAI API, model, application.yaml, generateSql(), buildPrompt(), input tables, recent queries, SQL queries, maxTokens(500), input-token costs

6.2.1 Testing the integration

This subsection shows how to configure an OpenAI integration test and why it checks basic SQL properties instead of an exact generated query.

Key terms: SqlGeneratorService, test profile, application-test.yaml, OpenAI API key, generated SQL, SELECT statement, not 100% deterministic

6.3 Developing the accuracy verification framework

The section motivates using the BIRD benchmark to verify text-to-SQL accuracy and identifies two input and prompting choices that may improve generated SQL.

Key terms: BIRD benchmark, BIRD dataset, text-to-SQL, schema of the tables, description of each column, Let’s think step by step, reasoning models, chain of thoughts

6.3.1 Delving into the bird-bench dataset

The section explains how BIRD’s development dataset supplies reference questions, SQL, and schemas for evaluating a text-to-SQL service.

Key terms: BIRD benchmark, dev.json, question, evidence, SQL, gold standard, db_id, difficulty, simple, moderate

6.3.2 Methods for comparing expected vs the actual generated SQL and their trade-offs

This section compares ways to evaluate generated SQL against ground-truth SQL, then explains a simple word-overlap measure chosen for the verification framework.

Key terms: Text-Based Methods, Structural (AST-Based) Methods, Semantic Methods, Embedding-Based Semantic, Execution-Based Semantic, Overlap-Based SQL Query Similarity, Jaccard similarity, intersection, union, non-keyword tokens

6.3.3 Submitting the queries to the SQL-generator-service

The section explains how the SQL-generator benchmark keeps evaluation data separate from prompt examples, submits queries, reports results, and interprets an overlap score.

Key terms: dev.json, examples.json, data contamination, few-shot prompt context, ValidateSqlGeneratorPrecision, CREATE TABLE, schema, Overlap Similarity, db_id/difficulty, gold-standard answer

6.4 Trade-offs with input-context size

This section explains how to add example SQL queries to a text-to-SQL prompt without contaminating the evaluation used to judge their effect.

Key terms: SqlGeneratorQueryRequest, nLastQueries, context, few-shot examples, data contamination, dev.json, examples.json, dbId, gold SQL

6.4.1 Accuracy vs context-size

The section interprets SQL-similarity benchmark results and shows that adding recent queries to the LLM’s context improves accuracy, but with diminishing returns.

Key terms: p50, p90, p99, similarity, summary file, zero last queries, query history, context size, diminishing returns, sweet spot

6.4.2 Context-size vs cost of running the model

The section combines measured accuracy, context size, and token-based cost to choose a text-to-SQL configuration and estimate its cost at production traffic.

Key terms: context size, tokens, p50 accuracy, accuracy-to-cost ratio, 316 queries, 100 requests per second, cost per day

6.4.3 Migration to another AI provider - Google Gemini

The section adds Google Gemini as a selectable text-to-SQL provider while preserving OpenAI support, then checks that Gemini is actually used.

Key terms: LanguageModelClient, SqlGeneratorServiceImpl, generateText(), ai.provider, configuration default, runtime fallback, precision metrics, Google Genai Java SDK, GeminiLanguageModelClient, Vibe Engineering

6.4.4 Deploying the application to Google Cloud Run

The section shows how to guide an LLM to generate a secure Google Cloud Run deployment, then build, configure, and deploy the text-to-SQL service.

Key terms: Google Artifact Registry, Container Registry, Google Cloud Secret Manager, guardrails, Dockerfile, Maven, gcloud builds submit, gcloud run deploy, Secret Manager, --set-secrets

6.4.5 Verifying the text-to-SQL running in the cloud

The section verifies the deployed text-to-SQL service through a REST request and Cloud Run logs, then explains why an overloaded-provider error reaches the user.

Key terms: REST API, user_prompt, Google console logs, gemini-2.5-flash, 503, free tier, runtime fallback, strict configuration default, benchmarking data contamination, isolated routing

6.4.6 Switch the AI provider

Shows how to check that the deployed application can switch from Gemini to OpenAI without being locked into one AI provider.

Key terms: vendor lock-in, AI_PROVIDER, Secret Manager, AI provider

6.5 Summary

Where this is used later

Concepts explained in this chapter

← Chapter 5: From autocomplete to agent fleets: the road to Continuous AI Chapter 7: Vibe performance engineering: when assumptions mislead →

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.