A benchmark measures a system against defined inputs and expected results. A high score answers a specific evaluation question; it does not, by itself, establish production readiness. section 6.3.3 section 8.1.5
A useful benchmark makes its inputs, comparison method, and reported results explicit. For an LLM-based SQL generator, the evaluation prompts and expected SQL must remain separate from the examples supplied in the model’s prompt. Otherwise, the benchmark can be contaminated by answers appearing in the model’s context. section 6.3.3
A SQL-generation benchmark can follow this workflow:
- Load evaluation prompts and expected SQL, and keep the frozen few-shot examples in a separate pool. section 6.3.3
- Extract table schemas, submit each evaluation prompt with its schema, and collect the generated SQL. section 6.3.3
- Compare generated and expected SQL using overlap similarity. Save per-query results and summarize the 50th, 90th, and 99th percentiles by database and difficulty. section 6.3.3
Listing 6.18 Precision measurement framework - the main logic
List<DbInput> dbInputs = loadDbInput(); #A
DevJsonQueryExtractor.setDbInputs(dbInputs); #B
List<DbInput> inputSubset = dbInputs.stream() #C
.filter(extractDBs())
.collect(Collectors.toList());
System.out.println("Will run the SQL generation for " + inputSubset.size() + " number of inputs.");
File tempDirectory = new File(System.getProperty("java.io.tmpdir")); #D
Path filePath = Files.createTempFile(
tempDirectory.toPath(),
"sql_generator_precision_result_last_"+NR_OF_LAST_QUERIES_INCLUDED + "_queries",
"_" + Instant.now().toEpochMilli() + ".csv"
);
System.out.println("The results will be saved to: " + filePath);
#E
Path summaryFilePath = Files.createTempFile(
tempDirectory.toPath(),
"sql_generator_precision_summary_last_"+NR_OF_LAST_QUERIES_INCLUDED + "_queries",
"_" + Instant.now().toEpochMilli() + ".csv"
);
System.out.println("The summary will be saved to: " + summaryFilePath);
try (CSVWriter writer = new CSVWriter(new FileWriter(filePath.toString()));
CSVWriter summaryWriter = new CSVWriter(new FileWriter(summaryFilePath.toString()))) {
#F
String[] header = {
"db_id",
"difficulty",
"question",
"expected",
"actual",
"overlap_similarity"
};
writer.writeNext(header);
#G
String[] summaryHeader = {
"db_id",
"difficulty",
"p50_overlap_similarity",
"p90_overlap_similarity",
"p99_overlap_similarity"
};
summaryWriter.writeNext(summaryHeader);
#H
List<SqlGenerationResult> results = new ArrayList<>();
String finalBaseUrl = baseUrl;
inputSubset.stream()
.map(input -> processDbInput(input, finalBaseUrl))
.forEach(result -> {
showResultAndSave(result, writer);
results.add(result);
});
generateAndSaveSummary(results, summaryWriter); #I
Inspect the benchmark’s main logic for loading inputs and setting up result files. section 6.3.3
Interpret scores against what the benchmark actually tests. HumanEval runs generated solutions against tests for isolated Python problems; EvalPlus adds edge-case tests but retains the isolated-problem setting. SWE-bench uses issues from existing repositories, making it closer to development in an existing codebase. section 8.1.4
Even strong results do not establish domain understanding, resilience to adversarial inputs, or maintainability. Use benchmark results as evidence about measured tasks, not as a substitute for production review and verification. section 8.1.5
Performance depends on the workload
An account-search comparison used warm-up and checked performance across implementations. In one custom run with 10,000 accounts, the single-threaded method took 25 ms, versus 63 ms for parallel streams and 30,398 ms for the concurrent method. These results challenge the assumption that adding threads necessarily improves this workload; they are not a universal ranking of the approaches. section 7.1.2
Listing 7.7 The custom benchmark results
Single-threaded time: 25 ms Parallel stream time: 63 ms Concurrent time: 30398 ms Optimized time: 406 ms
See the reported timings for the custom account-search run. section 7.1.2





























