Data leakage occurs when held-out information reaches a model through its inputs or training-like context. In evaluation, giving the model an answer it is supposed to generate can inflate the result and make the measurement useless. section 6.4
A tempting shortcut is to draw few-shot examples from the same dataset used for evaluation. If the prompt includes the current test query’s expected SQL, the model can reproduce the answer rather than solve the task. Keep evaluation prompts and answers separate from the frozen pool used for examples. section 6.4 section 6.3.3
Guarding a SQL-generator benchmark
The benchmark loads evaluation prompts and expected SQL from dev.json and draws examples from a separate examples.json pool. It selects examples for the relevant database and filters out the current gold SQL after normalization. The filter adds protection if the pools accidentally overlap; it does not establish that every other form of leakage has been eliminated. section 6.4 section 6.3.3
Listing 6.19 Extracting example queries from a frozen pool
public static List<String> getExampleQueries(String dbId, int n, String sqlToExclude) {
if (exampleEntries == null || exampleEntries.isEmpty() || n <= 0) {
return new ArrayList<>();
}
String normalizedExclude = normalizeSql(sqlToExclude);
return exampleEntries.stream()
.filter(entry -> dbId.equals(entry.getDbId()))
.map(DbInput::getSql)
.filter(sql -> sql != null && !sql.trim().isEmpty())
.filter(sql -> normalizedExclude == null || !normalizedExclude.equals(normalizeSql(sql)))
.limit(n)
.collect(Collectors.toList());
}
The extractor shows the database filter, normalized gold-SQL exclusion, and limit on the number of examples. section 6.4
Strong metrics are a reason to check the evaluation pipeline, not proof that it is clean. For example, stable cross-validation accuracy and a high F1 score on medical text warrant a leakage sanity check, even when results look consistent across folds. section 4.6.5











