liveBook · Manning

Calibrated Pass@1

As explained in Introduction to Generative AI, Second Edition

Calibrated Pass@1 is a code-generation metric measuring the confidence-adjusted rate at which a model produces a working answer on its first attempt.

When selecting an LLM for code generation, calibrated Pass@1 estimates how often it produces a working answer on its first attempt while adjusting for confidence. section 10.19

The metric supports more informed comparisons, but evaluation results can shift with details such as formatting changes, so leaderboard scores require careful interpretation. section 10.19

Table 10.1 A leaderboard for code-generation benchmark BigCodeBench

Rank

Model

Pass@1

Year

1

Gemini-Exp-1206

62.4

2024

2

DeepSeek-V3

62.2

2024

3

Llama-4-Maverick

61.4

2025

4

GPT-4o

61.1

2024

A BigCodeBench leaderboard illustrates how calibrated Pass@1 compares code-generation models, with scores ranging from 61.1 to 62.4 in the listed results. section 10.19

How Introduction to Generative AI, Second Edition covers it

How deep it goes
Defines the confidence-adjusted first-attempt success rate and notes evaluation sensitivity without detailing its calculation.
Examples
Uses BigCodeBench’s Complete–Full split and a leaderboard comparing Gemini-Exp-1206, DeepSeek-V3, Llama-4-Maverick, and GPT-4o.
What you can do after
Evaluate and compare LLMs for code generation using confidence-adjusted first-attempt performance.
How it connects
Connects code generation with confidence, benchmarking, model comparison, and transparent reporting.
Cover of Introduction to Generative AI, Second Edition
liveBookmanning.com
Read Introduction to Generative AI, Second Edition on liveBook Get the book at manning.com

This explainer was generated by Manning's knowledge graph from the text of Introduction to Generative AI, Second Edition. The section links point to the parts of the book it draws on.