liveBook · Manning

HumanEval

As explained in LLM Evaluation and Alignment, The Foundational Ideas

A benchmark dataset of programming problems used to evaluate code-generation systems by executing generated solutions against tests.

HumanEval is a benchmark for evaluating code-generation systems with executable tests. It presents 164 Python programming problems, each containing a function signature, docstring, and doctest cases. section 1.1.1

Evaluate a generated solution through four steps:

  • Provide the function signature and docstring to the model. section 1.1.1
  • Generate a completion for the function body. section 1.1.1
  • Execute the completion in a sandboxed runtime against the supplied tests. section 1.1.1
  • Record pass only when every test succeeds; otherwise record fail. section 1.1.1

Checking nearby numbers

The has_close_elements problem asks whether two numbers lie within a threshold. Its doctests expect False for [1.0, 2.0, 3.0] with threshold 0.5 and True for a list containing 2.8 and 3.0 with threshold 0.3.

How LLM Evaluation and Alignment, The Foundational Ideas covers it

How deep it goes
Defines a four-step execution workflow but stops short of measuring readability, efficiency, or broader code quality.
Examples
Uses the has_close_elements function with numeric lists, thresholds, and expected Boolean doctest outputs.
What it assumes
Starts from function signatures and docstrings, then makes the generation, execution, and pass/fail workflow explicit.
What you can do after
Apply it via sandboxed execution to determine whether generated Python solutions pass all supplied tests.
How it connects
Connects code generation with verifiable tasks, doctest cases, sandboxed execution, and open-ended evaluation.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Evaluation and Alignment, The Foundational Ideas. The section links point to the parts of the book it draws on.