HumanEval is a benchmark for evaluating code-generation systems with executable tests. It presents 164 Python programming problems, each containing a function signature, docstring, and doctest cases. section 1.1.1
Evaluate a generated solution through four steps:
- Provide the function signature and docstring to the model. section 1.1.1
- Generate a completion for the function body. section 1.1.1
- Execute the completion in a sandboxed runtime against the supplied tests. section 1.1.1
- Record pass only when every test succeeds; otherwise record fail. section 1.1.1
Checking nearby numbers
The has_close_elements problem asks whether two numbers lie within a threshold. Its doctests expect False for [1.0, 2.0, 3.0] with threshold 0.5 and True for a list containing 2.8 and 3.0 with threshold 0.3.
