An AI agent is a decision-maker that interacts with an environment to achieve a goal by taking actions and receiving feedback as rewards. A coding agent may browse documentation, edit files, and run tests. section 1.1.1
Evaluate an agent by checking whether it:
- Selects the correct tool or API at each step. section 1.1.1
- Uses the tool appropriately while pursuing its goal. section 1.1.1
- Produces outcomes that can be tested objectively, such as passing code tests. section 1.1.1
Coding agent
A coding agent edits a file and runs its tests. Passing tests provide a reward signal; failing tests provide no reward and expose a verifiable error. section 1.1.1
Listing 1.1 Verifiable rewards example (HumanEval)
def has_close_elements(numbers: List[float], threshold: float) -> bool:
“””Check if in given list of numbers, are any two numbers
closer to each other than given threshold.
>>> has_close_elements([1.0, 2.0, 3.0], 0.5)
False
>>> has_close_elements([1.0, 2.8, 3.0, 4.0, 5.0, 2.0], 0.3)
True
Use the HumanEval example to see how executable tests turn an agent-produced coding task into a reproducible reward check. section 1.1.1













































