liveBook · Manning

Code coverage

As explained in LLM Evaluation and Alignment, The Foundational Ideas

The extent to which a test suite exercises relevant code paths, inputs, or behaviors.

Code coverage measures how far a test suite exercises relevant code paths, inputs, or behaviors. Passing every supplied test does not prove correctness: unanticipated edge cases may remain untested. Verifications are only as strong as the cases they include. section 1.1.1

Interpret coverage alongside the limits of executable evaluation:

  • Use deterministic unit tests and exact string matching for objectively checkable behavior, but treat test results as partial evidence rather than a complete quality measure. section 1.1.1
  • Add cases for edge conditions that the initial suite does not anticipate; narrow suites can let incorrect implementations pass. section 1.1.1
  • Separate functional coverage from qualities such as readability, efficiency, maintainability, clarity, and contextual appropriateness, which executable checks may not capture. section 1.1.1

A narrow function test

A HumanEval-style test checks whether has_close_elements returns the expected result for selected number lists and thresholds. Those examples establish behavior for the tested inputs, but they do not establish behavior across every possible list, threshold, or edge case. section 1.1.1

Listing 1.1 Verifiable rewards example (HumanEval)
def has_close_elements(numbers: List[float], threshold: float) -> bool:
    “””Check if in given list of numbers, are any two numbers
    closer to each other than given threshold.
    >>> has_close_elements([1.0, 2.0, 3.0], 0.5)
    False
    >>> has_close_elements([1.0, 2.8, 3.0, 4.0, 5.0, 2.0], 0.3)
    True

Inspect the runnable function specification and its example assertions to see how executable cases define only part of the tested behavior. section 1.1.1

How LLM Evaluation and Alignment, The Foundational Ideas covers it

How deep it goes
Defines coverage gaps and their consequences, but does not provide a formal coverage formula or systematic test-generation procedure.
Examples
Uses HumanEval-style function tests and contrasts alternative FizzBuzz implementations with different nonfunctional qualities.
What you can do after
Diagnose test suites by finding untested edge cases and separating functional correctness from broader software qualities.
How it connects
Connects code coverage with unit testing, edge cases, verifiable evaluation, and multiple valid solutions.

Also explained in

Compare how 22 books explain code coverage →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Evaluation and Alignment, The Foundational Ideas. The section links point to the parts of the book it draws on.