liveBook · Manning

Software testing

As explained in LLM Evaluation and Alignment, The Foundational Ideas

Software testing is the systematic verification that software behaves correctly against specified requirements and test cases.

Unit tests make some software tasks verifiable: correctness has a definable answer, tests can check it automatically, and the criteria are objective. Code quality, readability, and efficiency may still require judgment. section 1.1.1

A test-based evaluation of generated code proceeds in four steps:

Listing 1.1 Verifiable rewards example (HumanEval)
def has_close_elements(numbers: List[float], threshold: float) -> bool:
    “””Check if in given list of numbers, are any two numbers
    closer to each other than given threshold.
    >>> has_close_elements([1.0, 2.0, 3.0], 0.5)
    False
    >>> has_close_elements([1.0, 2.8, 3.0, 4.0, 5.0, 2.0], 0.3)
    True

The doctests for has_close_elements show how input examples and expected Boolean results can check a function. section 1.1.1

How LLM Evaluation and Alignment, The Foundational Ideas covers it

How deep it goes
Details a four-step test workflow and notes that tests do not capture code quality, readability, or efficiency.
Examples
Uses sorting correctness and HumanEval doctests for has_close_elements as examples.
What it assumes
Assumes a definable correct answer, automatic checking, and objective criteria for a task to be verifiable.
What you can do after
Apply unit tests to check generated code automatically and record whether all tests pass.
How it connects
Connects unit testing with AI evaluation, code generation, and open-ended evaluation.

Also explained in

Compare how 67 books explain software testing →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Evaluation and Alignment, The Foundational Ideas. The section links point to the parts of the book it draws on.