liveBook · Manning

Code coverage

As explained in Vibe Engineering

Code coverage is a metric reporting the proportion of source-code lines, branches, or paths executed during testing.

Code coverage reports the proportion of source-code lines, branches, or paths executed during testing. High coverage can still conceal untested behavior: execution shows that code ran, not that assertions verified the right result. section 8.3.4

AI-generated coverage is especially risky when the same process produces both implementation and tests. Correlated tests may reproduce the implementation’s assumptions, exercise familiar cases, and miss adversarial inputs or edge cases. section 8.3.4 section 8.3.7

Use coverage as evidence, not as a release decision:

  • Review tests for requirements and expected behavior, rather than accepting high percentages as proof of correctness. section 8.3.7
  • Add cases for malformed input, boundary conditions, and mixed or multiline data that can traverse existing lines while exposing defects. section 8.3.4
  • Review AI-generated tests for validity and false positives before using them to support migration work. section 2.4.1
  • Combine coverage with property-based testing, mutation testing, contract testing, characterization testing, and runtime observability when the system warrants stronger evidence. section 8.9

A misleading 95% result

A markup parser can report roughly 95% coverage after tests for simple H1, H2, H3, and bold cases. That result does not establish correct handling of #### H4, a header without a space, or multiline input. The tests execute the parser, but they do not specify enough behavior to expose its blind spots. section 8.3.4

Listing 8.6 First set of test case
def parse_markup(text: str) -> str:
    result = text
    
    # Headers
    lines = result.split('\n')
    for i, line in enumerate(lines):
        if line.startswith('###'):
            lines[i] = f'<h3>{line[3:].strip()}</h3>'
        elif line.startswith('##'):
            lines[i] = f'<h2>{line[2:].strip()}</h2>'
        elif line.startswith('#'):
            lines[i] = f'<h1>{line[1:].strip()}</h1>'
    result = '\n'.join(lines)
    
    result = re.sub(r'\*\*(.+?)\*\*', r'<strong>\1</strong>', result)
    result = re.sub(r'\*(.+?)\*', r'<em>\1</em>', result)
    result = re.sub(r'\[(.+?)\]\((.+?)\)', r'<a href="\2">\1</a>', result)
    
    return result
 
 
def test_headers():
    assert parse_markup("# H1") == "<h1>H1</h1>"
    assert parse_markup("## H2") == "<h2>H2</h2>"
    assert parse_markup("### H3") == "<h3>H3</h3>"
 
def test_formatting():
    assert parse_markup("**bold**") == "<strong>bold</strong>"
    assert parse_markup("*italic*") == "<em>italic</em>"
 
def test_links():
    assert parse_markup("[a](b)") == '<a href="b">a</a>'

Inspect the parser and its first tests to see how straightforward examples can exercise code without testing adversarial behavior. section 8.3.4

How Vibe Engineering covers it

How deep it goes
Explains the metric and its verification gap, but does not detail coverage-tool internals or calculation procedures.
Examples
Uses a markup parser whose simple header tests achieve high coverage while missing malformed headers and multiline behavior.
What you can do after
Evaluate/understand: distinguish code execution from behavioral verification and demand tests that express independent requirements.
How it connects
Connects coverage with unit testing, property-based testing, mutation testing, contract testing, characterization testing, and observability.

Also explained in

Compare how 22 books explain code coverage →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Vibe Engineering. The section links point to the parts of the book it draws on.