liveBook · Manning

Software testing

As explained in Vibe Engineering

An executable procedure that evaluates software behavior against expected results or conditions.

A passing test shows that software met an expected condition for a particular case; it does not establish correctness for every case. This distinction matters when AI-generated code handles a convincing happy path but fails on neighboring inputs. section 8.1.3 section 8.2

Tests execute behavior against expectations. Unlike static checks for syntax, types, or conventions, they can check whether a given input produces the required output. Write those expectations as contracts: specify what the software must do without tying the test to how it does it. section 8.2.6 section 8.3.7

A parser that passes the easy cases

A generated markup parser may convert simple bold text, headers, and links correctly while breaking nested formatting or links with parentheses. Test both the ordinary conversions and combinations or malformed inputs that challenge the parser’s assumptions. section 8.2 section 8.8.6

Listing 8.1 Initial version of markup parser in Python
def parse_markup(text: str) -> str:
    result = text
    
    lines = result.split('\n')
    for i, line in enumerate(lines):
        if line.startswith('###'):
            lines[i] = f'<h3>{line[3:].strip()}</h3>'
        elif line.startswith('##'):
            lines[i] = f'<h2>{line[2:].strip()}</h2>'
        elif line.startswith('#'):
            lines[i] = f'<h1>{line[1:].strip()}</h1>'
    result = '\n'.join(lines)
    
    import re
    result = re.sub(r'\*\*(.+?)\*\*', r'<strong>\1</strong>', result) # Bold
    result = re.sub(r'\*(.+?)\*', r'<em>\1</em>', result)  # Italic  
    result = re.sub(r'\[(.+?)\]\((.+?)\)', r'<a href="\2">\1</a>', result)  # Links
    
    return result

Inspect the parser’s regular expressions and header handling alongside the cases they can miss. section 8.2

Use testing as a disciplined verification process:

  • Before migrating legacy code, establish a baseline with unit, integration, end-to-end, and performance tests where applicable. section 2.4
  • Ask an AI tool to identify and generate missing tests, then run and review them. Generated tests can be invalid, poorly designed, or produce false positives. section 2.4.1
  • Describe requirements independently of the implementation. High coverage can exercise code without specifying the right behavior; snapshots can preserve wrong behavior. section 8.3 section 8.3.7
  • Probe failure paths as well as expected cases, including empty, malformed, or malicious inputs and unavailable dependencies. section 8.8.6
Cursor
MemberRepositoryTest.java:
@ApplicationScoped
public class MemberRepositoryTest {
    @Inject
    @Mock
    private EntityManager em;
    @InjectMocks
    private MemberRepository memberRepository;
    @BeforeEach
    void setUp() {
        MockitoAnnotations.openMocks(this);
    }
    @Test
    void testFindById() {
        Member mockMember = new Member();
        mockMember.setId(1L);
        when(em.find(Member.class, 1L)).thenReturn(mockMember);
        Member result = memberRepository.findById(1L);
        assertNotNull(result);
        assertEquals(1L, result.getId());
        verify(em).find(Member.class, 1L);
    } // remaining tests omitted
 
RemoteMemberRegistrationIT.java:
  @Test
    public void testGetAllMembers() throws Exception {
        HttpRequest request = HttpRequest.newBuilder(getHTTPEndpoint())
                .GET()
                .build();
        HttpResponse response = HttpClient.newHttpClient().send(request, HttpResponse.BodyHandlers.ofString());
        Assert.assertEquals(200, response.statusCode());
        // Parse the JSON response
        JsonReader jsonReader = Json.createReader(new StringReader(response.body().toString()));
        JsonArray members = jsonReader.readArray();
        Assert.assertNotNull("Response should be a JSON array", members);
        
        // Log the members for debugging
        log.info("Found " + members.size() + " members");
        for (int i = 0; i < members.size(); i++) {
            JsonObject member = members.getJsonObject(i);
            log.info("Member " + i + ": " + member);
        }

Compare generated repository and integration tests when reviewing a migration baseline. section 2.4.1

Testing is one control, not a substitute for all others. An agent with production credentials can cause harm through its actions even when the software has excellent tests; permissions, environment boundaries, and review are separate safeguards. section 1.1.4

How Vibe Engineering covers it

How deep it goes
Explains test generation, review, execution, and contract design, including coverage limits and the need for other safeguards.
Examples
Examines a markup parser’s edge cases and generated repository and integration tests for a migration.
What you can do after
Practice/perform testing by specifying expected behavior, reviewing generated tests, and probing failure cases.
How it connects
Connects testing with static verification, code coverage, property testing, and access control.

Also explained in

Compare how 67 books explain software testing →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Vibe Engineering. The section links point to the parts of the book it draws on.