Chapter 8: Building effective tests with generative AI
The chapter introduces why testing matters and how generative AI can make writing thorough tests faster.
After this chapter you will be able to
- Value of sustained testing. Explain why writing tests is worth the time even when development is moving quickly.
- Generative AI testing assistance. Describe how generative AI can help developers write tests faster and suggest tests they might otherwise miss.
- Benefits of AI-generated tests. Explain how generative AI can make test development more productive by providing scaffolding and helping developers create more tests in less time.
- Testing risks and safeguards. Distinguish why some concerns about AI-generated code have less impact on tests from the risks that still require care.
- Focused unit test checks. Explain that unit tests check one piece of functionality at a time, often a Python function or class, by supplying inputs and asserting the expected output.
- Copilot test generation options. Describe the ways this section says developers can use Copilot to build or generate tests, and name the advantages it lists without explaining how those features work.
- Tabnine test generation options. Describe how Tabnine can be used for test generation and identify the advantages listed here, including its IDE or chat interfaces, application-aware context, model choices, and code-security options.
- Blackbox AI testing features. Describe how Blackbox AI can be used in an IDE or chat interface, and identify the features the section highlights for testing.
- Default testing framework. State which Python testing framework the tools tend to use by default, and distinguish it from the framework the author plans to request.
- Python testing framework choice. Explain why the author chooses to explore pytest with AI testing tools while recognizing unittest as an established alternative. Describe the pytest features that motivate that choice.
- Specific prompts and human review. Explain why the author moved from Copilot’s /tests shortcut to specific prompts and reviewed the generated tests rather than accepting them as correct.
- Isolated in-memory database. Describe why the author creates an in-memory SQLite database for tests and how the Copilot-generated setup was corrected.
- Reusable database fixture. Explain how the pytest fixture replaces repeated database connection setup across the tests.
- Meaningful test assertions. Describe how the author turned preliminary database checks and Copilot suggestions into tests with assertions suited to the application.
- Tabnine test planning workflow. Describe how Tabnine uses an existing test file to build a test plan with suggested tests alongside existing ones.
- Generated test refinement. Explain how the author changed Tabnine's first generated test to match the project's testing setup and checked the result.
- Context informed test generation. Explain how Blackbox AI used the application’s code as context to generate tests for the Questions class, and describe how the author checked whether the generated tests ran.
- Framework choice in test prompts. Distinguish the results of asking generally for unit tests from explicitly asking Blackbox AI for pytest tests in this example.
- Testing tool tradeoffs. Distinguish GitHub Copilot, Tabnine, and Blackbox AI by the strengths and limitations the section identifies, and explain why the choice depends on your needs.
- Human guided test generation. Describe the recommended sequence for generating AI-assisted tests and explain where developer review and guidance are needed for adequate coverage.
What it builds on
- Testing and debugging (chapter 1) · Introduces testing and debugging as part of an AI-assisted development workflow.
- Creating the database (chapter 5) · The application’s database.
- Pulling a set of questions (chapter 5) · The application functionality for pulling a set of questions.
Section by section
Introduction
The chapter introduces why testing matters and how generative AI can make writing thorough tests faster.
- Value of sustained testing: Explain why writing tests is worth the time even when development is moving quickly.
Why it matters: This is the reason to keep testing even under pressure to ship features. - Generative AI testing assistance: Describe how generative AI can help developers write tests faster and suggest tests they might otherwise miss.
Why it matters: AI assistance can reduce routine writing without giving up the goal of thorough tests.
Key terms: tests, reliable software, generative AI, boilerplate, test cases, coverage
8.1 Why use generative AI for testing?
This section explains why AI-generated tests can improve development speed and feedback while requiring care with their risks.
- Benefits of AI-generated tests: Explain how generative AI can make test development more productive by providing scaffolding and helping developers create more tests in less time.
Why it matters: Testing can be a productive use of AI even for developers who prefer to write their own application code. - Testing risks and safeguards: Distinguish why some concerns about AI-generated code have less impact on tests from the risks that still require care.
Why it matters: The case for AI-assisted testing depends on its narrower task, not on assuming generated tests are risk-free.
Key terms: AI-generated scaffolding, test generation, faster feedback loop, unit test, context, code review, careful prompt design
8.2 What are unit tests?
Defines unit tests as focused checks of individual pieces of code and explains how they help catch bugs before release.
- Focused unit test checks: Explain that unit tests check one piece of functionality at a time, often a Python function or class, by supplying inputs and asserting the expected output.
Why it matters: This gives you a way to judge whether a generated test checks a specific behavior. - Early bug detection limits: Explain why unit tests help developers catch bugs before publishing, while not covering everything that can go wrong.
Why it matters: Finding a failing input before release is better than discovering it when a user encounters it.
Key terms: unit tests, functions, classes, assert, bugs, reasonably stable
8.3 The tools we’ll use for Python testing
Introduces the AI tools the chapter will use for test generation and previews a comparison of their approaches.
8.3.1 Github Copilot
This subsection identifies ways to use GitHub Copilot for test generation and briefly lists its advantages.
- Copilot test generation options: Describe the ways this section says developers can use Copilot to build or generate tests, and name the advantages it lists without explaining how those features work.
Why it matters: These options help you recognize how Copilot might fit into a testing workflow.
Key terms: GitHub Copilot, chat-style interface, IDE, API, agent mode
8.3.2 Tabnine
The section introduces Tabnine as a test-generation tool and lists ways to use it and advantages it offers.
- Tabnine test generation options: Describe how Tabnine can be used for test generation and identify the advantages listed here, including its IDE or chat interfaces, application-aware context, model choices, and code-security options.
Why it matters: These options help you recognize what Tabnine offers before using it to generate tests.
Key terms: Tabnine, test generation, IDE, chat interface, local models
8.3.3 Blackbox AI
This subsection introduces Blackbox AI’s IDE and chat options and highlights its context-aware chat and test-focused agents.
- Blackbox AI testing features: Describe how Blackbox AI can be used in an IDE or chat interface, and identify the features the section highlights for testing.
Why it matters: These features explain why Blackbox AI is considered as a tool for writing tests.
Key terms: Blackbox AI, VS Code Chat, context, Customized agents
8.4 Writing unit tests with generative AI
This introduction notes that the AI tools tend to generate unittest tests by default and previews a comparison with pytest.
- Default testing framework: State which Python testing framework the tools tend to use by default, and distinguish it from the framework the author plans to request.
Why it matters: This tendency gives you a useful starting expectation when asking an AI tool to generate tests.
Key terms: unittest, pytest
8.4.1 unittest or pytest?
Compares unittest and pytest and explains why the chapter uses pytest to explore AI-generated tests.
- Python testing framework choice: Explain why the author chooses to explore pytest with AI testing tools while recognizing unittest as an established alternative. Describe the pytest features that motivate that choice.
Why it matters: This explains the framework choice behind the chapter’s AI-generated test examples.
Key terms: unittest, pytest, plain asserts, test discovery, fixture system
8.4.2 Using Copilot for test generation
The section walks through using GitHub Copilot to build pytest tests, correcting its suggestions and refining the test setup and assertions.
- Pytest test file setup: Describe how the section sets up pytest testing and names the test file for the Questions class.
Why it matters: The test file must be discoverable before pytest can run the tests Copilot helps write. - Specific prompts and human review: Explain why the author moved from Copilot’s /tests shortcut to specific prompts and reviewed the generated tests rather than accepting them as correct.
Why it matters: A plausible-looking generated test can target the wrong framework or an application method that does not exist. - Isolated in-memory database: Describe why the author creates an in-memory SQLite database for tests and how the Copilot-generated setup was corrected.
Why it matters: The database setup lets the author test realistic database-dependent behavior while protecting production data. - Reusable database fixture: Explain how the pytest fixture replaces repeated database connection setup across the tests.
Why it matters: Centralizing setup keeps multiple tests cleaner and avoids updating the same connection code in each one. - Meaningful test assertions: Describe how the author turned preliminary database checks and Copilot suggestions into tests with assertions suited to the application.
Why it matters: Assertions make the tests check expected behavior rather than merely run queries or accept plausible generated code.
Key terms: pytest, virtual environment, test_, GitHub Copilot, /tests, unittest, prompt, context, in-memory database, SQLite
8.4.3 Using Tabnine for test generation
This section demonstrates how Tabnine proposes tests from existing code and tests, then lets the developer refine a generated test to fit the project's pytest setup.
- Tabnine test planning workflow: Describe how Tabnine uses an existing test file to build a test plan with suggested tests alongside existing ones.
Why it matters: The plan gives you test ideas while showing how they relate to tests you already have. - Generated test refinement: Explain how the author changed Tabnine's first generated test to match the project's testing setup and checked the result.
Why it matters: A generated test can be useful after you adapt it to the project's fixtures and run it. - Tabnine model choice: Describe why the author mentions Tabnine's selectable models for test generation.
Why it matters: Model choice gives you another way to respond when a test-generation result does not suit your needs.
Key terms: test shortcut, existing test file, test plan, suggested tests, unittest, pytest, Mock, fixture, in-memory database, test ordering
8.4.4 Applying Blackbox AI for test generation
This section shows how Blackbox AI uses application context to generate tests for the Questions class, then explains how to steer its results with a more specific prompt or agent.
- Context informed test generation: Explain how Blackbox AI used the application’s code as context to generate tests for the Questions class, and describe how the author checked whether the generated tests ran.
Why it matters: Application context can make a simple test request useful, but generated tests still need to be run and checked. - Framework choice in test prompts: Distinguish the results of asking generally for unit tests from explicitly asking Blackbox AI for pytest tests in this example.
Why it matters: Naming the framework helps steer generated tests toward the form you intend to run. - More specific Blackbox requests: Describe the two ways the author suggests making a Blackbox AI test request more specific when an initial result is unsatisfactory.
Why it matters: These options give you a way to redirect Blackbox when a broad request does not produce the tests you want.
Key terms: Blackbox AI, Chat with Your Code Files, Questions class, in-memory database, unittest, pytest, pytest.fixture, assert, prompt, Blackbox AI agent
8.4.5 Which tool should you use for testing?
This section compares three AI testing tools and gives a human-guided workflow for generating effective tests.
- Testing tool tradeoffs: Distinguish GitHub Copilot, Tabnine, and Blackbox AI by the strengths and limitations the section identifies, and explain why the choice depends on your needs.
Why it matters: Comparing tradeoffs helps you try a tool that fits your testing needs rather than assuming one is best for everyone. - Human guided test generation: Describe the recommended sequence for generating AI-assisted tests and explain where developer review and guidance are needed for adequate coverage.
Why it matters: Developer guidance turns quickly generated tests into a suite that checks the intended behavior and important edge cases.
Key terms: GitHub Copilot, Tabnine, Blackbox AI, cloud-based, on-premises, AI test generation workflow, scaffold, fixtures, edge cases, boundary conditions
8.5 Summary
Concepts explained in this chapter
- test generationTest generation is the automated or semi-automated creation of software tests and their test data or expected outcomes.
- pytestpytest is a Python testing framework that supports concise test writing, test discovery, fixtures, and extensible execution.
- test fixtureA test fixture is reusable setup data, state, or environment configuration required to execute a test.
- GitHub CopilotCopilot Chat is a conversational generative AI tool that answers questions and assists with software-related tasks across web, desktop, and mobile interfaces.
- testA test case is a defined input, execution condition, and expected result used to evaluate software behavior.
- Blackbox AIBlackbox AI is an AI-powered software development assistant that provides code generation, code assistance, and related programming support.
- TabnineTabnine is an AI-powered programming assistant that provides context-aware code suggestions in supported code editors.
- evaluation suiteA collection of software tests maintained and executed together to evaluate an application or component.
- generative modelingArtificial intelligence that produces new content, such as text, source code, documentation, or tests, from learned patterns and input instructions.
- prompt engineeringPrompt engineering is the systematic design and refinement of instructions and context provided to an AI system to elicit desired outputs.
- contextContext is the surrounding conversation history, instructions, assumptions, and other information that determines the meaning and intent of an input.
- scaffoldApplication scaffolding is the automated generation of a basic application structure, including starter code, files, and directories.
- boilerplate codeBoilerplate code is repetitive, standard code required for common implementation tasks but contributing little distinctive application behavior.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.