Cover of Software Testing with Generative AI

What you will learn in Software Testing with Generative AI

by Mark Winteringham

Read it on liveBook Get the book at manning.com

Chapter by chapter

Chapter 1: Enhancing testing with large language models

1.1 Recognizing the effect of AI tools on testing and development · 1.1.1 Data generation · 1.1.2 Automated test building · 1.1.3 Test design · 1.2 Delivering value with LLMs · 1.2.1 A model for delivering value · 1.2.2 Using both human and AI abilities · 1.2.3 Being skeptical of LLMs · 1.3 Summary

What you learn in chapter 1 →

Chapter 2: Large language models and prompt engineering

2.1 LLMs explained · 2.2 Avoiding the risks of using LLMs · 2.2.1 Hallucinations · 2.2.2 Data provenance · 2.2.3 Data privacy · 2.3 Improving results with prompt engineering · 2.4 Examining the principles of prompt engineering · 2.4.1 Principle 1: Write clear and specific instructions · 2.4.2 Tactic 1: Use delimiters · 2.4.3 Tactic 2: Ask for structured output · 2.4.4 Tactic 3: Check for assumptions · 2.4.5 Tactic 4: Few-shot prompting · 2.4.6 Principle 2: Give the model time to “think” · 2.4.7 Tactic 1: Specify the steps to complete the task · 2.4.8 Tactic 2: Instruct the model to work out its own solution first · 2.5 Working with various LLMs · 2.5.1 Comparing LLMs · 2.5.2 Examining popular LLMs · 2.6 Creating a library of prompts · 2.7 Solving problems by using prompts · 2.8 Summary

What you learn in chapter 2 →

Chapter 3: Artificial intelligence, automation, and testing

3.1 The value of testing · 3.1.1 A different way of thinking about testing · 3.1.2 A holistic approach to testing · 3.2 How tools help with testing · 3.2.1 Automation bias · 3.2.2 Being selective with tooling · 3.3 Knowing when to use LLMs in testing · 3.3.1 Generative capabilities · 3.3.2 Transformation capabilities · 3.3.3 Enhancing capabilities · 3.3.4 LLMs in use in testing · 3.4 Summary

What you learn in chapter 3 →

Chapter 4: AI-assisted testing for developers

4.1 Examining the rise of the automated developer · 4.2 Pairing with LLMs · 4.2.1 Analyzing ideas · 4.2.2 Analyzing code · 4.2.3 Recognizing that a simulation is better than nothing at all · 4.3 Building in quality with AI assistance · 4.4 Creating our first TDD loop with LLMs · 4.4.1 Preparing the work · 4.4.2 Loop 1: Save a timesheet entry · 4.4.3 Loop 2: Retrieve a timesheet entry · 4.4.4 Loop 3: Calculating times for a project · 4.4.5 Refactoring code · 4.5 Improving documentation and communication with LLMs · 4.5.1 Generating code comments · 4.5.2 Generating release notes · 4.6 Maintaining a balance with code assistants · 4.7 Summary

What you learn in chapter 4 →

Chapter 5: Test planning with AI support

5.1 Defining test planning in modern testing · 5.1.1 Test planning, LLMs, and area of effect · 5.2 Focused prompts with the use of models · 5.2.1 Weak prompts mean weak suggestions · 5.2.2 What are models and why can they help · 5.3 Combining models and LLMs to assist test planning · 5.3.1 Creating a model to identify prompts · 5.3.2 Experimenting with different model types · 5.4 LLMs and test cases · 5.4.1 Having a healthy skepticism of generated risks and test cases · 5.5 Summary

What you learn in chapter 5 →

Chapter 6: Rapid data creation using AI

6.1 Generating and transforming data with LLMs · 6.1.1 Prompting LLMs to generate simple data sets · 6.1.2 Transforming test data into different formats · 6.2 Processing complex test data with LLMs · 6.2.1 Using format standards in prompts · 6.2.2 SQL exports as prompt guides · 6.3 Setting up LLMs as test data managers · 6.3.1 Setting up an OpenAI account · 6.3.2 Connecting to OpenAI · 6.4 Benefiting from generated test data · 6.5 Summary

What you learn in chapter 6 →

Chapter 7: Accelerating and improving UI automation using AI

7.1 Rapidly creating UI automation · 7.1.1 Setting up a project · 7.1.2 Creating our initial check with ChatGPT support · 7.1.3 Filling in gaps from generated code · 7.2 Improving existing UI automation · 7.2.1 Updating state management to use an appropriate layer · 7.2.2 Getting into the groove with AI tools · 7.3 Summary

What you learn in chapter 7 →

Chapter 8: Assisting exploratory testing with artificial intelligence

8.1 Organizing exploratory testing with LLMs · 8.1.1 Augmenting identified risks with LLMs · 8.1.2 Augmenting charter lists with LLMs · 8.2 Using LLMs during exploratory testing · 8.2.1 Establishing an understanding · 8.2.2 Creating data requirements for a session · 8.2.3 Exploring and investigating bugs · 8.2.4 Using LLMs to assist exploratory testing · 8.3 Summarizing testing notes with LLMs · 8.4 Summary

What you learn in chapter 8 →

Chapter 9: AI agents as testing assistants

9.1 Understanding AI agents and LLMs · 9.1.1 What defines an AI agent? · 9.1.2 How an agent works with LLMs · 9.2 Creating an AI Test Assistant · 9.2.1 Setting up our dummy AI agent · 9.2.2 Giving our AI agent functions to execute · 9.2.3 Chaining tools together · 9.3 Moving forward with AI test assistants · 9.3.1 Examples of AI test assistants · 9.3.2 Handling the challenges of working with agents · 9.4 Summary

What you learn in chapter 9 →

Chapter 10: Introducing customized LLMs

10.1 The challenge with LLMs and context · 10.1.1 Tokens, context windows, and limitations · 10.1.2 Embedding context as a solution · 10.2 Embedding context further into prompts and LLMs · 10.2.1 RAG · 10.2.2 Fine-tuning LLMs · 10.2.3 Comparing the two approaches · 10.2.4 Combining RAG and fine-tuning · 10.3 Summary

What you learn in chapter 10 →

Chapter 11: Contextualizing prompts with retrieval-augmented generation

11.1 Extending prompts with RAG · 11.2 Building a RAG setup · 11.2.1 Building our RAG framework · 11.2.2 Testing our RAG framework · 11.3 Enhancing data storage for RAG · 11.3.1 Working with Vector databases · 11.3.2 Setting up a vector-database-backed RAG · 11.3.3 Testing a Vector-database-backed RAG framework · 11.3.4 Going forward with RAG frameworks · 11.4 Summary

What you learn in chapter 11 →

Chapter 12: Fine-tuning LLMs with business domain knowledge

12.1 Exploring the fine-tuning process · 12.1.1 A map of the fine-tuning process · 12.1.2 Goal setting · 12.2 Executing a fine-tuning session · 12.2.1 Preparing data for training · 12.2.2 Preprocessing and setup · 12.2.3 Working with fine-tuning tools · 12.2.4 Setting off a fine-tuning run · 12.2.5 Testing the results of a fine-tune · 12.2.6 Lessons learned with fine-tuning · 12.3 Summary

What you learn in chapter 12 →

Concepts explained in this book

promptA prompt is an input instruction or context supplied to a language model to guide its generated output.
language modelA large language model is a machine-learning model trained on extensive text data to generate and transform language from an input prompt.
testA test case is a defined set of inputs, conditions, and expected outcomes for evaluating a particular software behavior.
delimiterA delimiter is a character or sequence of characters that marks the boundary between portions of text or data.
fine-tuningFine-tuning is the process of adapting a pretrained model to a task or domain by training it further on targeted data.
contextThe relevant background information, constraints, and requirements provided to guide interpretation of a testing request.
retrieval-augmented generationRetrieval-augmented generation is a generation technique that retrieves relevant external information and supplies it to a language model as context.
hallucinationA hallucination is a false or unsupported output generated by a large language model that is presented in a clear and authoritative manner.
quality characteristicAn aspect of a system's quality used to organize and evaluate testing risks.
few-shot learningFew-shot prompting is a prompting technique that supplies a small number of explicit examples to clarify instructions and guide a generative model's output.
context windowThe bounded amount of tokenized input and related context that a language model can process for a request.
prompt engineeringPrompt engineering is the deliberate design of inputs, instructions, constraints, and examples to influence a large language model toward a desired output.
preprocessingData preprocessing is the preparation and transformation of data into a form suitable for model training or fine-tuning.
test dataTest data is the input and supporting information provided to software under test to exercise specified behaviors and conditions.
riskThe possibility that an action, system, or technology will cause harm, loss, failure, or another undesirable outcome.
data flow diagramA diagram that represents how data moves through processes, components, and storage within a system.
tokenizationTokenization is the process of converting text into discrete tokens that a language model can process as numeric identifiers.
ChatGPTChatGPT is a conversational artificial-intelligence system that generates responses to user prompts using language-model technology.
accuracyAccuracy is a quality characteristic concerning whether software produces correct results and represents information correctly.
area of effect modelA book-specific model for keeping human judgment and control central when selecting and evaluating outputs from large language models.
test data generationTest data generation is the creation of input data sets for exercising software during testing activities.
corpusA corpus is a curated collection of documents or other information assembled for retrieval and analysis.
AI agentAn AI agent is a software system that uses artificial intelligence to pursue goals by interpreting information and taking actions.
tokenizerA tokenizer is a tool that converts text into tokens and commonly maps those tokens to numeric identifiers.
epochAn epoch is one complete iteration through all entries in a training dataset during model training.
consistencyConsistency is a quality characteristic concerning whether software and its data behave uniformly across equivalent inputs, operations, and uses.
AxolotlAxolotl is an open-source tool for configuring and running large-language-model fine-tuning workflows.
time-to-think principleThe time-to-think principle is a prompting practice that instructs a language model to review its proposed solution for completeness or correctness before producing the final output.
user storyA concise requirement statement describing a capability from a user's perspective and the value it provides.
datasetA dataset is a collection of related data items organized for analysis, processing, or use by a system.
templateA prompt template is a reusable prompt structure that combines fixed instructional text with fields populated from input data.
sentiment scoreA sentiment score in fine-tuning is a measure of how closely a model response aligns with the expected output.
refactoringRefactoring is the restructuring of existing source code to improve its internal design without changing its externally observable behavior.
parameterA learned numerical value that contributes to a model's statistical computations.
exploratory testingExploratory testing is an approach in which test learning, design, and execution proceed concurrently.
area of effectA model of the influence produced by the relationship among a person, their abilities, and the tools they use.
heuristicA practical rule or strategy for examining a system from a particular perspective without requiring a complete formal analysis.
risk assessmentThe systematic identification, evaluation, and prioritization of risks affecting a software system or testing activity.
garbage in, garbage outThe principle that poor or insufficient input tends to produce poor or insufficient output from a computational process.
testing storyA narrative testing report that organizes testing activities, observations, and findings into a richer account.
SQLA request for information or matching items from a data collection or information-retrieval system.
Cornell methodThe Cornell method is a note-taking system that organizes information into cues, notes, summaries, and related reflection or action sections.
testing notesTesting notes are informal records of test activities, observations, discoveries, and learning captured during or after testing.
CanopyCanopy is a framework for building retrieval-augmented generation applications with Pinecone's cloud vector database services.
PineconePinecone is a managed cloud platform for creating and operating vector databases and similarity-search indexes.
commentA code comment is an explanatory annotation embedded in source code that is generally ignored during program execution.
synthetic dataSynthetic data is artificially generated data designed to reproduce relevant characteristics of data from a real-world domain.
logical code sectionsLogical code sections are semantically coherent portions of source code partitioned according to program structure rather than arbitrary line boundaries.
semantic treeA semantic tree is a hierarchical representation of source-code constructs and their structural relationships.
mental modelAn internal conceptual representation through which a person understands a system, feature, or situation.
Apache Commons TextA Java library providing utilities for text processing, comparison, escaping, and related operations.
cosine distanceA distance measure derived from the cosine of the angle between vectors, where a lower value indicates greater directional similarity.
LangChainLangChain is a software library and toolset for developing applications that integrate with language models.
training datasetData supplied during model training so that the model can learn patterns or relationships.
test charterA test charter is a concise mission statement that defines the focus and purpose of an exploratory testing session.
formal modelAn explicitly defined model that uses established symbols, rules, or notation to represent a system.
SFDIPOTA mnemonic heuristic for examining a product through Structure, Function, Data, Interfaces, Platform, Operations, and Time perspectives.
Unified Modeling LanguageA standardized modeling language for specifying, visualizing, and documenting the structure and behavior of software systems.
check for assumptionsA prompting technique that instructs a large language model to recognize when required conditions are unmet and provide an explicit fallback instead of guessing.
state managementState management is the control and maintenance of the system conditions required for an operation or test.
UI automationUI automation is the automated interaction with and evaluation of a software system through its user interface.
foundational modelA foundational model is a pretrained machine-learning model that provides a general base for further adaptation or specialization.
test-driven developmentTest-driven development is a software development practice in which failing checks are created before production code, code is added to make them pass, and the code is then refactored while checks remain passing.
@Tool annotationA LangChain4j annotation that marks a method as an invocable tool and supplies natural-language metadata describing its operation.
tool chainingTool chaining is a technique in which an agent passes the result of one tool invocation as input to a subsequent tool invocation.
privacyCustomer privacy is the protection of information and confidentiality associated with people who use or receive a service.
PAOLOPAOLO is a mnemonic for generating test ideas about portrait orientation, audio, objects, landscape orientation, and overlays.
sentiment analysisSentiment analysis is the computational classification or scoring of text according to expressed sentiment or evaluative polarity.
shift leftShift-left testing is the practice of performing testing activities earlier in the software development lifecycle.
JSONJSON is a text-based data-interchange standard representing structured data with objects, arrays, values, and name-value pairs.
release notesA release note is a user-facing summary of changes, behavior, or other information about a software release.
parameter annotationA LangChain4j annotation that supplies natural-language context for a tool method parameter so a language model can extract its value from a prompt.
relevance checkingThe evaluation of whether retrieved information matches the subject and intent of a query.
HTTP payload contractA specification defining the structure, fields, types, and validity requirements of data carried in an HTTP request payload.
OpenAPI specificationOpenAPI is a standard for describing HTTP APIs and their operations, data structures, and constraints in a machine-readable format.
xml schemaXSD is a W3C standard for defining the structure, data types, and validation rules of XML documents.
controlControl is the degree of influence and insight available over a language-model process, its behavior, its deployment, and access to its outputs.
GeminiGemini is a family of generative artificial-intelligence models and conversational services.
GPT-4oGPT-4o is a multimodal generative model in OpenAI's GPT-4 family.
Hugging FaceHugging Face is a platform and community hub for sharing, accessing, training, and documenting machine-learning models and datasets.
LlamaLlama-2 is an open-source family of large language models released by Meta that can be downloaded and further trained.
test planningTest planning is the systematic preparation of how software testing will be conducted, including its scope, approach, and documentation.
OpenAI APIA web interface that provides programmatic access to OpenAI's artificial-intelligence models and services.
@FindBy annotationA Selenium page-object annotation that specifies how a web element is located.
PageFactoryA Selenium support library component that initializes and manages page-object fields annotated with element-location metadata.
testabilityThe degree to which a product's behavior and internal or external interfaces support effective testing.
xmlXML is a markup language and data-interchange standard for representing structured information with nested elements and attributes.
LangChain4jLangChain4j is a Java library for building applications that integrate with large language models.
automated UI checkAn automated UI check is a programmatic test that interacts with a user interface and verifies expected behavior or state.
test scriptA predefined sequence of instructions for executing a software test and checking its expected outcome.
vector databaseA vector database is a database specialized for storing and retrieving numerical vector representations using similarity-based search.
interoperabilityInteroperability is a quality characteristic concerning a system's ability to exchange and use information or work correctly with other systems.
securityA quality characteristic concerning protection against unauthorized access, misuse, alteration, disclosure, or loss.
JSON objectA JSON object is a data structure consisting of name-value pairs enclosed in braces.
format tacticA prompting technique that specifies the required representation or serialization format of a model's output.
time-to-think tacticA prompting technique that asks a generative model to check or reason about an output before producing it.
probability theoryProbability is a mathematical measure of the likelihood that an event or outcome occurs.
probability distributionA mathematical representation assigning probabilities to the possible values or outcomes of a variable.
reinforcement learning with human feedbackA model-alignment technique that uses human feedback to guide or refine a model's behavior.
unsupervised learningA machine-learning approach that trains a model on unlabeled data to discover implicit patterns or structure.
extensibilityThe capacity of a system to be expanded with additional capabilities or adapted to new requirements.
LLM platformA software platform that provides access to large language models and related capabilities.
qualityThe degree to which a language-model response is legible, useful, accurate, and appropriate for its intended purpose.
parameter countThe number of learned statistical parameters, or weights, in a machine-learning model.
explore with using templateA test-charter formulation template consisting of an exploration objective, its data or conditions, and the purpose of discovering a result.
autonomyThe capacity of a software system to select and perform actions toward a goal without an explicitly ordered sequence being prescribed for every action.
data transformationThe process of converting data from one structure, representation, or language into another while preserving its intended information.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.