Chapter 9. Automating acceptance criteria for non-UI requirements

Although they have their uses, web tests shouldn’t be the only tool in your automated acceptance testing toolbox. It’s important to know when to use them and when to look for alternative strategies. In this chapter, you’ll learn about other ways to automate your acceptance tests that don’t involve exercising the user interface (see figure 9.1).

9.1. Balancing UI and non-UI acceptance tests

The ideal balance between UI and non-UI tests will naturally vary from project to project, depending on the nature of the application being built, the features being developed, and the technologies used. For some requirements, web or UI tests will be a natural fit, but for others, non-UI-based testing is more appropriate. Still other tests may need a mixture of both approaches.

In this sort of application, automated web tests will typically be used to illustrate and verify the user’s journey through the application, to ensure that data is submitted to the server correctly, to check that form validation messages are displayed correctly, to see that results from the server are rendered accurately, and so on. The web tests also act as end-to-end tests, verifying the flow of information through the whole system. If the application is designed well, the business logic will be localized in the form of a well-defined service API that can be tested effectively using non-web tests.

9.2. When to use non-UI acceptance tests

However, because these acceptance criteria focus on the user experience and high-level outcomes, they deliberately gloss over some important underlying business logic. In particular, how do you determine if Joe is eligible or not? And depending on the scope of the feature, you may also need to explore other questions. How will the feature attract new Frequent Flyer members, and how will it earn revenue for the organization? This may involve some complex business rules. If you were to write automated web tests for each of these rules, it would slow down and add complexity to the test suite without adding a great deal of reporting value.

In fact, although you’ll need to have UI-based acceptance criteria that illustrate how users will interact with the application, and to demonstrate and verify any business logic that’s implemented within the UI layer itself, acceptance criteria that relate to business rules implemented on the server may not need to exercise the UI at all. In the rest of this chapter we’ll discuss ways to effectively automate acceptance criteria like these without the need for automated UI tests.

9.3. Types of non-UI automated acceptance tests

As you’ve seen, UI tests have their place when it comes to testing and illustrating user interactions with the system and the user experience as a whole, but it’s often inefficient to use them for more detailed business rules. There are a number of strategies that can be used to bypass heavyweight UI tests when automating your acceptance criteria, and many tools can be used to implement these approaches. Some of the more commonly used approaches include

Testing services remotely, such as by invoking web services and thus avoiding the UI layer entirely

9.4. Defining and testing nonfunctional requirements

Once you’ve expressed the performance requirements in these terms, you can automate. There are many open source and commercial load-testing tools, and most can be scripted. Popular open source options in the Java world include SoapUI (www.soapui.org), JMeter (http://jmeter.apache.org/), and The Grinder (http://grinder.sourceforge.net). These tools let you define and execute test scripts that simulate interactions with an application, such as requesting a web page, invoking a web service, and querying an EJB. To simulate load, they can run scripts on a number of remote machines as well as locally. They’re also relatively easy to automate and to run programmatically. This makes them easy to integrate into tools such as Cucumber and JBehave.

When Tom Howard needed to verify the performance of an application built around a messaging platform for a large electricity company, he chose to use a BDD approach.^[5] To do this, he used The Grinder and Cucumber-JVM to automate a BDD performance scenario similar to the one discussed here. The main Cucumber step invoked a Grinder script directly and waited for it to finish. The Grinder script ran a series of requests based on real production data from a number of worker machines to simulate production-like loads.

9.5. Discovering the design

BDD is an excellent way to design clean, reusable service APIs, as well as clean, reusable APIs at all levels of the application. Well-designed applications are typically organized into layers and components (in design patterns, this principle is often referred to as the separation of concerns). If you think of an application in terms of these layers and components, starting from the UI and working down, each layer is the “user” of the next layer down, and any code that uses a component is effectively a user of that component. When the user is application code (or, more precisely, the developer writing the application code) rather than a physical person, you’re effectively writing an API. In fact, any code that you write can be considered an API for someone else, even if that someone is yourself.

This process doesn’t stop with the step implementation. When you write the production code, you can do exactly the same thing, discovering what services you need from other components (which may not exist yet), and writing the code you’d like to have. This act of imagining the code that would serve you best is a powerful design practice: what information do you need to get the job done? What services do you need to call? Do they exist, or do they need to be created? What would be the most convenient way to obtain the data you need, and in what form? When you write this imaginary code, you’re providing an example of how you’d like to use an API that doesn’t yet exist. Because you haven’t written any implementation code yet, this example will not be polluted by preconceived ideas about what the technical solution should look like; rather, it will be driven by what would be easiest to use from the perspective of the developer using the API.

9.6. Summary

In this chapter you learned about different sorts of non-UI acceptance testing, including

In the next chapter, you’ll learn how BDD practices are also applicable at the coding level, and how this sort of lower-level BDD relates to Test-Driven Development and more traditional unit-testing practices.