Chapter 6. Unit testing continuously integrated code
This chapter covers
- Unit testing in a CI environment
- Continuously examining test coverage
- Test mocking
We’ll risk the opinion that without automated testing, CI would be obsolete, because CI’s main strength is that it shows how the changes you introduce into the code affect the software. The CI process should be designed to show you immediately when a change degrades the code quality. What better way to check for that kind of occurrence than to perform automated testing along with every source code change?
Automated software testing is a broad term. In this chapter, we’ll focus on one particular type of automated testing: unit testing. Unit testing lies somewhere toward the bottom of the common automated-test chain. We’ll get to the rest of the chain—integration, system, and acceptance testing—in chapter 7. But in this chapter, we’ll define what unit tests are and what purpose they serve in the CI process. We’ll take two popular testing frameworks, NUnit and Microsoft Unit Testing Framework (MSTest), and incorporate them into the CI process. Then you’ll learn how to mock things out to speed up your tests.
Before we jump in to create some unit tests, let’s define the term and look at the aspects that are important from the CI perspective. There’s a common misunderstanding about unit tests. They’re often associated with automated tests in general, but this assumption is incorrect. Unit tests are part of automated tests. Figure 6.1 shows the difference. As you can see, there’s a lot more to automated testing than just unit tests. (As we mentioned earlier, you’ll have to wait until chapter 7 to see the rest.) For now, you’ll search for the smallest testable part of the application you’re building: a unit. Depending on your point of view, it may be a class or a separate method. Unit tests verify this smallest testable part of your application.
A well-designed unit test works with a fully isolated piece of code. It should test the smallest part of your software without dependencies on other classes or external resources. Sometimes unit tests are written by the software developer even before the actual code. They’re perfect material for application verification in the CI process. Let’s look at how to use them.
When you’re designing unit tests for the CI process, you have to keep some simple rules in mind. The most important in the CI context are as follows:
- Make your unit tests fast and unambiguous.
- Have your unit tests rely on as few dependencies as possible.
- Let errors drive your unit tests.
Unit test should be fast. One unit test should run in a fraction of a second with no delays and no time-expensive operations. Each small piece of code should be tested in complete isolation, meaning the test shouldn’t have any access to external resources. It shouldn’t write to the hard drive, nor should it require network connections. If you have code that does that, you’ll have to mock it as part of the test. (We’ll discuss mocks later in this chapter.)
To illustrate the suspicious-looking rule “Let errors drive your unit tests,” we’ll revise a Samuel Beckett saying, “Ever tried. Ever failed. No matter. Fail again. Fail better,” and say “Ever tried. Ever failed. No matter. Try again. Fail no more.” We strongly believe that when it comes to unit tests, you shouldn’t fail more than once. This means there’s no excuse for not writing a test for every bug. You should be doing error/defect-driven development. Every time someone finds a bug in your code, you write a test for it, fix it, and let the test work from then on. It’ll function as a regression test in the future. You’ll be sure that particular bug is fixed for good, and your build will never again fail because of that bug.
Let’s jump right in, take the financial library you’ve worked with in earlier chapters, and create a simple unit test for it using NUnit and MSTest. You’ll integrate the tests with the CI servers. To demonstrate the mocking of functionality in unit tests, you’ll extend the financial calculator to perform some I/O operations that you can mock. But before we get into mocking, you need some code to test, and you need to write some tests for the code.
In chapter 1, we introduced a small application that’ll stay with you through your journey with CI. It’s your friend the leasing/credit calculator. It can calculate the credit rate according to several input variables such as contract duration and interest. But before we dive into some mathematical details of finance, let’s change the calculator a little by flattening the structure. For better clarity, you’ll keep all the projects in your solution at one level. If you like the structure with external SVN references, feel free to keep the project that way; but here you’ll modify it. From now on, you’ll have one solution named CiDotNet with some projects inside, including the calculation core in a project named CiDotNet.Calc (it contains basically what the Framework external SVN reference repository had). The Windows calculator is in the project CiDotNet.WinCalc, the web calculator is in CiDotNet.WebCalc, and the Silverlight calculator is in CiDotNet.SilverlightCalc. The sources provided with this book include a ready-to-use project.
Let’s start with the calculation core and its mathematical details. This information isn’t necessary from the CI point of view, but it’s important to fully understand the unit tests that will follow. If you’re a unit testing specialist, please feel free to skip the next section.
It’s time to add code to the project so you have something to unit test. Open the class library project CiDotNet.Calc and add a new class named FinanceHelper, as shown next.
This code seems to include lots of cryptic methods and values, but it’s much easier to understand than you can tell at first glance. Single Payment Present Value (SPPV) is the present value of money received in the future at a given interest rate. Single Payment Future Value (SPFV) is the future value of money paid in the future at a given interest rate. Uniform Series Present Value (USPV) is the payment required each period to achieve the future value. And Uniform Series Future Value (USFV) is the future value of a uniform payment.
All of these values are used in the last and most important calculation: the monthly payment, with the public method CalculateRate(). It takes as parameters all the necessary data to make the calculation, as shown in table 6.1.
Table 6.1. Parameters to the CalculateRate() method
| Parameter |
Description |
|---|---|
| periods | Number of periods you want to carry the burden of the loan |
| ppy | Periods per year—for example, 12 for monthly payments |
| interest | How much the bank charges you (the interest rate) |
| presentValue | How much money you need right now |
| finalValue | How much money you need at the end of the loan |
| mode | Whether the bank calculates interest income at the beginning of the calculation period or at the end |
The CalculateRate() method uses the periodic interest rate (annual interest divided over the number of periods in a year) and compound period rate (payments in a month). You may consider the decimal data type for use with money-related calculations. You may even want to use your own Money type. We won’t deal with these issues, to make the case simpler. After all, we’re chasing the perfect CI process and not financial issues. This small financial library is a perfect fit for the first test case. You’ll test it with NUnit.
NUnit (www.nunit.com) is a legend in the unit testing world of .NET. It’s one of the oldest automated testing frameworks for .NET and was originally a clone of the Java test library JUnit. NUnit has the responsibility of running unit tests and providing feedback about which tests pass and which ones fail. You’ll see that NUnit is easy to use. The easiest way to install it is to download the zip file and extract the core of the testing framework from NUnit-Version\bin\net-2.0\framework into your tools directory.
Next, you have to decide where to put the code you write for the unit test. There are two possible locations for your unit test code: together with the code you’re about to test, or in another project. Both approaches have their plusses and minuses. Putting all the code together lets you test the private members, but creates a dependency on the unit testing framework. We prefer using separate library classes for the sake of cleanly separating test and production code. This way, you can easily drop the test DLLs while building the release on the CI server. For this example, you’ll go this way.
It’s a good idea to use a pattern for the test projects’ names. We like to name them after the project they’re testing and then add the suffix .Test. For the example, this yields the name CiDotNet.Calc.Test.
Further, the test should correspond with the structure of the production code. The same folder structure and of course one test fixture per class is a good way to go. We encourage you to give this some thought; there’s no one best pattern for the unit test infrastructure; something else may work better for you. But keep in mind that your test suite will eventually grow to hundreds or thousands of test cases.
Now you need to create the actual unit test. Add a new class library project to your solution, and name it CiDotNet.Calc.Test. Add a reference to the CiDotNet.Calc project and then to the nunit.framework.dll. The Finance.cs class lies in the Math subdirectory of the production project, so create a FinanceTestFixture.cs file in the Math directory of the test project. Add the following code to this new class.
NUnit uses reflection to dig the test methods from the test library. It uses attributes to find what it needs. First, you need to decorate the test class with the [TestFixture] attribute, which tells NUnit that it’s found a class containing tests. All the test methods must be public voids and have the [Test] attribute. In the test code, you can do everything that’s possible in .NET. In the CalculateRate() method, you name the calculation parameters in local variables and fill them with values. You then define the ExpectedRate variable and assign it the value that you expect to be returned from the calculation. The ActualRate variable will be set with the actual calculation value from the Finance library.
A test needs something that tells it whether it was a success or a failure. This is called an assertion. The Assert.AreEqual method is part of the NUnit framework. It compares the ExpectedRate to the ActualRate. If they’re equal, the test passes. If not, the test fails.
You can execute the test a few ways. One of them is to use the GUI test runner that comes with NUnit, nunit-x86.exe. This is a program that lets you interactively run your tests and gives immediate feedback on the results. You’ll find it in the NUnitVersion\bin\net-2.0 folder in the NUnit zip archive. Because the CI unit test process needs to run with no user interaction, you won’t need it on the CI server. But you’ll use it now to demonstrate NUnit’s testing capabilities. The source code included with this book contains more unit tests for you to browse and learn from.
Launch the NUnit GUI test runner (see figure 6.2). Select File > Open Project, search for CiDotNet.Calc.Test.dll, and open it. NUnit will load the DLL and prepare everything for the tests.
Figure 6.2. The CiDotNet.Calc.Test assembly is loaded into the NUnit GUI test runner and ready to execute.
The left pane shows the assembly and the test methods you’ve written. You can run all the tests together or mark separate tests to execute them independently. To start the test, click Run. The tests will run, and the results will be displayed in the NUnit GUI (see figure 6.3).
Figure 6.3. If all the tests pass, you see a green bar in the right pane. The left pane shows a check mark inside a green circle next to each passing test.
Green means the tests passed and everything is all right. What you don’t want to see is red, which means the tests failed; or yellow, which indicates that at least one test wasn’t run.
Let’s make test results more colorful by creating one failing and one omitted test. In doing so, you’ll learn some other NUnit attributes. Copy the CalculateRate() test, paste it into the same class, and change the name to IgnoreTest(). If you decorate it with the [Ignore] attribute (in addition to the [Test] attribute), NUnit skips execution and shows a yellow result when you run the test.
Copy the test again, change the name to FailOnPurpose(), and decorate it with the [ExpectException] attribute. This informs the NUnit framework that you expect the tested code to cause an exception. Run the test DLL in the GUI test runner, and you’ll get the colorful output shown in figure 6.4.
Figure 6.4. If a single test fails, you get a red result in the right pane. The left pane shows an X inside a red circle for failing tests, which bubble up all the way to the top-level assembly. One test, CalculateRate(), passed. The IgnoreTest() method didn’t run, so it displays a question mark inside a yellow circle.
All the GUI tests are of course useless in the CI environment. The CI server isn’t as smart as you are and can’t use GUI tools. You need something that you’ll be able to start from a build script—something that will perform the tests and save the output in a text file. To do this, you can use a command-line test runner. You’ll hook it up to CruiseControl.NET (CCNet).
If you want to integrate the unit tests with your CI server, you’ll use a command-line tool and script the process in the build script. NUnit comes with a suitable command-line test runner. Add the nunit-console.exe file and all the dependencies (nunit-console-runner. dll, nunit-console.exe, nunit.core.dll, nunit.core.interfaces.dll, nunit.framework.dll, and nunit.util.dll) from the NUnit zip file that you downloaded earlier to the tools directory. To execute the tests you created earlier, issue the following command:
The console test runner will perform all the tests, as shown in figure 6.5.
In chapter 3, you chose MSBuild as your build engine of choice. You need NUnit to run from within the MSBuild script. There’s an MSBuild Community Task (see chapter 3) to run NUnit that you can use, but you’ll now execute it using the exec task as follows.
As you can see, you start the Test target that uses the exec task to execute the nunitconsole.exe application, providing it with the property that contains the DLL to test. The /xml parameter tells NUnit to create an XML report file. You’ll use this file on the CI server to integrate the test results within the feedback mechanism.
The build script contains the target named Build, which compiles and rebuilds the whole solution. You can use it directly as a build script on the CI server. If your project resides in a revision-control system (we described how to put it there in chapter 2) and you’re still using the CruiseControl.NET configuration from chapter 3, then you’re good to go. Update the MSBuild script according to listing 6.3, check everything in, and your CI process will snap in and perform the build followed by the test.
Let’s quickly glance at the Web Dashboard to see if everything works correctly (see figure 6.6).
Figure 6.6. CCNet Web Dashboard with a failing project. A red bar under the project is a quick indicator that something’s wrong.
You can see a lot on the CCNet Dashboard page. It integrates easily with various test tools. The test tool must be able to produce XML-formatted output; CCNet applies an XSL transformer to the report to show it on the Dashboard. NUnit can produce XML output. Running NUnit as in listing 6.3 produces a test report called NUnitReport.xml. On the CCNet server, this file needs to be integrated with the overall build report on the Dashboard page. To do so, you’ll have to modify the ccnet.config file and the definition for the CiDotNet project by changing the publishers tag:
Don’t forget to enable NUnit in the CCNet Dashboard Administrator function and include the xmllogger tag in the publishers tag. It includes the CCNet logs to the Dashboard page.
The XSL files we’ve talked about are defined in the dashboard.config file discussed in chapter 5. It’s usually located in C:\Program Files\CruiseControl.NET\webdashboard and contains a buildPlugins section. This section controls the build-level Dashboard page. To show the NUnit report formatted properly, it should contain this line:
The NUnit XSL transformer file is provided with CCNet. Similarly, there’s an XSL transformer for NUnit timings. It consumes the same XML report file to display different data.
If you apply the scenario we’ve just described, you’ll get an NUnit report like that shown in figure 6.7.
Figure 6.7. An NUnit report transformed from an XML file into a nice web page using an XSL stylesheet, and displayed on the CCNet Dashboard
We’ll deal with test analysis and code metrics in chapter 8. But one interesting code metric comes with unit testing: test coverage. Let’s look at that next.
Test coverage is the percentage of your code covered by tests. In this case, it’s the unit test. What does “code covered by tests” mean? It’s the measurement of how many lines of code are executed by the test code. Some teams strive to cover 100% of their source code lines with tests. Some teams settle for 50%. Covering all the code can be difficult and time consuming; in many cases, a number around 80% is about right.
The mother of all test coverage tools in .NET world used to be NCover. But it went commercial and costs about $200 in its classic version. If you want to do test coverage on the cheap and don’t mind a little manual work, a great open source alternative is available: PartCover (http://sourceforge.net/projects/partcover/). After installation, as usual, copy the necessary files to the project tools directory. All it takes to run the test coverage with PartCover is starting its command-line tool with the NUnit runner and some test assemblies, like this:
This code snippet is a part of an MSBuild script that checks the coverage on the calculator mathematical library, including only the namespace CiDotNet.Calc and excluding CiDotNet.Calc.Test. The output will be saved in the PartCoverReport.xml file. You can call this target in the DefaultTargets of your MSBuild project.
The integration of the report file with CCNet works as usual. You have to use an XSLT file on the XML output and integrate it with the CCNet Web Dashboard. PartCover comes with some XSLT files. Unfortunately, the files currently must be edited to work with CCNet, because the report file is integrated with the overall build-process report and extracted from there. The original files assume they’re working with separate files. We won’t discuss the required changes here; we hope the next version of PartCover comes with dedicated XSLT files. To make life easier for you, we’ve provided the corrected files with this book.
Copy the XSLT files to the xsl folder of your Dashboard installation. Go to dashboard.config, and extend the buildPlugins tag as follows.
Don’t forget to merge PartCoverReport.xml with the build log in the CCNet project configuration file ccnet.config.
You’re finished. Get it up and running, run the build, and you’ll see a report page similar to figure 6.8.
Figure 6.8. The PartCover report in the CCNet Web Dashboard. It shows the assembly test coverage and the coverage divided into separate classes. It’s easy to get to 100% coverage with so small a project, but you should try it with one of your own projects.
TeamCity comes with built-in functionality for NCover and PartCover. To use PartCover, you have to set it up on the Build Configuration page. First, enable NUnit Tests (mark the flag in New Unit Test Settings). Set it to run the tests from %system.team-city.build. workingDir%\CiDotNet.Calc.Test\bin\Release\CiDotNet.Calc.Test.dll.Go to the .NET Coverage section, choose PartCover from the drop-down list, and provide the path to the executables in the lib directory (%system.teamcity.build.workingDir%\lib\PartCover\PartCover.exe). In the Report XSLT test box, provide the following transformation:
That’s it. The next time your project builds, you’ll get a nice report about unit tests and test coverage.
NUnit was a big success in the .NET world, so big that Microsoft hired one of NUnit’s creators and developed its own unit testing framework.
Since Visual Studio 2003, Microsoft has had its own automated unit testing framework, commonly called MSTest. You’ll find it hidden in the Microsoft.VisualStudio.TestTools.UnitTesting namespace. The tools are fully integrated with Visual Studio and are available in all Visual Studio 2010 versions except the Express editions. It’s time to try it and see how it works.
Let’s take the same financial mathematical library you used with NUnit and create unit tests the Microsoft way. Open the Finance.cs file in Visual Studio, right-click somewhere in the text editor, and choose Create Unit Tests from the context menu (see figure 6.9).
Visual Studio browses through the code and finds all the methods worth creating unit tests for. If you don’t want to create tests for all the methods in your class, you have to choose the ones you want from the Create Unit Tests dialog box (see figure 6.10).
If you choose to create the test in a new project (a wise decision), you need to name it in the next dialog box. In this case, call it CiDotNet.Calc.MSTest; a naming convention will turn out to be important in a minute. In the newly created project, you’ll find a new class named after the class it will be testing, but with a Test suffix. The test method for the CalculateRate() method is shown next.
As you can see, the test method is given a [TestMethod()] 
attribute, and the actual test method looks similar to the NUnit test you wrote in the previous section. Now you need to get rid of the TODO comments and set all the variables. Don’t forget to erase the line Assert.Inconclusive("Verify the correctness of this test method."); 
. Even if your assertion passes, this line will make your test yellow.
To start the test, click the Run Tests in Current Context 
button on the toolbar, or choose Test > Run > Test in Current Context from the Visual Studio menu. The test runs directly in the Visual Studio GUI, and the results appear in the Test Results window (see figure 6.11).
Figure 6.11. MSTest results (bottom pane) for the test code (top pane). As you can see, the test passed.
Creating unit tests with MSTest is as easy as it is with NUnit. You can turn on test coverage for MSTest, too. To do so, you have to open the Local.testsettings file in Visual Studio (it’s with the solution items); see figure 6.12.
Figure 6.12. Turning on test coverage for MSTest. In the test settings, you have to enable Code Coverage for a given assembly (you can get to the configuration settings by double-clicking the Code Coverage row).
After enabling the test coverage, you have to decide what assemblies should be instrumented. You can do this by clicking the Configure button shown in figure 6.12. If for some reason you decide to strongly sign the assembly that contains the tests, you must enable re-signing. To do so, choose the re-signing key at the bottom of the configuration window.
We encourage you to further explore MSTest. One good resource is the Microsoft Press book Software Testing with Visual Studio 2010. But right now, let’s see how to add the MSTest test to your continuous build process on Team Foundation Server (TFS) 2010.
Integrating tests with TFS 2010 is easy. If the source of your project is already under TFS Version Control, as we described in chapter 2, and you followed the project naming convention with the Test suffix, you’re almost done. You check in your new project, and TFS will do the test work for you. Let’s examine why TFS does this.
In Team Explorer in Visual Studio, select your build. Right-click it, and from the context menu, choose Edit Build Definition. In the resulting dialog box, click the Process tab; you’ll see a definition like that shown in figure 6.13.
Figure 6.13. The build definition with the Automated Tests assembly matching pattern in the Basic area. All assemblies containing the word test search for automated tests.
In the Basic area of the Process tab is an Automated Tests property. This property tells TFS the assemblies from which to run the automated tests.
By default, the definition is “Run tests in assemblies matching **\*test*.dll using settings from $/MyFirstProject/CiDotNet.Ch5/Local.testsettings.” If you followed the pattern and named the test project to contain the Test suffix, your tests will be performed. Your continuous build should execute all the tests from your test library because it matches the pattern **\*test*.dll.
To see the build results from the context menu of your build definition, choose View Builds from the Builds folder of your team project in Team Explorer, and open the last-performed build (see figure 6.14). At the bottom of the report is the executed test count. Click it, and you’ll see the detailed test report.
Integrating MSTest with other CI servers isn’t as straightforward as with its natural habitat, TFS. Let’s see how can you do so with CCNet and TeamCity.
Sometimes you need to go against the flow and integrate MSTest with a third-party CI server. You have to go against the flow because, unfortunately, MSTest isn’t a framework—it’s part of Visual Studio and the TFS environment. This means you can’t take only the MSTest executables and run them friction-free on the build server. You have to install Visual Studio on the build server, do a lot of manual hacking to make it work without Visual Studio, or use third-party test runners for MSTest. By installing Visual Studio on the build machine, you’re going against the rule of a vanilla server that has as few external dependencies as possible. It also means you must purchase an additional license for Visual Studio, which increases your costs.
If you want to go the hacker way and make MSTest run without Visual Studio, you can use an external tool such as Gallio.
We’ll assume that you got your tests running on the build machine one way or another, and show you how to integrate MSTest with CCNet and TeamCity. As usual, you begin by extending the MSBuild script, as shown here.
All you do is use the command line to run the MSTest runner installed on the build agent to produce an MSTestReport.trx file. The .trx file is nothing more than an XML file. To add the test results to CCNet, you apply an XSLT file and configure the Dashboard to show the results. You already know the drill, so we won’t discuss it here. More interesting is the integration with TeamCity, which gives us the opportunity to discuss TeamCity service messages.
Service messages in TeamCity are commands you can pass from the build script to TeamCity itself. For example, you can tell TeamCity that you have a .trx file with an MSTest report for it to transform and show on the build page. You send a service message to TeamCity by outputting it to the standard output in the build script. If you’re using MSBuild, you use the simple line
TeamCity can interpret this message and has the built-in ability to process the MSBuild report files. So if it gets an importData command of type mstest, it searches a given path for the report to transform and then displays the results. Make sure you add the .trx file to the project artifacts. Add the previous message line to your build script, and you’ll get a TeamCity Test Details page as shown in figure 6.15.
Figure 6.15. The standard TeamCity Test Details page showing the results of the MSTest run on the build server
As you can see, integrating unit tests with the CI server is a straightforward task. It should be. Getting the unit tests to run every time the project is integrated is essential for a good CI process. Now it’s time to wrap up our discussion of unit testing.
We’ve covered a lot of ground in this chapter. Congratulations for making it all the way through! Writing good unit tests is an art of its own, and we’ve merely glossed over the surface. If you want to go deeper and master unit testing, look at Roy Osherove‘s great book, The Art of Unit Testing (Manning, 2009).
You’ve written a simple unit test and seen how to isolate the tests from external resources. But most important, you know why you need unit tests in the CI process and how to incorporate them into your CI server of choice.
Unit tests are the fastest way to ensure that the code you’re producing maintains some given level of excellence. Writing tests and running them inside your CI process lets you discover every fault as quickly as possible. Unit tests must run quickly—running a single unit test should take a fraction of a second. Remember that the entire test run, together with other CI activities, shouldn’t last longer than 5 to 10 minutes.
Together with unit tests comes a very useful software metric called test coverage. You saw how test coverage can show you how much of your code is tested. The more code you test, the higher the code quality.
Depending on your CI server, you can incorporate unit tests into the build script as you’ve done using MSBuild and CruiseControl.NET, or use built-in features of the CI server, like the CI setup in TFS 2010. We tend to believe that controlling everything in the build script is a better way to do it. We like to have total control over the build process, and MSBuild lets us do so. Of course, TFS relies on MSBuild under the hood, and you can take control; but it isn’t as obvious as using the build script from the beginning.
The general output from the suite of unit tests is always binary. All the tests pass, or the entire CI process is broken. This information, although important, is only part of the story. You can run many other tests—integration, acceptance, and system tests, for example—to check whether your code is operating properly. Those tests are the subject of the next chapter.