Erik Dietrich's blog series on unit testing is my favourite introduction to test-driven development (TDD). It is concise, enjoyable to read, and unlike a lot of introductions to TDD that only describe how to write tests, his series focuses on practical considerations like naming tests, test structure, build integration, and making the business case for writing tests.

One of the things he encourages is that a test should test only one thing:

When you’re running a bunch of unit tests, you’re generally going to see their result in a unit test runner grid that looks something like a spreadsheet. Or perhaps you’ll see it in a report. If when you’re looking at that, you see a failure next to IsPrime_Returns_False_For_12 then you immediately know, at a glance, that something went wrong for the case of 12. If, instead, you see a failure for Test_A_Bunch_Of_Primes, you have no idea what happened without further investigation. Another problem with the looping approach is that you’re masking potential failures. In the method above, what information do you get if the method is wrong for both 2 and 17? Well, you just know that it failed for something. So you step through in the debugger, see that it failed for 2, fix that, and move on. But then you wind up right back there because there were actually two failures, though only one was being reported.

I generally test one thing per test method and, for the majority of the tests that I write, this approach is perfectly natural.

Eric goes on to describe how, for a method that generates prime numbers, for example, it is unnecessary to test for every possible prime number; it is sufficient to test for a few prime numbers along with the boundary cases (e.g., the largest and smallest prime number, depending on the data type). There are some cases, however, particularly when writing functional tests (e.g., testing something for every data type) or security tests (e.g., whitelist or blacklist testing), where it is important to enumerate all possible cases in order to prevent regressions and ensure uniform test coverage. If there is a large set of test cases, although desirable, it can become difficult, and even error prone, to maintain one test method per test case.

When there are enough combinations that writing a test method for each test case becomes burdensome, the approach often taken, as Erik alluded to, is to define one test method that iterates over all possible test cases. This avoids the tedious nature of writing many, nearly identical tests, and the combinations are enumerated programmatically, so there is no chance of accidentally omitting a test case. Also, if the test is iterating over a collection, for example, and the collection is later expanded to include more elements, there is no danger of forgetting to add the corresponding test cases.

With this approach, inevitably there is a debate between 1) failing the test on the first failure or 2) running all of the tests with the hope that the pattern of test case failures might help quickly resolve the bug or regression. In my experience, most people tend to prefer the second approach since it provides a more complete picture. With the second approach, however, the test must resort to logging messages for each test case and then assert failure once at the end of the test, if one or more test cases failed. Determining what happens if the test fails involves wading through a log file. Or, alternatively, to print out a summary at the end of the test, we must complicate the test by keeping track of the outcome of each individual test case.

Generating Tests Programatically

There are times when I want the simplicity of having one test method for each test case, but also have the benefits that come from generating the tests programmatically, to avoid the tedious nature of writing these tests by hand, and the danger of missing test cases. Visual Studio T4 Text Templates can be used to generate test methods in this situation. T4 Text Templates are WYSISYG text mixed with C# or Visual Basic control blocks that get executed at compile time. They are used to generate text files programmatically. Since the templates simply generate a text file, they could be used to generate tests or scripts for pretty much any test framework, but they integrate best with writing and running tests in the programming languages supported by Visual Studio.

Consider these two enumerations.

public enum Colour
{
    Red,
    Orange,
    Yellow,
    Green,
    Blue,
    Violet
}

public enum Shape
{
    Circle,
    Square
}

The following template file, ColouredShapesTests.tt, uses C# control blocks to iterate over the Colour and Shape enumerations to generate C# test methods that test drawing all combinations of the coloured shapes.

<#@ template debug="false" hostspecific="false" language="C#" #>
<#@ assembly name="System.Core" #>
<#@ assembly name="$(TargetPath)" #>
<#@ assembly name="$(SolutionDir)packages\\Drawing\\Drawing.dll" #>
<#@ import namespace="System.Linq" #>
<#@ import namespace="System.Text" #>
<#@ import namespace="System.Collections.Generic" #>
<#@ import namespace="Drawing" #>
<#@ output extension=".Generated.cs" #>
using Drawing;
using Microsoft.VisualStudio.TestTools.UnitTesting;

namespace DrawingTests
{
    [TestClass]
    public partial class ColouredShapesTests
    {<#
foreach(var colour in System.Enum.GetNames(typeof(Colour)))
{
    foreach(var shape in System.Enum.GetNames(typeof(Shape)))
    {#>

        [TestMethod]
        public void Draw<#= string.Format("{0}{1}", colour, shape) #>()
        {
            DrawingTest(<#= string.Format("Colour.{0}", colour) #>, <#= 
                string.Format("Shape.{0}", shape) #>);
        }
<#
    }
}#>
    }
}

The <# control blocks contain C# statements for flow control and the <#= control blocks contain C# statements that output strings that will be written to the output file. This template will generate a file called ColouredShapesTests.Generated.cs that contains a partial class with all of the test methods.

using Drawing;
using Microsoft.VisualStudio.TestTools.UnitTesting;

namespace DrawingTests
{
    [TestClass]
    public partial class ColouredShapesTests
    {
        [TestMethod]
        public void DrawRedCircle()
        {
            DrawingTest(Colour.Red, Shape.Circle);
        }

        [TestMethod]
        public void DrawRedSquare()
        {
            DrawingTest(Colour.Red, Shape.Square);
        }

        [TestMethod]
        public void DrawOrangeCircle()
        {
            DrawingTest(Colour.Orange, Shape.Circle);
        }

        [TestMethod]
        public void DrawOrangeSquare()
        {
            DrawingTest(Colour.Orange, Shape.Square);
        }

        [TestMethod]
        public void DrawYellowCircle()
        {
            DrawingTest(Colour.Yellow, Shape.Circle);
        }

        [TestMethod]
        public void DrawYellowSquare()
        {
            DrawingTest(Colour.Yellow, Shape.Square);
        }

        [TestMethod]
        public void DrawGreenCircle()
        {
            DrawingTest(Colour.Green, Shape.Circle);
        }

        [TestMethod]
        public void DrawGreenSquare()
        {
            DrawingTest(Colour.Green, Shape.Square);
        }

        [TestMethod]
        public void DrawBlueCircle()
        {
            DrawingTest(Colour.Blue, Shape.Circle);
        }

        [TestMethod]
        public void DrawBlueSquare()
        {
            DrawingTest(Colour.Blue, Shape.Square);
        }

        [TestMethod]
        public void DrawVioletCircle()
        {
            DrawingTest(Colour.Violet, Shape.Circle);
        }

        [TestMethod]
        public void DrawVioletSquare()
        {
            DrawingTest(Colour.Violet, Shape.Square);
        }
    }
}

The number of test methods generated in this example is purposely small, but it is easy to see how this approach can be very powerful if the number of combinations is large.

I like to use partial classes so that all of the supporting code (e.g., the DrawingTest method) can be included in a separate file. This keeps the template as simple as possible. For this example, the remainder of the partial class, from the file ColouredShapesTests.cs, is as follows.

using Drawing;
using System;

namespace DrawingTests
{
    public partial class ColouredShapesTests
    {
        public void DrawingTest(Colour colour, Shape shape)
        {
            // Test code goes here
        }
    }
}

The template can easily become quite complex, to the point where it can be hard to maintain. There are some Visual Studio plugins that offer syntax highlighting and even template debugging that can make life easier. However, I recommend keeping the template as simple and as straightforward as possible. If used judiciously, the T4 Text Templates in Visual Studio can be a useful tool for generating test methods, offering the benefits of enumerating the test methods programmatically, while at the same time maintaining the advantages of having a single test method per test case.