In Defence of Testing One Thing Per Test Method

Mar 8, 2015

In my last post, I described why I prefer tests that test only one thing. Naturally, this can lead to a lot of test methods with a fair amount of repetitive code. I presented a technique for programmatically generating tests when doing so by hand would be tedious or error prone. I shared this technique because I thought it was interesting, but, truth be told, I have used this technique exactly once.

Most often, test cases vary slightly from one to another, but are different enough that they cannot be generated from a template. When this is the case, I certainly understand why people object to writing one test method per test case. It can be tedious and often involves a lot of boilerplate code.

One reason why people object to writing so many tests, in my opinion, is that they are too focused on the test writing itself, rather than appreciating the full life-cycle of the tests. Like a lot of the code that we write, the majority of the time will be spent living with this code — seeing the results of the tests each time they run with the build, troubleshooting test failures, extending test classes or frameworks to cover new functionality, etc. — not writing the code itself. Tests are a lot more effective in this regard if they are structured, well named, reliable, and maintainable, and, in my opinion, nothing keeps the focus on these qualities better than having one test case per test method.

Another reason why I think people shy away from writing a test method per test case is that they are not leveraging their development environment to its fullest. They are literally writing all these tests by hand. When faced with all these tests to write, they try to make one method test as many things as possible, or forego writing some test cases altogether. They are not using macros or code snippets for repetitive tasks. They don't leverage the keyboard shortcuts of their text editor. They avoid regular expressions. They don't have a go-to scripting language.

In this post, I will provide an example of how to transform test code to cover additional test cases and ease the burden of having to write each test. The foundation for doing this are well structured tests with descriptive names.

Naming Tests

When I first started writing tests, I tried to do a good job naming them, but I was never really sure how to name them. Some names described what the test did, while others were more organizational categories.

My approach to naming tests changed when I read Erik Dietrich's blog series on unit testing. As Erik describes:

As you get comfortable with unit tests, you’ll start to give them titles that describe correct functionality of the system and you’ll use them as kind of a checklist for getting code right. It’s like your to-do list. If every box is checked, you feel pretty good. And you can put checkboxes next to statements like Throws_Exception_When_Passed_A_Null_Value but not next to Test_Null.

When I started naming tests like checklist items, I was hooked. Simply describing the Arrange, Act, and Assert portions of the test became the basis of the test name. This gave me a pattern for naming tests. I no longer had to think up good test names, I just followed the pattern. It also provided a lot of consistency, especially as other members of the team I was working on adopted the same approach. Also, when looking at test results, these checklist names were very helpful in understanding test failures.

Being disciplined about naming tests in this manner means that, generally, the test can test only one thing. It becomes very difficult to name a test that tests more than one thing. I find this a helpful guideline: if I'm struggling to name a test, I'm probably not clear on the one thing that it is testing.

This approach will result in a lot of test cases. But they will be well structured and the intent of each test will be clear. They will be easy to maintain and refactor. When they fail, it is easy to determine what went wrong.

Another benefit that comes from these well structured and descriptively named tests is that they develop patterns that are easy to transform, which can save a lot of time in generating additional test cases. This is best demonstrated with an example.

Transforming Tests to Generate Additional Tests

Let's revisit some C# tests for testing database CRUD operations that I presented in a earlier blog post. Consider these two tests, named like checklist items, for the GetRecord method.

[TestClass]
public class DatabaseGetRecordTests
{
    private readonly Database _database = new Database("testdb.domain.tst");

    public TestContext TestContext { get; set; }

    [TestMethod]
    public void GetRecordReturnsNullIfRecordDoesNotExist()
    {
        Assert.IsNull(_database.GetRecord(TestContext.TestName));
    }

    [TestMethod]
    public void GetRecordReturnsRecord()
    {
        string name = TestContext.TestName;

        using (new DatabaseRecordContext(name, _database))
        {
            var record = _database.GetRecord(name);
            Assert.AreEqual(name, record.Name);
        }
    }
}

Say that the database client also supports asynchronous methods and that we want to include the equivalent tests for the asynchronous methods, in addition to the synchronous methods. It can be tempting to extend the existing tests to also include the asynchronous calls, since the Arrange and Assert portions of the test are identical.

[TestClass]
public class DatabaseGetRecordTests
{
    private readonly Database _database = new Database("testdb.domain.tst");

    public TestContext TestContext { get; set; }

    [TestMethod]
    public void GetRecordReturnsNullIfRecordDoesNotExist()
    {
        Assert.IsNull(_database.GetRecord(TestContext.TestName));

        // Get it again to test aync method
        Assert.IsNull(_database.GetRecordAsync(TestContext.TestName).Result);
    }

    [TestMethod]
    public void GetRecordReturnsRecord()
    {
        string name = TestContext.TestName;

        using (new DatabaseRecordContext(name, _database))
        {
            var record = _database.GetRecord(name);
            Assert.AreEqual(name, record.Name);

            // Get it again to test async method
            record = _database.GetRecordAsync(name).Result;
            Assert.AreEqual(name, record.Name);
        }
    }
}

This makes the intent of each test less clear. This approach also breaks down and significantly complicates the test if we need to do more Arranging after the first Assert.

Because the synchronous GetRecord tests are well structured, it is not hard to transform this test class into another one that tests the asynchronous methods. A simple approach is to copy this class and then use the find and replace feature of the text editor to find all of the GetRecord references and replace them with GetRecordAsync. The compiler will remind us that we need to wait for the asynchronous methods to complete by adding a .Result to the GetRecordAsync calls, and we are done.

[TestClass]
public class DatabaseGetRecordAsyncTests
{
    private readonly Database _database = new Database("testdb.domain.tst");

    public TestContext TestContext { get; set; }

    [TestMethod]
    public void GetRecordAsyncReturnsNullIfRecordDoesNotExist()
    {
        Assert.IsNull(_database.GetRecordAsync(TestContext.TestName).Result);
    }

    [TestMethod]
    public void GetRecordAsyncReturnsRecord()
    {
        string name = TestContext.TestName;

        using (new DatabaseRecordContext(name, _database))
        {
            var record = _database.GetRecordAsync(name).Result;
            Assert.AreEqual(name, record.Name);
        }
    }
}

That was easy. But it can be even easier if we write a script to do this work. For this example I will use Windows PowerShell, but you could use any scripting language of your choice, including the scripting features of your text editor. The approach is as follows.

Get the content of the file containing the GetRecord tests.
Replace any _database.GetRecord calls with _database.GetRecordAsync.
Append .Result to any _database.GetRecordAsync calls.
Modify any public void methods that start with GetRecord to start with GetRecordAsync.
Modify the test class name from DatabaseGetRecordTests to DatabaseGetRecordAsyncTests.
Output the resulting test to a new file DatabaseGetRecordAsyncTests.cs.

(Get-Content .\DatabaseGetRecordTests.cs) -Replace "_database.GetRecord","$&Async" -Replace "_database.GetRecordAsync\(.*?\)","$&.Result" -Replace "public void GetRecord","$&Async" -Replace "public class DatabaseGetRecord","$&Async" | Set-Content .\DatabaseGetRecordAsyncTests.cs

This script can be even more generic so that instead of just transforming tests for GetRecord, it can effortlessly transform similarly structured tests for CreateRecord, DeleteRecord, EditRecord, etc.

(Get-Content .\DatabaseGetRecordTests.cs) -Replace "_database.[A-z]*?Record","$&Async" -Replace "_database.[A-z]*?RecordAsync\(.*?\)","$&.Result" -Replace "public void [A-z0-9]*?Record","$&Async" -Replace "public class [A-z]*?Record","$&Async" | Set-Content .\DatabaseGetRecordAsyncTests.cs

Take, for example, a test class for the synchronous CreateRecord method.

[TestClass]
public class DatabaseCreateRecordTests
{
    private readonly Database _database = new Database("testdb.domain.tst");

    public TestContext TestContext { get; set; }

    [TestMethod]
    public void CreateRecordCreatesRecord()
    {
        string name = TestContext.TestName;
        var record = _database.CreateRecord(name);
        Assert.AreEqual(name, record.Name);
    }

    [TestMethod, ExpectedException(typeof(DatabaseException))]
    public void CreateRecordWithEmptyNameThrowsException()
    {
        var record = _database.CreateRecord(string.Empty);
    }

    [TestMethod, ExpectedException(typeof(DatabaseException))]
    public void CreateRecordWithNullNameThrowsException()
    {
        var record = _database.CreateRecord(null);
    }

    [TestMethod, ExpectedException(typeof(DatabaseException))]
    public void CreateRecordWithNameTooLongThrowsException()
    {
        var name = new string('a', 101);
        var record = _database.CreateRecord(name);
    }

    [TestMethod]
    public void CreateRecordWithMaximumLengthNameCreatesRecord()
    {
        var name = new string('a', 100);
        var record = _database.CreateRecord(name);
    }

    [TestMethod]
    public void CreateRecordWithMinimumLengthNameCreatesRecord()
    {
        var name = new string('a', 1);
        var record = _database.CreateRecord(name);
    }
}

This class can be transformed with the same script to yield all of the tests cases for the CreateRecordAsync method.

[TestClass]
public class DatabaseCreateRecordAsyncTests
{
    private readonly Database _database = new Database("testdb.domain.tst");

    public TestContext TestContext { get; set; }

    [TestMethod]
    public void CreateRecordAsyncCreatesRecord()
    {
        string name = TestContext.TestName;
        var record = _database.CreateRecordAsync(name).Result;
        Assert.AreEqual(name, record.Name);
    }

    [TestMethod, ExpectedException(typeof(DatabaseException))]
    public void CreateRecordAsyncWithEmptyNameThrowsException()
    {
        var record = _database.CreateRecordAsync(string.Empty).Result;
    }

    [TestMethod, ExpectedException(typeof(DatabaseException))]
    public void CreateRecordAsyncWithNullNameThrowsException()
    {
        var record = _database.CreateRecordAsync(null).Result;
    }

    [TestMethod, ExpectedException(typeof(DatabaseException))]
    public void CreateRecordAsyncWithNameTooLongThrowsException()
    {
        var name = new string('a', 101);
        var record = _database.CreateRecordAsync(name).Result;
    }

    [TestMethod]
    public void CreateRecordAsyncWithMaximumLengthNameCreatesRecord()
    {
        var name = new string('a', 100);
        var record = _database.CreateRecordAsync(name).Result;
    }

    [TestMethod]
    public void CreateRecordAsyncWithMinimumLengthNameCreatesRecord()
    {
        var name = new string('a', 1);
        var record = _database.CreateRecordAsync(name).Result;
    }
}

Conclusion

Implementing one test method per test case and following the Arrange, Act, Assert pattern will result in well structured, reliable, and maintainable tests. But it will also result in a lot of very similar test methods. Rather than writing all of these test methods by hand, or being tempted to cut corners and omit tests or complicate tests by testing more than one thing per test method, try and transform the code programmatically to yield all of the test cases.