Avoid Handling Exceptions in Negative Tests
Unit tests that provide coverage for every exception a method can throw are invaluable. When writing these tests, the developer needs to carefully consider (or reconsider) the exceptions being thrown, along with how they will ultimately be handled by the caller. This exercise alone often results in improvements to the design of the method under test. Perhaps even more valuable in the long term, however, these tests codify the exception contracts, allowing one to refactor at will without introducing regressions.
I find, however, that moving from developing code to writing these tests can be problematic. I’ve seen C# tests like the following:
[TestMethod] public void SomeFunctionWithNullArgumentThrowsException() { try { SomeFunction(null); } catch (Exception) { } }
This test is written from the point of view of the developer, calling the method and handling the exception as it might be in production code. But what happens if SomeFunction()
doesn’t throw an exception? The test still passes. This test no longer protects against regressions. In fact, the test doesn’t test anything. The test is not even checking for a specific exception. So then one might end up doing something like this:
[TestMethod] public void SomeFunctionWithNullArgumentThrowsException() { try { SomeFunction(null); } catch (ArgumentNullException) { return; } catch (Exception ex) { Assert.Fail("Unexpected exception {0}", ex); } Assert.Fail("We should never get here!") }
I’ve been tempted to write tests like this myself.
Catch statements in tests can also hide changes to exceptions since they will also catch a base class of the thrown exception. For example, this test would still pass if it caught ArgumentException
rather than ArgumentNullException
. This is usually not a problem if the test catches the most specific exception, but there are cases where it can hide subtle changes. I want any change to the exception to fail the test. The failing test may prevent a regression that would break the backwards compatibility of an API or simply act as a reminder that we also need to update calling code or documentation.
The key to improving this test is to stop thinking like a developer. Think like a tester. Use the built-in exception handling mechanisms of the testing framework to make the test more concise and reliable. Usually this involves decorating the fact that the test throws an exception so that it will be handled by the testing framework. For a Visual Studio unit test this accomplished with the ExpectedException
attribute. I like how this attribute decorates the test as a negative test so that anyone can tell at a glance that the test is a negative test. I also like how the attribute forces one to be specific about the exception; the test will fail even if the thrown exception inherits from the expected exception. Look how simple the test is now:
[TestMethod] [ExpectedException(typeof(ArgumentNullException))] public void SomeFunctionWithNullArgumentThrowsException() { SomeFunction(null); }
Even if you want to catch the exception to make assertions about some of its properties, like the exception message, be sure to rethrow the exception. This keeps the test reliable, concise, still clearly decorated as a negative test, and avoids the problems that can result from catching a base exception.
[TestMethod] [ExpectedException(typeof(ArgumentNullException))] public void SomeFunctionWithNullArgumentThrowsException() { try { SomeFunction(); } catch (ArgumentNullException ex) { Assert.IsTrue(ex.Message.Contains("something")); throw; } }
Write negative tests for each exception thrown by a method. When writing these tests, change your mindset from developer to tester and avoid handling exceptions. If an exception goes unhandled the test will fail and that is a good thing.