Writing Acceptance Tests for Collections: Two Approaches

I have written a few articles describing how I like to write automated acceptance tests with descriptive test names that read like a specification. These tests not only ensure the quality of the software, but they also act as documentation, and they become a focal point for collaboration. I have found this useful for communicating with fellow software developers, as well as other colleagues, like product managers, technical writers, and technical support staff.

Most of the time, writing automated acceptance tests in this manner feels very natural to me, and helps me define exactly what the application should do. When it comes to testing collections, like sets, lists, and maps, however, I often feel torn as to what to do. The reason is that most testing frameworks provide great libraries for comparing collections in various ways, making it easy to write test assertions. Most testing frameworks also provide a clear error if the test fails, for example, reporting the one item that was missing from a large collection. As a programmer, using these collection assertions is attractive. It makes writing these tests easy. The tests are concise and they are less prone to error. However, the tests no longer read like a specification, so the communicative nature of these tests is lost.

To demonstrate, consider a service for publishing and subscribing to messages from fleet of Internet of Things (IoT) devices. Each device—identified by a unique, 10-digit, alphanumeric identifier—publishes information on distinct topics. The topics are described by a Uniform Resource Identifier (URI), typical of publish-subscribe messaging systems. For example, device v2juqbkkq7 publishes information about device registration using the URI /device/registration/v2juqbkkq7, and status information using the URI /device/status/v2juqbkkq7. Clients can subscribe to messages for a specific URI, or a set of messages by using a wildcard (e.g., /device/status/* for the status messages from all devices).

If I want to test that a subscriber can list all of its subscriptions, I can use a collection assertion. Here is a test, written in Scala, that creates some subscriptions for a random device identifier, then verifies that listing the subscriptions returns only the expected subscriptions.

class SubscriberTests extends FlatSpec with Matchers {
  val endpoint = "messaging.test"
  val randomId = new RandomId()

  "getSubscriptions" should "list the subscriptions for the subscriber" in {
    val id = randomId.next()

    val subscriptions = List(
      s"/device/registration/$id",
      s"/device/activation/$id",
      s"/device/deactivation/$id",
      s"/device/status/$id",
      s"/device/errors/$id"
    )

    val service = MessagingService(endpoint)
    val subscriber = service.addSubscriber("getSubscriptionsTest")
    subscriber.addSubscriptions(subscriptions)

    subscriber.getSubscriptions() should contain theSameElementsAs subscriptions
  }
}

There is nothing contractual about the subscriptions in this test—they are just random—so there is no value in communicating the specific subscriptions in the test name. But now consider a logging application that should subscribe to a specific set of URIs. In this case, the set of subscriptions used by the application is contractual and something I would like to communicate in the test name. Using a collection assertion, like above, would be convenient, but then I will have to name the test something like: The Logging Service has the expected subscriptions. This does not make the test specification self-documenting, or useful for sharing with colleagues.

In this situation, I prefer to enumerate each assertion in a separate test. Note that it is also important to assert that the size of the collection matches what is expected, as this ensures that there are not more subscriptions, in addition to the expected ones. I usually do this by comparing the two collections—in this case using the diff method—as it will conveniently report the items that do not match, should the test fail.

class LoggingServiceTests extends FlatSpec with Matchers {
  val endpoint = "messaging.test"

  val expectedSubscriptions = List(
    "/device/registration/*",
    "/device/activation/*",
    "/device/deactivation/*",
    "/device/status/*",
    "/device/errors/*"
  )

  val service = MessagingService(endpoint)
  val actualSubscriptions = service.getSubscriptions("LoggingService")

  "The Logging Service" should s"have ${expectedSubscriptions.size} subscriptions" in {
    actualSubscriptions.diff(expectedSubscriptions) should be(List.empty)
  }

  expectedSubscriptions.foreach { subscription =>
    it should s"have a subscription for $subscription" in {
      actualSubscriptions should contain(subscription)
    }
  }
}

Using this approach, the tests read like a specification that is useful for sharing with colleagues.

LoggingServiceTests:
The Logging Service
- should have 5 subscriptions
- should have a subscription for /device/registration/*
- should have a subscription for /device/activation/*
- should have a subscription for /device/deactivation/*
- should have a subscription for /device/status/*
- should have a subscription for /device/errors/*

As demonstrated by these two examples, I do not think there is one right answer for testing collections. Remaining torn about which approach to employ is a good thing. When you write a test involving a collection, ask yourself the question: does this test describe a specification that would be useful to communicate with other people? If the answer is no, use the convenient collection assertions. If, rather, the answer is yes, there is value in taking the time to make your acceptance tests communicate the specification.