Sharing Tests and Generating Tests in Scala
I have written before about how I like an automated test to test only one thing, have one clear assertion, and have a descriptive name that reads like a checklist item in a specification that can be understood by someone who is not a programmer, like a product manager. This makes the tests reliable and easy to maintain, but it also makes the test results descriptive. Tests like this are an executable specification that becomes invaluable for guiding development, testing and maintaining applications, as well as communicating interfaces, contracts, and behaviours with colleagues, stakeholders, or customers.
This approach, however, can lead to a large number of very similar tests. When writing these tests, there is a tendency to copy and paste the previous test and modify whatever minor aspect is required to generate the subsequent test, which can be error prone and lead to a lot of duplication. I have provided examples of using context classes, in C#, C++, and Scala, to help streamline some of this duplication and, at the same time, provide for clearer, faster, and more reliable tests. I have also explored code generation and metaprogramming techniques for generating suites of automated tests. In this article, I want to explore how Scala makes sharing tests easy, reducing duplication and increasing expressiveness. I also want to show how tests in Scala can be easily generated, just by writing idiomatic Scala, without having to rely on external tools.
Sharing Tests in Scala
As an illustrative example, consider an HTTP service that maps a 12-character alphanumeric identifier, used by some legacy application, to a Universally Unique Identifier (UUID), used by a new application. Assume that the set of legacy IDs will never change, so the service only supports the HTTP GET method and the HTTP PUT, POST, and DELETE methods are not required. This service is described by the following HTTP route, implemented using Akka HTTP. The ID is resolved to a UUID using the ResolveID method, which is left to the imagination of the reader.
import akka.http.scaladsl.model.MediaTypes.`application/json`
import akka.http.scaladsl.model._
import akka.http.scaladsl.server.Directives._
import akka.http.scaladsl.server.{MethodRejection, RejectionHandler, Route}
object IdRoute {
def apply(): Route = {
def rejectionHandler =
RejectionHandler.newBuilder()
.handleAll[MethodRejection] { _ =>
complete(HttpResponse(StatusCodes.MethodNotAllowed))
}
.handleNotFound {
complete(HttpResponse(StatusCodes.NotFound))
}
.result()
val route =
handleRejections(rejectionHandler) {
path("^[A-Za-z0-9-]{12}$".r) { id =>
get {
ResolveId(id) match {
case Some(result) =>
complete(HttpEntity(ContentType(`application/json`),
s"""{"uuid":"$result"}"""))
case None =>
complete(HttpResponse(StatusCodes.BadRequest))
}
}
}
}
route
}
}
If the ID is resolved, the UUID is returned in a JSON response message with the HTTP status 200 for OK. If the ID is not found, the response is 400 to indicate a Bad Request. If the path in the URL is anything other than a 12-character ID, the response is 404 to indicate that the HTTP route was Not Found. For all HTTP methods other than HTTP GET, the response is 405 for Method Not Allowed.
I will start by writing some negative test for this route by testing a GET without an ID. I will use the Akka HTTP Route TestKit to unit test the route and the FreeSpec test specification, since it provides a lot of flexibility for structuring readable test reports. The first test asserts that the HTTP status 404 is returned and the second test asserts that the body of the response is empty.
import Routes.IdRoute
import akka.http.scaladsl.model._
import akka.http.scaladsl.server.Route
import akka.http.scaladsl.testkit.ScalatestRouteTest
import org.scalatest.{FreeSpec, ShouldMatchers}
import scala.util.Random
class IdRouteTests extends FreeSpec
with IdRouteBehaviours
with ScalatestRouteTest
with ShouldMatchers {
def randomId(length: Int = 12) =
Random.alphanumeric.take(length).mkString
val route = Route.seal(IdRoute())
"GET" - {
"with no ID should" - {
"return 404 Not Found" in {
Get("/") ~> route ~> check {
status should equal(StatusCodes.NotFound)
}
}
"return an empty body" in {
Get("/") ~> route ~> check {
responseAs[String] should be(empty)
}
}
}
}
}
These tests are very clear and straightforward. They will be easy to maintain and easy to debug should they ever fail. However, if I now want to write a negative test for an ID that is too short (11 characters) or too long (13 characters), I essentially need to repeat these tests. I could define a method that includes these two assertions and call it for each test, but this encourages naming tests like "Test GET Not Found for no ID", or worse, "Test no ID", rather than expressing each assertion individually. A test method named like this doesn't communicate anything about the actual behaviours being tested. I could also cut corners and just test each response for the 404 status code and forget about checking each one, consistently, for an empty body.
Alternatively, Scala makes it easy to share tests by including more than one test in a method, typically defined in a trait, using the behaves like pattern for writing tests. This allows me to define a set of behaviours that describe a Not Found response and then consistently apply this set of behaviours as the expected result, all while maintaining one assertion per test, along with descriptive test names.
Here is a trait that defines a set of behaviours for Not Found, Method Not Allowed, and OK responses.
trait IdRouteBehaviours {
this: FreeSpec with ScalatestRouteTest with ShouldMatchers =>
def notFoundRequest(request: HttpRequest, route: Route) = {
"return 404 NotFound" in {
request ~> route ~> check {
status should equal(StatusCodes.NotFound)
}
}
"return an empty body" in {
request ~> route ~> check {
responseAs[String] should be(empty)
}
}
}
def methodNotAllowedRequest(request: HttpRequest, route: Route) = {
"return 405 MethodNotAllowed" in {
request ~> route ~> check {
status should equal(StatusCodes.MethodNotAllowed)
}
}
"return an empty body" in {
request ~> route ~> check {
responseAs[String] should be(empty)
}
}
}
def okRequest(request: HttpRequest, route: Route) = {
"return 200 OK" in {
request ~> route ~> check {
status should equal(StatusCodes.OK)
}
}
"return content type application/json" in {
request ~> route ~> check {
contentType should equal(ContentTypes.`application/json`)
}
}
"return a JSON body with a UUID" in {
request ~> route ~> check {
responseAs[String] should fullyMatch regex
"""^\{"uuid":"[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}"\}$""".r
}
}
}
}
With these behaviours defined, it is easy to write a set of tests, both positive and negative, for the GET method. These tests just focus on the HTTP response behaviours, assuming another set of tests verify the correct mapping of IDs to UUIDs.
"GET" - {
"with no ID should" - {
val request = Get("/")
behave like notFoundRequest(request, route)
}
"with an 11 character ID should" - {
val request = Get("/" + randomId(11))
behave like notFoundRequest(request, route)
}
"with a 13 character ID should" - {
val request = Get("/" + randomId(13))
behave like notFoundRequest(request, route)
}
"with a trailing / after the ID should" - {
val request = Get("/" + randomId() + "/")
behave like notFoundRequest(request, route)
}
"with a 12 character ID should" - {
val request = Get("/" + randomId())
behave like okRequest(request, route)
}
"with a lowercase ID should" - {
val request = Get("/" + randomId().toLowerCase)
behave like okRequest(request, route)
}
"with an uppercase ID should" - {
val request = Get("/" + randomId().toUpperCase)
behave like okRequest(request, route)
}
}
These tests generate a descriptive set of test results: an executable specification. I like to use test reports like this as a focal point for collaborating with my colleagues, since they characterise the system and describe what we are building.
Generating Tests in Scala
One challenge with automated test tools that generate tests at compile time is that it becomes difficult to generate tests programmatically. Consider how Visual Studio uses attributes to define tests in C# and macros to define tests in C++. There is no easy way to generate test code, or even test names, dynamically. One has to rely on external tools for code generation or metaprogramming, as I've written about previously.
Since Scala tests are generated at run-time rather than at compile-time, it makes generating test cases straightforward and it can be done just by writing Scala, without having to rely on external tools.
To write tests for the HTTP methods that are not supported and should return Method Not Allowed, I can define a list of HTTP methods, iterate over the list using foreach, generate a portion of the test name dynamically, and, finally, test the response against a set of behaviours that constitute a Method Not Allowed response.
val methodsNotAllowed = List(
Post("/" + randomId()),
Put("/" + randomId()),
Patch("/" + randomId()),
Delete("/" + randomId()),
Options("/" + randomId()),
Head("/" + randomId())
)
methodsNotAllowed.foreach { request =>
s"${request.method.name} should" - {
behave like methodNotAllowedRequest(request, route)
}
}
This generates 12 descriptive tests from only a few lines of code. The tests are uniform and easy to maintain. It is easy to see what is being tested, because the unique aspect of each test is not lost among repeated boilerplate code. Generating tests in this manner can be especially valuable when you want to be exhaustive, for example, when writing tests to verify a whitelist used for input validation.
For completeness, I have included the entire test class below. Try using these two techniques, sharing a set of behaviors across tests and generating tests programmatically, to reduce duplication, improve maintainability, and generate tests that read like an executable specification.
class IdRouteTests extends FreeSpec
with IdRouteBehaviours
with ScalatestRouteTest
with ShouldMatchers {
def randomId(length: Int = 12) =
Random.alphanumeric.take(length).mkString
val route = Route.seal(IdRoute())
"GET" - {
"with no ID should" - {
val request = Get("/")
behave like notFoundRequest(request, route)
}
"with an 11 character ID should" - {
val request = Get("/" + randomId(11))
behave like notFoundRequest(request, route)
}
"with a 13 character ID should" - {
val request = Get("/" + randomId(13))
behave like notFoundRequest(request, route)
}
"with a trailing / after the ID should" - {
val request = Get("/" + randomId() + "/")
behave like notFoundRequest(request, route)
}
"with a 12 character ID should" - {
val request = Get("/" + randomId())
behave like okRequest(request, route)
}
"with a lowercase ID should" - {
val request = Get("/" + randomId().toLowerCase)
behave like okRequest(request, route)
}
"with an uppercase ID should" - {
val request = Get("/" + randomId().toUpperCase)
behave like okRequest(request, route)
}
}
val methodsNotAllowed = List(
Post("/" + randomId()),
Put("/" + randomId()),
Patch("/" + randomId()),
Delete("/" + randomId()),
Options("/" + randomId()),
Head("/" + randomId())
)
methodsNotAllowed.foreach { request =>
s"${request.method.name} should" - {
behave like methodNotAllowedRequest(request, route)
}
}
}