Contract Tests vs Functional Tests

Contract tests focus on the messages that flow between a consumer and provider, while functional tests also ensure that the correct side effects have occured. For example, imagine an endpoint for a collection of /orders that accepts a POST request to create a new order. A contract test would ensure that the consumer and provider had a shared and accurate understanding of the request and response required to create an order, while a functional test for the provider would ensure that when a given request was made, that an Order with the correct attributes was actually persisted to the underlying datastore. A contract test does not check for side effects.

A more subtle distinction is required when it comes to contract testing interactions that don't have side effects, like validation error responses.

Imagine that we have a simple User Service that allows Consumers to register new users, typically with a POST request containing the details of the created user in the body.

A simple happy-path scenario for that interaction might look like:

Given "there is no user called Mary"
When "creating a user with username Mary"
POST /users { "username": "mary", email: "...", ... }
Expected Response is 200 OK

Sticking to happy-paths is a risk of missing different response codes and potentially having the consumer misunderstand the way the provider behaves. So, let's look at a failure scenario:

Given "there is already a user called Mary"
When "creating a user with username Mary"
POST /users { "username": "mary", email: "...", ... }
Expected Response is 409 Conflict

So far so good, we're covering a new behaviour, with a different response code.

Now we've been talking to the Team managing the User Service and they tell us that username have a maximum length of 20 characters, also they only allow letters in the username and a blank username is obviously not valid. Maybe that's something we should add in our contract?

This is where we get on the slippery slope... it's very tempting to now add 3 scenarios to our contract, something like:

When "creating a user with a blank username"
POST /users { "username": "", email: "...", ... }
Expected Response is 400 Bad Request
Expected Response body is { "error": "username cannot be blank" }
When "creating a user with a username with 21 characters"
POST /users { "username": "thisisalooongusername", email: "...", ... }
Expected Response is 400 Bad Request
Expected Response body is { "error": "username cannot be more than 20 characters" }
When "creating a user with a username containing numbers"
POST /users { "username": "us3rn4me", email: "...", ... }
Expected Response is 400 Bad Request
Expected Response body is { "error": "username can only contain letters" }

We've gone past the contract testing at this point, we're actually testing that the User Service implements the validation rules correctly: this is functional testing, and it should be covered by the User Service in its own codebase.

What is the harm in this... more testing is good, right? The issue here is that these scenarios are going too far and create an unnecessarily tight contract - what if the User Service Team decides that actually 20 characters is too restrictive for username and increases it to 50 characters? What if now numbers are allowed in the username? Any Consumer should be unaffected by any of these changes, unfortunately the Users Service will break our Pact just by loosening the validation rules. These are not breaking changes, but by over-specifying our scenarios we are stopping the User Service Team from implementing them.

So let's go back to our scenarios and instead choose just one simple example to test the way the User Service reacts to bad input:

When "creating a user with an invalid username"
POST /users { "username": "bad_username_that_breaks_some_rule_that_you_are_fairly_confident_will_not_change", ... }
Response is 400 Bad Request
Response body is { "error": "<any string>" }

Subtle, but so much more flexible! Now the User Service Team can change (most) of their validation rules without breaking the Pact we give them... we don't really care about each individual business rule, we only care that if we send something wrong, then we understand the way the User Service responds to us.

When writing a test for an interaction, ask yourself what you are trying to cover. Contracts should be about catching:

  • bugs in the consumer

  • misunderstanding from the consumer about end-points or payload

  • breaking changes by the provider on end-points or payload

In short, your Pact scenarios should not dig into the business logic of the Provider but should stick with verifying that Consumer and Provider have a shared understanding of what requests and responses will be. In our example of validation, write scenarios about how the validation fails, not why the validation fails.