Writing Documentation via Contract Testing

Phil Sturgeon

Apr 19, 2018 — 5 min read

Part of my day job is trying to get folks to write API documentation, so the
hordes of new developers joining the company on a weekly basis are not stuck
trying guess at
contracts
by poking around years old code. Convincing API developers to take the time to
write specs has involved dangling a myriad of carrots.

One was a continuous integration pipeline to convert their in-repo OpenAPI specs
to read-only Postman Mirrors (because who wants to maintain Postman collections
manually?!!), and some folks are excited to use JSON Schema to offer
client-side
validation
(now that we’ve solved the JSON Schema >< OpenAPI issue.)

The carrot we’ll be discussing here is contract testing: the art of validating
the shape of data provided by the API is what you think it is.

Folks are regularly asking "how can we keep our documentation up to date”, and
we have covered a few solutions to
that,
one of which is using JSON Schema. Unfortunately thinking of it this way leads
to responses like "ok greeeeeat, when I have finished spending a week writing up
documentation, I get to write a bunch more stuff to ensure its up to date.
Yaaaay! 🙄”

Well that’s a fairly backwards way to think about the whole thing. I tell
people: API specifications are a really powerful way to get contract testing,
and they just so happen to render as documentation too. Once you’ve confirmed
your code conforms to the contract, you know your documentation is correct, so
it’s pretty much been done for free.

What is Contract Testing

A lot of applications just check that the endpoint doesn’t entirely crap out,
which is the most basic sort of unit test:

it 'should return HTTP OK (200)' do
  get "/users/#{subject.id}"
  expect(response).to have_http_status(:ok)
end

Sure ok, that’s better than literally nothing, but we need to know what fields are there, that those fields have a specific type, and we need to make sure that doesn’t randomly change!

Some folks will add some specific checks to their unit tests, maybe something like include_json:

it "has basic info about user" do
  expect(subject).to include_json(
    id: 25,
    email: "john.smith@example.com",
    name: "John"
  )
end

Others will bump this off to BDD-style testing with Cucumber or similar tools, which is essentially doing the exact same thing:

Feature: User API

Scenario: Show action
    When I visit "/users/1"
    Then the JSON response at "first_name" should be "Steve"
    And the JSON response at "last_name" should be "Richert"
    And the JSON response should have "username"
    And the JSON response at "username" should be a string
    And the JSON response should have "created_at"
    And the JSON response at "created_at" should be a string
    And the JSON response should have "updated_at"
    And the JSON response at "updated_at" should be a string
    ... ugh it goes on and on ...

At this point it is essentially contract testing, but its very laborious, and is a whitelist. If a new field shows up and we’ve not written a bunch of assertions for its type, we have no way of knowing if its correct. Clients might start using it, later it could change, and our test suite would never know.

Something else to consider, is that writing these assertions for each field sucks. You want your unit tests to confirm the shape of the response under all sorts of scenarios, but you do not want to copy these lines over and over again. This leads to folks shoving some assertions into the first unit test, then just crossing their fingers that other scenarios output the correct shape. 🤷‍♂️

JSON Schema for "DRY” Contract Testing

Don’t Repeat Yourself, and define reusable schemas/components/user.json files, populated by JSON Schema:

{
  "$schema": "https://json-schema.org/draft-06/schema#",
  "title": "Foo",
  "type": "object",
  "properties": {
    "id": {
      "readOnly": true,
      "type": "string",
      "example": "123"
    },
     "uuid": {
       "type": "string",
       "format": "uuid",
       "example": "50f50f52-0d41-4a08-85ea-56423b2803c8"
     },
     "email": {
       "type": "string",
       "example": "john@example.com"
     },
     "name": {
       "type": "string",
       "example": "john.smith"
     },
     "status": {
       "type": "string",
       "example": "inactive"
     },
     "created_at": {
       "type": ["string", "null"],
       "format": "date-time",
       "example": "2018-04-09T15:45:44.358Z"
     },
  },
  "required": [
    "email",
    "name",
    "uuid"
  ]
}

Learn more about the basic JSON Schema keywords over here:

The basics - Understanding JSON Schema 1.0 documentation
_When learning any new language, it's often helpful to start with the simplest thing possible. In JSON Schema, an empty…_spacetelescope.github.io

Armed with a bunch of reusable contracts, the only thing to do is check if those contracts match the code. Ruby users can do this with json_matchers by Thoughtbot.

Put this config wherever you handle your RSpec setup (or check out their alternative approaches):

require "json_matchers/rspec"

JsonMatchers.schema_root = "docs/components/schemas"

Then it’s simply a case of adding new tests for that context, or adding the expect line to your existing tests:

it 'should return HTTP OK (200)' do
  get "/users/#{subject.id}"
  expect(response).to have_http_status(:ok)
end

it 'should conform to user schema' do
  get "/users/#{subject.id}"
  expect(response).to match_json_schema('user')
end

That’s it as far as contract testing goes. Run your tests, tweak the responses and JSON Schema, try and get it to fail. Remove a required field and it’ll moan. Typecast an int to a string and it’ll get sad.

If you need help creating these JSON Schema files, there’s a few hacks to speed you up.

What about Docs?

Often early stage OpenAPI documentation is a lot of paths and maybe some properties, but folks often leave the schemas "until later” (never).

responses:
    200:
      description: OK

With the schema missing you lose all benefit of API specifications: you have no example values, attribute tables, enums for status fields, types, etc. Luckily the new JSON Schema fill exactly that gap!

  responses:
    200:
      description: OK
      content:
        application/json:
          schema:
            $ref: ./components/schemas/user.json

You’ll need to watch out for the divergence issue,
or use
json-schema-to-openapi-schema
to convert this JSON Schema & OpenAPI combo to "pure OpenAPI" in some build/CI
step, but this works juuuust fine.

Then it’s just a case of turning that OpenAPI into beautiful
documentation.

Give it a try and let me know what you think. I really like this simple,
powerful, approach to keeping code up-to-date with your API specs.

JSON Schema