> ## Content Index
> Fetch the complete content index at: https://apisyouwonthate.com/llms.txt
> Use this file to discover other available public pages before exploring further.

# Writing Documentation via Contract Testing
- URL: https://apisyouwonthate.com/blog/writing-documentation-via-contract-testing/
- Published: 2018-04-19T13:06:00.000Z
- Updated: 2023-03-24T12:32:50.000Z
- Author: Phil Sturgeon
- Tags: #Import 2023-03-24 12:19

Part of my day job is trying to get folks to write API documentation, so the  
hordes of new developers joining the company on a weekly basis are not stuck  
trying [guess at contracts](https://apisyouwonthate.com/blog/commit-to-api-contracts)  
by poking around years old code. Convincing API developers to take the time to  
write specs has involved dangling a myriad of carrots.

One was a continuous integration pipeline to convert their in-repo OpenAPI specs  
to read-only Postman Mirrors (because who wants to maintain Postman collections  
manually?!!), and some folks are excited to [use JSON Schema to offer client-side validation](https://apisyouwonthate.com/blog/json-schema-client-side-validation)  
(now that we’ve solved the JSON Schema >< OpenAPI issue.)

The carrot we’ll be discussing here is contract testing: the art of validating  
the shape of data provided by the API is what you think it is.

Folks are regularly asking "how can we keep our documentation up to date”, and  
we have [covered a few solutions to that,](https://apisyouwonthate.com/blog/keeping-documentation-honest)  
one of which is using JSON Schema. Unfortunately thinking of it this way leads  
to responses like "ok greeeeeat, when I have finished spending a week writing up  
documentation, I get to write a bunch more stuff to ensure its up to date.  
Yaaaay! 🙄”

Well that’s a fairly backwards way to think about the whole thing. I tell  
people: API specifications are a *really* powerful way to get contract testing,  
and they just so happen to render as documentation too. Once you’ve confirmed  
your code conforms to the contract, you know your documentation is correct, so  
it’s pretty much been done for free.

## What is Contract Testing

A lot of applications just check that the endpoint doesn’t entirely crap out,  
which is the most basic sort of unit test:

```bash
it 'should return HTTP OK (200)' do
  get "/users/#{subject.id}"
  expect(response).to have_http_status(:ok)
end

```

Sure ok, that’s better than literally nothing, but we need to know what fields are there, that those fields have a specific type, and we need to make sure that doesn’t randomly change!

Some folks will add some specific checks to their unit tests, maybe something like [include\_json](https://relishapp.com/waterlink/rspec-json-expectations/docs/json-expectations/include-json-matcher?ref=apisyouwonthate.com):

```
it "has basic info about user" do
  expect(subject).to include_json(
    id: 25,
    email: "john.smith@example.com",
    name: "John"
  )
end

```

Others will bump this off to BDD-style testing with Cucumber or similar tools, which is essentially doing the exact same thing:

```
Feature: User API

Scenario: Show action
    When I visit "/users/1"
    Then the JSON response at "first_name" should be "Steve"
    And the JSON response at "last_name" should be "Richert"
    And the JSON response should have "username"
    And the JSON response at "username" should be a string
    And the JSON response should have "created_at"
    And the JSON response at "created_at" should be a string
    And the JSON response should have "updated_at"
    And the JSON response at "updated_at" should be a string
    ... ugh it goes on and on ...

```

At this point it is essentially contract testing, but its very laborious, and is a whitelist. If a new field shows up and we’ve not written a bunch of assertions for its type, we have no way of knowing if its correct. Clients might start using it, later it could change, and our test suite would never know.

Something else to consider, is that writing these assertions for each field sucks. You want your unit tests to confirm the shape of the response under all sorts of scenarios, but you do not want to copy these lines over and over again. This leads to folks shoving some assertions into the first unit test, then just crossing their fingers that other scenarios output the correct shape. 🤷‍♂️

## JSON Schema for "DRY” Contract Testing

Don’t Repeat Yourself, and define reusable `schemas/components/user.json` files, populated by JSON Schema:

```json
{
  "$schema": "https://json-schema.org/draft-06/schema#",
  "title": "Foo",
  "type": "object",
  "properties": {
    "id": {
      "readOnly": true,
      "type": "string",
      "example": "123"
    },
     "uuid": {
       "type": "string",
       "format": "uuid",
       "example": "50f50f52-0d41-4a08-85ea-56423b2803c8"
     },
     "email": {
       "type": "string",
       "example": "john@example.com"
     },
     "name": {
       "type": "string",
       "example": "john.smith"
     },
     "status": {
       "type": "string",
       "example": "inactive"
     },
     "created_at": {
       "type": ["string", "null"],
       "format": "date-time",
       "example": "2018-04-09T15:45:44.358Z"
     },
  },
  "required": [
    "email",
    "name",
    "uuid"
  ]
}

```

*Learn more about the basic JSON Schema keywords over here:*

[**The basics - Understanding JSON Schema 1.0 documentation** \_When learning any new language, it's often helpful to start with the simplest thing possible. In JSON Schema, an empty…\_spacetelescope.github.io](https://spacetelescope.github.io/understanding-json-schema/basics.html?ref=apisyouwonthate.com "https://spacetelescope.github.io/understanding-json-schema/basics.html")[](https://spacetelescope.github.io/understanding-json-schema/basics.html?ref=apisyouwonthate.com)

Armed with a bunch of reusable contracts, the only thing to do is check if those contracts match the code. Ruby users can do this with [json\_matchers](https://github.com/thoughtbot/json%5Fmatchers?ref=apisyouwonthate.com) by Thoughtbot.

Put this config wherever you handle your RSpec setup (or check out their [alternative approaches](https://github.com/thoughtbot/json%5Fmatchers?ref=apisyouwonthate.com#usage)):

```ruby
require "json_matchers/rspec"

JsonMatchers.schema_root = "docs/components/schemas"

```

Then it’s simply a case of adding new tests for that context, or adding the expect line to your existing tests:

```ruby
it 'should return HTTP OK (200)' do
  get "/users/#{subject.id}"
  expect(response).to have_http_status(:ok)
end

it 'should conform to user schema' do
  get "/users/#{subject.id}"
  expect(response).to match_json_schema('user')
end

```

That’s it as far as contract testing goes. Run your tests, tweak the responses and JSON Schema, try and get it to fail. Remove a required field and it’ll moan. Typecast an int to a string and it’ll get sad.

If you need help creating these JSON Schema files, there’s [a few hacks to speed you up](http://localhost:3000/blog/creating-api-specifications-from-bullst?ref=apisyouwonthate.com).

#### What about Docs?

Often early stage OpenAPI documentation is a lot of paths and maybe some properties, but folks often leave the schemas "until later” (never).

```yaml
responses:
    200:
      description: OK

```

With the schema missing you lose all benefit of API specifications: you have no example values, attribute tables, enums for status fields, types, etc. Luckily the new JSON Schema fill exactly that gap!

```yaml
  responses:
    200:
      description: OK
      content:
        application/json:
          schema:
            $ref: ./components/schemas/user.json

```

*You’ll need to watch out for the [divergence issue](https://apisyouwonthate.com/blog/openapi-json-schema-divergence),* 
*or use* 
*[json-schema-to-openapi-schema](https://github.com/openapi-contrib/json-schema-to-openapi-schema?ref=apisyouwonthate.com)* 
*to convert this JSON Schema & OpenAPI combo to "pure OpenAPI" in some build/CI* 
*step, but this works juuuust fine.*

Then it’s just a case of [turning that OpenAPI into beautiful documentation](https://apisyouwonthate.com/blog/turning-contracts-into-beautiful-documentation).

Give it a try and let me know what you think. I really like this simple,  
powerful, approach to keeping code up-to-date with your API specs.

**JSON Schema**

- JavaScript  
  - [jest-json-schema](https://github.com/americanexpress/jest-json-schema?ref=apisyouwonthate.com) (by American Express 😲)
  - [chai-json-schema](https://www.chaijs.com/plugins/chai-json-schema/?ref=apisyouwonthate.com)
- PHP  
  - [estahn/phpunit-json-assertions](https://github.com/estahn/phpunit-json-assertions?ref=apisyouwonthate.com)
- Ruby  
  - [thougtbot/json\_matchers](https://github.com/thoughtbot/json%5Fmatchers?ref=apisyouwonthate.com)

*Tell us about your API description-based testing and what languages and* 
*frameworks you're using on* 
*[@apisyouwonthate](https://twitter.com/apisyouwonthate?ref=apisyouwonthate.com) or [our Slack](https://slack.apisyouwonthate.com/?ref=apisyouwonthate.com).*