Use OpenAPI Overlays Today

It's been a while since OpenAPI v3.1 came out, an Project Moonwalk (OAS v4.0) is still in discussion and design phase, but that doesn't mean there's nothing new in the world of OpenAPI. Overlays are one of the OpenAPI Initiatives newest offerings, and I've already been using them to solve problems.

Overlays are a way to "patch" an OpenAPI document, by pointing to a specific part of it with a JSONPath, then using "actions" to update or remove parts of the document. Why would anyone want to do this?

Technical Writers

Perhaps you're a technical writer trying to improve the OpenAPI-based documentation, but the OpenAPI is being generated from source code, and you don't fancy learning 5 different programming languages, web frameworks, and OpenAPI annotation frameworks. Fair enough!

Instead of having your changes overridden, you can put all of your expanded guides and descriptions into an overlay, and have CI patch them together before deploying to your documentation hub.

# Export OpenAPI from code (this is how Huma does it)
go run . openapi > openapi.yaml

Then when you've got the openapi.yaml, you can create a second document called overlays.yaml, which contains all the changes you'd like to apply.

# overlays.yaml
overlay: 1.0.0
info:
  title: Add descriptions to tags
  version: 0.0.1
actions:
  - target: '$.tags[?(@.name=="Order")]'
    description: Provide more information for Order tag.
    update:
      description: >
        The Order resource represents a single order for trees, which can be fulfilled by one or more
        deliveries. Orders are created by the [Protect Earth team](https://protect.earth/contact) and
        are used to track the progress of your order from creation to delivery.
  - target: '$.tags[?(@.name=="Organization")]'
    description: Provide more information for Organization tag.
    update:
      description: >
        The Organization resource represents a single organization, which can be a charity, business,
        or other entity. Organizations are created by the [Protect Earth team](https://protect.earth/contact)
        and are connected to each of your Orders.

I wrote more about how revolutionary overlays can be for technical writers for Bump.sh, and I suggest you check that out if you're more interested in some of the ideas for what you could overlay, e.g. visual improvements for various docs tools, adding code samples, etc.

Efficient Technical Writing Processes for API Documentation

How can engineering and technical writing teams worth together on creating fantastic API documentation without stepping on each others toes?

Bump.sh Docs & GuidesPhil Sturgeon API Design and Reforestation

White Label APIs

My green tech company Green Turtle produces a back-office system for funding tree planting and other biodiversity work, which has an API for integrating with funding partners like Ecologi. Multiple clients use this, including my own charity Protect Earth, but other clients too.

At first I was deploying the generic API documentation, but increasingly this was problematic as the contact details, introductions, and server URLs were all generic rubbish. I wanted to customize it all to be more specific to the client, and overlays worked perfectly for this.

overlay: 1.0.0
info:
  title: Overlay to customise API for Protect Earth
  version: 0.0.1
actions:
  - target: '$.info'
    description: Update description and contact for our audience.
    update:
      description: >
        Protect Earth's Tree Tracker API will let you see what we've been planting and restoring all
        around the UK, and help support our work by directly funding the trees we plant or the sites
        we restore.
        To get involved [contact us and ask for an access token](https://protect.earth/contact) then
        [check out the API documentation](https://protect.earth/api).
      contact:
        name: Protect Earth Support
        url: https://protect.earth/contact
        email: help@protect.earth

  - target: '$.servers.*'
    description: Remove all other servers so we can add our own.
    remove: true

  - target: '$.servers'
    description: Pop our server into the empty server array.
    update:
      - description: Production
        url: https://api.protect.earth/

Now I can deploy client-specific API documentation, so they can share that with their partners without having to explain what a Green Turtle is.

What tools support OpenAPI Overlays?

Overlays is still marked as "experimental", but considered stable enough for tools to start implementing. And implement they have!

openapi-overlays-js

As always Lorna Mitchell was there doing amazing things, and she's put together a handy JavaScript library openapi-overlays-js which handles overlays brilliantly.

At first this seemed like a bit of a hackathon project, but it's being actively maintained and has a few contributors. So far it doesn't have a CLI, but the Bump.sh folks have helped out there!

Bump CLI

The bump-cli wraps this up to help their users use the functionality when deploying their documentation, but anyone can use the cli for any reason.

npm install -g bump-cli 

# apply the overlay and create a new file
bump overlay openapi.yaml overlays.yaml > openapi.public.yaml

# or conviently deploy docs with overlays applied
bump deploy openapi.yaml 
    --doc my-doc 
    --token my-token 
    --overlay overlays.yaml

Instead of just lifting it with quiet thanks, the team have been sending fixes/functionality back, which is what you want to see.

Speakeasy

The Speakeasy team have also built overlay support into their CLI. The Speakeasy CLI is written in Go which will excite some people, but that makes it a little tricker to install in GitHub Actions which is what I was looking for.

# macOS or linux
brew install speakeasy-api/homebrew-tap/speakeasy

# Windows
choco install speakeasy

# Apply Overlay changes and make a new file
speakeasy overlay apply -s openapi.yaml -o overlays.yaml > openapi.public.yaml

They've gone for a really advanced implementation here, offering a few commands:

• speakeasy overlay apply - Given an overlay, construct a new specification by extending a specification and applying the overlay, and output it to stdout.
• speakeasy overlay compare - Given two specs, output an overlay that describes the differences between them.
• speakeasy overlay validate - Given an overlay, validate it according to the OpenAPI Overlay specification

I managed to get this working on GitHub Actions with the following workflow file:

# .github/workflows/deploy.yml
name: Deploy API documentation

on:
  push:
    branches:
      - main

jobs:
  deploy-doc:
    if: ${{ github.event_name == 'push' }}
    name: Deploy API documentation
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v3

      - name: Set up Homebrew
        id: set-up-homebrew
        uses: Homebrew/actions/setup-homebrew@master

      - name: Install Overalls dependencies
        run: |
          brew install speakeasy-api/homebrew-tap/speakeasy

      - name: Apply Overlays to customise OpenAPI
        working-directory: ./api
        run: |
          speakeasy overlay apply -s openapi.yaml -o overlays.yaml > openapi.public.yaml

      - name: Deploy API documentation
        run: <whatever command you use to deploy>
        

If you'd like to play with Speakeasy's fantastic SDK generation then you can get two for one jumping through these hoops, but if you find working with NPM easier I'd definitely go for the Bump.sh CLI approach.

Do we want more JSONPath in our lives?

If you're new to JSONPath, this might seem like a lot to take on, and if you are familiar with JSONPath you might be groaning at the thought of working with it more.

JSONPath is a query language that can be used to extract data from JSON documents, and readers might be familiar witht Spectral which uses it to help identify sections of JSON/YAML documents (most commonly OpenAPI and AsyncAPI) to apply linting rules to.

Much like Markdown in the days before CommonMark, there are a few different JSONPath definitions knocking around which are similar but different in problematic ways.

1. JSONPath "The Blog Post" - Written by Stefan Gössner in 2007.
2. jsonpath.com - An online evaluator which as far as I can tell matches the blog post.
3. JSONPath-Plus - A popular (but now abandoned) fork which expands on the original specification to add some additional operators.
4. Nimma - A fork of JSONPath Plus created by the Stoplight team for Spectral to handle more advanced use cases. A list of caveats can be found here.
5. RFC 9535: JSONPath - An IETF proposed standard that attempts to solve this whole mess by defining one implementation, confusing people like me who are used to JSONPath-Plus and Nimma.

If you are building tooling that uses low-level JSONPath tooling, make sure that tooling is based on RFC 9535, like these tools:

• JsonPath.Net in C#
• jpt in Ruby
• serde_json_path in Rust

Or at least "plans on" aligning with it as Nimma does.

Assuming everything can align on the standard, is JSONPath "good" or "bad"? Well, it can take a bit of getting used to, but it's incredibly powerful. I accept that it's not for non-technically minded people, but I already know of a lot of excited technical writers.

For example, a slightly bonkers JSONPath, but I really enjoyed this generic action which would look for specific server names to remove:

# overlays.yaml
overlay: 1.0.0
info:
  title: Overlay to customise API for Protect Earth
  version: 0.0.1
actions:
  - target: '$.servers[?(@.description=="Development" || @.description=="Staging")]'
    description: Remove Development and Staging servers but leave anything else.
    remove: true

The whole approach of trying to simply name all the objects in OpenAPI falls a bit flat when you are trying to filter and query like this, and you'll only end up inventing a DSL if you try and avoid it, so I say lets stick to a standard instead of making stuff up, and if that standard is JSONPath then that's fine enough with me.

The Future of Overlays

It seems like the specification is settling down and tooling vendors are happily starting to adopt it, but there is one change being discussed which I think could make it a lot better.

Currently you can only update or remove, but update is unclear in its intention when it comes to trying to add things. It's like the difference between PUT and PATCH, should this be a total replacement, where missing things are removed, or should it be a merge, where provided values are mixed in to what is already there?

There is a proposal to ditch update, and define two new operations add and replace. These two new actions have their semantics lifted from RFC 6902: JSON Patch, and would help end users and tooling vendors have a lot more clarity in what they're doing.

Switch from add to replace/update properties by lornajane · Pull Request #32 · OAI/Overlay-Specification

I’m proposing this change as a response to the ongoing discussion in #30 . Using “add” can be confusing and leaves us without being able to support some use cases, such as setting an array to a par…

GitHubOAI

Still, don't wait for everything to be perfect. Get out there and start using this. Build tools for this. Give feedback and bug reports for anything that doesn't work, and comment back here to let everyone know what you've using overlays for!

Further Reading

• Augmenting OpenAPI with Filters & Overlays by me for Bump.sh
• Efficient Technical Writing Processes for API Documentation by me for Bump.sh
• Improve Existing OpenAPI Descriptions by Lorna Mitchell