Turn HTTP Traffic into OpenAPI with Optic

This article is sponsored by Optic, in that they're paying for our time to review the tool, but they're not paying for what we write. If it's rubbish we'll let you know.

Documenting an API with OpenAPI can take a long time when you're starting from scratch. Some API developers use the API Design-first strategy to make their OpenAPI first, then they use that to easily make mocks and docs. API developers who missed that step are now stuck playing catch up, for their OpenAPI-based documentation/dev-portals. Some tool vendors are trying to throw AI at it, but there's a better way: capturing HTTP traffic.

I wrote about creating OpenAPI from HTTP traffic with Akita, but a new tool has popped up that's making things even easier: Optic CLI. I used this tool for a client of Green Turtle, to create an OpenAPI of their legacy black box API that nobody understood, then used that OpenAPI to spec out the work for their new API. Super handy, and since then they've made some improvements.

Optic wrote a simple Document your API in 10 minutes (Step-by-step guide) guide which can get you a long way, but to slightly repeat it:

npm install -g @useoptic/optic

Yeah sorry it's NPM, but that's massively popular and a lot of the best OpenAPI Tools are built in JavaScript so it makes sense.

In this article we're going to learn how the Optic tool works, and use it to build an OpenAPI document for the Mastodon API. You can change the example commands as we go to point to your API, and by the end you'll have a functioning OpenAPI document for your API without having to do any real work at all.

Making OpenAPI for any API

If you just want to practice doing this, you can pick any API, so I'm picking the Mastodon API because it's "new" and interesting, and the Twitter API costs a few bajillion dollars to use now because neoliberal tech bros ruin everything.

Create an empty OpenAPI document

Optic does not generate OpenAPI from scratch, it only patches existing description documents, which means you need to make one to start it off. This seems a bit confusing at first, but it's a huge benefit, because you can run through this process over and over and over, and it will keep improving your OpenAPI document with all the new details as it learns more about the API.

The oas new command will write an empty OpenAPI document to disk.

optic oas new openapi.yaml

Setup TLS (Optional)

Capturing traffic over https:// can be tricky, because it's all encrypted. This is good (you don't want every proxy reading everything) but bad if you literally need your API is running on HTTPS and you literally a proxy like Optic to read everything.

If the API you want to capture is running on http:// because maybe you're recording traffic via localhost, docker, or your API is just super insecure, then you can skip this section and move onto capturing that plan text traffic.

If the API you want to capture is running on https:// then you'll need to set up a man-in-the-middle attack. Other tools need a lot of setup to do this, and you have to teach those tools where the other tools are running, but Optic can wedge itself into the users system settings for network proxies. This means you can snoop on any API, even the production API from your computer. You can do it for literally anything, which is how we're going to record the Mastodon API, just by running a command.

Once the shock of the convenience wears off, you might be thinking "hang on a minute, I'm not sure I want this tool seeing all my traffic..." but it's only going to be running when you run the command, and it's not reporting anything to any servers anywhere. You'll see how it works as we go.

To start off with, setup the TLS logic.

sudo optic oas setup-tls

If you have permissions to run this command you should see a prompt asking you to click on one of your machines existing users. If not, add a user, and its your usual computer login username and password.

Once that's going through successfully you should see something like this.

Trusting Cert. This may take a few seconds. If you see a Keychain prompt appear, enter your password
Certificate trusted. 'oas capture' can now see traffic sent to https hosts when Optic is running

Let's do it.

Capture traffic from the API

Optic can capture traffic from a service you are running locally, or from staging/production. Point Optic at the hostname you want to capture traffic from:

$ optic oas capture openapi.yaml https://mastodon.green/api/

 » Proxy running on https://localhost:8000. System proxy updated
 help  Press [ Enter ] to finish capturing requests
⠼ 0 requests captured

At this point my computer knows to shove all traffic through localhost:8000, which you can confirm by looking at Settings > Network > Proxies. So long as your browser is detecting system proxy settings instead of using its own configuration then you should be capturing requests.

To test its working, head over to https://mastodon.green/philsturgeon and start clicking around on every button on the website. Fill out every form. Edit things. Delete things. Try and get as much API functionality involved as possible, so that Optic can build up the best representation of a superset of all the traffic is spots.

After a while I felt like I'd done everything I could, so I went back to the Optic CLI tab and hit Enter to end the session.

It let me know to run optic oas verify openapi.yaml, which lets you see all the traffic its recorded, then as I was happy with that I ran this command to save the output:

optic oas verify openapi.yaml --document all

This tells Optic to populate the openapi.yaml document with everything it knows about your API! Mine looked like this:

openapi: 3.1.0
info:
  title: Untitled service
  version: 1.0.0
x-optic-path-ignore:
  - "**/*.+(ico|png|jpeg|jpg|gif)"
paths:
  /v1/custom_emojis:
    get:
      responses:
        "200":
          description: 200 response
          content:
            application/json; charset=utf-8:
              schema:
                $ref: "#/components/schemas/GetApiV1Custom_emojis200ResponseBody"
                
# ... snip ... 
components:
  schemas:
    GetApiV1Custom_emojis200ResponseBody:
      type: array
      items:
        type: object
        properties:
          shortcode:
            type: string
          url:
            type: string
          static_url:
            type: string
          visible_in_picker:
            type: boolean
          category:
            type: string
        required:
          - shortcode
          - url
          - static_url
          - visible_in_picker
          - category

Brilliant! That's a whole lot of OpenAPI I don't need to write, and Optic is even making components to keep your OpenAPI Dry.

One problem I had was that I ended up loads of excess URLs, like /settings and /packs/{pack}/153-c1b3066c820d0c5de2e2.chunk.js which I am not interested in.

Getting rid of these is easy enough. For starters I should add my server URL, with the base path of /api set to let Optic know it should ignore things outside of that path. I can add that in openapi.yaml, then delete all the irrelevant endpoints.

servers:
  - description: Production
    url: https://mastodon.green/api

With that servers definition added I removed all the '/api` prefixes from the paths, and reran capture. Due to the way Optic patches the OpenAPI document, I had no problems rewriting the OpenAPI, and it immediately picked up on what I was trying to do.

Optic has inferred a lot of the what, but it cannot infer why, and this is where you cannot avoid spending a bit of time getting descriptions written. If you have a big enough team maybe a Technical Writer can handle this for you, maybe using a fancy GUI like Stoplight Studio, or maybe just going right into the YAML. Thankfully however you do it, Optic will happily patch that resulting OpenAPI, and the work will not be overridden.

Sharing OpenAPI to Optic Cloud

Once your OpenAPI is in a state you're happy to share, you can take the openapi.yaml and get it working with any of the many OpenAPI documentation tools, but Optic has that covered too.

optic login 

This will pop open your browser, and there is a "sign in with GitHub" flow. Then you'll be asked to create a Personal access token for Optic, which you can paste that the CLI.

Then run this command to upload the OpenAPI document to their servers:

$ optic api add openapi.yaml

Adding API /src/openapi.yaml
✔ Untitled service is now being tracked.
  View history: https://app.useoptic.com/organizations/7727d0f3-dba3-4c52-8b32-046187d60239/apis/sdCx0sapGyPyohJiKdOqB

Take a look at the Mastodon API documentation Optic created.

Thoughts and Feedback

This is still a very new tool, and you might find a few rough edges as you go. I found a few bugs using the tool, but Aidan Cunniffe and the Optic team are on our Slack channel and released bug fixes quickly.

Optic has mixed results with different browsers, and I could not get things working with Firefox. Thankfully Google Chrome works just fine, so I could easily switch browsers for this work. You can also attack the API directly with Postman, Curl, Insomnia, Paw, etc., which is pretty handy if you don't have a frontend you can use to make the requests.

My biggest gripe with the quality of the OpenAPI being output at this point is the lack of examples. Optic will happily look at actual JSON values coming through to infer the type might be e.g.: "string", but it forgets the value it saw entirely instead of saving it as an example. There's a lot of ways to handle examples in OpenAPI, but Optic does not seem to leverage any of them. No media type examples, schema examples, or property examples. Without any examples, the documentation it produced can feel rather lacking, so I really hope that gets added in the future.

All in all this is a really handy tool which can quickly get you an OpenAPI document, without having to litter your code with annotations which may or may not be accurate. You'll definitely want to do some work to get it up to scratch before going live, but it reduces the human error of having to manually type thousands of lines of YAML out, or click 1000 buttons in a GUI. I don't think of this as a replacement to the API Design First approach, but it's a brilliant second best when you've got to get some OpenAPI out fast.

Beyond the scope of this article, but Optic does a bunch of other cool change detection stuff too, so it's worth having a poke around the documentation to see if any of that can fit in with your API workflow.