> ## Content Index > Fetch the complete content index at: https://apisyouwonthate.com/llms.txt > Use this file to discover other available public pages before exploring further. # Turn HTTP Traffic into OpenAPI with Optic - URL: https://apisyouwonthate.com/blog/turn-http-traffic-into-openapi-with-optic/ - Published: 2023-03-27T17:42:29.000Z - Updated: 2023-12-28T11:20:34.000Z - Description: Capture real HTTP traffic from production or anywhere else, and create OpenAPI from it, for documentation, mocks, SDKs, or contract testing. - Author: Phil Sturgeon - Tags: openapi, http, api-design, #phil-sturgeon *This article is sponsored by* [*Optic*](https://www.useoptic.com/?ref=apisyouwonthate.com)*, in that they're paying for our time to review the tool, but they're not paying for what we write. If it's rubbish we'll let you know.* Documenting an API with OpenAPI can take a long time when you're starting from scratch. Some API developers use the API Design-first strategy to make their OpenAPI first, then they use that to easily make mocks and docs. API developers who missed that step are now stuck playing catch up, for their OpenAPI-based documentation/dev-portals. Some tool vendors are trying to throw AI at it, but there's a better way: capturing HTTP traffic. I wrote about [creating OpenAPI from HTTP traffic with Akita](https://apisyouwonthate.com/blog/creating-openapi-from-http-traffic), but a new tool has popped up that's making things even easier: Optic CLI. I used this tool for a client of [Green Turtle](https://www.greenturtle.io/?ref=apisyouwonthate.com), to create an OpenAPI of their legacy black box API that nobody understood, then used that OpenAPI to spec out the work for their new API. Super handy, and since then they've made some improvements. Optic wrote a simple *Document your API in 10 minutes (Step-by-step guide)* guide which can get you a long way, but to slightly repeat it: ```bash npm install -g @useoptic/optic ``` Yeah sorry it's NPM, but that's massively popular and a lot of the best [OpenAPI Tools](https://openapi.tools/?ref=apisyouwonthate.com) are built in JavaScript so it makes sense. In this article we're going to learn how the Optic tool works, and use it to build an OpenAPI document for the Mastodon API. You can change the example commands as we go to point to your API, and by the end you'll have a functioning OpenAPI document for your API without having to do any real work at all. ## Making OpenAPI for any API If you just want to practice doing this, you can pick any API, so I'm picking the Mastodon API because it's "new" and interesting, and the Twitter API costs a few bajillion dollars to use now because neoliberal tech bros ruin everything. ### Create an empty OpenAPI document Optic does not generate OpenAPI from scratch, it only patches existing description documents, which means you need to make one to start it off. This seems a bit confusing at first, but it's a huge benefit, because you can run through this process over and over and over, and it will keep improving your OpenAPI document with all the new details as it learns more about the API. The `oas new` command will write an empty OpenAPI document to disk. ```bash optic oas new openapi.yaml ``` ### Setup TLS (Optional) Capturing traffic over `https://` can be tricky, because it's all encrypted. This is good (you don't want every proxy reading everything) but bad if you literally need your API is running on HTTPS and you literally a proxy like Optic to read everything. If the API you want to capture is running on `http://` because maybe you're recording traffic via localhost, docker, or your API is just super insecure, then you can skip this section and move onto capturing that plan text traffic. If the API you want to capture is running on `https://` then you'll need to set up a man-in-the-middle attack. Other tools need a lot of setup to do this, and you have to teach those tools where the other tools are running, but Optic can wedge itself into the users system settings for network proxies. This means you can snoop on any API, even the production API from your computer. You can do it for literally anything, which is how we're going to record the Mastodon API, just by running a command. Once the shock of the convenience wears off, you might be thinking "hang on a minute, I'm not sure I want this tool seeing all my traffic..." but it's only going to be running when you run the command, and it's not reporting anything to any servers anywhere. You'll see how it works as we go. To start off with, setup the TLS logic. ``` sudo optic oas setup-tls ``` If you have permissions to run this command you should see a prompt asking you to click on one of your machines existing users. If not, add a user, and its your usual computer login username and password. ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2023/03/optic-perms.png) On macOS the CLI spawns an account picker interface when you run this command. Once that's going through successfully you should see something like this. > Trusting Cert. This may take a few seconds. If you see a Keychain prompt appear, enter your password > Certificate trusted. 'oas capture' can now see traffic sent to https hosts when Optic is running Let's do it. ### Capture traffic from the API Optic can capture traffic from a service you are running locally, or from staging/production. Point Optic at the hostname you want to capture traffic from: ``` $ optic oas capture openapi.yaml https://mastodon.green/api/ » Proxy running on https://localhost:8000. System proxy updated help Press [ Enter ] to finish capturing requests ⠼ 0 requests captured ``` At this point my computer knows to shove all traffic through localhost:8000, which you can confirm by looking at Settings > Network > Proxies. So long as your browser is detecting system proxy settings instead of using its own configuration then you should be capturing requests. To test its working, head over to `https://mastodon.green/philsturgeon` and start clicking around on every button on the website. Fill out every form. Edit things. Delete things. Try and get as much API functionality involved as possible, so that Optic can build up the best representation of a superset of all the traffic is spots. After a while I felt like I'd done everything I could, so I went back to the Optic CLI tab and hit Enter to end the session. It let me know to run `optic oas verify openapi.yaml`, which lets you see all the traffic its recorded, then as I was happy with that I ran this command to save the output: ``` optic oas verify openapi.yaml --document all ``` This tells Optic to populate the `openapi.yaml` document with everything it knows about your API! Mine looked like this: ```yaml openapi: 3.1.0 info: title: Untitled service version: 1.0.0 x-optic-path-ignore: - "**/*.+(ico|png|jpeg|jpg|gif)" paths: /v1/custom_emojis: get: responses: "200": description: 200 response content: application/json; charset=utf-8: schema: $ref: "#/components/schemas/GetApiV1Custom_emojis200ResponseBody" # ... snip ... components: schemas: GetApiV1Custom_emojis200ResponseBody: type: array items: type: object properties: shortcode: type: string url: type: string static_url: type: string visible_in_picker: type: boolean category: type: string required: - shortcode - url - static_url - visible_in_picker - category ``` Brilliant! That's a whole lot of OpenAPI I don't need to write, and Optic is even making components to [keep your OpenAPI Dry](https://blog.stoplight.io/keeping-openapi-dry-and-portable?ref=apisyouwonthate.com). One problem I had was that I ended up loads of excess URLs, like `/settings` and `/packs/{pack}/153-c1b3066c820d0c5de2e2.chunk.js` which I am not interested in. Getting rid of these is easy enough. For starters I should add my server URL, with the base path of `/api` set to let Optic know it should ignore things outside of that path. I can add that in `openapi.yaml`, then delete all the irrelevant endpoints. ```yaml servers: - description: Production url: https://mastodon.green/api ``` With that servers definition added I removed all the '/api\` prefixes from the paths, and reran capture. Due to the way Optic patches the OpenAPI document, I had no problems rewriting the OpenAPI, and it immediately picked up on what I was trying to do. Optic has inferred a lot of the what, but it cannot infer *why*, and this is where you cannot avoid spending a bit of time getting descriptions written. If you have a big enough team maybe a Technical Writer can handle this for you, maybe using a fancy GUI like Stoplight Studio, or maybe just going right into the YAML. Thankfully however you do it, Optic will happily patch that resulting OpenAPI, and the work will not be overridden. ## Sharing OpenAPI to Optic Cloud Once your OpenAPI is in a state you're happy to share, you can take the `openapi.yaml` and get it working with any of the many [OpenAPI documentation tools](https://openapi.tools/?ref=apisyouwonthate.com#documentation), but Optic has that covered too. ``` optic login ``` This will pop open your browser, and there is a "sign in with GitHub" flow. Then you'll be asked to create a Personal access token for Optic, which you can paste that the CLI. Then run this command to upload the OpenAPI document to their servers: ``` $ optic api add openapi.yaml Adding API /src/openapi.yaml ✔ Untitled service is now being tracked. View history: https://app.useoptic.com/organizations/7727d0f3-dba3-4c52-8b32-046187d60239/apis/sdCx0sapGyPyohJiKdOqB ``` Take a look at the [Mastodon API documentation](https://app.useoptic.com/organizations/7727d0f3-dba3-4c52-8b32-046187d60239/apis/sdCx0sapGyPyohJiKdOqB?ref=apisyouwonthate.com) Optic created. ## Thoughts and Feedback This is still a very new tool, and you might find a few rough edges as you go. I found a few bugs using the tool, but [Aidan Cunniffe](https://twitter.com/aidandcunniffe?ref=apisyouwonthate.com) and the Optic team are on our [Slack channel](http://slack.apisyouwonthate.com/?ref=apisyouwonthate.com) and released bug fixes quickly. Optic has mixed results with [different browsers](https://www.useoptic.com/docs/oas-reference/client-guides?ref=apisyouwonthate.com), and I could not get things working with Firefox. Thankfully Google Chrome works just fine, so I could easily switch browsers for this work. You can also attack the API directly with Postman, Curl, Insomnia, Paw, etc., which is pretty handy if you don't have a frontend you can use to make the requests. My biggest gripe with the quality of the OpenAPI being output at this point is the lack of examples. Optic will happily look at actual JSON values coming through to infer the type might be e.g.: "string", but it forgets the value it saw entirely instead of saving it as an example. There's a lot of ways to [handle examples in OpenAPI](https://phil.tech/2020/openapi-examples/?ref=apisyouwonthate.com), but Optic does not seem to leverage any of them. No media type examples, schema examples, or property examples. Without any examples, the documentation it produced can feel rather lacking, so I really hope that gets added in the future. All in all this is a really handy tool which can quickly get you an OpenAPI document, without having to litter your code with annotations which may or may not be accurate. You'll definitely want to do some work to get it up to scratch before going live, but it reduces the human error of having to manually type thousands of lines of YAML out, or click 1000 buttons in a GUI. I don't think of this as a replacement to the API Design First approach, but it's a brilliant second best when you've got to get some OpenAPI out fast. Beyond the scope of this article, but Optic does a bunch of other cool change detection stuff too, so it's worth having a poke around the [documentation](https://www.useoptic.com/docs/?ref=apisyouwonthate.com) to see if any of that can fit in with your API workflow.