# APIs You Won't Hate > The largest community for API Devs on the web. Public Ghost content for AI and LLM tooling. This file includes a bounded export of public pages first, then recent public posts. Append `.md` to any post or page URL to get the content in Markdown (for example, `/example-post.md`). ## Pages ### Books for API Developers URL: https://apisyouwonthate.com/books/ Last updated: 2023-03-25T13:32:53.000Z ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2023/03/build-apis-you-wont-hate.jpg) #### Build APIs You Won't Hate Everyone and their dog wants an API, so you should probably learn how to build them. [Get the book](https://apisyouwonthate.com/books/build-apis-you-wont-hate/) --- ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2023/03/surviving-other-peoples-apis--1-.jpg) #### Surviving Other People's APIs Designing the world's most beautiful API is only half the story, somebody needs to interact with it! [Get the book](https://apisyouwonthate.com/books/surviving-other-peoples-apis) ## ### Build APIs You Won't Hate URL: https://apisyouwonthate.com/build-apis-you-wont-hate-page/ Last updated: 2023-03-25T13:26:54.000Z ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2023/03/build-apis-you-wont-hate-1.jpg) Everyone and their dog wants an API, so you should probably learn how to build them. Tasked with building an API for your company but don't have a clue where to start? Taken over an existing API and hate it? Built your own API and still hate it? This book is for you. This book has been up on the LeanPub top 10 for most of its lifetime, and was the #5 Best Selling Book of 2015, so it can't be too bad. [Buy eBook with Shopify](https://apisyouwonthate.myshopify.com/cart/46864794762:1?channel=buy%5Fbutton&ref=apisyouwonthate.com) Also available on these sites *(but they take a bigger cut of sales):* [Amazon US](https://www.amazon.com/Build-APIs-You-Wont-Hate/dp/0692232699/?ref=apisyouwonthate.com) [Amazon UK](https://www.amazon.co.uk/dp/0692232699/?ref=apisyouwonthate.com) [Amazon FR](https://www.amazon.fr/Build-APIs-You-Wont-Hate/dp/0692232699/?ref=apisyouwonthate.com) [Amazon DE](https://www.amazon.de/Build-APIs-You-Wont-Hate/dp/0692232699/?ref=apisyouwonthate.com) --- API development is increasingly common for server-side developers thanks to the rise of front-end JavaScript frameworks, iPhone applications, and API-centric architectures. It might seem like grabbing stuff from a data source and shoving it out as JSON would be easy, but surviving changes in business logic, database schema updates, new features, or deprecated endpoints can be a nightmare. After finding many of the existing resources for API development to be lacking, Phil learned a lot of things the hard way through years of trial and error. This book aims to condense that experience, taking examples and explanations further than the trivial apples and pears nonsense tutorials often provide. Phil worked primarily as an API developer for the last three years. One horror was managing an API built in FuelPHP by a freelancer at the million dollar startup he joined. It was utilizing a then deprecated ORM which had been hacked to death by the previous developer, so took the time to delete that mess and build the next version in Laravel, leveraging it's simple routing, database migrations, schema, seeding, etc. When the following major version of the API was built no rewrite was required, and both managed to live side-by-side on the same "API" servers. By passing on some best practices and general good advice you can hit the ground running with API development, combined with some horror stories and how they were overcome/avoided/averted. This book will discuss the theory of designing and building APIs in any language or framework, with this theory applied in PHP-based examples. Some of the more advanced topics covered here are endpoint testing, embedding data objects in a consistent and scalable manner, paginating responses (including embedded objects) and hypermedia "HATEOAS" controls. ## Get *Build APIs You Won't Hate* ### You'll be building better APIs in no time [Buy the book!](https://apisyouwonthate.myshopify.com/cart/46864794762:1?channel=buy%5Fbutton&ref=apisyouwonthate.com) ### Surviving Other People's APIs URL: https://apisyouwonthate.com/surviving-other-peoples-apis-page/ Last updated: 2024-09-23T20:27:36.000Z ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2023/03/surviving-other-peoples-apis--1--2.jpg) Designing the world's most beautiful API is only half the story, somebody needs to interact with it! API Developers focus so much on designing and building their APIs, yet often we seem to forget the folks on the other end of the line. You, the frontend and backend developers trying to integrate our data and functionality into your own work, often get left with junky docs, or are just assumed to know how things are going to work. [Preorder on Leanpub](https://leanpub.com/surviving-other-peoples-web-apis?ref=apisyouwonthate.com) --- Frontend engineers work magic. They deal with all sorts of awful browser stuff, and have to deal with all sorts of ginormously complex problems. Most of those problems seem to be browser compatibility and Webpack, but there's a lot of documentation around how most of these things work. One area where frontend engineers are often left entirely in the dark? Interacting with APIs. Following the success of [Build APIs You Won’t Hate](https://apisyouwonthate.com/books/build-apis-you-wont-hate), this book will take a non-academic, easy-to-read approach to some pretty complex topics around HTTP interactions, versioning, client-caching, state management, differences between how you interact with RPC, REST and GraphQL, using JSON Schema for local validation, and all sorts of other awesome stuff that nobody ever bothered to mention. This book is aimed at frontend engineers (web, mobile, whatever) and backend engineers that find themselves talking to other APIs a lot, which is probably most of us these days. ## Preorder *Surviving Other People's APIs* ### Get instant access to the book as we write it [Preorder on Leanpub](https://leanpub.com/surviving-other-peoples-web-apis?ref=apisyouwonthate.com) ### Join the APIs You Won't Hate Slack Community URL: https://apisyouwonthate.com/community/ Last updated: 2024-01-17T02:00:57.000Z Join the APIs You Won't Hate Slack Community _This page is for subscribers only._ ### APIs You Won't Hate (the podcast) URL: https://apisyouwonthate.com/podcast-bak/ Last updated: 2024-12-18T12:28:59.000Z ## Got a question for us? We'd love to hear from you! [Submit your question for the show](mailto:mike@apisyouwonthate.com), and we'll do our best to answer on an upcoming episode. ## Thanks for listening! [ ![goodpods top 100 open source podcasts](https://storage.googleapis.com/goodpods-images-bucket/leaderboard_badges/technology_open-source_top1_month.png) ](https://goodpods.com/leaderboard/top-100-shows-by-category/technology/open-source?indie=false&period=month&ref=apisyouwonthate.com#54048873) [Goodpods Top 100 Open Source Podcasts ](https://goodpods.com/leaderboard/top-100-shows-by-category/technology/open-source?ref=apisyouwonthate.com) [ ![goodpods top 100 php podcasts](https://storage.googleapis.com/goodpods-images-bucket/leaderboard_badges/overall_top1_week.png) ](https://goodpods.com/leaderboard/top-100-shows-by-category/technology/php?indie=false&period=week&ref=apisyouwonthate.com#54052094) [Goodpods Top 100 PHP Podcasts ](https://goodpods.com/leaderboard/top-100-shows-by-category/technology/php?ref=apisyouwonthate.com) [ ![goodpods top 100 ruby podcasts](https://storage.googleapis.com/goodpods-images-bucket/leaderboard_badges/technology_ruby_top1_month.png) ](https://goodpods.com/leaderboard/top-100-shows-by-category/technology/ruby?indie=false&period=month&ref=apisyouwonthate.com#54061732) [Goodpods Top 100 Ruby Podcasts ](https://goodpods.com/leaderboard/top-100-shows-by-category/technology/ruby?ref=apisyouwonthate.com) ### About APIs You Won't Hate URL: https://apisyouwonthate.com/about/ Last updated: 2023-03-25T13:31:25.000Z API development is a topic very close to our hearts. APIs You Won't Hate started out as a book, with founder [Phil Sturgeon](https://phil.tech/?ref=apisyouwonthate.com) pouring everything API related he knew, all the problems he faced, all the design decisions he wish he thought about earlier. Phil soon teamed up with cofounder [Mike Bifulco](https://mikebifulco.com/?ref=apisyouwonthate.com), a developer advocate and startup founder, to build the APIs You Won't Hate community. Since the first book, APIs You Won't Hate has expanded to include many articles about API development, a podcast, several additional books, and a fantastic [community](https://apisyouwonthate.com/community) of API developers. Our goal is simple: provide a space for this brilliant community to debate and share experiences knowledge with other smart people. APIs You Won't Hate is dedicated to learning, writing, sharing ideas and bettering understanding of API practices. Together we can eradicate APIs we hate. ### Our Code of Conduct URL: https://apisyouwonthate.com/conduct/ Last updated: 2023-03-26T15:34:12.000Z ## Our Pledge In the interest of fostering an open and welcoming environment, we as contributors and maintainers pledge to make participation in our project and our community a harassment-free experience for everyone, regardless of age, body size, disability, ethnicity, sex characteristics, gender identity and expression, level of experience, education, socio-economic status, nationality, personal appearance, race, religion, or sexual identity and orientation. ## Our Standards Examples of behavior that contributes to creating a positive environment include: - Using welcoming and inclusive language - Being respectful of differing viewpoints and experiences - Gracefully accepting constructive criticism - Focusing on what is best for the community - Showing empathy towards other community members Examples of unacceptable behavior by participants include: - The use of sexualized language or imagery and unwelcome sexual attention or advances - Trolling, insulting/derogatory comments, and personal or political attacks - Public or private harassment - Publishing others’ private information, such as a physical or electronic address, without explicit permission - Other conduct which could reasonably be considered inappropriate in a professional setting ## Our Responsibilities Project maintainers are responsible for clarifying the standards of acceptable behavior and are expected to take appropriate and fair corrective action in response to any instances of unacceptable behavior. Project maintainers have the right and responsibility to remove, edit, or reject comments, commits, code, wiki edits, issues, and other contributions that are not aligned to this Code of Conduct, or to ban temporarily or permanently any contributor for other behaviors that they deem inappropriate, threatening, offensive, or harmful. ## Scope This Code of Conduct applies within all project spaces, and it also applies when an individual is representing the project or its community in public spaces. Examples of representing a project or community include using an official project e-mail address, posting via an official social media account, or acting as an appointed representative at an online or offline event. Representation of a project may be further defined and clarified by project maintainers. ## Enforcement Instances of abusive, harassing, or otherwise unacceptable behavior may be reported by contacting the project team at [phil+coc@apisyouwonthate.com](mailto:phil+coc@apisyouwonthate.com). All complaints will be reviewed and investigated and will result in a response that is deemed necessary and appropriate to the circumstances. The project team is obligated to maintain confidentiality with regard to the reporter of an incident. Further details of specific enforcement policies may be posted separately. Project maintainers who do not follow or enforce the Code of Conduct in good faith may face temporary or permanent repercussions as determined by other members of the project’s leadership. ## Attribution This Code of Conduct is adapted from the [Contributor Covenant](https://www.contributor-covenant.org/?ref=apisyouwonthate.com), version 1.4, available at [https://www.contributor-covenant.org/version/1/4/code-of-conduct.html](https://www.contributor-covenant.org/version/1/4/code-of-conduct?ref=apisyouwonthate.com). For answers to common questions about this code of conduct, see [https://www.contributor-covenant.org/faq](https://www.contributor-covenant.org/faq?ref=apisyouwonthate.com). ### Support APIs You Won't Hate by becoming a member URL: https://apisyouwonthate.com/membership/ Last updated: 2025-03-05T11:06:00.000Z Our dedicated team of three has been working tirelessly to build a community resource for API Developers. APIs You Won't Hate produces **books, blog posts, videos, podcasts, a bi-monthly newsletter, a Slack community, and open-source software**—all with the aim of fostering growth and innovation within the API community. Your support means the world to us. It enables us to continue producing high-quality content, maintaining our platforms, and developing new and innovative resources for API developers worldwide. By becoming a monthly member, you're helping to ensure the longevity and sustainability of APIs You Won't Hate, allowing us to reach new heights and expand our offerings. ## Free membership perks By signing up for our **free membership**, you'll get access to: - 5,400+ of your API-building peers in our [Slack Community](https://apisyouwonthate.com/community), the largest on the web - The *APIs You Won't Hate* Newsletter, 2 editions each month ## Paid membership perks As a token of our appreciation, ***paid* members** receive all of the above, plus exclusive access to: - Discounts on [**books**](https://apisyouwonthate.com/books) we publish, like [*Surviving Other People's APIs*](https://apisyouwonthate.com/books/surviving-other-peoples-apis/)and the upcoming [*Build APIs You Won't Hate 2: Oops I Did It Again*](https://leanpub.com/build-apis-you-wont-hate-2/?ref=apisyouwonthate.com)*.* - Access to members-only videos - New articles emailed straight to your inbox - A personal shout-out on the podcast --- # Our Community, by the numbers - **API Devs in our Slack community :** 5,400+ - **Newsletter subscribers:** 2,500+ - **Monthly unique website visitors:** 6,500 - **Podcast Subscribers:** 500-1,000 (estimated) ## Monthly membership and sponsor income Thank you to everyone who has supported APIs You Won't Hate since our inception. In the spirit of transparency, we've decided to share our monthly income from memberships and [sponsors](https://apisyouwonthate.com/sponsor). 💰 As of Jan 17, 2024, we make ****$26.50/mo** from paid memberships. We make ****$1500/mo** through sponsorships here and on openapi.tools ****MRR: $1526.49** We love this community dearly, and appreciate your support. Thanks so much for making APIs You Won't Hate what it is today! [Become a member](https://apisyouwonthate.com/#/portal) If you're working on a product that is geared toward API designers and developers, we'd love to talk to you about [sponsoring APIs You Won't Hate](https://apisyouwonthate.com/sponsor). ❤️ Thank you for your support! ## Support APIs You Won't Hate When you become an member, you'll get access to members-only content while directly supporting our work. Your support helps us to keep making resources for the API community. [Become a member today](https://apisyouwonthate.com/#/portal) ### API Glossary URL: https://apisyouwonthate.com/api-glossary/ Last updated: 2024-05-16T20:27:47.000Z The API Glossary is a resource for educational purposes, and an attempt to align API tooling vendors on a single vocabulary for talking about the same thing with the same words instead of confusing users with a million names for the same thing. ## API [](https://github.com/apisyouwonthate/api-glossary?ref=apisyouwonthate.com#api) The general definition "Application Program Interface" can mean a lot of things, with different types of interface. OpenAPI is specifically talking about HTTP-based APIs, which in general is anything REST, RESTish, and many generic types of RPC. ## API-First [](https://github.com/apisyouwonthate/api-glossary?ref=apisyouwonthate.com#api-first) When APIs first started getting popular, a lot of companies started flopping them out as a marketing ploy. These APIs were generally not particularly useful, rarely performant, and frustrating to work with. "API First" is a concept that teams should build the APIs first, and make them something they would want to work with, therefore solving useability issues, performance issues, etc. and supposedly creating a better experience for their customers. ## API Description [](https://github.com/apisyouwonthate/api-glossary?ref=apisyouwonthate.com#api-description) Aliases include: "API Definition" or "API Contract". An API Description is meta-data about an API, explaining what endpoints, resources, HTTP methods, headers, query parameters, etc. exist. Descriptions are written in a particular "API Description Format" (e.g.: [OpenAPI](http://spec.openapis.org/oas/v3.0.2?ref=apisyouwonthate.com), [JSON Schema](https://json-schema.org/?ref=apisyouwonthate.com), [RAML](https://raml.org/?ref=apisyouwonthate.com), [WSDL](https://www.w3.org/TR/wsdl/?ref=apisyouwonthate.com), etc.) and will usually be contained in an "API Description Document" (e.g.: `openapi.yaml`, `schemas/payment.json`) unless the authors decided to use annotations instead. ## API Design [](https://github.com/apisyouwonthate/api-glossary?ref=apisyouwonthate.com#api-design) Planning an API specifically using API Descriptions, or a tool which generates / exports API descriptions. When done before coding begins, it is known as API Design First, which is conceptually different to [API First](https://github.com/apisyouwonthate/api-glossary?ref=apisyouwonthate.com#api-first) but not mutually exclusive. More on [API Design First vs Code First](https://www.apisyouwonthate.com/blog/api-design-first-vs-code-first/?ref=apisyouwonthate.com). ## API Specification [](https://github.com/apisyouwonthate/api-glossary?ref=apisyouwonthate.com#api-specification) In the past the term API Specification was one of many terms to describe what most major tooling vendors call an [API Description Document](https://github.com/apisyouwonthate/api-glossary?ref=apisyouwonthate.com#api-description). It was the file that users would write to describe *their* API. Ask 100 people and you will get 100 answers, but consensus is forming around API Specifications being an umbrella term for various Specifications involved with API development, and more specific categories exist within the umbrella term. Message Formats like JSON-API, HAL, Siren, etc. all have a specification. Data Formats like JSON, YAML, etc all have a specification. Transfer Protocols like SPDY, HTTP/2, HTTP/3, etc have a specification. Authentication Strategies like OAuth 2, OpenID, etc all have a specification. Seeing as there as so many types of API specification floating around, to disambiguate its best to just avoid the term and use one of the more specific categories, or if talking about your own `openapi.yaml` just call it the description document. ## Bundle [](https://github.com/apisyouwonthate/api-glossary?ref=apisyouwonthate.com#bundle) Aliases include: "External Inlining". Bundling pulls in external `$refs` from different files, or URLs, and puts them into the `components` object. This is done to create a single OpenAPI file, which is easier to share, especially with tooling that does not support resolving external files. If tooling does not support `$ref` at all, then an alternative to bundling is required: [dereferencing](https://github.com/apisyouwonthate/api-glossary?ref=apisyouwonthate.com#dereference). ## Callbacks [](https://github.com/apisyouwonthate/api-glossary/blob/master/README.md?ref=apisyouwonthate.com#callbacks) Most programming languages have a concept of a callback which is a function passed as an argument to another function, which is run after another function has finished. The exact same concept exists in APIs, where a URL can be passed to be alerted when an action is complete. This is usually performed via a Webhook, so the term Webhook would be more common than Callabck, but OpenAPI specifically talks about Callbacks. *More about OpenAPI callbacks on* [*Swagger Docs: Callbacks*](https://swagger.io/docs/specification/callbacks/?ref=apisyouwonthate.com)*.* [](https://github.com/apisyouwonthate/api-glossary?ref=apisyouwonthate.com#callbacks) ## Code Generation [](https://github.com/apisyouwonthate/api-glossary?ref=apisyouwonthate.com#code-generation) ## Contract Testing [](https://github.com/apisyouwonthate/api-glossary?ref=apisyouwonthate.com#contract-testing) Confirming that an API is accepting and outputting what it says it is can take many forms. Most commonly, when you hear the term Contract Testing in relation to OpenAPI, people are talking about producer contract testing. This means the API development team (or some test/QA team nearby) have implemented a test suite that will [take an API description and compare it to the actual data](https://apisyouwonthate.com/blog/writing-documentation-via-contract-testing/). Conceptually contract testing is the same thing as [data validation](https://github.com/apisyouwonthate/api-glossary?ref=apisyouwonthate.com#data-validation), but done for different purposes at different parts of the life-cycle. The same tools could be used under the hood. There is also client-side contract testing, where you write out the contract in a client test suite then make calls to the API to see if it matches. ## Converter [](https://github.com/apisyouwonthate/api-glossary?ref=apisyouwonthate.com#converter) A set of tools that allow you to convert OpenAPI descriptions to another description format, such as API Blueprint, HAR, RAML, etc., or vice versa! Some will also convert to HTML which is usually for making [reference documentation](https://github.com/apisyouwonthate/api-glossary?ref=apisyouwonthate.com#reference-documentation). ## Data Instance [](https://github.com/apisyouwonthate/api-glossary?ref=apisyouwonthate.com#data-instance) ## Data Validation [](https://github.com/apisyouwonthate/api-glossary?ref=apisyouwonthate.com#data-validation) Often referred to simple as "validation" when there are many types of validation. Data validation is basically taking some real data (maybe an entire HTTP Message with a body, or just the body) and comparing it to the API description. This is then used to power things like [contract testing](https://github.com/apisyouwonthate/api-glossary?ref=apisyouwonthate.com#contract-testing), [gateway validation](https://github.com/apisyouwonthate/api-glossary?ref=apisyouwonthate.com#gateway-validation), and [server-side validation](https://github.com/apisyouwonthate/api-glossary?ref=apisyouwonthate.com#server-side-validation). ## Dereference [](https://github.com/apisyouwonthate/api-glossary?ref=apisyouwonthate.com#dereference) Aliases include: "Dereferencing", "Internal Inlining" or "Transclusion". All `$ref`s and replaced with their values à la copy & paste. No more `$ref`'s exist in the file/object representation. If you have 10 operations referencing the same model 10 times, you now have 10 different models. ## Description Validation [](https://github.com/apisyouwonthate/api-glossary?ref=apisyouwonthate.com#description-validation) Aliases include: "Schema Validation". A very different type of validation to [data validation](https://github.com/apisyouwonthate/api-glossary?ref=apisyouwonthate.com#data-validation). Description validation focuses on making sure the [description](https://github.com/apisyouwonthate/api-glossary?ref=apisyouwonthate.com#api-description) is correct against the OpenAPI Specification itself. When a validator moves beyond checking the document is valid, and into custom rules, design opinions, style, etc. the term "linting" is used. *A list of description validators is available on* [*OpenAPI.Tools*](https://openapi.tools/?ref=apisyouwonthate.com#description-validation)*.* ## Documentation [](https://github.com/apisyouwonthate/api-glossary?ref=apisyouwonthate.com#documentation) The most common meaning of documentation in OpenAPI-world is [reference documentation](https://github.com/apisyouwonthate/api-glossary?ref=apisyouwonthate.com#reference-documentation), but there is a lot more to good API documentation than just that. Guides, tutorials, and other types of how-to content is often combined with reference documentation to provide the best developer experience. This combination along with "sign up for API tokens" type functionality is often referred to as a Developer Portal. ## Gateway Validation [](https://github.com/apisyouwonthate/api-glossary?ref=apisyouwonthate.com#gateway-validation) Data Validation leverages in an API Gateway is known as gateway validation, and it will stop invalid requests from wasting application resources. AWS, Tyk, Express Gateway, etc. support some level of OpenAPI or JSON Schema gateway validation. ## Links [](https://github.com/apisyouwonthate/api-glossary?ref=apisyouwonthate.com#links) Links can mean a lot of things in the concept of computing and APIs, but in OpenAPI there is a [Link Object](http://spec.openapis.org/oas/v3.0.2?ref=apisyouwonthate.com#link-object), which represents a possible design-time link for a response. This can be handy for making API documentation a bit less RPC and a bit more REST, by giving hints as to next available actions for any response. It gets a bit more like Hypermedia Controls (a.k.a HATEOAS) if used at runtime, as once a client has a response, the headers, and data instance can be combined with the links to figure out next available actions. ## Linting [](https://github.com/apisyouwonthate/api-glossary?ref=apisyouwonthate.com#linting) OpenAPI linter like [Spectral](https://stoplight.io/open-source/spectral?ref=apisyouwonthate.com) can confirm if API descriptions are valid, but also if they match predetermined rules, like a style guide. Style Guides can be defined by API Governance teams at larger companies, or be shared, like the ones on [openapi-contrib/style-guides](https://github.com/openapi-contrib/style-guides?ref=apisyouwonthate.com). ## Mocking [](https://github.com/apisyouwonthate/api-glossary?ref=apisyouwonthate.com#mocking) A fake server that takes a description document as input, then routes incoming HTTP requests to example responses or dynamically generates examples. *A list of mock servers is available on* [*OpenAPI.Tools*](https://openapi.tools/?ref=apisyouwonthate.com#mock-servers)*.* ## OAS [](https://github.com/apisyouwonthate/api-glossary?ref=apisyouwonthate.com#oas) This is just shorthand for the [OpenAPI Specification](https://github.com/apisyouwonthate/api-glossary?ref=apisyouwonthate.com#openapi), which is [Markdown files on the internet](https://github.com/OAI/OpenAPI-Specification/tree/master/versions?ref=apisyouwonthate.com) defining how each version of OpenAPI should work. ## OpenAPI [](https://github.com/apisyouwonthate/api-glossary?ref=apisyouwonthate.com#openapi) Aliases include "OpenAPI Specification". The new name of the [API Description](https://github.com/apisyouwonthate/api-glossary?ref=apisyouwonthate.com#api-description) Format, which is defined in an [API Specification](https://github.com/apisyouwonthate/api-glossary?ref=apisyouwonthate.com#api-specification). It used to be called [Swagger](https://github.com/apisyouwonthate/api-glossary?ref=apisyouwonthate.com#swagger). ## The OpenAPI Initiative [](https://github.com/apisyouwonthate/api-glossary?ref=apisyouwonthate.com#the-openapi-initiative) The [OpenAPI Initiative](https://www.openapis.org/?ref=apisyouwonthate.com) (OAI) are the group in control of the development of the OpenAPI Specification ([OAS](https://github.com/apisyouwonthate/api-glossary?ref=apisyouwonthate.com#oas)). ## Open API [](https://github.com/apisyouwonthate/api-glossary?ref=apisyouwonthate.com#open-api) An open API is a public API. When referring to the [OpenAPI Specification](https://github.com/apisyouwonthate/api-glossary?ref=apisyouwonthate.com#oas), community or the [OpenAPI Initiative](https://github.com/apisyouwonthate/api-glossary?ref=apisyouwonthate.com#the-openapi-initiative), there is no space in "OpenAPI". ## Parameters [](https://github.com/apisyouwonthate/api-glossary?ref=apisyouwonthate.com#parameters) Parameters is a general catch-all term for headers, path parameters, query string parameters, and in OpenAPI v2 it also included the request body (like form data). *More about* [*Parameters*](http://spec.openapis.org/oas/v3.0.2?ref=apisyouwonthate.com#parameter-object) *in the OpenAPI v3.0 specification.* ## Reference Documentation [](https://github.com/apisyouwonthate/api-glossary?ref=apisyouwonthate.com#reference-documentation) Very similar to the sort of "Classes, Methods, Constants" documentation you're used to seeing for code libraries, modules, packages, explaining the various inputs and outputs. Reference Documentation is a rendering of the API Description in HTML (or maybe a PDF) so slightly less technical people can figure out how to work with the API. ## Reference [](https://github.com/apisyouwonthate/api-glossary?ref=apisyouwonthate.com#reference) A [JSON Reference](https://tools.ietf.org/html/draft-pbryan-zyp-json-ref-03?ref=apisyouwonthate.com), which can point to a JSON structure in a different location. They may point to objects inside of the current file, but may also refer to other, possibly remote files. They live inside the Reference Object, with the key `$ref`. Here are some examples: - Reference Object: `{"$ref": "https://example.com/api/openapi.yaml#/components/schemas/Pet"}` - JSON Reference: `https://example.com/api/openapi.yaml#/components/schemas/Pet` - [JSON Pointer](https://tools.ietf.org/html/rfc6901?ref=apisyouwonthate.com) (to the location in the referenced file): `/components/schemas/Pet` ## Resolve [](https://github.com/apisyouwonthate/api-glossary?ref=apisyouwonthate.com#resolve) Looking for the value found at the end of a [$ref](https://github.com/apisyouwonthate/api-glossary?ref=apisyouwonthate.com#reference), but no changes are made to the file or object being resolved. ## Schema [](https://github.com/apisyouwonthate/api-glossary?ref=apisyouwonthate.com#schema) Aliases include "Data Model". A schema is metadata, which describes the data type, and other properties about the ata like a specific format, and validation rules. JSON Schema is one example of a schema, but OpenAPI has it's own flavour of JSON Schema which it uses for the various locations a `schema` keyword can exist. Schema is most commonly associated with describing the body of a HTTP request or response, but it can also describe various [parameters](https://github.com/apisyouwonthate/api-glossary?ref=apisyouwonthate.com#parameters) like headers or path parameters. ## SDK [](https://github.com/apisyouwonthate/api-glossary?ref=apisyouwonthate.com#sdk) "Software Development Kits" are a generic term in computer-land but in the context of Web APIs and OpenAPI in particular, they usually mean some sort of client library for other developers to interact with an API at a programming language level, and not a HTTP library level. *A list of SDK generators is available on* [*OpenAPI.Tools*](https://openapi.tools/?ref=apisyouwonthate.com#sdk)*.* ## Server-side Validation [](https://github.com/apisyouwonthate/api-glossary?ref=apisyouwonthate.com#server-side-validation) Using the concept of [data validation](https://github.com/apisyouwonthate/api-glossary?ref=apisyouwonthate.com#data-validation) to More on [Server-side Validation](https://www.apisyouwonthate.com/blog/server-side-validation-with-api-descriptions/?ref=apisyouwonthate.com). ## Swagger [](https://github.com/apisyouwonthate/api-glossary?ref=apisyouwonthate.com#swagger) Historically, "Swagger" was the original name of OpenAPI Specification (OAS). It was called Swagger when it was released in 2011, and when SmartBear acquired Swagger Specification they kept the name for a while, and made a bunch of tools with the name Swagger in it. Swagger Editor, Swagger Inspector, SwaggerHub, etc. Once the Swagger specification was given to [Open API Initiative](https://github.com/apisyouwonthate/api-glossary?ref=apisyouwonthate.com#the-openapi-initiative) in 2016, the name was changed to OpenAPI. Now, the word "Swagger" is just part of the SwaggerHub brand of tooling. The specification is "OpenAPI" (not OpenAPI/Swagger). ## Posts ### APIs Are Having a Moment URL: https://apisyouwonthate.com/newsletter/apis-are-having-a-moment/ Last updated: 2026-09-01T15:47:26.000Z API's are once again having their moment in the sun; people don't just want to go to a web app; they want your data to plug into an MCP or this graph or that connector. OpenAPI specs and design-first are also being dragged front and centre. Well-defined relationships and good documentation are a focal point when Agents get involved. The interesting trend I am seeing is the push to do insights and aggregations inside the API so they can be tested and validated, rather than letting Agents draw their own conclusions. I've suddenly found time-series and statistical APIs a focus point. While I find the negative effects of AI often overwhelming, I do enjoy the push for better, more complete APIs. \-- Alexander --- ## The API Roundup *API News, links, and tools from around the web* ### [Use Arazzo ](https://apisyouwonthate.com/newsletter/throwaway-tooling/) A JaveScript/TypeScript toolkit for Arazzo. It lets you validate documents and run workflows against live APIs. It has a great CLI and an MCP compiler on the roadmap. If you're still unsure what Arazzo is, we've got you covered with a nice [intro blog](https://apisyouwonthate.com/blog/introducing-arazzo-describe-api-workflows/). ### [Karve, a native Windows workspace for your .http files](https://karve.dev/?utm%5Fsource=apisyouwonthate&utm%5Fmedium=newsletter&utm%5Fcampaign=classified) Karve gathers the .http files scattered across your repos into one native Windows workspace without moving them. Run requests and inspect responses without opening an IDE. There’s a 15-day free trial, and if you keep it, you pay once. `sponsored` ### [MCP Is The New REST](https://www.youtube.com/watch?v=4EK3usamWOc&list=PLGZGLmEvGK8I&index=9&ref=apisyouwonthate.com) Like I said in the intro API's are having a new moment in the sun, in part due to MCP, but is wrapping your API in MCP the right way? Andy put together a great talk at GopherCon about what MCP is and how TypeFrom is looking beyond REST as MCP becomes the new integration surface. Plus, it has Back to the Future memes. What more could you ask for? ### [A Deep Dive Into Hono](https://flaviocopes.com/how-hono-is-built/?ref=frontenddogma.com) Hono is a lightweight web framework built on web standards. Honestly, one of my favourites when building on Cloudflare. Check out this awesome post on how a request runs through Hono via routing, middleware, context, web standards, and runtime adapters. The quote at the start really sticks with me: > I like frameworks that disappear when I look closely at them ### [OpenAPI React Query Codegen](https://socket.dev/blog/openapi-react-query-codegen-npm-compromise?ref=apisyouwonthate.com) A codegen tool for turning OpenAPI specs into React Query code was hit with a supply chain attack. Socket has a great write-up covering the ten compromised versions that were published. Maybe we should go back to building every inch of our software ourselves to avoid security issues 😂. ### [FastAPI Conf '26](https://fastapiconf.com/?ref=apisyouwonthate.com) The FastAPI Conf is coming to Amsterdam this October (honestly the best month of the year). You can still get your tickets now, with great speakers from Google DeepMind, OpenAI, and my favorite Space Telescope Science Institute. Who doesn't like space and telescopes....? --- ## APIs You Won't Hate *The latest from the team at API's You Won't Hate.* ### Join us on the Podcast We are lining up the next series of APIs You Won't Hate (the [podcast](https://apisyouwonthate.com/tag/podcast/)) - and we're looking for guests! If you (or someone you know) is working on an interesting API project, product, or initiative, [drop us a line](mailto:mike@apisyouwonthate.com). --- ## From the Community *Articles written and shared in our free* [*Slack community*](https://slack.apisyouwonthate.com/?ref=apisyouwonthate.com)*.* ### [Can MCP Server Finally Get Their Own OpenAPI](https://apichangelog.substack.com/p/can-mcp-servers-finally-get-their-own-openapi) What does an MCP server do? You don't know until you connect and run a command. Maybe it's time they get their own documentation standard. Bruno is trying out [mcpdesc](https://mcpdesc.org/?ref=apisyouwonthate.com) and explains why. ## Support APIs You Won't Hate When you become an member, you'll get access to members-only content while directly supporting our work. Your support helps us to keep making resources for the API community. [Become a member today](https://apisyouwonthate.com/#/portal) Thanks so much to our members: **Kin**, **Juxt, Alex R, Nolan S, Brandon K, Frank, James D, Bill D,** and **Abdelhadi D**. Your support means the world to us! ✌️ Until next time, Alexander, Phil & Mike ### Throwaway Tooling URL: https://apisyouwonthate.com/newsletter/throwaway-tooling/ Last updated: 2026-08-17T12:50:39.000Z Over the last month, I have noticed an explosion of new tooling and libraries. I attribute a lot of this to LLM-driven development. What is also interesting is seeing these tools gain traction for a short time and then, for some, fizzle out. Combine this with the burnout we are seeing across the open source community I wonder where this leads. Can I safely rely on a tool? Is this something I can recommend? People are working to [solve burnout issues in OpenSource](https://stewardshiplab.org/?ref=apisyouwonthate.com), but even if that were solved, the increase in tooling is still an interesting question. Sure, more competition and ideas are a good thing, and throwaway software has its place, but there are still downsides here. I also wonder if it's time to add age and release frequency to things like [OpenAPI.tools](https://openapi.tools/?ref=apisyouwonthate.com). An interesting time to be building software, for sure. \-- Alexander ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2026/08/darksvixsquare-1.jpg) ## Never deal with flaky webhook integrations again. If you're getting pinged at 4 in the morning for failed webhook deliveries, this is for you. Svix handles retries, idempotency, and security. Qualified startups get $12,000 in free credits. [Get Started for Free](https://link.svix.com/APIsywh?ref=apisyouwonthate.com) ## The API Roundup *API News, links, and tools from around the web* ### [Doing GraphQL Is Governance by Default](https://apievangelist.com/2026/08/06/doing-graphql-is-governance-by-default/?ref=apisyouwonthate.com) A reflection on GraphQL and what actually makes it better than REST. GraphQL forces you to describe every type, every field and relationship. You can't ship a query until you do this. For REST, on the other hand, you can ship an endpoint straight away. Governance is something you must choose to do. The post raises a good point; however, I've seen many people half-ass a GraphQL schema, which led to major rewrites. Humans will always find a way to be lazy 😜. ### [Fastify V6 ](https://github.com/fastify/fastify/releases/tag/v6.0.0-alpha.0?ref=apisyouwonthate.com) A popular Node.js web framework, Fastify version six, is now in Alpha. You can see a full list of what will be included in the [V6 Planning Issue](https://github.com/fastify/fastify/issues/6834?ref=apisyouwonthate.com) on GitHub. Great to see conversations in here about reducing the MB shipped as well ### [Miasma Supply Chain Attack on AsyncAPI](https://www.asyncapi.com/blog/miasma-supply-chain-attck?ref=apisyouwonthate.com) Can't believe I missed this one, but there was a supply chain attack on AsyncAPI back in July. This was a well-coordinated attack that used PR spam and exploited an unmerged security fix. The corrupted version was live for about four hours before it was taken down. Read the blog post for the full details and takeaways from the incident. ### [API Days](https://www.apidays.global/?ref=apisyouwonthate.com) API Days has a few conferences coming up. They're in India in August and September; they are heading to Canada and England, and finally Australia in October. For some of these, they are still accepting talk submissions, so if you have something in mind, there's time to submit before it closes. ### [Browser to API](https://www.skills.sh/browserbase/skills/browser-to-api?ref=apisyouwonthate.com) An interesting skill from Vercel's skill registry that consumes a browser trace to build an OpenAPI spec. Very useful for capturing an undocumented API, schemas or generating a coverage report. ### [OpenAPI Overlays for MCP and AI-Agent Enrichment](https://apievangelist.com/2026/08/11/openapi-overlays-for-mcp-and-ai-agent-enrichment/?ref=apisyouwonthate.com) An interesting use for Overlays: creating better context for agents. Your base OpenAPI spec stays untouched, and you apply an overlay at build time so an MCP generator picks it up. A great way to avoid maintaining two documents. --- ## APIs You Won't Hate *The latest from the team at API's You Won't Hate.* ### [OpenAPI.tools](https://openapi.tools/?ref=apisyouwonthate.com) Our repository of great tools for building with OpenAPI. It's an Open Source, community maintained list built on Astro. If you're building with OpenAPI, check it out - and if your team builds tools for OpenAPI, [pull requests](https://github.com/apisyouwonthate/openapi.tools?ref=apisyouwonthate.com) are always welcome. --- ## From the Community *Articles written and shared in our free* [*Slack community*](https://slack.apisyouwonthate.com/?ref=apisyouwonthate.com)*.* ### [APIs You (or Agents) Won't Hate](https://confengine.com/conferences/apidays-india-2026/register/selection?ref=apisyouwonthate.com#workshop-378-info) Frank Kilcommins shared his upcoming workshop at API's Day India on using specifications like OpenAPI, Overlay and Arazzo to design and govern APIs for developers and AI consumers. It's next week, so cutting it close, but there are still a few seats left. ## Support APIs You Won't Hate When you become an member, you'll get access to members-only content while directly supporting our work. Your support helps us to keep making resources for the API community. [Become a member today](https://apisyouwonthate.com/#/portal) Thanks so much to our members: **Kin**, **Juxt, Alex R, Nolan S, Brandon K, Frank, James D, Bill D,** and **Abdelhadi D**. Your support means the world to us! ✌️ Until next time, Alexander, Phil & Mike ### Quick: Expose Every Endpoint! URL: https://apisyouwonthate.com/newsletter/quick-expose-every-endpoint/ Last updated: 2026-08-06T14:06:08.000Z I find myself in a weird bind recently. AI seems to be causing everyone to throw good engineering out the window. Design, planning, architecture, and efficient software are gone in the pursuit of AI. I feel this pressure even more so with API Design, from shoving all your endpoints into MCP, to shoving multiple endpoints into one endpoint to expose via MCP, and finally exposing your DB directly to any agent. Wherever you sit on the AI spectrum, good design, sound technical choices and focusing on good outcomes and product will get you much further than ramming AI into everything as fast as possible. Take [Google Satellite AI](https://www.theverge.com/tech/973943/google-earth-ai-image-generation-deepfake-tool?ref=apisyouwonthate.com) for example, what a colossally stupid idea. \-- Alexander ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2026/08/darksvixsquare-1.jpg) ## Never deal with flaky webhook integrations again. If you're getting pinged at 4 in the morning for failed webhook deliveries, this is for you. Svix handles retries, idempotency, and security. Qualified startups get $12,000 in free credits. [Get Started for Free](https://link.svix.com/APIsywh?ref=apisyouwonthate.com) ### [I am Forking Spectral](https://apievangelist.com/2026/07/29/i-am-forking-spectral/?ref=apisyouwonthate.com) Maintenance on Spectral has been fading since SmartBear acquired Spotlight, and Kin Lane has the numbers to prove it. The repository is down from 157 issues closed in 2021 to only 10 so far in 2026\. OpenSource is rough right now, but when backed by a large org, we should expect more. After a supply chain attack, Kin has forked the repo and kept his long-standing invitation to donate it open. ### [The API Report Card](https://supergood.ai/api-report-card?ref=apisyouwonthate.com) All enterprise software graded on whether you can integrate with it or not. A human scores each platform across six areas: Existence, Access, Coverage, Auth, Docs & DX, and Stability. Fun fact: according to the site, most fail. ### [An HTTP Request's Journey Through Node](https://200ms.thenodebook.com/?ref=apisyouwonthate.com#act-0-prologue) Ishtmeet Singh does not disappoint with this fantastic blog post. An interactive journey through a HTTP request in Node triggered by someone clicking buy. An interactive clock so you can track the milliseconds through every step. A wonderful read over my morning coffee. Thank you for this, Ishtmeet. ### [Fetch Needs Error Codes](https://www.jasnell.me/posts/fetch-needs-error-codes?ref=apisyouwonthate.com) For those of you who don't know the frustration of getting a TypeError in JavaScript on a fetch request, I envy you. James Snell covers this gap in fetch and explains that with error signalling in HTTP/2 and HTTP/3, it is time for fetch to support error codes. There is an active proposal with TC39 in Stage 1 right now. ### [httpretty](https://github.com/henvic/httpretty?ref=apisyouwonthate.com) A delightful package that prints the HTTP requests of your Go applications in your terminal. Simple and easy to set up, with nicely formatted outputs. Support has recently been added for multipart bodies, colour status, and more. --- ## From the Community *Articles written and shared in our free* [*Slack community*](https://slack.apisyouwonthate.com/?ref=apisyouwonthate.com)*.* ### [AI Agents Don't Break Your APIs](https://netapinotes.com/ai-agents-dont-break-your-apis-they-just-stop-compensating-for-your-docs/?ref=apisyouwonthate.com) AI Agents aren't breaking your API's they can't compensate for your poor documentation like humans can. Another killer blog post from Matthew Reinbold on how documentation is affecting agents and humans alike, and a simple campaign to update your API docs is not going to fix it. ## Support APIs You Won't Hate When you become an member, you'll get access to members-only content while directly supporting our work. Your support helps us to keep making resources for the API community. [Become a member today](https://apisyouwonthate.com/#/portal) Thanks so much to our members: **Kin**, **Juxt, Alex R, Nolan S, Brandon K, Frank, James D, Bill D,** and **Abdelhadi D**. Your support means the world to us! ✌️ Until next time, Alexander, Phil & Mike ### Extending OpenAPI URL: https://apisyouwonthate.com/newsletter/extending-openapi/ Last updated: 2026-07-15T11:33:52.000Z If you use OpenAPI long enough, you may eventually hit a roadblock. Honestly, the main response I see is to give up and walk away, but did you know OpenAPI can be extended? [Kin Lane has a great breakdown](https://apievangelist.com/2026/06/26/tyk-and-the-quiet-superpower-of-extending-openapi/?ref=apisyouwonthate.com) of an API Gateway extending the specification. OpenAPI also has a registry for all [types of extensions](https://spec.openapis.org/registry/index.html?ref=apisyouwonthate.com). Registering something you built here also helps the OpenAPI team decide what to adopt in their roadmaps. \-- Alexander --- ### [Building Gin: Simple Over Easy](https://manualmeida.dev/articles/gin-simple-over-easy/?ref=apisyouwonthate.com) Did you know Gin was built to power a social network? Gin is one of the best HTTP frameworks for Go, created by Manu Martínez-Almeida. Manu walks us through what inspired him to build Gin and how it has since grown without him. A great read on how aiming for simplicity, not complexity, made things easy to use and for others to build on. ### [Finally, a Great Email API](https://upyo.org/?ref=apisyouwonthate.com) I honestly f\*\*\*ing hate plugging in email providers to a backend. You normally end up switching providers, encountering runtime issues, or dealing with breaking API changes. Well, no more. Finally, a unified API that runs in all JS runtimes and swapping providers is a single-line change; how I have longed for this. ### [Seeking CMMS people](mailto:apisyouwonthate.com@gregwolanski.com?subject=%5BAPIs%20You%20Won't%20Hate%5D%20CMMS%20People) Do you work with CMMS (**not** *CMS* 😄) APIs? Greg Wolanski is working on a research project and would love to learn from your experience. [Email Greg to schedule a paid interview](mailto:apisyouwonthate.com@gregwolanski.com?subject=%5BAPIs%20You%20Won't%20Hate%5D%20CMMS%20People). `sponsored` ### [Postgres to API with Prest](https://github.com/prest/prest?ref=apisyouwonthate.com) Do you need a simple API layer on top of your Postgres? Well, say hello to Prest. Smash out a quick REST API on top of your DB, a nice, quick, low-code solution. ### [Discovering features using HTTP OPTIONS](https://evertpot.com/discovering-features-with-http-options/?ref=apisyouwonthate.com) I have talked to many developers who have no idea what `OPTIONS` actually is. It has a few uses, but its main one is to help clients find out what is possible here. Evert Pot gives us a delightful walk-through on everything `OPTIONS` can do. An easy, fun read. Thank you, Evert. I enjoyed this one a lot. ### [Leaf Wiki](https://github.com/perber/leafwiki?ref=apisyouwonthate.com) Not strictly API focused, but a wonderful, easy-to-deploy docs or wiki. A single Go binary using SQLite and Markdown. No external database, it just works. They have a wonderful [little demo](https://demo.leafwiki.com/home?ref=apisyouwonthate.com) you can check out as well. ### [FastAPI Cloud](https://fastapicloud.com/?ref=apisyouwonthate.com) If you're out of the loop, a while back, FastAPI launched its own cloud platform for deploying and scaling your FastAPI applications. Recently, their website and docs got a makeover. Last time I checked in on this, it was still in beta and has come a long way since then. 🗞️ ****Have something to share?** Classified ads are short, text-only placements available for [$100 per issue](https://workspace.passionfroot.me/apisyouwonthate/packages/5cc9cd89-9bbc-430f-b81a-2536fdbe8429/book?ref=apisyouwonthate.com). ## APIs You Won't Hate *The latest from the team at API's You Won't Hate.* ### [The Strangler Fig Pattern: How to Rebuild Everything](https://mikebifulco.com/newsletter/stangler-fig-pattern?ref=apisyouwonthate.com) A crossover from Mike's 💌 *Tiny Improvements* newsletter, he talks about a software engineering pattern used when it's time to rebuild everything... responsibly. This one's broader than APIs - architecture, framework, the kitchen sink - it all fits in here. --- ## From the Community *Articles written and shared in our free* [*Slack community*](https://slack.apisyouwonthate.com/?ref=apisyouwonthate.com)*.* ### [Prototype to Operator](https://bsky.app/profile/mikebifulco.com/post/3mq2kgy4itc2c?ref=apisyouwonthate.com) Our very own Mike recently shared his thoughts on extended support in our [Slack group](https://apisyouwonthate.slack.com/archives/C0HGJS734/p1783431112604419?ref=apisyouwonthate.com). So many docs, SDKs, and APIs focus on getting you started but not on long-term use, scaling, and high-use gotchas. Personally, I chalk this up to open-source contributors having limited time, or to new customers being a priority for the business, not existing ones. He is right; however, I wish more places had time or resources for this level of care. ## Support APIs You Won't Hate When you become an member, you'll get access to members-only content while directly supporting our work. Your support helps us to keep making resources for the API community. [Become a member today](https://apisyouwonthate.com/#/portal) Thanks so much to our members: **Kin**, **Juxt, Alex R, Nolan S, Brandon K, Frank, James D, Bill D,** and **Abdelhadi D**. Your support means the world to us! ✌️ Until next time, Alexander, Phil & Mikes ### Query Is Here URL: https://apisyouwonthate.com/newsletter/http-query-is-here/ Last updated: 2026-07-01T11:58:17.000Z The new [HTTP QUERY](https://www.rfc-editor.org/rfc/rfc10008.html?ref=apisyouwonthate.com) method is here, so stop shoving read-only operations into `POST` because your query strings get messy. With `QUERY` officially being supported in OpenAPI 3.2 and many web frameworks now supporting it either directly or via custom methods, adoption has never been easier. A common misuse today is complex filters and searches being shoved into a `POST`, that works, but it hides the real semantics. `POST` does not tell clients, caches, observability tools, or retry logic that the operation is safe and idempotent. Start using `QUERY` today. \-- Alexander --- ### [OpenAPI Python](https://npmx.dev/package/@hey-api/openapi-python?ref=apisyouwonthate.com) HeyAPI is at it again, delivering more OpenAPI goodness, this time with a Python Code Generator. Their new package converts OpenAPI into production-grade SDKs or Pydantic models. Honestly, at the rate I bring up HeyAPI, I should get myself a t-shirt. ### [Unified API](https://www.unipile.com/?utm%5Fsource=linkedin&utm%5Fcampaign=passionfroot%5Fvictor-baggio) One API for LinkedIn, Instagram, Messaging, Email & Calendar - `sponsored` ### [GraphQL 17](https://graphql.org/blog/2026-06-15-introducing-graphql-js-v17/?ref=apisyouwonthate.com) Being the first major release since October 2021, GraphQL is dropping some goodies in this latest release. First class AbortSignal support, broader schema validation coverage, `GraphQLHarness` for customisation and more. A great release that really expands GraphQL. 🗞️ ****Have something to share?** Classified ads are short, text-only placements available for [$100 per issue](https://workspace.passionfroot.me/apisyouwonthate/packages/5cc9cd89-9bbc-430f-b81a-2536fdbe8429/book?ref=apisyouwonthate.com). ### [OpenAPI Is the Unit of API Governance](https://apievangelist.com/2026/06/25/openapi-is-the-unit-of-governance/?ref=apisyouwonthate.com) A great post by Kin Lane about how your OpenAPI is the Unit of Governance and how this is increasingly important with AI. Humans can push through bad descriptions; LLMs will take it at face value. > The `info.description`, the operation `summary` and `description`, the parameter descriptions, the schema property descriptions — this is where engineering and product actually negotiate what the API is. ### [FastAPI Now Serves Frontend](https://fastapi.tiangolo.com/tutorial/frontend/?ref=apisyouwonthate.com) FastAPI is a web framework for building API's with Python with a focus on performance. Thanks to FastAPI's hard work, you can now serve static frontend apps with support for React with Vite, TanStack Router, Astro, Vue, Svelte, Angular, Solid, and any others that produce a static build. ### [HTTP Cookie Parser Faster and Slimmer](https://github.com/jshttp/cookie/releases/tag/v2.0.0?ref=apisyouwonthate.com) Cookie is a Node library for parsing and serialising HTTP Cookies and recently landed a killer release. Cookie is now ESM only and has some nice performance improvements. Being part of [E18E](https://e18e.dev/?ref=apisyouwonthate.com) this really made my heart sing. Feels like so many libraries in the Node/Express ecosystem have been getting upgrades recently. ### [Socket-Activation for a Go HTTP service](https://poweruser.blog/socket-activation-for-a-go-http-service-part-1-on-linux-with-systemd-0e530ed463a3?ref=apisyouwonthate.com) I removed and added this post to the newsletter a few times this week. It's not strictly API-focused, but I found it so interesting that it's worth sharing. Chris walks us through setting up an HTTP service that only spins up and uses its full resources once you hit the endpoint. Might have limited uses, but a great read and learning exercise. --- ## APIs You Won't Hate *The latest from the team at API's You Won't Hate.* ### [Newsletter Feedback ](https://apisyouwonthate.slack.com/archives/C0B76423SQM?ref=apisyouwonthate.com) We have been running the API's Newsletter for a few years now and would love to hear from our readers. What do you want to see more of, less of, and what are we missing? We have set up a Slack channel in our community, so head on over and leave us some feedback. We can't wait to hear from you. --- ## From the Community *Articles written and shared in our free* [*Slack community*](https://slack.apisyouwonthate.com/?ref=apisyouwonthate.com)*.* ### [Portman](https://www.getportman.com/?ref=apisyouwonthate.com) Can't believe I missed this one. One of our community members was looking for API testing tools to ensure a service meets its API contract. Portman was shared as a way to automate contract and variation testing running locally and in your CI/CD. ## Support APIs You Won't Hate When you become an member, you'll get access to members-only content while directly supporting our work. Your support helps us to keep making resources for the API community. [Become a member today](https://apisyouwonthate.com/#/portal) Thanks so much to our members: **Kin**, **Juxt, Alex R, Nolan S, Brandon K, Frank, James D, Bill D,** and **Abdelhadi D**. Your support means the world to us! ✌️ Until next time, Alexander, Phil & Mike ### Go Faster with HeyAPI URL: https://apisyouwonthate.com/newsletter/go-faster-with-hey-api/ Last updated: 2026-06-18T01:08:13.000Z Even in 2026, I often get challenged about the point of maintaining an OpenAPI Spec. Even when explaining the importance of planning and designing APIs, you can still get pushback. One argument that never fails is tooling integrations like frontend code generation from OpenAPI specs. Tools like HeyAPI solve real issues that developers can't ignore. Building a modern React App using TanStack you hit many speedbumps. TypeScript types drifting from the backend contract, query keys inconsistent across apps, mutations and hooks repeated throughout the codebase, and backend changes triggering many, many frontend updates. AI is also exacerbating these issues. There is a great talk from [React Miami by Delvin Duldulao on this very topic](https://www.youtube.com/watch?v=HotpF1Dw9ng&ref=apisyouwonthate.com), totally worth a watch. Design better APIs and empower frontend teams to go faster with an OpenAPI Spec and tools like HeyAPI. \-- Alexander --- ## The API Roundup *API News, links, and tools from around the web* ### [PatchBay: Find an API](https://patchbay.cc/?ref=apisyouwonthate.com) A wonderful find that stumbled into my feed this week. Browse over 3000 APIs and find the right one for your project. They also have their own API and llms.text endpoints to make working with the data easier. Love projects like this. ### [Heimdall](https://github.com/gojek/heimdall?ref=apisyouwonthate.com) With a cool name like Heimdall, why even bother finding out what it does? Just start using it. Heimdall is an HTTP client for Go that specialises in making many requests. It comes with a bunch of built-in features to control and retry failing requests. It sees all. ### [Talos: Open Source API Key Server](https://github.com/ory/talos?ref=apisyouwonthate.com) Issue, verify and revoke API keys at scale with Talos. Comes with admin and self-surface setups, built ready for cloud-native environments. Getting API keys right is hard, and Talos comes with so much built in that it makes handling them easy. ### [Drift](https://github.com/oxidecomputer/drift?ref=apisyouwonthate.com) Apparently, Oxide has some of the best OpenAPI tooling in the ecosystem. While I was digging to find out if this was the case, I stumbled upon their OpenAPI tool, Drift. It detects changes between two OpenAPI docs. Classify changes as backwards-incompatible, forward-incompatible, incompatible, trivial, or unhandled. Definitely worth checking out. ### [Node.js 24 vs 25 vs 26 Complete Benchmark](https://www.repoflow.io/blog/node-js-24-vs-25-vs-26-complete-benchmark?ref=apisyouwonthate.com) Want to know how different versions of Node stack up? With so many APIs built with Node, it's definitely worth reviewing benchmarks for each version. Node.js has undergone some amazing changes over the last few versions. Personally, I feel like the competition from Bun and Deno lit a fire under the maintainers and community. --- ## APIs You Won't Hate *The latest from the team at API's You Won't Hate.* ### [Does your company build API tools?](https://github.com/sponsors/apisyouwonthate?ref=apisyouwonthate.com) APIs You Won't Hate is built to help the API developer community share knowledge and raise the bar for the tech we use. We're looking for more corporate backers through [GitHub Sponsors](https://github.com/sponsors/apisyouwonthate?ref=apisyouwonthate.com) to help keep the site running, pay for maintenance and server costs, and to continue our mission. If you'd like to talk about a sponsorship, [reach out](mailto:mike@apisyouwonthate.com)! --- ## From the Community *Articles written and shared in our free* [*Slack community*](https://slack.apisyouwonthate.com/?ref=apisyouwonthate.com)*.* ### [Work Sucks](https://netapinotes.com/work-sucks/?ref=apisyouwonthate.com) Mathew reflects on the changes AI is bringing to software engineering and large organisations. Personally, I love the line "Rented Capability Is Not Owned Competence". A good reflection on the challenges we are currently facing and what to focus on moving forward. ### [Jentic API Scorecard](https://github.com/jentic/jentic-api-scorecard?ref=apisyouwonthate.com) Score your OpenAPI document using a scorecard to assess its readiness for AI integrations. It scores your doc across five areas: Foundation Compliance, Developer Experience, AI Readiness, Agent Usability, Security, and AI Discoverability. An interesting tool, especially as I find more of the APIs I build being integrated into LLM workflows. Thanks for sharing, Frank Kilcommins. ## Support APIs You Won't Hate When you become an member, you'll get access to members-only content while directly supporting our work. Your support helps us to keep making resources for the API community. [Become a member today](https://apisyouwonthate.com/#/portal) Thanks so much to our members: **Kin**, **Juxt, Brandon K, Frank, James D, Bill D,** and **Abdelhadi D**. Your support means the world to us! ✌️ Until next time, Alexander, Phil & Mike ### All Devtools Die URL: https://apisyouwonthate.com/newsletter/all-devtools-die/ Last updated: 2026-06-01T11:38:53.000Z Stainless, a tool for turning your OpenAPI Spec into an SDK, was acquired by [Anthropic for over $300 million](https://www.stainless.com/blog/stainless-is-joining-anthropic/?ref=apisyouwonthate.com). Hopefully, the founders and employees are seeing a reward from the deal. However, one outcome of this acquisition is that Stainless's customers must migrate off the platform. This reminds me of the Statsig deal, acquired by OpenAI last year, only for the customers and tool to be sold to Amplitude this year, while Statsig staff stay at OpenAI. I know a lot of acquisitions end in the ending or degradation of the platform, but these newer AI deals leave me feeling a little sour. Will all good tools just be brought and shut down? Maybe it's just that watching our industry cannibalise itself has made me more cynical. \-- Alexander ## Support APIs You Won't Hate When you become an member, you'll get access to members-only content while directly supporting our work. Your support helps us to keep making resources for the API community. [Become a member today](https://apisyouwonthate.com/#/portal) ## The API Roundup *API News, links, and tools from around the web* ### [Arazzo1.1.0](https://spec.openapis.org/arazzo/v1.1.0.html?ref=apisyouwonthate.com) Arazzo's latest version just dropped with support for AsyncAPI, chained workflow execution, OpenAPI 3.2 alignment and more. If you're not familiar with Arazzo, it's a spec for describing how to use a series of API operations. Great to see this new spec mature and grow. Check out the release notes for the full specification. ### [Express Gets A Fresh Coat of Paint](https://expressjs.com/en/blog/2026-05-18-a-new-look-for-express/?ref=apisyouwonthate.com) I have adored watching the Express maintainers push more updates, but they have not stopped there. The team pushed a new update for their website and docs, and honestly, they look great. Their blog post about the release includes some info on Express's vision and values. With all this refreshed vigour, Express is no longer an older framework to migrate away from for your Node APIs. ### [Tracing HTTP Requests with Go Standard Library](https://blainsmith.com/articles/httptrace-with-go/?ref=apisyouwonthate.com) The Go Standard Library has a built-in tracing module. Turns out most developers have not used it. Blain talks us through how to use it and how its small surface makes it easy to use. Start using it today and find those slow HTTP calls. ### [The Production Playbook for Node.js Stream Leaks](https://frontendmasters.com/blog/the-production-playbook-for-node-js-stream-leaks/?ref=apisyouwonthate.com) Streaming is a great way to serve large data sets from an API. In Node, there are many ways streaming can cause leaks. Lucky Durgesh at Frontend Masters has put together a playbook for using streams in Node. If you're still not used to streaming via APIs, [Phil has got you covered with a great blog post](https://apisyouwonthate.com/blog/streaming-data-with-rest-apis/). ### [Axios to Fetch](https://app.codemod.com/registry/@nodejs/axios-to-whatwg-fetch?ref=apisyouwonthate.com) Still using Axios to call other services in your Node API? Well, now there is a handy codemod to move to `fetch`. Simply `npx codemod @nodejs/axios-to-whatwg-fetch` to remove and migrate in a pinch. ### [Fiber 3.3.0: Express styled Web Framework for Go](https://github.com/gofiber/fiber/releases/tag/v3.3.0?ref=apisyouwonthate.com) Fiber recently released a new version that adds support for server-sent events and host auth middleware. Fiber is an Express-inspired framework that offers high performance and a delightful developer experience. --- ## APIs You Won't Hate Articles written and shared in our free [Slack community](https://slack.apisyouwonthate.com/?ref=apisyouwonthate.com). ### [Newsletter Feedback ](https://apisyouwonthate.slack.com/archives/C0B76423SQM?ref=apisyouwonthate.com) We have been running the API's Newsletter for a few years now and would love to hear from our readers. What do you want to see more of, less of, and what are we missing? We have set up a Slack channel in our community, so head on over and leave us some feedback. We can't wait to hear from you. [Send us your feedback](https://apisyouwonthate.slack.com/?redir=%2Farchives%2FC0B76423SQM%3Fname%3DC0B76423SQM&ref=apisyouwonthate.com) Thanks so much to our members: **Kin**, **Juxt, Alex R, Nolan S, Brandon K, Frank, James D, Bill D,** and **Abdelhadi D**. Your support means the world to us! ✌️ Until next time, Alexander, Phil & Mike ### Save Money and use APIs URL: https://apisyouwonthate.com/newsletter/save-money-and-use-apis/ Last updated: 2026-05-15T14:33:25.000Z While I still have so many mixed feelings about AI, one of its killer abilities is pulling information from across apps, documents, and internal tools to find what I need. Honestly, this saves me days at work, but is this the most efficient way? A lot of AI Agents don't use API's, they use a web/apps visual interface, which burns significantly more tokens, based on [recent testing](https://www.theregister.com/ai-and-ml/2026/05/07/ai-vision-agents-use-45x-more-tokens-than-apis-in-benchmark/5231346?ref=apisyouwonthate.com). Turns out a good API with good documentation is what you need. Thankfully, everyone is great at building APIs... 🫠 \-- Alexander ## Support APIs You Won't Hate When you become an member, you'll get access to members-only content while directly supporting our work. Your support helps us to keep making resources for the API community. [Become a member today](https://apisyouwonthate.com/#/portal) ## The API Roundup *API News, links, and tools from around the web* ### [Plow: a HTTP(S) benchmarking tool](https://github.com/six-ddc/plow?ref=apisyouwonthate.com) Benchmarking tools are great and even better when written in Go 😉. Plow uses fasthttp under the hood and produces real-time summary statistics, a histogram of execution time and calculates percentiles. Displaying on the web and in your terminal. Seriously, check this out: ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2026/05/demo.gif) Gif of Plows real time output in the browser and terminal ### [Find My Way](https://github.com/delvedor/find-my-way?ref=apisyouwonthate.com) A remarkably fast HTTP router that's framework independent. How is it so fast? It uses a highly performant Radix Tree. If you are already using Fastify or Restify, then you may already be using it. Also comes with a nice [benchmark against other routers](https://github.com/delvedor/router-benchmark?ref=apisyouwonthate.com). ### [QUIC and HTTP/3 Come To Node.js (finally)](https://www.jasnell.me/posts/quic-comes-to-node?ref=apisyouwonthate.com) Who wants to use HTTP/2 anymore? We want HTTP/3, right? Well, it's now available in the `node:quic` module behind the flag `--experimental-quic`. James talks us through the module and how to use it. Be warned, it's highly experimental, but try it out today. ### [Testing in Ktor Server ](https://ktor.io/docs/server-testing.html?ref=apisyouwonthate.com) Every time I have to walk in the Java world, I reach for Ktor. Honestly, it's a delightful framework for building server applications in Kotlin. However, what about testing once you have that API set up? Thankfully, Ktor comes with a testing engine that runs the application call directly, without starting the full web server, helping you create fast-running tests. ### [The Indie Web Index API](https://theindex.fyi/api/docs?ref=apisyouwonthate.com) There's a rather cute [web index of indie sites](https://theindex.fyi/?ref=apisyouwonthate.com) covering tech blogs, search engines, RSS aggregators, and more. Recently, they also dropped a nice read-only API for pulling the same data. Not strictly API news, but it's wonderful to see projects like this exist in an increasingly closed-off web. --- ## APIs You Won't Hate Articles written and shared in our free [Slack community](https://slack.apisyouwonthate.com/?ref=apisyouwonthate.com). ### [Which of Fastify's OpenAPI Plugins is Best?](https://apisyouwonthate.com/blog/fastifys-openapi-plugins-which-are-best/) An in-depth test drive of several popular options for using Fastify with OpenAPI - written by Phil and chock full of good information. Fastify is a fantastic, popular, low-overhead Node.js framework, but how does the support for OpenAPI stack up? ### [Happy Birthday Mike](https://mikebifulco.com/?ref=apisyouwonthate.com) API's You Won't Hate was founded by Phil when he wrote his first book on APIs, but he quickly teaming up with Mike. It takes a lot of work to keep API's You Won't Hate running, covering our [website](https://apisyouwonthate.com/), [newsletter](https://apisyouwonthate.com/tag/newsletter/), [OpenAPI.tools](https://openapi.tools/?ref=apisyouwonthate.com) and [our community](https://join.slack.com/t/apisyouwonthate/shared%5Finvite/zt-3yh9f8p28-owgpmyHRX4tMNmNg~TG%5F7Q?ref=apisyouwonthate.com). Thank you for all your hard work, and happy birthday. --- ## From our Community Articles written and shared in our free [Slack community](https://slack.apisyouwonthate.com/?ref=apisyouwonthate.com). ### [Voiden, An Offline First API Client](https://github.com/VoidenHQ/voiden?ref=apisyouwonthate.com) Nikolas Dimitroulakis jumped into our Slack community to share his new open source API Client. Your API specs, tests, documentation and context all in executable markdown. Personally, I like that tagline Nikolas uses: "curl living in Obsidian files". Definitely worth checking out. --- Thanks so much to our members: **Kin L, Juxt, Alex R, Nolan S, Frank, Bill, James D, Rich, Ryan T, Umair, Abdelhadi, and Brandon**. Your support means the world to us! ✌️ Until next time, Alexander, Phil & Mike ### Which of Fastify's many OpenAPI plugins are the best? URL: https://apisyouwonthate.com/blog/fastifys-openapi-plugins-which-are-best/ Last updated: 2026-05-15T13:32:49.000Z Fastify took the JavaScript community by storm in 2016, offering a solid alternative to things like Express which were starting to feel a bit rough around the edges in a world where demands for APIs were growing, and HTTP was evolving rapidly. One of the core features is JSON Schema validated routing, so several times over the last decade of wanting to find frameworks with good support for OpenAPI I have wandered over, and been let down by what was on offer. There are three main ways API frameworks can integrate with OpenAPI. - [**Annotations, Comments, or Decorators**](https://apisyouwonthate.com/blog/code-first-how-to-generate-openapi-files-in-2024/#Annotations) \- Old school code-first approach of sprinkling some extra syntax around your code and hope that proximity leads to accuracy, which it generally doesn't. - [**OpenAPI-aware Frameworks**](https://apisyouwonthate.com/blog/code-first-how-to-generate-openapi-files-in-2024/#OpenAPI-aware-Frameworks) \- The framework is creating OpenAPI simply from the actual bare bones of the code, so when you register a `app.post('/foo/{id}')` and all the rest of the validation and logic, it will create that OpenAPI for you. The modern way to handle code-first. - [**OpenAPI Middlewares**](https://apisyouwonthate.com/blog/code-first-how-to-generate-openapi-files-in-2024/#move-to-api-design-first) \- The secret power of design-first is being able to reference that `openapi.yaml` in a [server-side validation middleware](https://apisyouwonthate.com/blog/server-side-validation-with-api-descriptions/) that uses the OpenAPI to provide powerful request validation (and often response contract testing) to reduce you writing all that code a second and third time. Which of these approaches does Fastify offer between official and community plugins? We're going to look at the following: 1. `@fastify/swagger` 2. `fastify-openapi-glue` 3. `eropple/fastify-openapi3` 4. Something I just slapped together in five minutes using `openapi-data-validator.js`. ## Package 1: @fastify/swagger The official plugin for Fastify, so the first place many will look. > A Fastify plugin for serving [Swagger (OpenAPI v2)](https://swagger.io/specification/v2/?ref=apisyouwonthate.com) or [OpenAPI v3](https://swagger.io/specification?ref=apisyouwonthate.com) schemas, which are automatically generated from your route schemas, or an existing Swagger/OpenAPI schema. The official `@fastify/swagger` plugin starts off showing its age just in the name. Swagger is the long dead name of OpenAPI, now only living on as a trademark for an ageing suite of SmartBear products, and generally abandoned by every modern tool in the [OpenAPI ecosystem](https://openapi.tools/?ref=apisyouwonthate.com). The tool mentions supporting "v2 or v3", and usually that means it does not support v3.1 or v3.2\. Thankfully after some digging around it does seem to support OpenAPI v3.1, but I had to scrabble around in the code looking for [commits like this](https://github.com/fastify/fastify-swagger/pull/707/changes/eeba45090738db115b53179317e26148057da23d?ref=apisyouwonthate.com) to find that out as it's just not been mentioned on the repository. Digging into the functionality now, there are two modes: "dynamic" and "static". Dynamic is where most of the functionality and documentation lays so let's start with that. ### **Dynamic mode - old school code-first** Here is the code sample from the README. ``` const fastify = require('fastify')() await fastify.register(require('@fastify/swagger'), { openapi: { openapi: '3.0.0', info: { title: 'Test swagger', description: 'Testing the Fastify swagger API', version: '0.1.0' }, tags: [ { name: 'user', description: 'User related end-points' }, { name: 'code', description: 'Code related end-points' } ], components: { securitySchemes: { apiKey: { type: 'apiKey', name: 'apiKey', in: 'header' } } } } }) fastify.put('/some-route/:id', { schema: { description: 'post some data', tags: ['user', 'code'], summary: 'qwerty', security: [{ apiKey: [] }], params: { type: 'object', properties: { id: { type: 'string', description: 'user id' } } }, body: { type: 'object', properties: { hello: { type: 'string' }, obj: { type: 'object', properties: { some: { type: 'string' } } } } }, response: { 201: { description: 'Successful response', type: 'object', properties: { hello: { type: 'string' } } }, default: { description: 'Default response', type: 'object', properties: { foo: { type: 'string' } } } } } }, (req, reply) => { }) await fastify.ready() fastify.swagger() ``` General OpenAPI metadata like `info`, `servers`, and `tags` are defined in the middleware registration, then the schema objects are defined on the route. The only thing OpenAPI-specific about this is the middleware registration, and the "write all the schemas in the routes file" experience is just how the JSON Schema validation already works in Fastly core. That means people who have never even heard of OpenAPI are already defining these schemas on their routes to get benefits like server-side JSON Schema validation. [Server-Side Validation with API DescriptionsValidation can mean a lot of things, but in API land it generally means figuring out if the data being sent to the API is any good or not. Validation can happen in a lot of different places - it can happen on the server, and it can happen in![](https://static.ghost.org/v5.0.0/images/link-icon.svg)APIs You Won't HatePhil Sturgeon![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/thumbnail/dual-controls.jpg)](https://apisyouwonthate.com/blog/server-side-validation-with-api-descriptions/) Defining all the schemas in the routes, regardless of whether that's a framework convention or not, puts it into the *Annotations, Comments, or Decorators* category. Yes those schemas will validate incoming HTTP requests, but the juice is really not worth the squeeze if this is how you have to do it. The cumbersome approaching of having it all in one routes file is going to lead to conflicts, and whilst you can DRY it up, mixing in $ref with JavaScript code is just getting weird. You cannot rely on linters like Spectral or Vacuum to help check if any of its even valid, let alone set up your own API style guides to sniff out problems during development phase. Working like this is ok for anyone who would rather use JSON Schema to Joi for example, but for the OpenAPI community it's inappropriate for anyone doing anything more than checking a box of "making some OpenAPI docs". There are two extra plugins to add to get docs out of this OpenAPI. One is the prehistoric Swagger UI that's mainly being updated by a slopbot, and the other is [Scalar's API Reference docs](https://scalar.com/products/docs/getting-started?ref=apisyouwonthate.com) which is infinitely more modern, and being actively developed by human beings. ``` npm install @scalar/fastify-api-reference ``` Wedge it into the routes somewhere. For me this was in `plugins/openapi.js` which is picked up by [@fastify/autoload](https://github.com/fastify/fastify-autoload?ref=apisyouwonthate.com). ``` import fp from "fastify-plugin"; import swagger from "@fastify/swagger"; import scalar from "@scalar/fastify-api-reference"; export default fp(async (fastify) => { await fastify.register(swagger, { openapi: { info: { title: "Train Travel API", description: "API for finding and booking train trips across Europe.", # ... snip ... } }, }); await fastify.register(scalar, { routePrefix: "/reference", }); }); ``` Very simple to add the last few lines, and thanks to some sort of magic with a hidden routing registry, the `scalar/fastify-api-reference` middleware has everything it needs from `fastify/swagger` to build out as much documentation as it can from the metadata, annotations, and schema provided in the source code. ![API documentation, using the Stripe-like three-column layout we've come to expect. This particular one Powered by Scalar.](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2026/05/image-1.png) Scalar documentation hosted on the same server as Fastify endpoints, which in this case was [http://localhost:3000/reference](http://localhost:3000/reference?ref=apisyouwonthate.com). This is simply done, but looking a bit sparse, so a lot more keywords and descriptions will need to be thrust into the routes code to make it actually useful as documentation. ### **Static mode - load in openapi.yaml** If dynamic mode makes you define everything in the routes file, and static lets you load in existing OpenAPI, is that going to help me validate requests from my existing fantastic `openapi.yaml`? Here's the code that shows how to make Fastify aware of an existing `openapi.yaml` that is quite rightly in the source code, and pass it off to Scalar to render beautifully. ```javascript const fastify = require("fastify")({ logger: true }); const swagger = require("@fastify/swagger"); const scalar = require("@scalar/fastify-api-reference"); fastify.register(swagger, { mode: "static", specification: { path: "./openapi.yaml", }, }); fastify.register(scalar, { routePrefix: "/reference", configuration: { title: "API Reference", }, }); fastify.post("/bookings", async (request, reply) => {}); fastify.post("/stations", async (request, reply) => {}); fastify.post("/trips", async (request, reply) => {}); fastify.listen({ port: 3000 }, (err) => { if (err) throw err; }); ``` Run that server and the `localhost:3000/reference` is now powered by the original `openapi.yaml` which has been through all sorts of tech writers (who didn't fancy mucking about learning JavaScript to contribute), so the docs are infinitely better fleshed out. ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2026/05/image.png) Sadly, this does not provide any request validation. It just serves the docs, and that is all. Once again, this comes down to the JSON Schema validation being a core feature of Fastify itself. They are not reading the OpenAPI and creating temporary schema/validation objects at build time or run time, which could then be used to validate just like if they were written all over the routes file. This one package could support design-first and code-first nicely if it just did that, but... it does not. ### Conflating OpenAPI Schema and JSON Schema Throughout all of this there are alarm bells ringing around using JSON Schema by default, then trying to shove that into OpenAPI v3.0\. Without boring you all with years of trouble, the two schema objects are [not actually compatible](https://apisyouwonthate.com/blog/openapi-json-schema-divergence/)! OpenAPI v3.0 schemas were a subset *and* a superset. It took a lot of work to [get them lined up in OpenAPI v3.1](https://apisyouwonthate.com/blog/openapi-v3-1-and-json-schema/). Unless Fastify leverage something like [json-schema-to-openapi-schema](https://www.npmjs.com/package/@openapi-contrib/json-schema-to-openapi-schema?ref=apisyouwonthate.com) and/or nudge people towards OpenAPI v3.1+, then this is going to be confusing. That confusion is split between everyone who just hasn't got around to noticing the problems yet, and the even more confusing situation of when the mistakes pop up. ## Package 2: fastify-openapi-glue So, Fastify wants somebody or something to go and build those Schema objects ey? Thankfully that's exactly the sort of thing computers are good at, so up steps [fastify-openapi-glue](https://www.npmjs.com/package/fastify-openapi-glue?ref=apisyouwonthate.com) by [seriousme](https://github.com/seriousme?ref=apisyouwonthate.com). Right off the bat, there's a lot to like. 1. "It aims at facilitating "design first" API development. 2. This project replaces [fastify-swaggergen](https://github.com/seriousme/fastify-swaggergen?ref=apisyouwonthate.com) to focus on the future of OpenAPI not old timey Swagger. 3. It seems to support (or at least accept) OpenAPI v3.1 [somewhat](https://github.com/seriousme/fastify-openapi-glue/blob/master/docs/schema2020.md?ref=apisyouwonthate.com). 4. It will build your schema objects so you don't have to! This all sounds a bit good to be true, so let's look at the code real quick then see what the validation looks like. ``` import Fastify from "fastify"; import scalar from "@scalar/fastify-api-reference"; import openapiGlue from "fastify-openapi-glue"; import fs from "node:fs"; import path from "node:path"; import { fileURLToPath } from "node:url"; import { Service } from "./service.js"; import { Security } from "./security.js"; const fastify = Fastify({ logger: true, ajv: { customOptions: { strict: false, // Allow custom formats without throwing errors }, }, }); const currentDir = path.dirname(fileURLToPath(import.meta.url)); const openApiPath = `${currentDir}/openapi.yaml`; const openApiContent = fs.readFileSync(openApiPath, "utf8"); await fastify.register(openapiGlue, { specification: openApiPath, serviceHandlers: new Service(), securityHandlers: new Security(), }); await fastify.register(scalar, { routePrefix: "/reference", configuration: { title: "API Reference", content: openApiContent, }, }); await fastify.listen({ port: 3000 }); ``` Pretty simple on the face of it. Point it at the `openapi.yaml` and something about running AJV in not-so-strict mode all fine, and Scalar API Reference is there again to show off the docs which is optional. More light needs to be shed on services and security handlers in a minute, but at this point let's see how validation works. Let's fire off an invalid `POST /bookings` request, that has an integer for `trip_id` instead of the UUID. ```bash curl -s -X POST http://localhost:3000/bookings \ -H "Content-Type: application/json" \ -d '{"trip_id": 123, "passenger_name": "John Doe"}' | jq . ``` Response: ```json { "statusCode": 400, "code": "FST_ERR_VALIDATION", "error": "Bad Request", "message": "body/trip_id must match format \"uuid\"" } ``` Sending in a real UUID however gets through the validation happily. ```bash curl -s -X POST http://localhost:3000/bookings \ -H "Content-Type: application/json" \ -d '{"trip_id": "4f4e4e1c-c824-4d63-b37a-d8d698862f1d", "passenger_name": "John Doe", "has_bicycle": true}' | jq . ``` Response: ```json { "id": "35f7f686-1b78-4489-9e2b-a4d9c07cd08c", "trip_id": "4f4e4e1c-c824-4d63-b37a-d8d698862f1d", "passenger_name": "John Doe", "has_bicycle": true, "has_dog": false, "links": { "self": "http://localhost:3000/bookings/35f7f686-1b78-4489-9e2b-a4d9c07cd08c" } } ``` Very nice, thank you for that. Saves me writing up 1000 if conditions or turning OpenAPI into `addSchema`, Joi, or any one of a thousand other "rewriting your contract out again and again" situations most of us API developers are trying to avoid. Is it just for `POST`? Nope! Handles `GET` and all the other HTTP methods nicely too. Let's see what happens if we just ask for all train trips in all or Europe without any query parameters like a city to go to or from. ```bash curl -s "http://localhost:3000/trips" | jq . ``` The response: ```json { "statusCode": 400, "code": "FST_ERR_VALIDATION", "error": "Bad Request", "message": "querystring must have required property 'origin'" } ``` Thank you! Fantastic. The one thing I glossed over to get to the functionality was the security and service classes. The security handler is simple enough. ``` export class Security { async OAuth2(req, scopes, schema) { // Demo-only auth: allow all requests so you can focus on validation behavior. return true; } } ``` It's easy enough to imagine checking some stuff and returning true or false in there. What about the services? Using the build in open-glue CLI command, you pass it an OpenAPI document and it generates a whole project folder for you, with empty stubs in the service handler and loads of huge comments that contain all the YAML of the OpenAPI that was passed;. A few problems immediately show up with this approach. 1. When I add more endpoints I cannot run this again without overriding code. 2. The stub code being generated has syntax errors. ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2026/05/image-2.png) 1. I don't want to do any of this! When one plugin decides that it is going to throw out all the usual way of working, it's a big jolt to the team who are expected to work with it. Suddenly the team of experienced Fastify developers are not writing Fastify routes and middlewares, they're writing "Fastify OpenAPI Glue Service handlers". What if they want to use some other plugin which hijacks things similarly extremely? This might not be a blocker for you, but it would be for me. I just want to let the OpenAPI that's already been defined handle request validation (and ideally response contract testing) in the framework as a vanilla experience instead of inventing a whole new paradigm inside the framework. ## Package 3: eropple/fastify-openapi3 A third-party extension has popped up that looks to replace fastify/swagger with a more modern-focused code-first system "whack your schema in the routes" approach, which advertises OpenAPI v3.1 support right off the bat. [GitHub - eropple/fastify-openapi3: Developer-friendly OpenAPI3 tooling for Fastify that’s easy to use.Developer-friendly OpenAPI3 tooling for Fastify that’s easy to use. - eropple/fastify-openapi3![](https://static.ghost.org/v5.0.0/images/link-icon.svg)GitHuberopple![](https://opengraph.githubassets.com/90b8b123a88ca985537db9c2c7cf33fb8b9eac6899ef0d2eed1f76d32b435663/eropple/fastify-openapi3)](https://github.com/eropple/fastify-openapi3?ref=apisyouwonthate.com) This library does take a few opinionated stances, such as requiring the use [@sinclair/typebox](https://github.com/sinclairzx81/typebox?ref=apisyouwonthate.com). ## Honourable Mentions ### Hey-API The Fastify integration from new popular SDK generator Hey-API [looks pretty good](https://heyapi.dev/openapi-ts/plugins/fastify?ref=apisyouwonthate.com) at first... ``` const fastify = Fastify(); const serviceHandlers: RouteHandlers = { createPets(request, reply) { reply.code(201).send(); }, listPets(request, reply) { reply.code(200).send([]); }, showPetById(request, reply) { reply.code(200).send({ id: Number(request.params.petId), name: 'Kitty', }); }, }; fastify.register(glue, { serviceHandlers }); ``` ... but it's just a [wrapper](https://github.com/hey-api/openapi-ts/blob/main/examples/openapi-ts-fastify/src/server.ts?ref=apisyouwonthate.com) around `fastify-openapi-glue` and still seems to want you to have a one-off generation of handlers which are non-standard Fastify. ### PayU/openapi-validator-middleware This is the only other entry on the [Fastify Ecosystem](https://fastify.dev/docs/latest/Guides/Ecosystem/?ref=apisyouwonthate.com) page which mentions OpenAPI is `PayU/openapi-validator-middleware`, and sadly this is a tool I had to kick off [OpenAPI.Tools](https://openapi.tools/?ref=apisyouwonthate.com) years ago for inactivity. The last release was Feb 28, 2022, and the tool does not support v3.1\. The last supported version of Fastify was v3, and we are in a v5 world now. [GitHub - PayU/openapi-validator-middleware: Input validation using Swagger (Open API) and ajvInput validation using Swagger (Open API) and ajv. Contribute to PayU/openapi-validator-middleware development by creating an account on GitHub.![](https://static.ghost.org/v5.0.0/images/link-icon.svg)GitHubPayU![](https://opengraph.githubassets.com/010ba27e00ba84896a0fb56be5bd9fdbb115cb69a218247fa17c6a6375e4c5ba/PayU/openapi-validator-middleware)](https://github.com/PayU/openapi-validator-middleware?ref=apisyouwonthate.com#fastify) It's a shame because the approach was exactly what is actually needed, and no more. The requests are validated against OpenAPI in middleware, letting Fastify developers continue to write their code like Fastify developers do, only without needing to spam OpenAPI-kinda-but-not-really all over the source code, and keeping the OpenAPI development out of the way so that Text/GUI editors can be used to manage it, source code can leverage it, contract testing can use it across the. test suite, and if you really need to pull in docs and emit them from the API you can. ## Conclusion Obviously a lot of hard work has gone into all of these tools over the years, and OpenAPI is not an easy space for individual developers or small teams to keep up with (especially with OpenAPI v3.1 introducing breaking changes...) but this usually just means we need better collaboration and more reliance on shared utility packages instead of everyone reinventing the sausage single individually. Based on whats on offer, if you prefer code-first maybe see how far you can get with `@fastify/swagger` or `eropple/fastify-openapi3`. If you're design-first, maybe the approach taken by `fastify-openapi-glue` is not so bad once you're used to it? Wait, I have an idea, ## Package 4: Slap a middleware together yourself If you agree that all you really need is a middleware, there are quite a few out there which are not listed on the [Fastify Ecosystem](https://fastify.dev/docs/latest/Guides/Ecosystem/?ref=apisyouwonthate.com) page, but could work either out of the box or with a bit of tinkering. I've got my eye on [openapi-data-validator.js](https://github.com/Authress-Engineering/openapi-data-validator.js/?ref=apisyouwonthate.com) from Authress Engineering and friend of the community [Warren Parad](https://github.com/wparad?ref=apisyouwonthate.com). ``` curl -X POST http://localhost:3000/bookings \ -H "Content-Type: application/json" \ -d '{"passenger_name": "Jane Doe"}' | jq . ``` Got it working quite nicely. ``` { "statusCode": 400, "error": "Bad Request", "message": "missing required property request.body.trip_id", "errors": [ { "path": ".body.trip_id", "message": "must have required property 'trip_id'", "fullMessage": "missing required property request.body.trip_id" } ] } ``` All that took was this: ``` import Fastify from "fastify"; import { createRequire } from "node:module"; import { join, dirname } from "node:path"; import { fileURLToPath } from "node:url"; const require = createRequire(import.meta.url); const { OpenApiValidator } = require("openapi-data-validator"); const specPath = join(dirname(fileURLToPath(import.meta.url)), "openapi.yaml"); const fastify = Fastify({ logger: true }); const validate = new OpenApiValidator({ apiSpec: specPath }).createValidator(); fastify.addHook("preHandler", async (request, reply) => { try { await validate({ method: request.method, route: request.routeOptions.url, headers: request.headers, query: request.query, body: request.body, path: request.params, }); } catch (err) { return reply.status(err.status ?? 400).send({ statusCode: err.status ?? 400, error: "Bad Request", message: err.message, errors: err.errors, }); } }); fastify.post("/bookings", async (request, reply) => { return reply.status(201).send({ id: crypto.randomUUID(), trip_id: request.body.trip_id, passenger_name: request.body.passenger_name, has_bicycle: request.body.has_bicycle ?? false, has_dog: request.body.has_dog ?? false, }); }); await fastify.listen({ port: 3000 }); ``` Perfectly vanilla Fastify setup, all conventions intact, but I can rely on OpenAPI being validated as pre-hook before my route handlers are even touched, meaning they don't need anywhere near as much manual validation inside the handler. For me, this would do the job perfectly. If I wanted to publish docs I'd simply publish directly to [Scalar](https://scalar.com/products/docs/getting-started?ref=apisyouwonthate.com) with [Github Action](https://scalar.com/products/docs/deployment/github-actions?ref=apisyouwonthate.com) or CLI/CI, instead of awkwardly hosting it in the API itself. Same goes for SDKs, I'd trigger a new build of Speakeasy on merge to `main` and keep SDK logic out of this individual API, which could be one of many. Perhaps somebody could wrap this approach up ☝️ and create `fastify-openapi-middleware`, and remove the old middleware off the [Fastify Ecosystem](https://fastify.dev/docs/latest/Guides/Ecosystem/?ref=apisyouwonthate.com) page as it's long dead. Let me know if you do! 🫡 ### Quality Matters URL: https://apisyouwonthate.com/newsletter/api-quality-matters/ Last updated: 2026-05-04T11:32:46.000Z I stumbled across a new tool this week, Hey API; however, it wasn't just the tool that caught my eye. A contribution from one of their community members led to [significant performance improvements](https://bsky.app/profile/heyapi.dev/post/3mkm2swnd3s2t?ref=apisyouwonthate.com) when parsing OpenAPI. Then, Hey API follows up with [recognition and financial compensation.](https://bsky.app/profile/heyapi.dev/post/3mkjourcssk2g?ref=apisyouwonthate.com) Amazing to see the hard work recognised and rewarded. We are currently living in an era where people want us to stop caring about quality. To ship 100x more, to merge without checking and spend more tokens. It's really great to see the focus on quality and end-user experience in the OpenAPI space. \-- Alexander --- ## The API Roundup *API News, links, and tools from around the web* ### [OpenAPI Support Comes to kin-openapi](https://github.com/getkin/kin-openapi/releases/tag/v0.136.0?ref=apisyouwonthate.com) It's got Phil excited, and it got us excited as well. kin-openapi has finally shipped OpenAPI v3.1 support. Great news: this now gets them off the legacy page on [openapi.tools](https://openapi.tools/?ref=apisyouwonthate.com). The leap from 3.1 to 3.2 is not as difficult, so hopefully we will see this soon. ### [Swagger launches support for OpenAPI 3.2.0](https://swagger.io/blog/swagger-launches-support-for-openapi-3-2-0/?ref=apisyouwonthate.com) Another OpenAPI upgrade hit us this week: Swagger rolled out support for OpenAPI 3.2\. While I was getting excited about this release, a little unease hit me when they talked about AI being a core part of their workflow. I get it, everyone is scrambling to talk about how they're AI-native now, but really, that means nothing. The quality of the output and how well it works are what matter. ### [Hey API](https://heyapi.dev/?ref=apisyouwonthate.com) OpenAPI to Typescript in seconds. Honestly, I'm wondering where this has been all my life. Hey, API takes your OpenAPI spec and turns it into a TS SDK or an HTTP client for Fetch, Angular, Axios, and more. However, they don't stop there; you can create plugins for Zod, TanStack Query, etc. Honestly, I'm shoving this into my work projects tomorrow. ### [Error translation in Go services](https://rednafi.com/go/error-translation/?ref=apisyouwonthate.com) Are you accidentally exposing SQL errors to your Go API's clients? It can be easy to let these things slip in any system. Jump into a nice, easy-to-read post on how to catch those pesky errors on the service boundary, turn them into domain errors and stop your internals leaking. ### [Fetch-Extras](https://github.com/sindresorhus/fetch-extras?ref=apisyouwonthate.com) Writting `fetch` functions in JavaScript can sometimes be a little time-consuming and repetitive. Especially when you find yourself building the same wrapper in every project. Enter Fetch-Extras, which allows you to compose just the extra features you need over `fetch`. No need to learn a new framework, it's just `fetch`. Works in Browsers, Node.js, Deno, Bun and Cloudflare Workers. Super easy `fetch` clients. --- ## APIs You Won't Hate Articles written and shared in our free [Slack community](https://slack.apisyouwonthate.com/?ref=apisyouwonthate.com). ### Support APIs You Won't Hate APIs You Won’t Hate exists to help developers build better APIs. We are independent, deeply practical, and run a team of 3 people supporting a growing community. If our work has helped you or your team, there are two ways to support us: 🫶 Support us as a [****GitHub Sponsor**](https://github.com/sponsors/apisyouwonthate/dashboard?ref=apisyouwonthate.com), or upgrade your subscription to become a paid [****Community Member**](https://apisyouwonthate.com/newsletter/how-do-you-paywall-an-api/) Thanks as always for your support - it means the world to us! --- ## From our Community Articles written and shared in our free [Slack community](https://slack.apisyouwonthate.com/?ref=apisyouwonthate.com). ### [OpenAPI Changes Suck](https://quobix.com/articles/openapi-changes-sucks/?ref=apisyouwonthate.com) A reflection on the first version of openapi-changes by its author, Quobix. While being a little too harsh on himself, Quobix talks us through the new version and the changes he has made. The React to Web Components part also caught my eye. Can't wait to try out the new version. ## Support APIs You Won't Hate When you become an member, you'll get access to members-only content while directly supporting our work. Your support helps us to keep making resources for the API community. [Become a member today](https://apisyouwonthate.com/#/portal) Thanks so much to our members: **Kin**, **Juxt, Alex R, Nolan S, Brandon K, Frank, James D, Bill D,** and **Abdelhadi D**. Your support means the world to us! ✌️ Until next time, Alexander, Phil & Mike ### How DO you Paywall an API? URL: https://apisyouwonthate.com/newsletter/how-do-you-paywall-an-api/ Last updated: 2026-04-16T01:01:57.000Z In our Slack Community last week, one of our members was looking for a paywall to drop in front of their API, and some interesting replies came in. The first suggestion was to use Stripe; however, this would mean building the developer portal and other tooling yourself. Phil dropped in Zuplo, a solid choice that ticks a lot of boxes. Turns out another member of our community has a startup building exactly this (more details in our community section). Someone also suggested Fienta, which is a platform for selling tickets, no idea how you use that to gate an API... where there's a will, there's a way, I suppose? \-- Alexander ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2026/03/Pasted_Image_3_2_26__12_20---PM.png) ## Agent skills for OpenAPI and SDK development Agent skills by Speakeasy: targeted instructions for your favorite AI coding assistants. [Get started](https://www.speakeasy.com/blog/release-agent-skills?utm%5Fsource=apisywh?utm%5Fmedium=email) ## The API Roundup *API News, links, and tools from around the web* ### [The Best CORS Middleware Lib in Go?](https://github.com/jub0bs/cors?ref=apisyouwonthate.com) We all know what CORS is, right? How it works, how to set it up and how to configure the headers? The best CORS lib in Go released version 1.0.0 last week. The best thing about this CORS lib is that it is easy to set up and very hard to misuse. If you're building an API in Go, check it out. ### [GraphQL Over HTTP](https://graphql-over-http.mbonnin.net/draft/?ref=apisyouwonthate.com#sec-Overview) GraphQL has released a draft specification for consuming GraphQL over HTTP. The goal here is to map GraphQL semantics to their HTTP counterparts, leading to better integration with clients and tools. Depending on how it's executed, it might finally address one of GraphQL's main criticisms. ### [NodeJS Streams](https://nodejs.org/docs/latest/api/stream%5Fiter.html?ref=apisyouwonthate.com) Since Phil's first blog post on streaming data via REST APIs, I've been keeping an eye on Node's new stream-based API. An experimental version has finally landed, but requires a flag to use it: `--experimental-stream-iter`. This new version of Streams is built on iterators, unlike the current version's class structure. ### [Hummingbird 2](https://bsky.app/profile/joanniso.bsky.social/post/3mj53zjq7sk27?ref=apisyouwonthate.com) A recent post on Bluesky said response times for Hummingbird 2 had been reduced by 25%. Great, but what is Hummingbird? After a quick Kagi search, I find it's a [Swift server framework](https://hummingbird.codes/?ref=apisyouwonthate.com). Swift is a beautiful language to code in, and I can't wait to have a play and try to build a few APIs with it. ### [Docus](https://docus.dev/en?ref=apisyouwonthate.com) A wonderful tool for writing docs with Markdown and Nuxt. Comes with built-in navigation, full text search and internationalisation support. No OpenAPI support yet, unfortunately, but it could be good as your overall doc site with your OpenAPI docs embedded. ### [Why we replaced Node.js with Bun for 5x throughput](https://trigger.dev/blog/firebun?ref=apisyouwonthate.com) An interesting write-up about how Trigger.dev sped up a service by switching from Node to Bun and found a memory leak in Bun. What caught my eye, though, was the difference between how Node with Fastify or Express handles request-response and how Bun does. Something to watch out for if you move your own Node-powered APIs. --- ## Support APIs You Won't Hate When you become an member, you'll get access to members-only content while directly supporting our work. Your support helps us to keep making resources for the API community. [Become a member today](https://apisyouwonthate.com/#/portal) ## APIs You Won't Hate Articles written and shared in our free [Slack community](https://slack.apisyouwonthate.com/?ref=apisyouwonthate.com). ### 🎙️[Stigg: Infrastructure for pricing models with Anton Zagrebelny](https://apisyouwonthate.com/podcast/stigg-infrastructure-for-pricing-models-with-anton-zagrebelny/) This interview is a throwback from a few years ago, but APIs cofounder and pod host Mike Bifulco interviewed Anton Zagrebelny from Stigg About their API Monetization product. Worth a listen if you're thinking about pricing models! ## From our Community Articles written and shared in our free [Slack community](https://slack.apisyouwonthate.com/?ref=apisyouwonthate.com). ### [FrontGate](https://www.frontgate.app/?ref=apisyouwonthate.com) FrontGate is a new startup from Jack Skinner — a Dev Portal in a box for your API. The key feature is that FrontGate does not proxy your API traffic, so you maintain operational separation. Currently in early access, with free sign-up available in a few months. Thanks so much to our members: **Kin**, **Juxt, Alex R, Nolan S, Brandon K, Frank, James D, Bill D,** and **Abdelhadi D**. Your support means the world to us! ✌️ Until next time, Alexander, Phil & Mike ### Slop APIs URL: https://apisyouwonthate.com/newsletter/slop-apis/ Last updated: 2026-04-02T07:15:38.000Z The recent petition to [ban AI code in Node.js](https://github.com/indutny/no-ai-in-nodejs-core?ref=apisyouwonthate.com) caught my eye this week and stirred up more complex feelings about the changes in our industry. I see so many posts about focusing on the spec and design, and letting AI do the work. However, in my day-to-day, I see more slop, less thought, worse design. It's almost as if giving people a lazy button has led them to either checking out or working in an echo chamber, thinking everything is solid gold. **Don't let your API design skills atrophy; keep them sharp and functional.** AI or no AI, strong fundamental skills lead to better APIs. \-- Alexander ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2026/03/Pasted_Image_3_2_26__12_20---PM.png) ## Agent skills for OpenAPI and SDK development Agent skills by Speakeasy: targeted instructions for your favorite AI coding assistants. [Get started](https://www.speakeasy.com/blog/release-agent-skills?utm%5Fsource=apisywh?utm%5Fmedium=email) ## The API Roundup *API News, links, and tools from around the web* ### [GoDoc Live statically analyses your Go HTTP services](https://github.com/syst3mctl/godoclive?ref=apisyouwonthate.com) A wonderful tool for Go APIs. Build an interactive docs site and OpenAPI spec with GoDoc Live. It uses the same packages that the Go compiler uses to parse your source code and extract every route, parameter, request body, and more, then generate the site. ### [REST API vs GraphQL](https://devops-daily.com/games/rest-vs-graphql?ref=apisyouwonthate.com) A comparison of a REST API and a GraphQL API with visual breakdowns and trade-offs over different call types. Honestly, it looks great; however, that's where my adulation ends. Each test basically treats GraphQL as inherently better, overlooking benefits like HTTP/2 and HTTP/3, caching, API design, and how you use the API. I could easily knock out each test in reverse. There are real trade-offs between the two, but honestly, these tests are very misleading. ### [Testing unary gRPC services in Go](https://rednafi.com/go/testing-unary-grpc-services/?ref=apisyouwonthate.com) An in-depth guide to testing gRPC services, both the method logic itself and the HTTP server and gRPC layer. A great guide covering what each testing level gives you, and I love the section on testing errors. ### [Va Vite](https://github.com/cyco130/vavite?ref=apisyouwonthate.com) Vite has taken the JavaScript world by storm over the last few years, dominating the frontend. Va Vite allows you to build server-side rendered applications; however, it also supports other server applications not related to SSR. With the introduction of Void0 from the Vite team (a JavaScript version of Laravel or Ruby on Rails), I wonder if this is still relevant. Fun tool either way. ### [Execute API workflows from YAML](https://strefethen.github.io/arazzo-cli/?ref=apisyouwonthate.com) A wonderful CLI tool that was added to our [OpenAPI.tools](https://openapi.tools/?ref=apisyouwonthate.com) site this week. Write your API workflow using the Arazzo spec, run it and then debug as needed. Honestly, I love this tool. ### [Spry](https://spry.medz.dev/guide/openapi?ref=apisyouwonthate.com#docs-ui) Spry, a Dart framework for building modern servers that you can deploy anywhere, recently added support for a UI for your OpenAPI spec via Scalar. Add a Scalar instance to your OpenAPIConfig, and you get a generated docs route that loads your spec. ## APIs You Won't Hate Articles written and shared in our free [Slack community](https://slack.apisyouwonthate.com/?ref=apisyouwonthate.com). ### [Anvil is the Data Company that makes PDFs a Breeze](https://apisyouwonthate.com/podcast/anvil-is-the-data-company-that-makes-pdfs-a-breeze) Mang-Git Ng, founder and CEO of document tools company Anvil sat down with Mike to talk about PDFs in the age of LLMs, and making delightfully simple products. --- ## From our Community Articles written and shared in our free [Slack community](https://slack.apisyouwonthate.com/?ref=apisyouwonthate.com). ### [Four Open-Source Agentic Authorisation Alternatives](https://apichangelog.substack.com/p/four-agentic-authorization-alternatives) Bruno shared his article on different authorisation patterns for agents using an API, allowing you to remove a human from the loop. While I've recently been doing more and more of the work myself and trusting the agents less, it's still a great read with some options if this style of workflow is for you. ## Support APIs You Won't Hate When you become an member, you'll get access to members-only content while directly supporting our work. Your support helps us to keep making resources for the API community. [Become a member today](https://apisyouwonthate.com/#/portal) Thanks so much to our members: **Kin**, **Juxt, Alex R, Nolan S, Brandon K, Frank, James D, Bill D,** and **Abdelhadi D**. Your support means the world to us! ✌️ Until next time, Alexander, Phil & Mike ### Anvil is the Data Company that makes PDFs a breeze URL: https://apisyouwonthate.com/podcast/anvil-is-the-data-company-that-makes-pdfs-a-breeze/ Last updated: 2026-04-02T06:19:48.000Z ## Notes - Mang-Git Ng on LinkedIn: [https://www.linkedin.com/in/manggit/](https://www.linkedin.com/in/manggit/?ref=apisyouwonthate.com) - Anvil - [https://www.useanvil.com/](https://www.useanvil.com/?ref=apisyouwonthate.com) - Anvil on GitHub - [https://github.com/anvilco](https://github.com/anvilco?ref=apisyouwonthate.com) - Anvil on YouTube - [https://www.youtube.com/@useanvil](https://www.youtube.com/@useanvil?ref=apisyouwonthate.com) - Anvil on TikTok - [https://www.tiktok.com/@useanvil](https://www.tiktok.com/@useanvil?ref=apisyouwonthate.com) - Anvile on Insta - [https://www.instagram.com/useanvil/](https://www.instagram.com/useanvil/?ref=apisyouwonthate.com) - spectaql - docs generation: [https://github.com/anvilco/spectaql ](https://github.com/anvilco/spectaql ?ref=apisyouwonthate.com) - Anvil is Hiring: Wellfound/Angellist: [https://wellfound.com/company/useanvil](https://wellfound.com/company/useanvil?ref=apisyouwonthate.com) - Build vs Buy Podcast: [https://open.spotify.com/show/68VESxJNOw2GDP8Lcq8bdw?si=e5d3927f742744aa](https://open.spotify.com/show/68VESxJNOw2GDP8Lcq8bdw?si=e5d3927f742744aa&ref=apisyouwonthate.com) #### Transcript Mang-git Ng and Mike Bifulco \[00:00:00\] ****Mike Bifulco:** Hello and welcome back to APIs You Won't Hate. I am mike Biko, one of the co-founders of APIs You Won't Hate here sitting down for a chat with my new friend, Manu ing from Anvil. Manu, thanks for joining today. Why don't you tell me a little bit about yourself, and I'd love to hear the elevator pitch for Anvil too. ****Mang-git Ng:** Awesome. Thanks for having me on the podcast today. Mike, it's. It's always a pleasure to do these. So a little bit about myself. Like Mike, Mike mentioned my name's Span ing. I am a founder, the founder of Anvil, and prior to that I worked as a software engineer for many years. So previously I had worked at companies like Dropbox Style Pad Flexport, and then also a small company called Loom. Not the video recording one that everybody knows about, but it, they did own loom.com and then that was acquired into Dropbox. So been in the startup space for quite some time. My background that kind of led me to Anvil was both personal and professional. I will say in my personal life I was just, dealing with adult stuff, like applying for a mortgage and \[00:01:00\] getting infinite numbers of PDFs to fill out that I had to then send. Over email to bankers so that I could get a quote for a mortgage rate. So that was painful. And then in my professional life working at Dropbox, there was obviously a lot of documents at Flexport. What you don't realize is that most shipping is just generating documents along the way with some data that you have. Flexport system was essentially a system of record for the data and it would generate the correct documents as your shipment, cross borders and got onto different ships. And every time there was a PDF that needed to be created somebody had to hand code a template of that PDF in HTML, and then we'd render it and then generate the pdf. So that's why we created Anvil. Anvil is, frankly speaking the. Easiest way to build software for documents, whether that's generating PDFs, filling out PDFs, collecting e-signatures, or the entire document process where you gotta collect information, generate a \[00:02:00\] certain set of documents based on some rules, and send it through E-signature. And the nice thing is that Anvil's completely embeddable white labeled and has a very robust API, which is why we're on the show today. ****Mike Bifulco:** right on. Lots to get into there. Really interesting. I know Flexport does a lot of Wild logistics stuff. I guess it totally makes sense that you're generating PDFs left and right 'cause there's bills of record the whole way through for shipping that are really important. That's fascinating. So you were, I guess in, in the world before Anvil existed, I'd imagine you felt the pain of Those PDFs in one way or the other. What was the thing that made you go Hey, like I'm a smart engineer, I should really dive into trying to make this a better thing. ****Mang-git Ng:** Smart engineer might be an overstatement. I was an engineer. ****Mike Bifulco:** Fair enough. Yeah. ****Mang-git Ng:** I've met very many engineers that are much smarter than I am. Let's put it that way. I think it was just, I knew I wanted to work on my own startup, my own project at some point. So that was always in the cards for me. I had been a part of many startups before starting Anvil, right? Like Dialpad, I was like number \[00:03:00\] seven Lu I was number seven as well. And then Flexport and Dropbox was there during the growth stage is when it was like 500 people. So gotta see that spectrum. ****Mike Bifulco:** Sure. ****Mang-git Ng:** so that's maybe the starting point. And then I was really looking for a problem space that I felt was both. Large enough and painful enough, and also impactful enough that really would be something I'd be willing to commit to for the next, five, 10, however many years to tackle. So I think those are probably the two driving forces. And then it was just really happenstance, like I was going through a period of my life when I was starting a company, where I just had a lot of documents to do. Like I was doing all the life stuff that you don't do when you're working, 'cause you put it off. And so it just was like an aha moment where everything clicked together. I think hindsight's 2020 always, but ****Mike Bifulco:** Sure. ****Mang-git Ng:** I. ****Mike Bifulco:** In my experience, I've been lucky to meet a lot of founders by way of podcast interview and, having friends With similar, Mental states to my own, who found themselves in The founder seat one way \[00:04:00\] or the other. And There are a Few buckets that founders tend to fall into. And, There's always something interesting talking to people who've like really felt the problem that they're trying to solve before, before solving it because it's so much more deeply. Like You feel it to your core, but you know when it's success because past you would've loved this, To an extent. Yeah. That's really cool. I, I'd imagine a lot of the folks listening to this, I'd imagine virtually everyone on Earth has touched A PDF in some way. I'd imagine some of the people listening to this have tried to do what you described before with PDF templates or pulling structured data from a document or putting structured data into a document. my experience Is somewhat limited. But I can tell you that, I still have nightmares about trying to deal with some of the Biggest company in the world, like Their APIs and tools for building PDFs. Tell me about What the process is like. What's the hello world for someone, Onboarding with Anv look like ****Mang-git Ng:** Yeah. So typically the first thing you do is just sign up. You go grab a PDF form that you're trying to automate. So typically it's a blank form, right? We actually ask you to upload blank forms. \[00:05:00\] And you upload it into Anvil. Anvil will automatically run through the PDF and use computer vision to find all the blanks for you. Does a pretty good job. I'll say not perfect, but does a very good job. And then we'll run it through an LLM to label and tag all the fields for you. So it'll do things like. Give each field a readable name. It'll give each field a specific type. So we have address types, social security number types, short text, long text, all that kind of stuff, check boxes. And it will also associate the fields together. So address fields typically are compound fields. There's line one, line two, city, state, zip. Anvil will associate those together so that when you go to populate this PDF, it's like a data structure that makes sense. It's not just a bunch of fields that are not interrelated, even though they should be. So that's pretty much it. If you have a schema already that you're trying to map to A PDF, you can pass along the keys \[00:06:00\] and we will also try to associate your schema to specific fields on the PDF automatically. That just makes it easy for you to normalize data across multiple documents. And then you make an API request. We automatically generate a restful endpoint for you to fill this out. There's a payload of data that gets auto generated and you just make a request to that endpoint with your API key. And you should receive a filled out PDF. ****Mike Bifulco:** I'm a little, Gobsmacked right now. Having had tried to build something on, DocuSign, APIs in particular, like The complexity that large companies can inject into a process like this can be pretty mind numbing. And it's hard to see when you're that big. I've worked for great big corporations, I've worked for n equals one corporations, and, No fault in either direction. But like that, what you've just described sounds like the dream I needed, When I was working on this the last time. And It's a really interesting tool. Like that is something that I think fundamentally, anyone whose business has some overlap into the real world where a printer might be involved, Can use this And has some value \[00:07:00\] for it. I'll actually maybe give you an example from my world. Outside of APIs, we won't hate or run a startup called Craftwork. We're a vertically integrated home painting company, and one of the things we deal with a lot is we get invoices and receipts from a paint supplier. So a paint store. We go and order paint for a project. It'll have The associated. Project that it's with. So it might be like Mike's house, And here's the 10 gallons of paint we ordered and the four paint brushes and the primer and all the other stuff that's needed there. Those per company are generally pretty standardized. However, that's like another form of oddly unstructured data in that it's an invoice that can have, One or 10 or 50 rows. And I don't necessarily have, let's call it like A blank version of that. In, in your world, is that Should I go And find a way to regenerate that sort of template with a blank? Is there a story for that with Anvil as well? Is that something that, Is a use case you're interested in? ****Mang-git Ng:** Yeah, absolutely. I think when it comes to extracting data out of PDFs, we don't actually do that today. Our thesis has always been, and I said we don't actually do that today. Actually, I don't know if we'll ever do it, let's be honest. Our \[00:08:00\] thesis has always been that we already live in a world of a lot of structured data. And instead of destructuring it and putting it into a PDF, and then having somebody pull that unstructured data out of a PDF with OCR and lLM text extraction methods it should just be structured the whole time. Like the PDF becomes an artifact of the process. And so one way that I would think about your problem in this ideal future is when an invoice is generated as a PDF the actual data that went into generating the PDF should be attached in the PDF as metadata that is structured. And so if you're like receiving the PDF instead of parsing it out with computer vision and whatnot and overcomplicating it with this Rube Goldberg machine that you've created you just go read the data out and parse it into your system. And so this is actually like a hack week project that I did, which was I really wanted to take that structured data. Concept to the next level with PDFs. And Anvil does a lot with \[00:09:00\] collecting data. We have a workflow tool, we call it. It collects data on the front end with a web form. It's dynamic. There's lots of logic, and then it generates the corresponding set of documents that you need, right? But really what I want to do is embed all that data as metadata alongside as the PDF, so that anybody could go pull it out if they knew it was a anvil generated document. Or hopefully this becomes like an open standard, and then PDFs literally become packets, right? It's like it's, there's like the internet packets and then there's like the packet that like the average, normal lay person would think of. It's like a packet of PDFs that then get passed around and the data can just be extracted. So that, that's our take on it. But we live in a messy world and there is a lot of a lot of. Unstructured data living in PDFs today, and I think there's lots of really great companies that are tackling it. From the, the legacy OCR companies and RPA companies to the newer crop of 'em that came along in this gen AI world. ****Mike Bifulco:** The idea that A PDF might have structured data embedded in it right now \[00:10:00\] makes me wanna go Dive into my hex editor And figure out what I've been missing this whole time. Like genuinely, that's never crossed my mind That's even possible ****Mang-git Ng:** they, the funny thing is they don't, most of them do ****Mike Bifulco:** Yeah. ****Mang-git Ng:** it was really, actually, I, as far as I know, nobody, the, all of the metadata that gets embedded into the PDF is not the data that's on the PDF. It's it's oh, this person signed the document. It's that kind of information. But why not take it a step further? I don't understand why nobody's really done it that way, but ****Mike Bifulco:** That's No different than putting an image in a document. That's crazy. That's such a good idea. I really like that. You also mentioned documents, Signing as well. That's something that you serve. So it feels like you've got a Pretty complete picture for Populate this thing and then go ask for a signature. Is that right? ****Mang-git Ng:** Yeah, absolutely. We found that was like a very natural point for Anvil to move towards after building the first version of Anvil. We really think of paperwork as. Both a mechanism for data transport, like moving data between individuals and organizations, but also a mechanism for describing \[00:11:00\] a business process, right? A lot of times a business operates on a set of documents and they have to be completed in a certain manner, in a certain order. And that essentially is a codification of what that company does and people leave, people come, but like the process stays the same. Yeah. ****Mike Bifulco:** It sounds like you've been through a few different phases. I don't know if you mentioned before, how old is the company? ****Mang-git Ng:** We're about seven and a half years so been around for a hot second. Back to the statement earlier about something I wanted to work on for five 10 or maybe even longer in terms of years. ****Mike Bifulco:** Yeah. I'm curious then, like You've evolved over the past seven years. What did your first crop of customers look like and how did you find them? And then what do they look like now and How has that changed? ****Mang-git Ng:** yeah. Our first crop of customers we were really targeting non-technical people. The whole premise was, Hey, we're gonna create a set of tools. We actually, so we actually built the workflow tool first. A set of tools that where you can take documents. And upload it \[00:12:00\] into Anvil and Anvil, then automatically create an online form for you that you can then share with people to collect the data. So the premise was TurboTax, anything. So our first customer was actually they're a great customer, still a customer of ours. First Ascent. They're now part of geo wealth. They are a giant, kinda like wealth management. Back office firms. So wealth advisors would choose them to manage all the documents and paperwork and they would handle the account opening documents, they would handle the transition, all that stuff. As well as like investment strategies and stuff like that. That's, that was our first customer. We effectively built Anvil around their use case. They were like a design partner, which was. Honestly a really lucky first experience, right? There was just so much about the way paperwork works and the way companies manage documents that, was so much more complicated than what we had anticipated. I think the most telling thing for me is like there's just a \[00:13:00\] lot of tribal knowledge that lives in individual's heads that. If you're trying to codify it and translate it into a workflow you really need to like, pull it out of 'em. Oftentimes they don't even realize it's tribal knowledge and because it was such a complex use case, we actually built a pretty robust system. Fast forward a couple years, what we realized was. A lot of our friends were building companies in these legacy industries and they were running into this problem where they had to generate documents and they kept coming to us asking us for an API. So we just decided to productionize our own internal APIs, expose 'em, make it easy for kinda like technology customers to use us. And nowadays the vast majority of our customers are actually like early to growth sage technology customers that find us. Read our APIs may maybe the lms the agents read our APIs and then implements us into their system. So yeah, we're pretty deeply embedded into organizations and we generally power like a pretty important stack and \[00:14:00\] their business. Yeah. ****Mike Bifulco:** I will say, yeah, definitely. As someone, Building tools for a old industry that's a early to growth stage company, I think that's like. I can draw a lot of circles around things I need that are document driven right now. It's a really interesting problem to solve. And I can imagine many types of companies that would be into that. I'm curious what you can tell me about how the software is built. So from architecture to, the types of engineers that work for you. What are, how are you building this? And, I dunno, what does your team look like? ****Mang-git Ng:** Yeah. So from a technology stack perspective, it's a pretty standard stack. It's no js. GraphQL endpoints and react on the front end. We also have a fairly robust like PDF library that we it's an open source library that we've then wrapped in this nice web experience. So there's a lot, a fair amount of Java in our code as well. For better or worse, and. Yeah so that's the joke is always oh, it makes sense that the ultimate enterprise file format is also written in the \[00:15:00\] ultimate enterprise language, which is Java. Of course. And so that's the stack. In terms of our team, we're a very lean team. We're a team of seven and it's about half engineering and. Half non-engineering. Everybody's fairly technical, I will say on our team. And I say half engineering 'cause I count myself as half an engineer, so that makes it exactly half. And yeah and so it's, we run pretty lean. The nice thing is our product is, generally speaking, very self-serve. We tried a heavy sales motion early on, but just given our ICP, which is engineers and product people. didn't want to get on sales calls. We just, they just wanted to have access to the docks. They wanted to have a dev sandbox. They wanted to kick the tires. And that's how a lot of our customers come to us. They start paying us 5 cents, 10 cents, a hundred dollars, $2,000 a month, and then it just becomes like a real, it becomes a real account. Yeah. ****Mike Bifulco:** Yeah, I, Am guilty of diving into your docs \[00:16:00\] and looking at things, as I've been like preparing for this interview. And, A few things stood out to me that are very bright green signals for me as a developer is, I'll start by saying that if you're listening to this podcast, you should go check out Anvil's docs pages. 'cause they're so well organized. They're beautiful, they're like in Thought out. They have nice API reference. It's like whoever worked on those, give them a high five for me. That's, What I would say is do that and then think of any other PDF, like API you can think of and go look at the difference. And As an engineer, I feel like you can feel the difference. If you're selling to engineers, in my eyes, that sells itself. I've, I'm also like really happy to see that, Your pricing is pretty transparent. Do you wanna talk a little bit about the pricing strategy? ****Mang-git Ng:** Yeah, absolutely. It is pretty much pay as you go. It's 10 cents A PDF generation. It's a dollar 50 for a completed signature packet. And I say completed because. All of the other, or not all, but 99% of the other e-sign providers, they charge you on send regardless of if the signatures were completed. And so we really want to align value there. And then it's a \[00:17:00\] dollar for a workflow submission. So a workflow is a web form that can be pretty complex or multiple web forms that then populate multiple PDFs. We do also have some kind of like platform fee type fee pricing. Specifically if you want like white labeling or some high, more advanced features, those are a monthly platform fee or an annual platform fee. And then enterprise is similar. It's a larger platform fee. You have more advanced features, you get more support, and then of course, as usage scales, you're able to buy discounted. Usage. So overall we want to scale with you. We wanna make it easy for you to start and not pay a bunch up front, but if you do start using us a lot, we don't want to be nickel and dimming you at retail prices, essentially. ****Mike Bifulco:** What I like about that is, It's predictable. If you can look at that and understand how it fits into your world as a team, building something, But also accessible enough, If I want to test it and see if, My first a hundred documents, Makes sense and work and fits my team's use case, that's not \[00:18:00\] a insurmountable thing, ****Mang-git Ng:** The first a hundred documents will actually be free. 'cause when you plop on a credit card, we give you a bunch of free credit for production usage. So I believe it's don't quote me on this. I believe it's a thousand PDF generations and. Like 50 workflows or 50 signatures. I'll have to double check, but it is pretty generous. We do want you to I think we want you to build as a developer and use the dev keys and all that stuff, and that's pretty much the full experience. You get access to pretty much everything When you flip into production, we remove the watermarks and stuff like that. We do want to give you enough credit to actually. Run it into production in a meaningful way so that you have confidence that you can build your business on top of this. ****Mike Bifulco:** Yeah, I like that too. I walked right into Your free tier as well. I, Promise I Didn't try to do that, but That's true. And so from what I remember, looking through your docs, You are GraphQL and rest endpoint Driven, right? So there's like Our API, Homies listening to the podcast will be really interested in what that all, Looks like. Do you have, is it, \[00:19:00\] Primarily like rest endpoints In GraphQL? Do you have, Client SDKs in, written in, Java, whatever c Node languages Or is it, Or documentation around Send a request that looks like this and this is what will happen. ****Mang-git Ng:** Yeah. So we do have client sks that we've written and we publish. They are, I believe there's a JavaScript one, a C one, there's a Python one. And we generally try to be pretty good about maintaining them. Occasionally we do get some people writing in and then we dive in there and go fix it, and we do ask if somebody has a fix, please feel free to open up pull request. So we do have a lot of pre-written libraries. They're mostly just like client wrappers. Our API is pretty hopefully pretty, pretty simple to understand and integrate with. And so I think that. If you're just making web requests it's pretty easy. The client libraries do nice things like handle multi-part uploads for you if you're uploading a large PDF to be templatized or handle, like automatically handle rate \[00:20:00\] limiting four, two nines for you. And using your encryption keys and stuff like that. So it's really just like nice helper functions to, to get you started faster. ****Mike Bifulco:** One of the other things I wanted to mention too is you have quite A broad list of like open source things published on Anil's, GitHub, which is really cool to see. And also nice that You're, Open to and willing to like, accept community driven, Changes and whatnot. Especially, Keeping up with lots of languages can be a challenge and, getting the, like developer flavored change Is often. Really useful, especially, If it's like not your specific native language, ****Mang-git Ng:** Yeah, absolutely. Absolutely. And I'll say, actually just a quick call out. If anybody out there is looking for a documentation reference docs generation system for GraphQL check out spectacle, that's S-P-E-C-T-A-Q-L that is the system we use and we op, we open source for generating our reference docs for ****Mike Bifulco:** very cool. ****Mang-git Ng:** It was a fork from a previous project that kind of got abandoned and we updated it. And yeah. Now whenever we push a \[00:21:00\] new endpoint or change something it just automatically compiles and creates a new reference stocks for our endpoints. ****Mike Bifulco:** I will make sure That shows up in the show notes. So if you're listening to this, you can scroll down to the description of your podcast and get right to that too. Can you tell me like Where your team is headed? What are the kinds of problems you're looking at next? ****Mang-git Ng:** Oh man. So many. I think there's one to start maybe revisiting the pricing question earlier. One thing we did last year was. Make any usage of Anvil from our online experience free. The goal is really to make Anvil accessible to non-technical people as well. So your ops team can send out a signature packet and have that be free. And so that kind of plays into what we're thinking about in 2026, which is we want to bring anvil to places that people do work. And this is I think, a pretty strong counter position against other e-sign providers. Other e-sign providers think of themselves as the platform and then say, Hey, come to us and we will be your one place to manage documents. And the fact of the matter is like nobody's working out of \[00:22:00\] DocuSign. They're working out of Salesforce, they're working out of HubSpot. And so what Anvil's trying to do is make it so they can connect Anvil to your Salesforce account. And then from within Salesforce, let's say you can just say, generate us a sales contract and automatically generates it. You review it with our embedded components all within Salesforce, and then send out for signature. So we wanna do this for Salesforce, HubSpot, all of these like touch points where teams generally get work done. And a big one of, for the, for that in the, forward looking is really MCP UI apps, right? ****Mike Bifulco:** Yeah, sure. ****Mang-git Ng:** chat, GPT, Claude Cowork where somebody can say, I need to get this document signed, upload an anvil's system automatically tags and labels. And they say here's a payload of data. Go fill it out. Send it out for signature. That's one aspect. The flip side of that is how do we prevent agents from just randomly signing a bunch of stuff like we need to know it's a human. So we got some pretty fun stuff fun projects up our sleeves to create \[00:23:00\] a frictionless way to identify an individual as a human, not an agent. More on that to come, but that's something we're working on probably in, in Q1, Q2 this year. And I think we'll be the first ones to really tackle that problem beyond like a. Hey, set up a video camera, a video call and notarize and have a notary or do one of those like crazy face scan things, right? That nobody seems to like doing. So that's another one. And then the last project, which is the bid project is goes back to how thinks of ourselves as a data company, not so much a PDF company. Like the data, just the PDF is an artifact and it just happens to be how the world operates. But one thing we do today is again, you can upload your keys when you're tagging A PDF, and we will automatically match your key schema to the schema of the document, right? So that makes it easy for you. Then create data payloads that are easy for you to manipulate in your code and have that rendered onto A PDF. With. \[00:24:00\] new project, what we're trying to do is go beyond mapping a schema to A PDF. Imagine a world where you could grab a payload of data in any schema and tell us the target schema. And an and anvil just automatically translates that data into your target schema. So you can use it in whatever system. It could be a PDF, it could be another application database, it could be your snowflake. So we already have this general concept in Anvil. It's just very PDF oriented. ****Mike Bifulco:** Sure. ****Mang-git Ng:** The goal is to build it out so that you can effectively install anvil in front of your application or, wherever and just use it as a pass through for transforming data on the fly to be consumed in another system. So that, that's really what we're looking to, to work on in this year and in the future. ****Mike Bifulco:** It's a really cool use of the tools and Expanding your borders a little bit. And becoming the translator between, Adjacent, Namespace is a really interesting idea. I have spent a lot of my career, and I'm sure a lot of \[00:25:00\] people listening to this have spent a lot of their career doing exactly that, where it's I don't know, we just get this. Garbage format from some COBOL program from 700 years ago, and We need to convert it into this new thing. And so I'm gonna spend the next, Nine weeks writing a, an intensive like JSON schema thing to translate things around. And, A lot of the world we're living in now makes that easier and more interesting, And more capable for Everyone to do. That's super cool. I really like ****Mang-git Ng:** Yeah. Did I say the words AI yet, or the word AI yet? The part of the reason of doing this is actually, if you think about is actually really good at these types of matching problems where it's you have this thing here and you have this thing here. And like a human could look at it and be like, oh yeah, that makes sense. This field obviously meet matches this key. But to do that derma deterministically is quite difficult. You'll never run out of rules to have to implement, but to do that with ai and lms, it's a lot. Easier. And so now we actually have this tool that allows us to do that, and we think we have a unique data set that allows us to do that better than anybody else. And we can actually fine tune \[00:26:00\] and tr, make these models work for this use case. That I think is actually a pretty good moat as we build up this feature. Yeah. ****Mike Bifulco:** Definitely. Yeah, that sounds really interesting. You'll have to send that my way as soon as it's available. I'd love to play around with it. ****Mang-git Ng:** I definitely will. ****Mike Bifulco:** Couple more small things for you. Is your team expanding? Are you looking to hire anytime soon? ****Mang-git Ng:** Yeah, absolutely. We are hiring specifically looking for engineers. We are planning on hiring two-ish, two to three-ish engineers, so if anybody's interested, please do check us out. I believe the jobs are posted on, found, also known as Angel List. Anybody has deep experience with AI ml and wants to be a full stack, just like a builder, I don't wanna say you're not just focused on AI and ml, but you can be building the entire application and be close to customers. Yeah, come check us out. We have some we have some open roles there. ****Mike Bifulco:** Very cool. Yeah, I will be sure to, Stick that link in the bottom here too. The last thing I wanted to ask you about is you host a \[00:27:00\] podcast as well. Can you tell me about that? Gimme the pitch for your show. ****Mang-git Ng:** Yeah. Thanks Mike. We host a podcast called Build First Buy and we get product leaders founders on our show to talk about their decisions when they're building a new company, creating a new company. Their decision making around whether to build something or buy something and specifically around technology. And I think there is a strong bias to building in Silicon Valley. I think one guest that came on made a very good point that like people in Silicon Valley are rewarded for building but fundamentally there are just things that you probably shouldn't. Build if you're really trying to move fast and launch. That's the conversation we have there. We've had 26, 27 episodes now with some really great leaders. Many of them are people that I've been fortunate enough to work with in the past. And yeah, would love for people to check it out. It's bill verse buy look it up with Anvil in the Google search. Or ask Claude to look it up for you or something. Yeah. And. Hopefully you \[00:28:00\] guys enjoyed listening to that podcast as well. ****Mike Bifulco:** Yeah, definitely. We'll make sure to link that through too. Build versus buy is the perpetual question, For all of us. And I agree with you. I think a lot of Silicon Valley rewards building when like money works to make things happen too, ****Mang-git Ng:** We see that now as well, right? Like the whole premise around like Clause is gonna build everything. You're just gonna prompt claw code and know SaaS is dead. Maybe we'll get there. Maybe that is the future. But I think as of the current state, Claude is great at building MVPs and actually, and great at editing existing code bases. I think that's what it's really good at and it is very powerful. I use it every day, but to productionize something to make it scalable and reliable and resilient and you honestly have to go through a couple fires and feel the pain. No. And then get a bunch of phone calls and then, yeah. So we, I think we've battle tested and battle hardened our system quite a bit at this point. ****Mike Bifulco:** You're speaking to my heart. I feel that deeply right now, without a doubt. Right on. Manding, thank you for joining. I really appreciate having you on the show. Before I let you go, where's the best \[00:29:00\] place to find you online and how do people find Anvil Online? ****Mang-git Ng:** Yeah. Best place to find me online is LinkedIn. Mange it. Just plot or just like Google, me, my name's unique enough that I'm, I think I'm the first two pages of results. I most active on LinkedIn, I will say. And then Anvil is use anvil.com. We also have a YouTube channel. We have a TikTok and Instagram, which you might think is funny for a B2B SaaS company, but there's some funny videos there. I would say we, we work with a really great agency. So check those out. Hopefully they're entertaining and yeah, hope to hear from you. I'll maybe on LinkedIn or on one of our social. ****Mike Bifulco:** Right on. That sounds great. We'll make sure to tag you there when this, Goes live too. Mange, thanks for joining me. It was really great getting to know you and super cool to chat about the product. I'm excited to see where it goes from here. ****Mang-git Ng:** Thanks for having me on the show. It was a lot of fun. ****Mike Bifulco:** Of course, we'll catch you soon. Take care. ### OpenAPI.tools is Growing URL: https://apisyouwonthate.com/newsletter/openapi-tools-is-growing/ Last updated: 2026-03-16T12:24:16.000Z If you missed the announcement at the start of the year, we launched a revamped version of our [OpenAPI.tools](https://openapi.tools/?ref=apisyouwonthate.com) website. The new version favours tools that support versions 3.1 and 3.2, pushing older tools to the back. However, over the last three months, PRs have been coming as more tools now support 3.1 and 3.2\. It been great to see the progress and the introduction of new tools. If you're looking for a tool for your OpenAPI workflow, the best place is [OpenAPI.tools](https://openapi.tools/?ref=apisyouwonthate.com). \-- Alexander ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2026/03/Pasted_Image_3_2_26__12_20---PM.png) ## Agent skills for OpenAPI and SDK development Agent skills by Speakeasy: targeted instructions for your favorite AI coding assistants. [Get started](https://www.speakeasy.com/blog/release-agent-skills?utm%5Fsource=apisywh?utm%5Fmedium=email) ## The API Roundup *API News, links, and tools from around the web* ### [Go: oapi-codegen](https://github.com/oapi-codegen/oapi-codegen/releases/tag/v2.6.0?ref=apisyouwonthate.com) oapi-codegen is celebrating its 7th birthday by cutting a release with a bunch of new features, bug fixes, documentation improvements, and more. If you have never heard of oapi-codegen, it's a great tool that takes OpenAPI specs and converts them into Go code for the server and client. ### [Native tRPC OpenAPI Support](https://github.com/trpc/trpc/discussions/7239?ref=apisyouwonthate.com) A long-time requested feature for tRPC has been OpenAPI JSON generation. An official discussion was just opened in the GitHub repository to gather feedback as they get closer to their beta release. Feedback is being sought, so if you have thoughts, now is the time to get involved. ### [Laravel OpenAPI CLI](https://laravel-news.com/laravel-openapi-cli?ref=apisyouwonthate.com) A new OpenAPI tool has just dropped, a CLI that turns OpenAPI specifications into dedicated Laravel Artisan Commands. Just register your spec in the CLI, and then you can call the endpoints like so `php artisan acme:get-orders --limit=10`. Check out the blog for the full setup. ### [Announcing Speakeasy MCP Platform](https://www.speakeasy.com/blog/release-speakeasy-mcp-platform?ref=apisyouwonthate.com) Speakeasy has dropped a new platform for building, deploying and connecting MCP servers. After identifying auth, observability, and distribution as the real blockers to shipping an MCP server to production, Speakeasy has released a platform to make the release of your MCP server as smooth as possible. ### [We Scanned 250 Node.js Repos for Blocking I/O.](https://stackinsight.dev/blog/blocking-io-empirical-study/?ref=apisyouwonthate.com) Your Node-powered API likely uses code that makes blocking synchronous I/O calls. Ko-Hsin Liang has delved into the JavaScript ecosystem and found many packages that are blocking your I/O, producing a great deep dive into what blocking the event loop means, fixes to the common mistakes they found, and how to find them in your own code. Make your API faster and stop unnecessarily blocking the event loop. --- ## APIs You Won't Hate Articles written and shared in our free [Slack community](https://slack.apisyouwonthate.com/?ref=apisyouwonthate.com). ### [🎙️ Unofficial APIs for Everything](https://apisyouwonthate.com/podcast/supergood-is-building-unofficial-apis-for-everything/) Founder Alex Klarfeld joins host Mike for a podcast interview, with an update on Supergood's product journey, reverse engineering APIs ethically, and automating tech for old industries --- ## From our Community Articles written and shared in our free [Slack community](https://slack.apisyouwonthate.com/?ref=apisyouwonthate.com). ### [Vacuum OpenAPI Linter Gets an Upgrade](https://quobix.com/vacuum/?ref=apisyouwonthate.com) Dave Shanley or `quobix` dropped into our Slack group to show off some great upgrades to the Go API Linter vacuum. It now supports an [open-collection](https://quobix.com/vacuum/commands/open-collection/?ref=apisyouwonthate.com) command to convert OpenAPI specs to OpenCollection formats and [apply-overlay](https://quobix.com/vacuum/commands/apply-overlay/?ref=apisyouwonthate.com). ## Support APIs You Won't Hate When you become an member, you'll get access to members-only content while directly supporting our work. Your support helps us to keep making resources for the API community. [Become a member today](https://apisyouwonthate.com/#/portal) Thanks so much to our members: **Kin**, **Juxt, Alex R, Nolan S, Brandon K, Frank, James D, Bill D,** and **Abdelhadi D**. Your support means the world to us! ✌️ Until next time, Alexander, Phil & Mike ### Supergood is building unofficial APIs for everything URL: https://apisyouwonthate.com/podcast/supergood-is-building-unofficial-apis-for-everything/ Last updated: 2026-03-16T11:59:41.000Z #### Transcript ### We 💚 Git-driven API clients URL: https://apisyouwonthate.com/newsletter/we-git-driven-api-clients/ Last updated: 2026-03-02T17:24:25.000Z As agents have made their way into my workflow at work, OpenAPI Spec Rules and Git-native API clients have become my most sought-after tools. Git-native clients are an easy win because they keep everything in the same repo, giving the agent more content. I get the most out of agents in projects with well-defined patterns, linting and tests. It becomes quite enjoyable to chat with the agent about changes, then see the OpenAPI spec update, lint it against the rules, confirm the output, and then watch the code change. If you slept on OpenAPI Rules and Git-native API clients, it's time to pick them up. [Bruno](https://www.usebruno.com/?ref=apisyouwonthate.com) has been my go-to client for a while now. \-- Alexander ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2025/08/Pasted_Image_8_4_25__7_14---AM.png) ## The fastest way to build production-ready MCP servers Gram by Speakeasy: turn your API platform into an AI platform. Create tools from OpenAPI, curate into custom toolsets, and deploy hosted MCP servers [Get started today](https://app.getgram.ai/?utm%5Fmedium=email&utm%5Fsource=apisywh) ## The API Roundup *API News, links, and tools from around the web* ### [Overlay](https://spec.openapis.org/overlay/v1.1.0.html?ref=apisyouwonthate.com) Version 1.1.0 of the Overlay Specification launched at the start of 2026, bringing several new features. The `copy` property for the Action Object allows any element in the spec to be copied or moved in the OpenAPI document. You can now also update primitive values such as strings, integers, and others directly, rather than the parent object, making Overlay documents easier to manage. Check out the full spec for all the details. ### [Go-Fault](https://github.com/lingrino/go-fault?ref=apisyouwonthate.com) A nifty Go middleware package that makes it super easy to reject incoming HTTP requests and respond with an HTTP error. Super handy for testing failure scenarios with its three types of injectors `SlowInjector`, `ErrorInjector`, and `RejectInjector`. ### [How to make an HTTP request in Node.js](https://nodejsdesignpatterns.com/blog/nodejs-http-request/?ref=apisyouwonthate.com) There are so many ways to make an HTTP request in Node; thankfully, Luciano has put together a detailed walkthrough covering everything from basic requests to timeouts, streaming, form data, and more. ### [JSON Schema as the Foundation for API Design](https://naftiko.io/blog/json-schema-as-the-foundation-for-api-design?ref=apisyouwonthate.com) Kin Lane talks with David Viesack about JSON Schema and how great it makes working with API specifications. The part that caught my eye was David's discussion of a tool they built that extracts schemas from OpenAPI documents, allowing testing of request bodies without the full API structure. Definitely worth reading in full. ### [MCP Is Just an API](https://apichangelog.substack.com/p/mcp-is-just-an-api) Bruno Pedro sits down with Emmanuel Paraskakis at API Days to talk through all the chatter on MCP making APIs irrelevant. Their main takeaway is that a good OpenAPI spec leads to a great MCP server with a good task success rate for the LLM. Watch the full video for all the details. ## APIs You Won't Hate Articles written and shared in our free [Slack community](https://slack.apisyouwonthate.com/?ref=apisyouwonthate.com). ### [Alternatives to Postman](https://apisyouwonthate.com/blog/http-clients-alternatives-to-postman/) On the off-chance you haven't seen our rundown of various API Clients to use as an alternative to postman, this one is worth a read through. Our love for Bruno and the new class of git-driven API Clients started here - maybe yours will too! --- ## From our Community Articles written and shared in our free [Slack community](https://slack.apisyouwonthate.com/?ref=apisyouwonthate.com). ### [OpenAPI and SDK Agent Skills](https://www.speakeasy.com/blog/release-agent-skills?ref=apisyouwonthate.com) Sagar Batchu, the co-founder of Speakeasy (one of our sponsors), posted about the release of Agent skills for OpenAPI and SDK generation. I have been really digging some of the released vibe-skills for helping colleges unfamiliar with libraries like TanStack Router, so it's great to see some skills around the OpenAPI Spec now. ## Support APIs You Won't Hate When you become an member, you'll get access to members-only content while directly supporting our work. Your support helps us to keep making resources for the API community. [Become a member today](https://apisyouwonthate.com/#/portal) Thanks so much to our members: **Kin**, **Juxt, Alex R, Nolan S, Brandon K, Frank, James D, Bill D,** and **Abdelhadi D**. Your support means the world to us! ✌️ Until next time, Alexander, Phil & Mike ### Is REST Dead... Again? URL: https://apisyouwonthate.com/newsletter/rest-is-dead-again/ Last updated: 2026-02-16T12:33:02.000Z Kelsey Hightower is an interesting person to follow. His genuine curiosity about different tech is very infectious, and this week, he dug into MCP and came up with a few thoughts. > What I'm reading from the popularity of MCP is that standardizing on REST for APIs may have been a mistake. Intent based APIs, backed by a discovery and example usage mechanism, was the way to go. JSON-RPC seems to have provided the foundation for new protocols like MCP to close the gap. > > — [Kelsey Hightower (@kelseyhightower.com)](https://bsky.app/profile/did:plc:7i3fhorekojhdjhkbln7q7gq?ref%5Fsrc=embed&ref=apisyouwonthate.com) [2026-02-12T20:38:45.077Z](https://bsky.app/profile/did:plc:7i3fhorekojhdjhkbln7q7gq/post/3meotwtdf2c2y?ref%5Fsrc=embed&ref=apisyouwonthate.com) This one is interesting, and some replies note that REST already includes these features, but most people don't implement them correctly. However, he makes a point about many APIs requiring multiple API calls to complete a task — hello, Arazzo — an interesting thread to look into. My final takeaway is that, initially, they said AI would replace web frameworks and let us write close to the browser, but that turned out not to be true, along with many other predictions. So, rather than make wild predictions, keep learning and see where things end up. \-- Alexander ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2025/08/Pasted_Image_8_4_25__7_14---AM.png) ## The fastest way to build production-ready MCP servers Gram by Speakeasy: turn your API platform into an AI platform. Create tools from OpenAPI, curate into custom toolsets, and deploy hosted MCP servers [Get started today](https://app.getgram.ai/?utm%5Fmedium=email&utm%5Fsource=apisywh) ## The API Roundup *API News, links, and tools from around the web* ### [Accountability, APIs, and the United States of Surveillance](https://netapinotes.com/accountability-apis-and-the-united-states-of-surveillance/?ref=apisyouwonthate.com) Matthew shared his article in our Slack community, but the message hit home hard as I read it, so it went to the top of the stack. In Matthew's own words: > This article looks at how familiar technical patterns - API access, identity resolution, data enrichment, and permissive authorization models -enable surveillance and intimidation at scale. Not through exotic technology, but through ordinary, well-designed systems doing exactly what they were built to do. It's an insightful editorial on the role of devs existing in a complicated world, and we highly recommend giving it a read. In case you were at all confused on where we stand: **FUCK ICE**. ### [The Rise and Rise of FastAPI](https://www.youtube.com/watch?v=mpR8ngthqiE&ref=apisyouwonthate.com) FastAPI is a very popular web framework for building APIs in Python. Well, the same incredible people behind the documentaries for Vite, React, Node, and so many more have done one on FastAPI. As with all Cult Repo documentaries, it's totally worth watching. ### [Vercel vs Netlify vs Cloudflare: Serverless Cold Starts Compared](https://punits.dev/blog/vercel-netlify-cloudflare-serverless-cold-starts/?ref=apisyouwonthate.com) A breakdown of cold starts across Cloudflare, Netlify, and Vercel, with some useful conclusions about how quickly APIs were served, warm vs cold requests, and more. With more and more serverless API being deployed, it's definitely worth a read. ### [JavaScript: Backend Frameworks](https://2025.stateofjs.com/en-US/libraries/back-end-frameworks/?ref=apisyouwonthate.com) JavaScript's annual survey results dropped, and the backend results are very interesting. ExpressJS is still at the top of the pack, with NestJS usage continuing to rise. Hono is near the top in interest and rated highly for satisfaction. ### [Hono CLI](https://blog.yusu.ke/hono-cli/?ref=apisyouwonthate.com) Hono is a delightful, lightweight web framework that, from personal experience, is a delight to use. Well, they have been hard at work and have released another tool to improve the developer experience. A CLI for developers and agents, with docs, search, trigger requests, optimise your app, and more. Can not wait to start making this part of my dev flow. ### [httpx](https://github.com/projectdiscovery/httpx?ref=apisyouwonthate.com) I love it when a tool I have not seen before lands on my desk, and this one's interesting. This command-line tool lets you run multiple HTTP probers simultaneously to test or verify information on web servers, URLs, and other HTTP elements. --- ## APIs You Won't Hate Articles written and shared in our free [Slack community](https://slack.apisyouwonthate.com/?ref=apisyouwonthate.com). ### [Alternatives to Postman](https://apisyouwonthate.com/blog/http-clients-alternatives-to-postman/) Our rundown on viable alternatives to Postman is doing the rounds again - more and more devs are changing the way they integrate tooling into their workflow for building and testing APIs, and tools like Bruno and Insomnia offer great devX without vendor lock-in. --- ## From our Community Articles written and shared in our free [Slack community](https://slack.apisyouwonthate.com/?ref=apisyouwonthate.com). ### [Apitally Now Supports OpenTelemetry Tracing](https://apitally.io/blog/tracing-release-announcement?ref=apisyouwonthate.com) Simon Gurcke has been showing off his indie project in our Slack community for a while now. He recently released an update that adds OpenTelemetry tracing, keeping your metrics, logs, and traces for your APIs in one place. Keep up the great work, Simon. ## Support APIs You Won't Hate When you become an member, you'll get access to members-only content while directly supporting our work. Your support helps us to keep making resources for the API community. [Become a member today](https://apisyouwonthate.com/#/portal) Thanks so much to our members: **Kin**, **Juxt, Alex R, Nolan S, Brandon K, Frank, James D, Bill D,** and **Abdelhadi D**. Your support means the world to us! ✌️ Until next time, Alexander, Phil & Mike ### Sustainable APIs URL: https://apisyouwonthate.com/newsletter/building-sustainable-apis/ Last updated: 2026-02-05T11:11:30.000Z The only thing I love more than a well-designed API is one that's built with performance and sustainability in mind. Tom Kerkhove from Microsoft's Azure API Management team appears in the latest episode of the [Enviorment Variables](https://shows.acast.com/environment-variables/episodes/azure-api-management?ref=apisyouwonthate.com). A great conversation about how well-designed APIs can reduce energy use and improve a system's efficiency. An excellent conversion on how caching, throttling, and observability improve reliability and developer experience. Really nailing how choices on your API layer can have a massive impact on cost, performance and sustainability at scale. \-- Alexander ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2025/08/Pasted_Image_8_4_25__7_14---AM.png) ## The fastest way to build production-ready MCP servers Gram by Speakeasy: turn your API platform into an AI platform. Create tools from OpenAPI, curate into custom toolsets, and deploy hosted MCP servers [Get started today](https://app.getgram.ai/?utm%5Fmedium=email&utm%5Fsource=apisywh) ## The API Roundup *API News, links, and tools from around the web* ### [Terrible Train APIs](https://bsky.app/profile/lcas.dev/post/3mb3di22hts2m?ref=apisyouwonthate.com) Who doesn't love an API and trains... turns out the people building the API for the trains. An excellent thread from Luc Casonato (creator of Fresh) on the APIs from two Italian high-speed train operators. We got XML in JSON, spelling mistakes, hardcoded passwords and more. ### [Clerk: API Keys Public Beta](https://clerk.com/changelog/2025-12-11-api-keys-public-beta?utm%5Fsource=cooper%5Fpress&utm%5Fmedium=newsletter&utm%5Fcampaign=api-keys&utm%5Fcontent=01-15-26&dub%5Fid=9RwfUFXg9mIuTunM) Clerk is way more than simplified Auth and React components; it's a complete user management setup, and it now includes API keys. With a simple flick of the switch, allow users to create, view and revoke API keys to access your API. Their backend SDK has a great setup, making scope, claims and expiration easy. Personally, I always loved using Clerk, and it's great to see this addition. ### [Express 5 Migration](https://app.codemod.com/registry?q=scope%3Aexpressjs&ref=apisyouwonthate.com) Express 5 has been out a while now, but we all know migration can be complex. Thankfully, there are some incredible Express Codemods to make your transition so much easier. ### [Cheat Sheet](https://github.com/pransh15/http-status-code-emoji-cheatsheet/blob/main/http-emoji-cheatsheet-1.pdf?ref=apisyouwonthate.com) Who doesn't love a good cheat sheet? Well, MDN has got you covered with a lovely PDF of all the different HTTP Codes with Emoji pairings. I have linked the PDF from GitHub for a nice high-resolution version. ### [Ktor 3.4.0 ](https://go.bsky.app/redirect?u=https%3A%2F%2Fkotl.in%2Fn31gtm&ref=apisyouwonthate.com) My favourite framework for building APIs in Kotlin has recently dropped a new release with their OpenAPI generation now fully ironed out. The releases also include a bunch of great features, including Zstd compression support, an HTTP request lifecycle plugin, and more. --- ## APIs You Won't Hate Articles written and shared in our free [Slack community](https://slack.apisyouwonthate.com/?ref=apisyouwonthate.com). ### [OpenAPI.Tools](https://openapi.tools/?ref=apisyouwonthate.com) It's been almost two weeks since we dropped our new OpenAPI.Tools website and we have already had some excellent contributions. Remember, if your favourite tool is missing or needs an update, it only takes a small PR. --- ## APIs You Won't Hate Community We love a good community submission. If you're an APIs You Won't Hate member, and you'd like to have your work featured in the community section of the newsletter, send us a message in our free [Slack community](https://slack.apisyouwonthate.com/?ref=apisyouwonthate.com) to get the ball rolling. ## Support APIs You Won't Hate When you become an member, you'll get access to members-only content while directly supporting our work. Your support helps us to keep making resources for the API community. [Become a member today](https://apisyouwonthate.com/#/portal) Thanks so much to our members: **Kin L, Juxt, Alex R, Nolan S, Brandon K, Frank, James D, Bill D, and Abdelhadi** . Your support means the world to us! ✌️ Until next time, Alexander, Phil & Mike ### A Shiny New OpenAPI Tools URL: https://apisyouwonthate.com/newsletter/openapi-tools-relaunch/ Last updated: 2026-01-15T13:21:08.000Z The stars aligned over the Christmas break, and [Mike](https://mikebifulco.com/?ref=apisyouwonthate.com), [Phil](https://philsturgeon.com/?ref=apisyouwonthate.com), and [I](https://alexanderkaran.com/?ref=apisyouwonthate.com) came together to bring you a revamped [OpenAPI Tools website](https://openapi.tools/?ref=apisyouwonthate.com). ![screenshot of oenapi.tools website](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2026/01/image.png) A fresh look for openapi.tools 🥳 The new version is jam-packed with goodness. It has pages for each category, with guides, articles, and reviews for each tool. We have added support for Overlays and Arazzo. Tools that don't support OpenAPI v3.1 or later are listed *only*on the new [legacy page](https://openapi.tools/legacy?ref=apisyouwonthate.com). We can't wait for you to check it out. The site is built with Astro, hosted on Netlify, and driven by a collection of markdown files representing each tool. If your favorite OpenAPI tools are missing, [a quick PR](https://github.com/apisyouwonthate/openapi.tools?ref=apisyouwonthate.com) is all it takes to add them. \-- Alexander ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2025/08/Pasted_Image_8_4_25__7_14---AM.png) ## The fastest way to build production-ready MCP servers Gram by Speakeasy: turn your API platform into an AI platform. Create tools from OpenAPI, curate into custom toolsets, and deploy hosted MCP servers [Get started today](https://app.getgram.ai/?utm%5Fmedium=email&utm%5Fsource=apisywh) ## The API Roundup *API News, links, and tools from around the web* ### [Fern is joining Postman](https://buildwithfern.com/post/postman-acquires-fern?ref=apisyouwonthate.com) Postman has been on a buying spree recently, acquiring Fern (past sponsor), a tool that creates beautiful docs and SDKs for your API. I always have mixed feelings about acquisitions, with some turning out to be great and others signalling a slow death of the company. A positive note is that the entire Fern team is joining Postman, which is very different from the recent tech industry acquisitions we have seen. Hopefully, this means they will continue to steer the product in the right direction. ### [Postman acquires liblab](https://blog.postman.com/postman-acquires-liblab/?ref=apisyouwonthate.com) Five minutes after I heard that Postman was acquiring Fern, I learned that liblab had also been gobbled up. Personally, I not heard of liblab before, but they also offer SDK generation from OpenAPI spec. Now I am honestly flummoxed as to why they would buy two? Perhaps the focus is on acquiring their customers and removing two of the top tools from the market? ### [Bruno Version 3](https://www.usebruno.com/?ref=apisyouwonthate.com) Bruno is a local-first, open-source API Client. Honestly, I love Bruno, and they just released version three, which includes a host of new goodies—a much-welcome UI refresh, YAML support, enhanced Git features, and now a built-in terminal. ### [GraphQL loves REST](https://mbonnin.net/2025-12-15%5Fgraphql%5Fover%5Frest/?ref=apisyouwonthate.com) A comparison between GraphQL and REST, comparing their strengths and weaknesses. While I disagree that latency is a REST issue, please don't make me tap the sign; design first is what matters. If you find your client making too many round-trip requests, your API design may need review, and should be [designed for better cacheability](https://apisyouwonthate.com/blog/api-design-basics-cacheability/), or you may be using the wrong solutions. Anyway, rant aside, the article examines persistent queries and how they address some of GraphQL's downsides. --- ## APIs You Won't Hate Articles written and shared in our free [Slack community](https://slack.apisyouwonthate.com/?ref=apisyouwonthate.com). ### [OpenAPI Tools](https://openapi.tools/?ref=apisyouwonthate.com) As I mentioned in the intro, OpenAPI Tools got a significant refresh. Is there a tool you're missing from the site? Check out the contribution guidelines for [adding a new tool](https://github.com/apisyouwonthate/openapi.tools/blob/main/CONTRIBUTING.md?ref=apisyouwonthate.com#-adding-new-tools). --- ## From our Community Articles written and shared in our free [Slack community](https://slack.apisyouwonthate.com/?ref=apisyouwonthate.com). ### [Arazzo Specification 1.0 - Cheat Sheet](https://jentic.com/extra/arazzo-specification-cheatsheet.pdf?ref=apisyouwonthate.com) Just before Christmas, Frank shared this great cheatsheet for the Arazzo spec in our Slack group. Thank you, Frank. ## Support APIs You Won't Hate When you become an member, you'll get access to members-only content while directly supporting our work. Your support helps us to keep making resources for the API community. [Become a member today](https://apisyouwonthate.com/#/portal) Thanks so much to our members: **Kin L, Juxt, Alex R, Nolan S, Brandon K, Frank, James D, Bill D, and Abdelhadi** . Your support means the world to us! ✌️ Until next time, Alexander, Phil & Mike ### Design First, AI Never URL: https://apisyouwonthate.com/newsletter/design-first-ai-never/ Last updated: 2025-12-15T14:12:05.000Z What a whirlwind of a year. AI has been pumped into everything, and the constant marketing push informs me weekly that my job is no longer needed. I have been searching for the right way to articulate the importance of good API design in the age of AI, and this post [from Oxide](https://rfd.shared.oxide.computer/rfd/0576?ref=apisyouwonthate.com#%5Fllms%5Fas%5Fwriters) hit close to the mark for me. LLMs are bad writers, and involving them early in the process can taint your output. Circling back to your APIs, you should focus on designing and writing good APIs flows yourself; use LLMs for review, not for design. As the year comes to an end, this will be the last newsletter. We're back on the 15th of January. From the whole team at APIs You Won't Hate, have an amazing Christmas Holiday and a fantastic New Year. See you in 2026! \-- Alexander, Phil and Mike ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2025/08/Pasted_Image_8_4_25__7_14---AM.png) ## The fastest way to build production-ready MCP servers Gram by Speakeasy: turn your API platform into an AI platform. Create tools from OpenAPI, curate into custom toolsets, and deploy hosted MCP servers [Get started today](https://app.getgram.ai/?utm%5Fmedium=email&utm%5Fsource=apisywh) ## The API Roundup *API News, links, and tools from around the web* ### [OpenAPI 3.3 ](https://github.com/OAI/OpenAPI-Specification/discussions?ref=apisyouwonthate.com) With OpenAPI 3.2 out in the wild, bolstering a substantial set of new features, attention is turning to version 3.3\. If you're keen to get involved or learn what the OpenAPI team is thinking for version 3.3, check out their discussion in the GitHub repo. ### [OpenAPI 3.2 is here](https://quobix.com/articles/openapi-3.2/?ref=apisyouwonthate.com) Is this your first time hearing about OpenAPI 3.2? If so, where have you been living, under a rock? If you want a complete lowdown on all the features, check out this great post from Quobix (Dave Shanley). Honestly, I love the excitement that comes through in his writing for the new version. ### [OpenAPI.NET: The Biggest Update Ever](https://devblogs.microsoft.com/openapi/openapi-net-release-announcements/?ref=apisyouwonthate.com) A few issues ago, I mentioned that OpenAPI.Net would receive OpenAPI V3.2 support in OpenAPI.Net V3\. However, I was selling them short; they also added new OpenAPI features to OpenAPI.Net v2\. (*Wow, every time I re-read that sentence, my tongue ties in knots* 😂) The short version is that V2 now supports the OpenAPI spec 3.1, and V3 supports 3.2\. Check out the post for all the new supported features. ### [State of URL parsing performance in 2025](https://www.yagiz.co/state-of-url-parsing-2025/?ref=apisyouwonthate.com) The author of cURL recently questioned the performance comparisons of Ada's URL parser. Yagix Nizipli jumped in and broke down the difference between cURL and Ada. In short, Ada delivers significant performance gains when parsing URLs, but we were not comparing apples to apples because Ada and cURL follow different specifications. Worth a read. ### [Gin is a very bad software library](https://eblog.fly.dev/ginbad.html?ref=apisyouwonthate.com#gin-is-a-very-bad-software-library) If you're not familiar with it, Gin is one of the many Go web frameworks used for APIs, full-stack web applications, and more. However, Efron Licht does not like Gin and comes to the table with a whole heap of reasons why. It's a long blog post to summarise, but he gives solid reasons why you might not need Gin. A good read, regardless of whether you love or hate it. ### [Modernising Reddit's Comment Backend Infrastructure](https://www.reddit.com/r/RedditEng/comments/1mbqto6/modernizing%5Freddits%5Fcomment%5Fbackend%5Finfrastructure/?ref=apisyouwonthate.com) Reddit discusses migrating its read and write endpoints from a Python monolith to Go microservices. While the read endpoints were relatively easy to migrate by comparing responses, the write endpoints were more complex. A solid approach if you ever have to migrate endpoints to another system. ### [How to make your API a catalyst for growth](https://www.stainless.com/blog/how-to-make-your-api-a-catalyst-for-growth?ref=apisyouwonthate.com) A more commercial-focused post on how to get the most out of your API for you and your business. How can an API win you more customers and deals while driving awareness of your organisation? It's time to learn how your API can grow your business. --- ## APIs You Won't Hate Articles written and shared in our free [Slack community](https://slack.apisyouwonthate.com/?ref=apisyouwonthate.com). ### [**Zero-Downtime Migration from Laravel Vapor to Laravel Cloud**](https://apisyouwonthate.com/blog/zero-downtime-migration-from-laravel-vapor-to-laravel-cloud/) Stuck on Laravel Vapor and dreaming of moving to Laravel Cloud? Phil has you covered with a handy guide to migrating without breaking anything. Open your favourite code editor and get ready to migrate and take advantage of Laravel Cloud's functionality. --- ## From our Community Articles written and shared in our free [Slack community](https://slack.apisyouwonthate.com/?ref=apisyouwonthate.com). ### [Speakeasy OpenAPI 3.2.0 Support](https://github.com/speakeasy-api/openapi/releases/tag/v1.12.0?ref=apisyouwonthate.com) Tristan from Speakeasy reached out to our Slack channel last week to let us know they have added support for OpenAPI 3.2 in their parsing library and CLI tool. Get ready to update Speakeasy. ## Support APIs You Won't Hate When you become an member, you'll get access to members-only content while directly supporting our work. Your support helps us to keep making resources for the API community. [Become a member today](https://apisyouwonthate.com/#/portal) Thanks so much to our members: **Kin L, Juxt, Alex R, Nolan S, Frank, Bill, James D, Rich, Ryan T, Umair, Abdelhadi, and Brandon**. Your support means the world to us! ✌️ Until next time, Alexander, Phil & Mike ### Zero-Downtime Migration from Laravel Vapor to Laravel Cloud URL: https://apisyouwonthate.com/blog/zero-downtime-migration-from-laravel-vapor-to-laravel-cloud/ Last updated: 2025-12-11T16:33:17.000Z For Laravel PHP users, [Laravel Vapor](https://vapor.laravel.com/?ref=apisyouwonthate.com) was the hottest new serverless hosting experience and we were all very excited. Over time the excitement around serverless for APIs seems to have died down, and a more regular hosting experience has popped up to replace it: [Laravel Cloud](https://cloud.laravel.com/?ref=apisyouwonthate.com). No longer serverless, and a whole lot more functionality baked in ready to go. So, how to migrate over without breaking everything? ## Step 1: Sign up on Laravel Cloud [Create an account](https://cloud.laravel.com/sign-up?ref=apisyouwonthate.com), it's free and does not take long. Make an Organization for your company/team (e.g. "Protect Earth"). ## Step 2: Create an application Laravel Cloud links up the application to deploy, turning pushes into deployment triggers and commit messages into deployment messages. ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2025/12/import-from-git-1.png) This is a lot more like other classic Platform as a Service systems (think Heroku, Netlify, Vercel, etc.) and is a lot easier than needing to run a CLI from a CI server / GitHub Action. Once you have hooked up the application, given it a name and picked a region, you will be taken to see your new application with all its core resources being deployed. ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2025/12/default-application-2.png) The first deployment will happen when you click the bit Deploy button, or when you push a commit. One good commit to push would be disabling Vapor deployments so we know we're only deploying to one location. ``` git rm .github/workflows/deploy-vapor.yaml git commit -m "disable laravel vapor deployments" git push origin main ``` That's going to keep Vapor up and running, which we'll need for a smooth transition. ## Step 3: Point application to AWS RDS The new application in Cloud is going to start up without a database, cache, websocket, or any other resources, so you can expect to see plenty of errors. Seeing as Laravel Vapor is a thin wrapper around AWS, all of those resources can be used over the public network, so hop over to Vapor, get into the database settings, and enable public access. Now you don't **need** to make it public, you can do it with [Vapor tunnels](https://isyadiarung.medium.com/connect-and-dump-sql-from-aws-rds-mysql-server-of-a-vapor-project-the-easy-way-using-heidisql-86b35f818803?ref=apisyouwonthate.com), but it's a whole lot easier to make it public and just delete it before you get hacked. Vapor automatically injects username and password as environment variables, but we'll need to copy those down and paste them into Cloud environment variables in Settings to let it know where this RDS database is running. ``` DB_HOST={rds-db}.{rds-region}.rds.amazonaws.com DB_PORT=3306 DB_DATABASE=vapor DB_USERNAME=vapor DB_PASSWORD={your-password} ``` To get these details, head over to the Vapor dashboard, go to Databases, and find the password behind the Show Password link. If you can [connect to the database locally](https://www.youtube.com/watch?v=G4OXytPY%5F6E&ref=apisyouwonthate.com), you can use the exact same settings to have the Cloud application talking to it. Pop those environment settings into the application settings, then click Save and Deploy. The application will hopefully be pulling data from the database, and you're well on your way to having a functioning application. ## Step 4: Reuse AWS S3 Storage Thousands of photos of trees. Could not migrate them all easily so pointed the application at that same bucket. ``` { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "s3:PutObject", "s3:PutObjectAcl", "s3:GetObject", "s3:GetObjectAcl", "s3:DeleteObject" ], "Resource": "arn:aws:s3:::your-bucket-name/*" }, { "Effect": "Allow", "Action": [ "s3:ListBucket" ], "Resource": "arn:aws:s3:::your-bucket-name" } ] } ``` Let the application know where to find the S3 bucket, and you're good to go. ``` FILESYSTEM_DISK=s3 AWS_ACCESS_KEY_ID= AWS_SECRET_ACCESS_KEY= AWS_DEFAULT_REGION=eu-west-2 AWS_BUCKET=your-bucket-name ``` Save and deploy, and see if images are showing up. ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2025/12/image-shows-up-2.png) Yaaay. Hello tree. ## Step 5: Setup Cache, Queues, Logs Dwight Watson has [a great post](https://www.dwightwatson.com/posts/migrating-from-laravel-vapor-to-laravel-cloud?ref=apisyouwonthate.com) about some of the changes they needed to make getting caching, queues, and logs working during their migration. I forgot to take notes on this section since I started these migrations in August so I'm glad this post is covering that. [Migrating from Laravel Vapor to Laravel CloudI was a day-one subscriber to \[Laravel Vapor\](https://vapor.laravel.com). I work on a number of app![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/icon/icon.png)![](https://www.dwightwatson.com/posts/migrating-from-laravel-vapor-to-laravel-cloud/opengraph-image?717e708f1ee038ec)](https://www.dwightwatson.com/posts/migrating-from-laravel-vapor-to-laravel-cloud?ref=apisyouwonthate.com) When you've twiddled enough of these options that you have a cache (whether its database or Laravel KV), and have a queue running (whether its on the same worker as the web application or another queue resource) it's time to give it a decent bit of testing, get QA involved if you can, and make sure its all above board before going live. ## Step 6: Switch over DNS Add the custom domain to Cloud and update DNS settings to point to it. See the Laravel Cloud documentation on [custom domains](https://cloud.laravel.com/docs/environments?ref=apisyouwonthate.com#custom-domains). Once this is done, give it enough time for DNS to propagate, and we can start taking down Laravel Vapor resources one at a time to start shrinking that AWS bill down to zero. This can be done a few weeks or months later if you've got other stuff to be doing. The sooner you do it the sooner you save money but at least go for a nice walk. Ride a bike. See a goat. Come back refreshed. ## Step 7: Create a new database In Laravel Cloud create a new empty database, either MySQL or Postgres. When it creates the database **do not deploy** yet. Cloud will still have the `DB_*` environment settings pointing to RDS even if you do deploy this new database, but click "Save only" to play it safe. The strategy for migrating the database is simple: export the old database, then import it over here. Admittedly this is not truly "zero downtime", because the second you take this dump the data is stale. I did this late at night, knowing we were not planting any trees at midnight, and any late orders for tree sponsorships could be copied over afterwards. Stick Vapor into maintenance mode for a few minutes as you do this, to turn off queues and workers to avoid unexpected effects happening on the old system. You could set up DB replication if you truly need zero downtime, or just go cowboy mode and do this step quickly. ## Step 8: Export the old database Exporting the database via the CLI can be hit or miss if this is your first go, but for MySQL users it should look a little something like this: ``` mysqldump --set-gtid-purged=OFF --single-transaction -h {rds-db}.{rds-region}.rds.amazonaws.com -P 3305 -u vapor -p vapor > dump.sql ``` Our RDS database is MySQL so I downloaded MySQL Workbench for a bit of assistance. Connect to the old database using the same credentials pasted into Laravel Cloud. ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2025/12/mysql-workbench-connection-2.png) Select every table you want to export, outputting both the table structure and the data that goes in them. ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2025/12/mysql-workbench-export-2.png) Depending on what sort of data you have, you may need to fiddle around with various settings to get it exporting smoothly. Here are the settings I've used. ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2025/12/mysql-good-export-settings-2.png) As someone who uses a lot of MySQL GIS, I had to enable `hex-blob` to get `POINT()` and `POLYGON()` values working properly. ## Step 9: Import the database dump As soon as you have the database dump ready, import them with a command that looks a bit like this. ``` $ mysql --init-command="SET SESSION FOREIGN_KEY_CHECKS=0;" -P 3306 -h db-{host}.eu-west-2.public.db.laravel.cloud -u -p main < dump.sql -vv ``` Once that's finished running, remove the hardcoded `DB_*` environment variables from the Cloud application that were pointing it to RDS. When these are removed, click "Save and Deploy". Give it another round of Q&A, and take it out of maintenance mode, or go to bed. ## Step 10: Migrate AWS S3 to Cloud (Cloudflare R2 kinda) This part could take 20 minutes or two months depending on many files you have. Laravel Vapor uses AWS S3, but Laravel Cloud uses Cloudflare R2\. These services are essentially the same thing, and whilst they share the same API (Flysystem can talk to both of them) it's not possible to transfer files from one to the other in any but the slowest and daftest of ways. Amazon S3 has all sorts of clever ways to transfer content between buckets. This won't work. Cloudflare R2 has a bulk import tool called [Super Slurper](https://developers.cloudflare.com/r2/data-migration/super-slurper/?ref=apisyouwonthate.com), but there's no way to use that for your managed R2 bucket. I had awscli installed so I downloaded the whole bucket. ``` mkdir backup-bucket brew install awscli aws configure aws s3 sync s3:// ./backup-bucket ``` This will clone the entire bucket into `.` This worked fine for Protect Earth as we only had 120,000 photos of trees to download, but another tree planting charity had a million and that was hecking slow. For the upload of these files, awscli is not going to work as the authentication will be all wrong. As a big fan of Cyberduck I cracked out their CLI utility `duck` and added the [Cloudflare R2 profile](https://docs.cyberduck.io/protocols/s3/cloudflare/?ref=apisyouwonthate.com) to help it quick the right language. ``` mv "~/Downloads/Cloudflare R2 Storage \(S3\).cyberduckprofile" "~/Library/Group Containers/G69SCX94XU.duck/Library/Application Support/duck/" ``` In the Cloud dashboard create and attach a new object storage. Name it whatever, and attach it to production with another name which could be whatever. I named both `cloud`. Create an access token for that bucket called `duck upload` or something, and we can use this to import the files. Click "View Credentials" on the Cloud dashboard and you'll see a bunch of `AWS_*` environment variables, even though this is for R2. ``` AWS_BUCKET=fls-f8efy7-b2e6-49db-b13e-d1bf4c342bcf AWS_DEFAULT_REGION=auto AWS_ENDPOINT=https://numbers-and-letters.r2.cloudflarestorage.com AWS_URL=https://fls-9fae57b9-b1e4-47db-b43e-c1bf4c342bcd.laravel.cloud AWS_ACCESS_KEY_ID=id AWS_SECRET_ACCESS_KEY=secret ``` As far as duck is concerned, the access key will be the username, and the secret will be the password. For the host look for `AWS_ENDPOINT=`. Grab the whole value, and cut off the `https://` part. So if it was `https://example.r2.cloudflarestorage.com` we just want `example.r2.cloudflarestorage.com`. Then add the `AWS_BUCKET` value after it, so if your bucket was `my-bucket` you'd end up with `example.r2.cloudflarestorage.com/my-bucket`. Now you can use the `duck` command to upload your files: ``` duck --verbose --retry --parallel --username {AWS_ACCESS_KEY_ID} --password {AWS_SECRET_ACCESS_KEY} --upload s3-cloudflare-r2://.r2.cloudflarestorage.com/fls-f8efy7-b2e6-49db-b13e-d1bf4c342bcf/ ./backup-bucket/ ``` this may take forever, and you might want to run it from a server, a raspberry pi, or some other always on device so it doesn't cancel every time you move anywhere for the new few months, but no dramas if it does cancel. Run duck again with `--existing skip` and it'll upload missing files, even doing a md5 checksum to make sure there were no broken uploads. You can skip the md5 if thats too slow by using `--nochecksum`. ## Finally Now you can delete whatever bits of Laravel Vapor you've not already turned off. Wipe out AWS RDS, all the databases, proxies, and snapshots. Do as much of it as you can through the Laravel Vapor interface to reduce clicking around. ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2025/12/delete-proxy-anytime-2.png) Remove all the images from all of the S3 bucket your user files were stored in and everything with `vapor` in the name too. Close down all the queues, remove any remaining Lambdas, and eventually you should be able to close down your AWS account entirely, and then close your Laravel Vapor account too. ## How is Laravel Cloud Generally I'm a big fan. My charity [Protect Earth](https://protect.earth/?ref=apisyouwonthate.com) is down from $120/month to $30/month and performance is better thanks to built in support for [Laravel Octane](https://cloud.laravel.com/docs/compute?ref=apisyouwonthate.com#laravel-octane). Most importantly for API developers, Laravel Cloud has [network caching enabled by default](https://cloud.laravel.com/docs/network?ref=apisyouwonthate.com), helping you focus on [designing APIs to more cacheable](https://apisyouwonthate.com/blog/api-design-basics-cacheability/) without needing to faff around setting up the mechanics yourself [like you did with Vapor](https://apisyouwonthate.com/blog/http-caching-for-laravel-vapor/). Finally the built in support for [Laravel Nightwatch](https://nightwatch.laravel.com/docs/guides/cloud?ref=apisyouwonthate.com) has provided a lot of the same use-cases covered by Sentry, but with one less price tag to drain the charity coffers. Let me know how the migration goes, and if you get stuck along the way [I can help you out](https://cal.com/philsturgeon?ref=apisyouwonthate.com). Drop a comment with improvements or feedback if I've missed anything! ### NestJS: Bad, or Really Bad? 😉 URL: https://apisyouwonthate.com/newsletter/nestjs-bad-or-really-bad/ Last updated: 2025-12-01T13:17:02.000Z If you fancy having a good giggle and reading something spicy, check out this post on NestJS from Reddit: The original post in [r/node](https://www.reddit.com/r/node/?ref=apisyouwonthate.com) states, "[NestJs is bad, change my mind](https://www.reddit.com/r/node/comments/1ov6xrd/nestjs%5Fis%5Fbad%5Fchange%5Fmy%5Fmind/?ref=apisyouwonthate.com)". To me, NestJS is like someone rebuilt Spring Boot in JavaScript, and I have mixed feelings about Spring Boot. Can you build complex applications and scale APIs with NestJS? Yes. Is it easy to use? Honestly, it depends on the team or the developers. My criticisms last time I used it were a lack of native ESM support and the focus on class validators and transformers. For me, Zod or TypeBox are a much nicer way to validate API requests. \-- Alexander ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2025/08/Pasted_Image_8_4_25__7_14---AM.png) ## The fastest way to build production-ready MCP servers Gram by Speakeasy: turn your API platform into an AI platform. Create tools from OpenAPI, curate into custom toolsets, and deploy hosted MCP servers [Get started today](https://app.getgram.ai/?utm%5Fmedium=email&utm%5Fsource=apisywh) ## The API Roundup *API News, links, and tools from around the web* ### [Simple HTTP, REST, and SSE client library for Go](https://resty.dev/?ref=apisyouwonthate.com) A Go library called Resty that makes it super easy to build a REST API. Honestly, if you're not picking this library on name alone, you might need to reflect on your engineering choices up until this point. Resty includes so much, including HTTP/3 support, method chaining, concurrent safety, middleware, and more. Stop wasting time and start building now. ### [Bruno Version 2.15.0](https://www.usebruno.com/?ref=apisyouwonthate.com) Bruno recently released a new version with a better test runner and improved import/export support, including support for tools like Postman and HTTP Streaming. If you have never heard of Bruno before, think of it as an offline, open-source version of Postman. Honestly, that description does not do it justice. Go check it out. ### [Portman CLI 1.0 - Better API testing](https://www.apideck.com/blog/announcing-portman?ref=apisyouwonthate.com#from-pr-to-portman-cli) Want better testing with your API? Say hello to Portman CLI. Feed in your OpenAPI docs, and Portman then transforms these into collections in PostOffice while also injecting contract and variation testing. If you use Postman heavily, this is a great tool that lets you leverage its advanced features with ease. ### [GitBook: Beautiful API Docs](https://gitbook.com/docs/api-references/openapi?ref=apisyouwonthate.com) Great tools that make documentation easy are the best. GitBook plugs straight into Git (who would have guessed) and has excellent support for OpenAPI. Integration with CI/CD, the ability to test your API in the docs, and the ability to describe ENUMs will make building good API docs easy. ### [OpenAPI won't make your APIs AI-ready. But Arazzo can.](https://bump.sh/blog/make-your-apis-ai-ready/?ref=apisyouwonthate.com) Arazzo allows you to explain multi-step API workflows. Bump.sh explains that while this is great for explaining to developers how to use your tools, it's also fantastic input for AI Agents. Win-win across the board. --- ## APIs You Won't Hate Articles written and shared in our free [Slack community](https://slack.apisyouwonthate.com/?ref=apisyouwonthate.com). ### 🎙️ [Building a Sustainable Future in APIs with Kin Lane](https://apisyouwonthate.com/podcast/building-a-sustainable-future-in-apis-with-kin-lane/) In the latest edition of our podcast, Kin Lane drops by to talk to Phil Sturgeon about his new startup, the changing landscape of API tech, why REST fundamentals are still important, and building sustainable API tools. --- ## From our Community Articles written and shared in our free [Slack community](https://slack.apisyouwonthate.com/?ref=apisyouwonthate.com). ### [API Schema Automation for Developers](https://www.appear.sh/?ref=apisyouwonthate.com) Thomas Peterson shared his startup in our tooling channel last week with a very interesting tool that helps teams work with internal APIs. It supports no vendor lock-in and detects schema changes based on traffic patterns. Definitely worth a look, not just because it's from Australia. Thanks for sharing, Thomas. ## Support APIs You Won't Hate When you become an member, you'll get access to members-only content while directly supporting our work. Your support helps us to keep making resources for the API community. [Become a member today](https://apisyouwonthate.com/#/portal) Thanks so much to our members: Kin L, Juxt, Alex R, Nolan S, Frank, Bill, James D, Rich, Ryan T, Umair, Abdelhadi, and Brandon. Your support means the world to us! ✌️ Until next time, Alexander, Phil & Mike ### Building a Sustainable Future in APIs with Kin Lane URL: https://apisyouwonthate.com/podcast/building-a-sustainable-future-in-apis-with-kin-lane/ Last updated: 2025-12-01T13:15:57.000Z ## Chapters 00:00 Introduction and Welcome 01:07 Catching Up with Kin Lane 02:35 Bloomberg and Spectral APIs 06:11 Frustrations with AI Pitches 07:26 AI in the API World 11:37 Challenges in API Development 16:30 The Evolution of API Tools 32:04 The Cycle of API Startups 41:25 Conclusion and Farewell ## Show Notes - Kin Lane - [personal website](https://kinlane.com/?ref=apisyouwonthate.com) - [API Evangelist](https://apievangelist.com/?ref=apisyouwonthate.com) - Follow Kin's startup [Naftico on LinkedIn](https://www.linkedin.com/company/naftiko/?ref=apisyouwonthate.com) - [Kin on GitHub](https://github.com/kinlane?ref=apisyouwonthate.com) - Kin on Socials: [Bluesky](https://bsky.app/profile/apievangelist.com?ref=apisyouwonthate.com) [Mastodon](https://mastodon.apievangelist.com/web/@api?ref=apisyouwonthate.com) [LinkedIn](https://www.linkedin.com/company/api-evangelist/?ref=apisyouwonthate.com) - Check out Kin's Podcast, API Evangelist Conversations ([Spotify](https://open.spotify.com/show/3Yd46TJhCvXvWX7UZEcFlO?ref=apisyouwonthate.com) & [Apple Podcasts](https://podcasts.apple.com/us/podcast/api-evangelist-conversations/id1765157921?ref=apisyouwonthate.com)) ## Support APIs You Won't Hate [Become a member](https://apisyouwonthate.com/#/portal) #### Transcript \[00:00:00\] Hello everybody. Welcome to APOs You Won't Hate. I'm Phil Sturgeon and I'm really excited this time to be joined by the Very Kin Lane, also known as the API Evangelist. How you doing kin? I'm doing very well today. Beautiful. Sunny day here in New York City. You're in New York City. I'm so jealous. I used to live there and I don't now. It makes me mad. How is it It misses you. It misses you and you should come back. You, me, directly told me directly. Have you got, have you got ice all up in New York City or are they staying outta there? Uh, I think they're kind of weighing, they're staying out for right now, but it's coming. I'm sure at some point here, I think they just picked other, picked on other second tier cities or still big, but to test things out and then, then they're gonna come here. I just feel like they're, they're gonna get to like the Bronx and wish they didn't. It's gonna be real interesting when they get there. Yeah. It'll be, it'll be, yeah. It's not, it's not gonna be pretty, but, oops. Let's go back to Portland. This was a bad choice. Everybody loves to hate Portland, so Portland's a good testing ground \[00:01:00\] when you're, you're, you're putting hate out there. Yeah. Anyway, that's, that's not exactly what we're here to talk about, but Excellent. Uh, it's been a while since I've seen you in person. You're at all the API conferences and I don't do those anymore, but, uh, Hey, what, what are you up to at the moment? So like you were super involved in open API for a bit. You were involved Postman for a bit. Like what's going on? What have you been up to? Yeah, so it has been a few years I would say. So I think when I last. Worked with you and talked. We, I was a co-director of the Open API initiative. Um, I think at the time I was starting up the Postman Open Technologies program and kind of doing the. I don't know the, how do you, how do you articulate the, the startup plus open source and open standards you invest in and you build homes for these places. And, and so I invested in open API Async, A-P-I-J-S-O schema, very keen to invest in spectral baby of yours. Mm-hmm. And, uh. And you and I kind of, you \[00:02:00\] know, connected again around. I mean, we've been on the conference circuit for a while, but I did four, finished four years at Postman and then I left and spent a year at Bloomberg standing up an API governance program there here in New York City. And spectral was, was pivotal to that and still remains. But now I've left and I'm doing a startup on on API integration and consumption and automation. That's very cool. Firstly, what the heck is Bloomberg doing with Spectral? They just got big APIs and they wanted a style guide or what's going on there? I've not heard about that one. Yeah, so I sprawling organization, lots of business domains that are very separate, like legally compliant, like buy side, can't talk to sell side. The government people can't talk. Like there's some legit reasons why there's different groups. Not all the other BS reasons. The organizations we've worked in don't talk to each other. That is really interesting. 'cause usually it's just like, this team is in Singapore or some shit, or, or it's just like, that team is run by Gary and I fucking hate Gary, but like the, the \[00:03:00\] lack of communication between teams is usually not like legally mandated wherever I've been, even, even at WeWork as much of a shit state as that was. That's quite funny. Yeah. No, and it's so, it's significant, but then there's still other, you know, common organizational issues, team topology. Tribal, you know, I really prefer to use the word tribe 'cause that's how people tend, not just at Bloomberg, everywhere, I think tend to each other. But, so map the landscape. Here's all the open API, uh, swagger, you know, it was half and half. And then come up with a base rule set and then I, you know, your, your rule set guidance out there. That is the always the place to start with. Um, and, and kind of assemble that base set of rules that comes with stoplight. But I think you, you've had several stories that have expanded on it in meaningful ways. Yeah. And so applied that base set of rules. Then set in motion, also governance of async, API, on top of that for the event driven as well as GraphQL. GraphQL was more traditional \[00:04:00\] linting and schema linting rather than, uh, spectral. But, uh, yeah, just applying a, a common sense rule set. I mean, minimum viable bar rule set. Across a 25,000 person org, you know, several hundred APIs. And then, uh, watch the, the chaos that ensues and all the reasons why people can't comply in the design reviews. Oh, man, we gotta get this out the door right now, man. Yeah. We ain't got time for this stuff. You know, you've heard all the reasons we gotta, we gotta push non-standard garbage to production right now that, that we'll be trapped with for a million years. But yeah, we couldn't possibly wait another day to fix that before it goes out. That would be bad. Yeah, that is, that is stressful. Yeah. But, but they're, they're doing the good work there. They're continuing, they're automating around it. Yeah. Very nice. And then I think going ahead first into this whole AI realm that everyone else is being forced into. So, so, yeah, but I left. Been almost a year now, and I'm doing my own startup now, trying to make sense of this madness and, but with the level of \[00:05:00\] control, I think that you, me and you and our old curmudgeonly aged demand and need, otherwise, I was out the door of tech going to look at do something entirely other like you have, so, yeah. Yeah, yeah. I mean, yeah. Most of my brain right now is focused on the fact that we have spent two years waiting for our next bit of land to come through, and as soon as like. One thing needs to happen. We've got half a million pounds just sitting in a bank account waiting to be given to this person, and as soon as they like do one thing, it's literally like send one email. We can just buy it. It's so fucking frustrating. We've been there for like three months and what I wanna do is go and like work with the beavers that are already on that site and design like a sweet habitat for them. They're gonna love it. I'm gonna love working with them. It's gonna be a nice time. But instead, yeah, just like hitting refresh on my inbox consistently is pretty stressful. But when I'm not hitting refresh on my inbox, I, well, when I am hitting refresh on my inbox, I'm seeing emails hitting the protector \[00:06:00\] inbox. Um, such as Rick from Current company. I won't mention. Just absolutely convincing me that I need this new AI startup, this AI startup is going to. Read all of the emails that hit my inbox in my own voice. I know y'all think I love the sound of my own voice, but god dammit, I actually do not wanna hear it all that much. Not, not like reading an email from an asshole in my voice as well is a weird pitch. So I get pretty frustrated with people sending these AI pitches sometimes to a PO he won't hate. Fair enough. You want us to talk about it, sending it to Protect Earth. However, does my head in, so I got this email the other day. I responded with, I'm not investing any time or effort thinking about the AI bubble, especially not on this email address you scraped online somewhere. If you thought about the address you were contacting, you would've noticed that email is, uh, the email address is a climate and nature charity, which is being absolutely devastated by compute, burning slot machines like the one you're working on. I'd recommend you look for another job opportunity promptly and consider getting into Green Tech. Linked to my article about \[00:07:00\] Green Tech, what is that and how to get into it if you'd like to do something constructive instead. Warm regards. Right. Um, so I am not necessarily the most reasonable person when it comes to talking about ai. Now, I, I can back all this up with a long chat. We can talk for fucking six hours about why it's a stupid idea. But I did really appreciate your blog post the other day where you had a few more useful things to say compared to that nonsense that I just wrote, to that poor person who's just trying to make a wage. You wrote, I entered into these conversations in the early part of the summer with much empathy for people doing artificial intelligence without much empathy for people doing artificial intelligence. You were the same as me and have come out the other side with a lot more empathy for what's happening and where we are headed. I still feel strongly that AI is overhyped and overblown and obfuscates some very dangerous realities around labor, climate change and copyright. But as I've learned more, I've softened the edge on how I talk with folks about this moment that we are in, and specifically regarding how we are going to address what is next without jeopardizing what we've already built. Now that \[00:08:00\] is pretty well worded. Do you wanna expand on that a whole bunch? Like what? What have you learned that makes you not think that everyone working on AI stuff is just being a silly money grabbing nutty. Well, I mean, I think the bullet points in that the, the labor, the job precarity kind of part of this, like talking to people, like when you talk to people behind the scenes, they're like, look, I'm totally with you. I'm, I'm, I'm anti ai. Please don't publish that or share that or let anybody know that. But, um, I have to, like, this is my job and I, I just can't afford losing my job right now. And, and so, you know, the AI thing from squeezing white. Mail mostly, or you know, in, in Silicon Valley all the way to Africa, to South America, that spectrum where, you know, some of us on the upper end are making, you know, a quarter of a million dollars a year all the way down to making a quarter. You know, every couple, you know, tasks that they do or respond to that labor squeeze is kind, is \[00:09:00\] real, you know, and mm-hmm. I mean, I think if the powers that be want, they would want us all like, you know, in, you know. Working just online, you know, for minimum viable and kind of that Uber, you know, price surge. Kind of like, oh, here's a demand to work. Click, click, click. And, and we'll make sure you can barely eat and you won't have healthcare. You know? So people are genuinely scared. And, and, and I don't wanna all, and once people talk to me, it's similar to you. You're not as scary as you, as you sound, you know, you're, you're actually a pretty compassionate guy. Once you talk to people and they realize I'm not as scary mean, um, I wanna hear like, what are their challenges? And they don't wanna lose their job in their healthcare, especially in the US healthcare, you know, like, don't. You lose that and then now there's, you're gonna lose your H one V ones. You're not gonna be, be here as an immigrant. So there's, there's a lot of other things that get in the way of me just being a complete asshole right up front. That's funny. Yeah. I mean, obviously the, um, in most companies there's the suits who, who don't really know \[00:10:00\] anything about anything, who have just heard that like this is the future. Um, 'cause there are, there, there's, there's two types of suits. There's the ones. That know that this is bullshit and are pushing it to make money, and there's the ones that. Don't know that this is bullshit and are desperately trying to keep up so that they can make some money. I mean, that's just, that's just capitalism right there. But I feel like, yeah, there's a lot of people, there's a lot of people in the C-suite just being like, wow, AI is gonna be the thing we all gonna do. Ai, shove it in everywhere, blah. Make some money. And, um, they're all British. Even if it's in Silicon Valley, that's just the money. That's the money guy voice that I do. I don't have another one. Yeah. So obviously the people being, being given their deliverables, like KPIs get set. Product targets are given, sales are are being told. You know, sales are always like defining the roadmap of just like we've been, we told everyone that we can do this and now you have to go figure it out. So there's a lot of layers of business in between kind of the person who's writing the feature. And I'm, I'm never usually mad at \[00:11:00\] them. It is just, uh. A lot of, I feel like basically my, my thing with ai, especially in the API scene, we could talk about AI in general. That's huge. But like in the API scene, there was an API days conference, I went to, I don't remember what it was. I think it was 2020 in Paris and it was like before they started doing AI tracks. And it was just like everyone was there that year talking about ai. Obviously that's why they got AI tracks after that. And it was just loads to loads of people being like. We are developing this tool, which means that you don't need to, you don't ever need to integrate with an API ever, again, you don't ever need to think about any of the data that's being handed around. You can just have a random API and, and you can generate a random front end for it based on vibes, and it would just be brilliant and absolutely fine. And he got the whole way through his 45 minute odd talk going on about how you never need to write integration code ever again. And it would all just work perfectly. And, and one, \[00:12:00\] the first question was like, so how does any of it work? Is it literally, can you just point it at any API like you've been saying, or like, do you have to build certain conventions into your APIs? Like. Well, you do need to like rewrite your API to follow these specific, very restrictive rules and guidance and everything else, right? And obviously this is what became the precursor for Mt p. They wanted to add a layer in between a perfectly good functioning API and whatever bullshit the slot machine required to function. And so that it was frustrating for a long time that like my very first introduction to it in the API world was someone lying through their teeth and then crumbling under the very first follow up question they got. But since then, people just seem to have got better at hiding some of that and, and it's never got better at functioning. It just seems to be they're better at selling you on the product and convincing you that you should try really hard. By which point you're kind of locked in. Haven't been delving into delving, uh, into the kind of products \[00:13:00\] as much lately to see if they've got any better. And I'm sure people are working hard on solving it. And I imagine that, that, that may well be what you are trying to do, right? Like, make this stuff work good. Because from the early days, I kind of went and then I just cut. I'm gonna stop paying attention. Like how, how do you see AI helping? Nicely with integrations. Like what, what do you do to make it be useful beyond the demo in a talk that gets people signing up to your product and then wandering off after they've paid you for a few months and it hasn't worked, but you still had them paying for a few months? Yeah, I'm less, um, I mean there are very few things I use AI for that are, are useful and practical. I generate some schemas. And I always, you know, I'll, I'll pass it a pricing page, URL or the content from a pricing page for a, a SaaS, and then I give it A-J-S-O-N schema and say, I want that pricing page in that schema. And I do that programmatically. I do that fairly \[00:14:00\] common. I'll do it with rate limit pages. I just have standard schema for those things that I do because I don't, I don't want just my docs to be. Machine readable. I want my pricing rate limits. I want the whole package, you know? Yeah. The whole experience to be, so I do use it for some of that, but beyond that, like I don't use it to edit stuff. I like writing, I like, and I, I like editing my shit even though I screw it up sometimes and mess because that's how I learn. So like I don't wanna off load that stuff. I guess, where I'm finally finding a market fit after, I mean, this is after a year of just going. I'm gonna go do something else 'cause I just don't see any sense in this. Like, and then talking to enough people and then realizing they've been doing interesting things at, at interesting companies in interesting industries. And then they just have some clown boss that came to them and said, Hey, you gotta do this. You gotta do MCP. And so I'm trying to help them do, 'cause they've got some mandate to do a copilot or something. Augments their tool so they can be like GitHub or Microsoft or something. \[00:15:00\] And my goal is to just make it as. Low impact. You can use your existing open APIs, how to maybe add vendor extensions, how to expand your, your spectral rule set a little bit. So how do you build on that existing work to meet this demand? It's less what can you do with ai? That's super cool. I've got a few use cases and I think you have one or two, but I don't at all feel compelled to like find interesting use cases for AI, for anybody. There's plenty of clowns and people out there who are, who are interested in doing that, but I am interested in helping us all. Keep, keep our jobs, maintain our businesses, not go bankrupt and, and yeah. Not totally mess up the world if possible. Yeah, absolutely. I've, I've been, I've been really enjoying kind of the, as much as I've kind of had this kind of, uh, uh, Luddite in the proper definition of it, uh, approach to ai, um, of just like. I don't hate technology. I hate when it completely replaces labor and, and makes everyone's \[00:16:00\] lives worse, which is what the ludic were about. Right? Thank you for that, by the way. Thank you for that real, real definition. Exactly that that needs to be said more because people just toss that term around like you ate computers. I'm like, I've defined my entire fucking life by computers. How dare you. Like I even run like a reforestation charity now that uses like a whole bunch of tech. We've literally got an API in the background that shows you where our trees are and like it powers the entire company. I don't hate tech. I hate ars assholes doing ars asshole shit for money with tech. Amen. I do occasionally find like a wonderful use of AI that actually brings me joy. So we were speaking on, I think it was the last episode with Tom from Wire Mock and they have a, a, a mocking system, which does a whole bunch of stuff, right? Like mocking does a million things. But they were talking about how they actually kind of edge into testing and edge into fudge fuzz testing. And the, the cloud offering can like use AI to create useless requests that you can see how your API handles. I was like, that's fucking genius. That's like create your request that tries to do this and like make a bunch of mistakes and like \[00:17:00\] obviously that last part is redundant. It was gonna do that anyway. That's what it does. Um, and, and so to use AI slop to be slop, um, and, and test how your API handles slop is brilliant because the requests that we're gonna hit your API for years and years and years, were gonna be. Poorly constructed by someone that didn't read the docs or didn't know you had docs. Or you didn't have docs, right. They were gonna be bad requests coming in, but now it's just used like this entire swath, this entire treasure trove of the entire internet of, of dumb mistakes to creatively cock up requests to your. API and you can see how your API responds to those. Like, that's obviously brilliant and that's not melting the planet. So like, I do like it when people find good things. I think a lot of the problem has for me is just, it's obviously exactly the same people who were really excited about crypto and that didn't go anywhere and they're really excited about NFTs and that didn't go anywhere, and they got really excited about ai and that's \[00:18:00\] not really, you know, the bubble hasn't blown yet. So they're all still really excited about it. But it, it, it is that. Seeing the exact same people, the exact same, like snake oil salesman, just keep pushing bollocks. So when someone can go, well, actually there is this very limited use case in which I think it can genuinely, really help. I'm like, brilliant, do it. I actually quite liked, um. Optic. Um, so before Optic got bought out by Postman, they were working on No, at They got bought by, oh, it was Atlassian, sorry. Yeah, yeah, yeah, yeah. It was, ah, postman bought a similar learning tool, which was Oh, Aita, Aita. Aita. Thank you very much. Yeah, I always confuse those two, but yeah, postman similar. Atlas Competitive. Yeah. And Optic had that Lynch, GPT. And, um, there was a little minute where I was like, this is actually pretty good. Like, I've written some rules that spectral couldn't handle and it was like out for any. Fields that have PII in the response or something like that. And it would, it would very inconsistently and, and non \[00:19:00\] determinative, but quite well occasionally spot some fields that I wouldn't have thought of, that I wouldn't have put into a big if field name equals array in spectral. So there have been some things that I think are pretty cool. Have you got, have you found any other little bits like that that, that you think it's really helping with? Yeah, I mean, similar ones that are, that are verifiable. So if you can link it to, you know, some sort of determinism With a JSO schema, I have quite a few, a toolbox of use cases. 'cause I, um, I have a, a specification called APIs, JSON, which is, is, is billed as API discovery and that's what most people think of it is. But for me, it's, mm-hmm. Discovery of APIs using crawlers. So that kind of discovery rather than developers discovering APIs. But I, so I crawl GitHub and I crawl Bing and I crawl Google via partners for APIs. And when I find docs and there's no open API present, I will auto generate an open API from that. But I validate it with the nice, the js ON \[00:20:00\] specification. And I also do that for pricing pages, rate limit pages. Because in my, I want the whole portal, the whole experience to be machine readable, so for discovery for other purposes. But it doesn't tend to align with API providers or why people think things should be machine readable. So it hasn't ever moved forward as fast, but I find it useful for that 'cause I can validate it with a J schema coming out of the output and that helps me. And I, I use that for, I generate specal rules sometimes too from that landscape. Mm-hmm. And then I validate 'em 'cause there's a J schema for spectral rules, which I'm sure you know because you've helped create. Right. And so like, you know, these things are, are validatable and you can make 'em deterministic. And they're low. Like I'm not using 'em in runtime. I'm using 'em in design time. So the, the risk level is lower. So for, yeah, low risk things I can make deterministic. Hell yeah. I'm on board. Okay. So where's the AI involved in that? So you're crawling, you're crawling through search engines and then mm-hmm. What are you \[00:21:00\] doing? You find an API and you start like generating APIs, Jason, with ai. So it's like. Is that, where does that AI get involved? I won't, so I won't generate the APIs json So here's the one that I just did with, it's a little inception level for Claude Gemini and chat GPT. Okay? Mm-hmm. So there's some, some of them have, uh, o open api, open AI has an open APIs for their api. I, that's, that was hard one to say. Yeah. Claude, I couldn't find one. Gemini has a discovery doc. So I needed to converge on open API I three one across that. So I helped do that. For me, the APIs, JSOI use as scaffolding, so I don't need it to generate that. It's pretty, pretty basic stuff. It's, but the individual properties, docs being that first one open API. So if there's not an open API or if there's often a swagger, how do I get to the open API three one. If there's a Ramel, if there's a, a Postman collection, if there's a Bruno \[00:22:00\] collection and insomnia, like how do I get to open API three, one, that helps me. But then with Claude, GE, and Gemini and, and OpenAI, I want their pricing and rate limits, their headers and all of that. And I want that all machine readable because I wanna automate. The usage and the AB testing across these. So you should be able to route to Gemini or Claude for different purposes. Yeah. And I wanna know the cost. What's this gonna cost? Is it a cost based decision? Is it quality base, that kind of thing. And so having the pricing, the rate limits, and um, so they all three have usage based pricing. The count you're in. Excuse me. User based pricing, the account, user account, you sign up, you pay monthly. They have usage on the API. Yeah, traditional cloud-based usage. And then they have model based pricing, and then they have a fourth, which is called tier. So how much money I load up in my credit card on cloud \[00:23:00\] or depends on determines. How my, my rate limits and my costs are. So there's this spread of costs that I wanna automate. And so using each of those services to do the little incremental pieces of automation, not the overall scaffolding. That's my vision. That's what I need. I'm finding value in that, but it's, it's me doing the homework all along the way. Yeah. Okay. Fun. That's pretty nice. I've not, yeah, I've not played around with too many of the specific providers. I mean like I think most of my AI interaction is like VS. Code has just wedged a bunch of it in there and every now and then it, it makes recommendations at me that are just absolutely horrific. And then I say, no, go away. Exactly. It spends a lot of time trying to help me write. Articles and I'm like, no, my, this, I can, I can make money by typing words don't type words for me. I will type better words than you anyway. Mm-hmm. But then sometimes I feel like it's just vomiting entire paragraphs stuff I wrote.\[00:24:00\] 'cause I feel like the open API. Writing about open API is a pretty niche section of the world, and like it will, it will recommend a paragraph. I'm like, that's my fucking paragraph, man. I wrote that. I'm sorry. Just the, the probabilistic realm. Dude, you've written a significant portion of the open API content, so thus it is filled, right? Yeah, it started like if you try and write about open API with a copilot on, it will, it will use British English and swear a lot anyway. That's probably enough about ai. I mean, I, I think we talked about it in general, but like what, specifically? What, what is your, uh, startup up to? What, what are you. What's your plan? Or is it early days? You don't really know. I mean, literally it's early days and so literally it's, it's finding that that approach that I can, can come at it without holding my nose or losing my soul and, and constantly build a startup and then go out and be the face of it. So that's been the last five, six months. 'cause I came into it with the same feeling as \[00:25:00\] you. And, and so I think I've, I've got that, but it's, um. You know, and it kind of speaks to specifically the MCP and the specs, I think is what more, more yours and mine's wheelhouse is. It's not, it's less about ai and for me it's about the power grab of the digital, of the digital bits, which. Has been going on for a long time. I mean, you know, postman, you know, open API stole swagger, you know, like that. Politics and business across those bits, Ramel, API Blueprint, everyone was fighting over those. The spec bits we landed on open API it's in the foundation, but you and I know they're still Smart Bear and, and MuleSoft kind of fingerprints on that struggle for the bits, but it's in the foundation. Yeah. Since then, you know, we've had type spec from Microsoft. Smithy less so from AWS, um, but Open API is the solid player. I mean, the big players out there have open API Specs and GitHub, plaid, Twilio, \[00:26:00\] Stripe, they all have open api. So that's the defacto standard. But then GraphQL came along, you know, there was that power grab around GraphQL, and this is the thing, and it's kind of shrunk back. People realize, you know, there's still a market, there's still use, but it's not the, the silver bullet. It was, it was gonna replace rest. We had a little with event driven along the side, but now we have MCP, which is just, you know, J-S-O-N-R-P-C and another kind of power grab at these digital bits. And so that's, that's what I'm trying to focus on, is like. I'm ta I'm trying to sell to customers who still have soap and still have whis, you know, and trying to not get the, to get them to not step away from their open API and their spectral governance that they've invested in for like, the last seven, eight years and chase this new shiny thing and how, right. But, but how can they deliver it without, without, uh, too much cost and too much shift. So it's in that wider. \[00:27:00\] Landscape, that 25 year view of the things that I'm trying to orient what's going on with CP and ai. Yeah, that makes a lot of sense. I mean the, yeah, there's definitely those companies that are like, yeah, still on soap and things that came before it, and they seven, seven or eight years thinking about switching to a new thing whilst they work through that and, and really spec it out like. Big companies slow roadmaps. I, I think a lot of us forget that that is a thing that exists. Like the, the thing for me that was always really annoying about GraphQL was that people, people were like throwing away a rest API they built six months ago, a year ago, two years ago, because they thought they needed to go run off and do GraphQL to do a thing, which was perfectly possible and reasonable and common to do with the rest. API be it sparse, failed sets or compound documents. Um. And so I always got really frustrated about like, if you are rest API, after a year or two. It's so bad you want to throw it out \[00:28:00\] completely. Redesigning everything in a brand new paradigm that you don't understand yet is not gonna make that better, right? Like it's, it's gonna be worse than what you currently have. And so I spent a lot of time trying to just get people to pump the brakes a little bit and, and yeah, like there. Just how, how do you build, how do you design and plan an API that's gonna be useful and usable for more than a year? 'cause I feel like half the people WeWork's happening, A lot of people went from rest to GraphQL. Running straight back to reco. And actually it was fine over there. I'm just gonna make this one tweak. Or just like adding a gz compression, a GIP compression header will probably help. Or switching the GS passer means that we don't need to completely switch to some other GRPC. There were these like small improvements that could be made, and I want people, I spent a lot of time focusing on helping people improve where they're at, but I was always trying to help them improve the rest so they didn't have to run off to something else or rewrite to another rest. But I, I do like the idea of like. \[00:29:00\] You are there saying, Hey, you're on soap right now. Rest is gonna be a good step forward for you. If you do it like this, please don't run off and just AI generate some slop like you've spent eight years thinking about how to build this new rest API that will solve all your problems. Go in, make API hot into a prompt. Isn't gonna, isn't gonna help. So yeah. That's a really funny use case. You talk to different people to me. Yeah. It's the, I mean, think of MCP as just GraphQL. It's just for a different, you know, consumer client audience. And as I see it, it's one tool in, in a large toolbox. Where rest is still the simplest, cheapest to do. Um, and it's gonna get to your widest audience so people can produce it or consume it. It's gonna, there's most tools to document and govern it. And then, yeah, there's, there's times you need events, but, uh, not everyone needs Kafka. You know, web hooks work real well. Um, yeah. And yeah, there's sometimes you have a specific team who knows the schema. It's a massive schema and, and their front end developers get \[00:30:00\] outta their way, give 'em a GraphQL endpoint. And I'm, I'm guessing there's situations where we want MCP, but the problem is, is every wave of these coming along, going, you've gotta rer your whole base and come over to our platform and it's the latest, you know, snake oil from whatever venture capital and people jump and then they don't realize that in, in three or four years, there's gonna be an exit. And the in notification sets in and, and you know, that's the way it is. Yeah, that is always the way. I think there's also like something we've both been pretty annoyed about is that in all of these like emerging markets, there's an immediate money grab of like cool hot startups with a brilliantly beautiful marketing page and a snazzy title that are just there to be like, we've solved every problem that you never know existed. And they come out and they make a bunch of really cool stuff and then they might settle down. By version three or four or five, they're like making some genuinely really useful stuff. And I've seen so many of these companies, like Optic, they, they were \[00:31:00\] on like version nine, uh, within a, within two years or something. They, they rewrote their stuff so many times. I have no idea how Aidan had time for this, but like they were making some really useful stuff. Uh, stoplight, like when I joined, they're on version four of the SaaS platform at least. And they'd rewritten Prism three or four times from Golan to. Type Js, uh, type script. And it was getting pretty useful. And like spectral was the third version they'd written. And it was a ripoff of two versions of specky before that. Like you kind of get this like quick burst of just like, we're really excited about stuff. How are we gonna make loads of things? Loads of things, loads of things, and it takes a little while to settle down. It becomes genuinely useful. And then you've got this golden little time where things are genuinely really useful. And then the massive corporates just swing by and, and snatch everything up. I was writing about API E closing down, apparently them putting a little banner on apiary. If anyone's not familiar listening, uh, on the podcast. 'cause there are other people here, not just me and you. I keep forgetting this. Um, API E kind of built that \[00:32:00\] a PA blueprint format. It was one of the three main rivals. Bef uh, you know, the open API Ramel A P Blueprint. They made a P vPrint pretty useful. Loved it. It was my first introduction to API descriptions and I was a big fan. They made that whole thing, and then that was like 10 years ago. I was working with that. 2016\. They recently put a banner up. It was apparently a bug. Wasn't intentional. It was a bug that said like, Hey, we're closing this down in September. Y'all better shove off. Apparently that was a bug, but it's like no one, no, no one's cat got on the keyboard and typed out that prompt, like they're clearly about to get rid of it, even if it wasn't meant to go live at that point. Yeah. So yeah, like obviously being bought out by Oracle, they've been closing down the. Paid pricing. They've been suggesting people move over to Oracle Cloud. They've been completely failing to write a new feature for almost a decade like these. An API I Apiary was amazing. Um, an API blueprint was amazing and the tool suite was amazing. Dread was amazing. All these things. And, and \[00:33:00\] that seems to just be on repeat throughout the API, especially open API world, which is call new startup. They go for a while, they get a few rounds of funding, they get really big, they get really useful, and then some giant boring corporate buys them. Merges them into a really unflattering platform where you can barely find that functionality and you've gotta pay five squillion dollars. To get anywhere near anything above the free plan that doesn't do anything before you can even start trying to find the functionality that used to be reasonably priced. Like, how do you feel about that? Why does that keep happening in APIs? Is that just capitalism and we're a bunch of mo little socialists on the internet or what's going on there? Yeah, I mean, it's just capitalism with, with venture capital tacked on and that velocity and, and not pumping the brakes, not slowing things down. I don't think we're gonna stop it. We're gonna change it. I mean, you know how taking on capitalism, we're not gonna win, bro. Like, we're not, there's no way. \[00:34:00\] But there's ways we can slow that down, change it, shift course, and, and do things and make, and still make the world a better place. And so a PA was a big deal in the moment. Like swagger wasn't a spec to begin with. It was a config file for the docs and the code gen. Which both suck like that. They were groundbreaking in the moment and then Apiary came along and was like, oh wait. Hey, how about if we create a good spec and not just a config schema and do this intentionally and create a set of tooling that's gonna help us be designed first. And it, so it was a big deal and, and it still resonates in the space that the concept of design first came from that, that realm. Yeah. And people like you and I championed that. So that's just the cycles. I mean, it's like, you know, whatever the tool is. SmartBear, I mean, SmartBear takes advantage of acquiring and kind of building a patchwork quilt that they've, things they've acquired. Then they don't do anything else with 'em when they go forward. MuleSoft was just \[00:35:00\] rocketing towards an acquisition, postman's on its own. You know, all of these have their own kind of trajectories, but yeah, like. Grabbing up anything nice in the space and owning it. That's how you buy relevancy. I mean, like SmartBear, if you can't innovate, you acquire relevancy, you acquire innovation. Yeah. So that's what you do. But we can slow 'em down with open source. But open source isn't a, a save all, you know, specky to spectral to. Wherever we're at today, like open source isn't gonna save our soul, but it's one way we can give back to the community along the way. And I, I like spending venture capital money on, on open source. I think that's a worthy cause trees spending it on trees if you can. Yeah. So it's the way things are. We have to live in it. I think we have to get creative about how we still not lose ourselves. I mean, it makes it hard to get excited about. New players in the space sometimes because I, I see new things come out \[00:36:00\] and I'll, I'll share them around like, um, you know, speakeasy do an incredibly good SEK generation where you don't have to install Java and, and scale it like recreating most of the kind of open source. Foundational layers of open API stuff that have just been completely unmaintained for so long, like swagger bars are being replaced. Brilliant. They, they've done loads of open source stuff and they're starting to roll out more and more products on top of that. Yeah. Bump just being like, here's one single CLI command you can run and your, your docs are all hosted, like these tooling providers are the underdog and they're scrappy and some of them are more early days than others, but like they're going and they're doing really useful stuff. And then, you know, I'm writing like, Hey, these things are great. You should go use these things. And then the more that you say, Hey, these are great, you should go use these things, the more the chance of them just getting acquired by an asshole company that fires everyone and like deletes the code increases and it, it feels like I'm just like the gentrification of \[00:37:00\] the API tooling space of just like it gets good for a little bit and then it's just, it's completely unaffordable and terrible. So it's, it's hard to, to be in that cycle, I guess is, is all I'm thinking. It's not like being the main example, like, you know, pouring that, that was, that was, I was an insider. I, I was working there for a long time. Product manager, project manager. Really, really like making those tools what they were. And then they've just been like, some of it was copied and pasted into, into a swagger, uh, a swagger hub. And then a lot of it's just being kind of ditched, right? And so it's hard whether you are working on it, advocating for it. Being just the person at your company that says, Hey, this thing's cool, we should use that. And then you are the person that's responsible for making this entire company reliant on something that's vanished six months later. It's really hard. I don't know how to feel about any of that. It just makes me angry. Yeah. Well you, me, you mentioned dread earlier, like I've been tempted to re, re revive dread like two or three times over since Oh, yeah. It's gone away. \[00:38:00\] Oh yeah. Hell yeah. And there's other tools that are similar to that. So, I mean, it kind of feels like, you know, like the business you're in is like. How do you plant trees on land? That there's all this history. I mean, there's so much history in the UK as far as who owns this land and who has access to this land and who can, let me tell you a story after we've recorded, who owns the resources on that land? Who doesn't? Yeah. You know, I know there's, there's just a lot of politics, business and politics, and it's the same in this virtual world we're building. So it's like, what I wanna try to figure out, and this is what I'm, I'm trying to do with my startup now. Be honest about with inve investors and go, look, we're building open source and we're gonna be building stuff that's gonna stick around and we're gonna build community. There's a, a trajectory that's going into Linux, CN CF Apache. It works with those existing tools. We know the standards that are out here, so we're paying attention to all the fences and landmarks and paths and trails and communities that you would be paying attention to with trees. How do we get \[00:39:00\] people giving a shit about this? But leave a mark. But acknowledge that in 10 years when that new, and I don't know, you know, e battery factory goes in and land priorities change. You know, we're gonna need to ship, you know, so it's like, yeah, there's no constant, I guess we just gotta do the best we can with what we have. Yeah, that makes sense. Good things can't last forever, but they can be handy for a while. Cool. I mean, on that note, we are probably about out of time, but I feel like I could talk to you about a million things forever. Where, where can people find more of you, and what is the name of this? That you're working on right now? I mean, as always, API evangelists will always be Kin Lane and be my, I'll never sell it to anybody. I'll, I'll bury it before it, you know, and I still write there and I still rant there, but I don't care about social media or traffic or anything, you know, it's just my, it's, it's kinda like your, your brand too. It just keep it. But I'm building, I'm building nco, which is Greek. It means Navy. We don't have a product yet. We just set up the company. It's a European \[00:40:00\] company. We set it up in France. Um, nice. And we're, you know, we'll be launching it here, coming up. Um, this, uh, this, this early winter, uh, late fall, but we don't have a site. You can find us on LinkedIn, but API Evangelists is where, where you can see what's going on right now. That's brilliant. I'll shove some links in the show notes for everyone listening and, and, uh, yeah, they can follow along. Thank you so much for joining us on the podcast. It's nuts that we've had however many episodes we've had. I, it'd be too boring to count and I've definitely lost track, but we've been going for years we've never had you on. That's nuts. So thank you for helping me rectify that. Any time, anytime. And, uh, keep up the good work you're doing, my friend. It's, uh, important stuff. Thank you very much. Cheer folks. Not folks. The, not the API stuff. The API stuff is important. The true stuff. None of this matters. The true stuff is important. Yeah, this is very true. Thank you very much. ### TanStack DB: No More Broken APIs URL: https://apisyouwonthate.com/newsletter/tanstack-db-no-more-broken-apis/ Last updated: 2025-11-17T16:19:48.000Z [TanStack DB](https://tanstack.com/blog/tanstack-db-0.5-query-driven-sync?ref=apisyouwonthate.com) will be a boon for REST APIs - even though it's a frontend library. Just as React's UI patterns led to GraphQL, complex UI requirements drive developers to create bespoke REST endpoints. TanStack DB changes this: components query, join, and filter a local DB instead of making network requests. The DB handles data population and state updates to your data source over your favourite protocol. This setup allows the frontend to query data exactly as needed, while keeping your REST API clean and decoupled. TanStack DB pairs naturally with sync engines, but its get and mutation structure works equally well with REST APIs \-- Alexander ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2025/08/Pasted_Image_8_4_25__7_14---AM.png) ## The fastest way to build production-ready MCP servers Gram by Speakeasy: turn your API platform into an AI platform. Create tools from OpenAPI, curate into custom toolsets, and deploy hosted MCP servers [Get started today](https://app.getgram.ai/?utm%5Fmedium=email&utm%5Fsource=apisywh) ## The API Roundup *API News, links, and tools from around the web* ### [Build Forms with JSON Schema](https://github.com/rjsf-team/react-jsonschema-form?ref=apisyouwonthate.com) Building a form that calls a REST API? Now you can use the JSON Schema from the API Spec to make forms. React JSON Schema Form provides a simple React component for building your forms: customisable, advanced validation, UI styling and more. Now you have even more use for your API Spec. ### [Mastering URLPattern for Cross-Platform Routing](https://jsdev.space/urlpattern-router-node24/?ref=apisyouwonthate.com) A native routing primitive has finally arrived in JavaScript, supported in both browsers and Node. With this new API, you get a framework-free router that can handle route matching with strict validation, named groups, and more. This blog covers all the background and how to use it in a Service Worker or Node. ### [Enterprise Ready REST API Microservice in Golang](https://github.com/rameshsunkara/go-rest-api-example?ref=apisyouwonthate.com) We all know that a REST API is the best API. Now there is a production-ready REST API template in GO. This killer template is packed with everything from health checks, logins, OWASP compliance, authentication, Docker readiness, and OpenAPI 3 specification—honestly, an incredible template. ### [Strongly Typed Web API Integration Tests](https://timdeschryver.dev/blog/write-strongly-typed-web-api-integration-tests-using-kiota-and-openapi?ref=apisyouwonthate.com) Tim Deschryver guides us through using Kiota, an OpenAPI client generator, to create strongly typed API clients for testing purposes. No need to handcraft your HTTP requests and pray you got the route right; instead, you get auto-complete method calls that match your API spec. ### [OpenAPI.Net Now Supports OpenAPI 3.2.0](https://github.com/microsoft/OpenAPI.NET/releases/tag/v3.0.0?ref=apisyouwonthate.com) The OpenAPI.Net package received a major release this week, being upgraded to support OpenAPI 3.2.0\. This release is a breaking change; please review the [migration guide](https://github.com/microsoft/OpenAPI.NET/blob/main/docs/upgrade-guide-3.md?ref=apisyouwonthate.com) before updating. --- ## APIs You Won't Hate Articles written and shared in our free [Slack community](https://slack.apisyouwonthate.com/?ref=apisyouwonthate.com). [JSON Streaming in OpenAPI 3.2.0](https://apisyouwonthate.com/blog/json-streaming-in-openapi-v3-2/) Learn how OpenAPI v3.2 helps describe JSON Streaming, and in the process find out more about what the heck JSON streaming even is. --- ## From our Community Articles written and shared in our free [Slack community](https://slack.apisyouwonthate.com/?ref=apisyouwonthate.com). ### [Phil Heads to API Days Amsterdam](https://www.apidays.global/events/amsterdam?ref=apisyouwonthate.com#agenda) Phil biked all the way to Amsterdam for API Days at the start of November. AI appears to be dominating the content of talks these days. Phil will soon be writing some blogs about his experience at the conference. ## Support APIs You Won't Hate When you become an member, you'll get access to members-only content while directly supporting our work. Your support helps us to keep making resources for the API community. [Become a member today](https://apisyouwonthate.com/#/portal) Thanks so much to our members: Kin L, Juxt, Alex R, Nolan S, Frank, Bill, James D, Rich, Ryan T, Umair, Abdelhadi, and Brandon. Your support means the world to us! ✌️ Until next time, Alexander, Phil & Mike ### Postman was Offline? URL: https://apisyouwonthate.com/newsletter/postman-was-offline/ Last updated: 2025-11-03T12:06:36.000Z During the AWS outage at the end of October, the [Postman app went down](https://status.postman.com/?utm%5Fsource=substack&utm%5Fmedium=email). If you're unaware, AWS was hit by DNS issues, taking down services running in the `us-east-1` region. What is crazy to me is that a HTTP Client stopped working due to an AWS outage. It's on my laptop; I should be able to make API calls and store any requests I have saved locally. The funny part is that a few days before the announcement, Postman shared that they had started working on an offline mode. How was this not the default setup? Honestly, I feel like many large companies have forgotten what it means to build good software. [Bruno](https://www.usebruno.com/?ref=apisyouwonthate.com), [Scalar](https://scalar.com/?ref=apisyouwonthate.com#api-client) and [curl](https://curl.se/docs/manpage.html?ref=apisyouwonthate.com) don't need an AWS to be up and running to call an API. \-- Alexander ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2025/08/Pasted_Image_8_4_25__7_14---AM.png) ## The fastest way to build production-ready MCP servers Gram by Speakeasy: turn your API platform into an AI platform. Create tools from OpenAPI, curate into custom toolsets, and deploy hosted MCP servers [Get started today](https://app.getgram.ai/?utm%5Fmedium=email&utm%5Fsource=apisywh) ## The API Roundup *API News, links, and tools from around the web* ### [A Complete Guide to HTTP Caching](https://www.jonoalderson.com/performance/http-caching/?ref=apisyouwonthate.com#utm%5Fsource=cassidoo&utm%5Fmedium=email&utm%5Fcampaign=change-your-life-today-dont-gamble-on-the-future) A well-designed API and HTTP caching work together exceptionally well, but how caching works and its interactions with CDNs and browsers can sometimes be a bit confusing. Jono Alderson has put together a fantastic post covering everything, and I mean everything. Cache headers, how they work, how different components in the network journey respond to the headers, modern browser behaviours, and a deep dive into CloudFlare. Honestly, an incredible resource, thank you, Jono. ### [Jsonriver](https://github.com/rictic/jsonriver?ref=apisyouwonthate.com) This incredible library for parsing JSON as it streams recently appeared in my feed. Fast, lightweight, and utilising only the standard features of JavaScript, it works in any JavaScript environment. We have been talking nonstop about how streaming can improve your APIs since OpenAPI Spec v.3.20 dropped, and now with jsonriver, you can easily use any streamed API. ### [The GraphQLConf 2025 Videos](https://graphql.org/conf/2025/schedule/?ref=apisyouwonthate.com) If you missed it, the annual GraphQL conference took place in September, and all the talks have now been published. If you check out the schedule pages, each talk is linked. You can also visit the official [YouTube channel](https://www.youtube.com/@GraphQLFoundationTalks/videos?ref=apisyouwonthate.com). ### [GraphQL Myths](https://jovidecroock.com/blog/graphql-myths/?ref=apisyouwonthate.com) An incredible read from Jovi De Croock on common GraphQL myths and how they persist because they describe the default configuration rather than the optimal setup. Jovi delves into Persisted operations and how they provide enhanced security, performance, debugging, and caching, while addressing some of GraphQL's criticisms. ### [Utoipa - Auto-Generated OpenAPI Documentation in Rust](https://github.com/juhaku/utoipa?ref=apisyouwonthate.com) Love building APIs in Rust? Well, do I have the library for you? Since Vite Oxc caught my attention, I have been exploring Rust and searching for an OpenAPI generator. Utopia simplifies API documentation by providing simple macros that generate it from your code. Worth checking out. ### [NodeJS V25](https://nodejs.org/en/blog/release/v25.0.0?ref=apisyouwonthate.com) I am super excited about this release of NodeJS. Why, you ask? Because it comes with some serious `JSON.stringify` performance improvements. Faster JSON means faster web APIs. ## APIs You Won't Hate Articles written and shared in our free [Slack community](https://slack.apisyouwonthate.com/?ref=apisyouwonthate.com). ### [Alternatives to Postman: Powerful HTTP/API Clients](https://apisyouwonthate.com/blog/http-clients-alternatives-to-postman/) Our guide shares a list of quick and easy graphical HTTP clients you can use if you're considering alternatives to Postman. --- ## From our Community Articles written and shared in our free [Slack community](https://slack.apisyouwonthate.com/?ref=apisyouwonthate.com). ### [Jentic OpenAPI Tools](https://pypi.org/project/jentic-openapi-tools/?ref=apisyouwonthate.com) Frank Kilcommins shared a new Python library he has shipped for parsing, validating, and transforming OpenAPI documents. Supporting multiple input formats and extensible via Python entry points, definitely worth checking out. ## Support APIs You Won't Hate When you become an member, you'll get access to members-only content while directly supporting our work. Your support helps us to keep making resources for the API community. [Become a member today](https://apisyouwonthate.com/#/portal) *Thanks so much to our members:* **Kin L, Juxt, Alex R, Nolan S, Frank, Bill, James D, Rich, Ryan T, Umair, and Abdelhadi**. Your support means the world to us! ✌️ Until next time, Alexander, Phil & Mike ### Goodbye Stoplight? URL: https://apisyouwonthate.com/newsletter/goodbye-stoplight/ Last updated: 2025-10-16T12:51:37.000Z When I first started using Stoplight four and a half years ago, it felt like a breath of fresh air. It made API documentation fun to design and build. All API work started with a PR using Stoplight. I finally felt like someone had built the perfect API tool. When SmartBear acquired Stoplight, I was a little nervous, and those fears are starting to seem like they have teeth. SmartBear has focused on integrating Stoplight features into API Hub, and work on Stoplight appears to be slowing. Personally, I really hope Stoplight does not die a slow death, but the warning signs are there. \-- Alexander ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2025/08/Pasted_Image_8_4_25__7_14---AM.png) ## The fastest way to build production-ready MCP servers Gram by Speakeasy: turn your API platform into an AI platform. Create tools from OpenAPI, curate into custom toolsets, and deploy hosted MCP servers [Get started today](https://app.getgram.ai/?utm%5Fmedium=email&utm%5Fsource=apisywh) ## The API Roundup *API News, links, and tools from around the web* ### [An Open Letter to Stoplight Users](https://www.linkedin.com/pulse/open-letter-stoplight-users-shane-o-connor-fyjme?ref=apisyouwonthate.com) The Scalar team shared their frustrations with the Stoplight transition in an open letter this month. Scalar offers excellent tools for building first-class API docs, but has taken it further with an offline API Client and more. Really impressed with the tools from Scalar. ### [Express Inspired Web Framework Written in Go](https://github.com/gofiber/fiber?ref=apisyouwonthate.com) Fibre is an Express-inspired web framework in Go. It is built on the fastest Go HTTP engine, with a focus on rapid development and performance. They are currently working on version three. There is a trend of JavaScript engineers picking up Go at some point, so now they have even more of a reason to do so. ### [API Design in Node.js, v5](https://frontendmasters.com/courses/api-design-nodejs-v5/?utm%5Fsource=email&utm%5Fmedium=nodeweekly&utm%5Fcontent=nodejsv5) Frontend Masters has some genuinely fantastic courses, and their API Design in Node got a fresh update. Ran by Scott Moss from Netflix. Learn REST API design practices by using Express, building routes, implementing schema validations, and more. ### [Upgrading from OpenAPI 3.1 to 3.2](https://learn.openapis.org/upgrading/v3.1-to-v3.2.html?ref=apisyouwonthate.com) Ready to upgrade to OpenAPI 3.2? The OpenAPI blog has a guide to help with the upgrade process and gives you a roadmap for gradually adopting new features. ### [API Days Australia, Amsterdam and Paris](https://www.apidays.global/events/paris?%5Fgl=1%2Aghmqcp%2A%5Fup%2AMQ..%2A%5Fga%2AODcyNTczMzI0LjE3NjAyNTk0MzY.%2A%5Fga%5FDPJV07TS8F%2AczE3NjAyNTk0MzUkbzEkZzEkdDE3NjAyNTk1NDAkajYwJGwwJGgw&ref=apisyouwonthate.com) API Days has several conferences coming up for the end of the year. Australia on October 29th, Amsterdam on November 6th and finally Paris on December 9th. The Paris conference really caught my eye with a Sustainable Tech and Green APIs track. Phil will be at Paris and Amsterdam, and you can guess which track he'll be in. --- ## APIs You Won't Hate The latest from the team at API's You Won't Hate. ### [OpenAPI Format: A GUI for Overlays](https://apisyouwonthate.com/blog/openapi-format-gui-for-overlays/) Overlays help users improve OpenAPI and AsyncAPI docs by adding extra content; however, they can sometimes be a little tricky to get your head around. Phil walks you through OpenAPI Format, a great GUI tool for working with Overlays. ### [Automatically Upgrade to OpenAPI v3.2](https://apisyouwonthate.com/blog/automaticly-upgrade-to-openapi-v3-2/) Phil is super excited about OpenAPI 3.2 and has written a handy guide on updating any OpenAPI docs to the latest version. He covers the extensions that are no longer needed and shows some convenient tools to make the migration easier. --- ## From our Community Articles written and shared in our free [Slack community](https://slack.apisyouwonthate.com/?ref=apisyouwonthate.com). ### [Tracking OpenAPI V3.2 Compatibility ](https://openapi.tools/?ref=apisyouwonthate.com) OpenAPI V3.2 has been out for a few weeks now, so you can imagine every tool has adopted it..... ok, I am being a little facetious. However, thanks to Phil, our tool tracker now tracks 3.2 compatibility. ## Support APIs You Won't Hate When you become an member, you'll get access to members-only content while directly supporting our work. Your support helps us to keep making resources for the API community. [Become a member today](https://apisyouwonthate.com/#/portal) *Thanks so much to our members:* **Kin L, Juxt, Alex R, Nolan S, Frank, James D, Rich, Ryan T, and Umair**. Your support means the world to us! ✌️ Until next time, Alexander, Phil & Mike ### Automatically Upgrade to OpenAPI v3.2 URL: https://apisyouwonthate.com/blog/automaticly-upgrade-to-openapi-v3-2/ Last updated: 2025-10-16T12:41:30.000Z You might have heard the news that [OpenAPI v3.2](https://spec.openapis.org/oas/v3.2.0?ref=apisyouwonthate.com) has been released. If you missed that news you should join our newsletter, because we would have let you know. [OpenAPI 3.2... Finally!The long-awaited launch of the newest version of the OpenAPI standard, plus JSON Streaming, Scaling API Workflows, a new RPC protocol, and a peek at Bluesky’s AT Protocol.![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/icon/apis-logo-square-17.png)APIs You Won't HateAlexander Karan![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/thumbnail/openapi-3.2.0-launch.jpg)](https://apisyouwonthate.com/newsletter/openapi-3-2-finally/) Whenever a new version of anything is launched, whether that's a description language or a programming language, it can be bittersweet news. On the one hand, there's handy new features that will make your life a whole lot easier, but on the otherhand you have to "do stuff" now. Many OpenAPI tooling creators are more familiar with that double-edged sword, and have set out to make the upgrade as easy as possible, with CLI tooling you can run against any OpenAPI document. ## What are the differences OpenAPI v3.2 is entirely additional, with no breaking changes. Mainly there are some extensions that are no longer needed. 1. Tags `x-displayName` can be handled with the new `summary` keyword. 2. Tags `x-tagGroups` can be handled with the new `parent` keyword. [Enhanced TagsFor API designers and writers wishing formalize their API in an OpenAPI Description document.![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/icon/favicon-16.ico)OpenAPI Documentation![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/thumbnail/OpenAPI_Logo_Pantone-1-1.png)](https://learn.openapis.org/specification/tags.html?ref=apisyouwonthate.com) Other than that... it's mostly changing `openapi: 3.1` to `openapi: 3.2` and looking to see if any of the new features are particularly useful. Some of you might still be stuck on OpenAPI v3.0 or god forbid the old "Swagger" 2.0 (🤢). Either of these tools will help you make that jump in one go. ## CLI: OpenAPI Format [OpenAPI Format](https://www.npmjs.com/package/openapi-format?ref=apisyouwonthate.com) is an amazingly useful CLI and web tool, and one that we used to help convert old timey API Blueprint to OpenAPI v3.1 in the [migration guide from Apairy to Bump.sh](https://apisyouwonthate.com/blog/goodbye-apiary-io/). [Goodbye Apiary.io, You’ll Be MissedToday we say farewell to a legend in the API documentation space as O.G. API design-first solution Apiary.io shuts its doors.![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/icon/apis-logo-square-20.png)APIs You Won't HatePhil Sturgeon![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/thumbnail/farewell-apiary--1-.webp)](https://apisyouwonthate.com/blog/goodbye-apiary-io/) The NodeJS CLI not only handles generally tidying up OpenAPI documents, but it can also [upgrade OpenAPI document versions](https://www.npmjs.com/package/openapi-format?ref=apisyouwonthate.com#cli-convertto-usage) from v3.0 to v3.2. ``` npm install openapi-format openapi-format openapi.json -o openapi-3.2.json --convertTo "3.2" ``` If you need to convert from Swagger 2.0 you can bung it through [swagger2openapi](https://www.npmjs.com/package/swagger2openapi?ref=apisyouwonthate.com) first, or have a go with the following. ## TypeScript: Scalar's OpenAPI Upgrader Scalar are a new player aiming to take Stoplight's place now that it's pretty much binned off after the acquisition by SmartBear Scalar is not only creating a full OpenAPI editing, API reference documentation, mocking, SDK integration, full lifecycle package, but they're also doing it with the same sort of dedication to open-source tooling that Stoplight had back in the day. Go and have a sift through the [scalar/scalar GitHub monorepo](https://github.com/scalar/scalar/tree/main/packages?ref=apisyouwonthate.com) if you're curious. There are countless packages that do all sorts of things, and the main work that needs doing is just documenting it all. I was rummaging about the other day and spotted [scalar/openapi-upgrader](https://www.npmjs.com/package/@scalar/openapi-upgrader?ref=apisyouwonthate.com), which does exactly what it says on the tin: upgrades OpenAPI documents all the way from 2.0 to 3.2. This package is baked into the larger [scalar/cli](https://www.npmjs.com/package/@scalar/cli?ref=apisyouwonthate.com), but at time of writing v1.3.3 of the package does have any interface to allow the OAS 3.2 upgrade to happen. It's still experimental and this will only get you to OAS 3.1. ``` $ npm install @scalar/cli $ scalar document upgrade openapi.yaml --output openapi.31.yaml ``` Whilst it's not ready for CLI use, the TypeScript interface may well be interesting to those building tooling, web interfaces, or complicated workflows that need a bit more than a CLI to work with. ``` import { upgrade } from '@scalar/openapi-upgrader' const OPENAPI_DOCUMENT = { swagger: '2.0', info: { title: 'Hello World', version: '1.0.0', }, paths: {}, } // We need to explicitly pass '3.2' to upgrade to OpenAPI 3.2 const document = upgrade(OPENAPI_DOCUMENT, '3.2') console.log(document.openapi) // Output: 3.2.0 ``` [Give it a whirl](https://www.npmjs.com/package/@scalar/openapi-upgrader?ref=apisyouwonthate.com) and get back in the comments with feedback, or open some issues to help the Scalar team make this excellent. ### OpenAPI Format: A GUI for Overlays URL: https://apisyouwonthate.com/blog/openapi-format-gui-for-overlays/ Last updated: 2025-10-16T12:33:08.000Z [Overlays for OpenAPI](https://docs.bump.sh/guides/openapi/specification/v3.1/extending/overlays/?ref=apisyouwonthate.com) and AsyncAPI are all the rage, helping all sorts of users improve API descriptions by “patching” in extra content. Tech writers improving bare bones OpenAPI with descriptions and tags to improve the resulting API documentation. Developers are popping code samples for generated SDKs in instead of relying on generic samples. The utility of this specification seems limitless, but overlays come with a steep learning curve. The OpenAPI Format Playground aims to make things easier with a graphical interface on building and applying overlays, so let’s take a look at how it works. ## What is OpenAPI Format [OpenAPI Format](https://github.com/thim81/openapi-format?ref=apisyouwonthate.com) started life as a NPM command-line tool, with the goal of helping to tidy up OpenAPI documents. It can remove unused components, sort various components and parameters alphabetically, upgrade documents from OpenAPI v3.0 to v3.1, and all kinds of other handy utilities to keep OpenAPI documents neat and tidy. Recently OpenAPI Format got the ability to apply overlays to OpenAPI documents, but they didn’t stop with a command-line interface for this. ## Introducing OpenAPI-Format Playground The maintainers of OpenAPI-Format built a Playground, a [web interface to help folks work with overlays in a visual way](https://openapi-format-playground.vercel.app/?ref=apisyouwonthate.com). Overlays are essentially a series of “actions” which can be applied to a document, in order to create a new document with the resulting changes. Actions need a target defined as a [JSONPath](https://docs.bump.sh/guides/openapi/jsonpath/?ref=apisyouwonthate.com) to point to a particular part of an OpenAPI document, and an action type of of “update” or “remove”. These actions can be used in combination to add new content, replace content, or filter out bits that aren’t wanted. Figuring out how to do all of this just from staring into an empty text editor can be a bit rough, so let’s try using the playground to guide us on our way. ## Creating Overlays with OpenAPI-Format Playground ### Step 1 - Add an OpenAPI document Copy in some OpenAPI into the box on the left. If you don’t have any handy why not use the [Train Travel API](https://github.com/bump-sh-examples/train-travel-api?ref=apisyouwonthate.com) by [Bump.sh](https://bump.sh/?ref=apisyouwonthate.com). ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2025/10/new-openapi-in-playground.png) ### Step 2 - Open the Overlay section Click the “*OpenAPI Overlay”* button and it will go to a different section. In here you can give your Overlay a name, something that describes what we are trying to do. For example: “Add descriptions to tags” is pretty clear about what is going to be achieved by the actions of this overlay. The version number can be whatever. ![](https://docs.bump.sh/images/guides/openapi-format-overlays/overlay-name.png) ### Step 3 - Play with JSONPath [JSONPath](https://apisyouwonthate.com/blog/jsonpath-and-openapi/) is hard and scary at first, but much like RegEx it can be mastered, especially with the help of a handy tool like this. To start with JSONPath you use the `$` as the root, then `.` to go to a property. The OpenAPI structure can be navigated like this, so something like `$.info` `$.paths` `$.tags` or `$.components` is a good start. When you type in a JSONPath in the Target field, the Target Preview will update, so you can be certain you are working on the right bit of the OpenAPI document. ![](https://docs.bump.sh/images/guides/openapi-format-overlays/overlay-target.png) To update a specific tag, JSONPath supports filters, which use a `?(condition)` syntax. Here I am looking through an array of objects for the one which has a name property equal to “Stations”: `$.tags[?(@.name == "Stations")]`. ![](https://docs.bump.sh/images/guides/openapi-format-overlays/overlay-target-better.png) The JSONPath worked, I can see the tag object is there in the target preview. Now to work on the action. [How to use JSONPath with OpenAPIJSONPath is a query language that can be used to extract data from JSON documents, and it’s becoming increasingly useful in the OpenAPI ecosystem.![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/icon/apis-logo-square-18.png)APIs You Won't HatePhil Sturgeon![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/thumbnail/openapi-jsonpath.jpg)](https://apisyouwonthate.com/blog/jsonpath-and-openapi/) ### Step 4 - Create the first Action [#](https://docs.bump.sh/guides/openapi/openapi-format-overlays/?ref=apisyouwonthate.com#step-4---create-the-first-action) Overlay actions come in two flavours: "update" or "remove". Update is basically a merge, taking the value provided and updating the target to include these values. If it’s an object it will add in the properties provided and override any that were there with the same name. To add/override the description for a specific tag, we can pass a new `description` property, along with any other properties from the [tags object](https://docs.bump.sh/guides/openapi/specification/v3.1/documentation/grouping-operations-with-tags/?ref=apisyouwonthate.com). ![](https://docs.bump.sh/images/guides/openapi-format-overlays/overlay-update.png) Remove will not need to take a value, it just removes the target object entirely. For example, removing the tag Payments. ![](https://docs.bump.sh/images/guides/openapi-format-overlays/overlay-remove.png) There’s a whole lot more than can be done than just mess with tags, but instead of getting stuck into advanced overlaying, let’s see how these two actions run. ### Step 5 - Compare the output Click the *“Apply Overlay”* button and it’s back to the main Playground screen. ![](https://docs.bump.sh/images/guides/openapi-format-overlays/apply-overlay.png) The main screen has OpenAPI input on the left and OpenAPI output on the right. The version on the right will have had the overlay applied as well as any other formatting rules, so we should see two changes to the `tags:` object: 1. Stations has a wonderful new description. 2. The Payments tag has vanished entirely. ![](https://docs.bump.sh/images/guides/openapi-format-overlays/output-difference.png) If its hard to see what has changed and you really want to be sure, you can click the *Show Diff* button to see the difference between the original OpenAPI document on the left, and the one with overlay changes on the right. Keep in mind that this will also show formatting changes, so there might be more changes than just the ones you made with the overlay. ![](https://docs.bump.sh/images/guides/openapi-format-overlays/overlay-show-diff.png) Now you can keep iterating on the overlays you want to build, keep trying them out in this playground, then when its working nicely go back to the Overlay screen and click *“Download Overlay”*. ### Bonus: Editing/Debugging Overlays Once overlays are out and about in the world they will sometimes need improving or debugging to make sure everything works as expected. OpenAPI Format will allow you to import an Overlay from a file or a URL, and you could even *“Switch to Code Mode”* and paste the raw YAML of an overlay into the editor. --- OpenAPI Format is a brilliant tool for helping teams get started with overlays, without forcing them to stare into an empty text-editor and try to learn Overlays and JSONPath all at once. Technical writers and others can use Playground to [create overlays to improve API documentation](https://docs.bump.sh/guides/technical-writing/efficient-tech-writing-process/?ref=apisyouwonthate.com) in countless ways. How will you use them? ### Stream GeoJSON in a HTTP/REST API URL: https://apisyouwonthate.com/blog/stream-geojson-rest-api/ Last updated: 2025-10-05T15:06:54.000Z When building an API for reforestation charity [Protect Earth](https://protect.earth/?ref=apisyouwonthate.com), I needed a way to show all the trees we were planting on a map. Thanks to an iOS application we used to photograph trees, we could record the species, and use the GPS to record the location, so we knew exactly where all those trees were. As the climate & nature crises ramped up, so did the site of our projects, with sites now having 10,000 or even 20,000 trees planted at a site. Soon it'll be even more. This growth is fantastic, but not if our software completely falls over. This broke our API because I never worked out how to paginate GeoJSON, and was too busy out in the field to mess around with it. Thankfully streaming has solved the problem perfectly, and it’s been a lot of fun to implement. Probably read this article before pressing on too much further if you're not familiar with what JSON Streaming is. [Streaming Data with REST APIsAre you forcing API clients to wait for every single byte of massive JSON collections to be sent from the server before letting them render data that’s ready already?![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/icon/apis-logo-square-16.png)APIs You Won't HatePhil Sturgeon![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/thumbnail/streaming-in-depth-2.jpg)](https://apisyouwonthate.com/blog/streaming-data-with-rest-apis/) ## Use Case It’s important to know where our trees have been planted so we can protect them, and also so we can make sure GPS hasn’t gone wonky and sent some off into the sea. I don’t want people having a certificate saying their tree is in one place if its actually miles from there. So if theres 20,000 pins to put on a map, we gotta put 20,000 pins on a map. ![Two trees have wandered down the hill from the planting site and gone for a cheeky pint in town.](https://bump.sh/website/images/blog/geojson-streaming/gone-wandering-to-town.png) A `/maps/parcel/abc` endpoint outputs the GeoJSON, and for smaller sites this is fine. Those 20,000 sites struggle, first with zero pins showing up for almost 30 seconds, then the browser panics as it receives a huge payload and tries to render all of that at once. I originally wondered about pagination. GeoJSON does not offer any standard pagination solution that I’m aware of, so trying to wedge it in with [Links headers](https://apisyouwonthate.com/blog/api-design-basics-pagination/) could have worked, but it feels weird to even try. Generally pagination is more of a user interface feature, allowing the optional fetching of as much of the collection as is needed for the user/client to get what they need, but its a laborious and flailing approach that feels more like long-polling than fetching a collection. The problem here is that the sheer size of the collection is stressing the memory of both the server and client, because it has to dealing with all of it at once. Dealing with a stream of items would solve that memory issue, and we can do that with [JSON Streaming](https://apisyouwonthate.com/blog/streaming-data-with-rest-apis/). ## JSON Streaming Streaming is how the rest of the web works. Much like a large image loading progressively, with a low-res quality showing quickly (the start of the stream) then as more data is loaded the image can load the higher quality representation a few chunks at a time. ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2025/10/Screen-Recording-2025-09-02-at-6.49.46---PM.gif) Streaming generic JSON is the same idea: putting chunks of JSON into the response so that the client can split it up. To avoid confusion over all the `{ }` and `[ ]` brackets that would be split over various lines there are standards and conventions. One standard is [RFC 7464: JSON Text Sequence](https://datatracker.ietf.org/doc/html/rfc7464?ref=apisyouwonthate.com), which not only pops a new JSON object on each line, but also requires special ASCII control characters to split them up so you can format them pretty or keep them in-line. It should look like this: ``` {"timestamp": "1985-04-12T23:20:50.52Z", "level": 1, "message": "Hi!"} {"timestamp": "1985-04-12T23:20:51.37Z", "level": 1, "message": "Hows it hangin?"} {"timestamp": "1985-04-12T23:20:53.29Z", "level": 1, "message": "Bye!"} ``` With the ASCII control characters visualized it would actually look a bit like this: ``` 0x1E{"timestamp": "1985-04-12T23:20:50.52Z", "level": 1, "message": "Hi!"}0x0A 0x1E{"timestamp": "1985-04-12T23:20:51.37Z", "level": 1, "message": "Hows it hangin?"}0x0A 0x1E{"timestamp": "1985-04-12T23:20:53.29Z", "level": 1, "message": "Bye!"}0x0A ``` Don’t worry, this will all start to make sense with some coming code samples. ## GeoJSON Streaming This is a post about GeoJSON, so let’s talk about that. Guess what, there’s another RFC that specifically talks about how to stream GeoJSON using JSON Text Sequence, and I bet you can guess the name: [RFC 8142: GeoJSON Text Sequence](https://datatracker.ietf.org/doc/html/rfc8142?ref=apisyouwonthate.com). It works the same way, but instead of returning a `FeatureCollection`, this is a stream of `Feature` objects, which is essentially a FeatureCollection right? ``` $ curl -XGET http://localhost/parcel/1 -i HTTP/1.1 200 OK Content-Type: application/geo+json-seq Cache-Control: no-cache, private Date: Tue, 09 Sep 2025 16:52:29 GMT {"type":"Feature","geometry":{"type":"Point","coordinates":[-3.331792,50.997009]},"properties":{"type":"tree"}} {"type":"Feature","geometry":{"type":"Point","coordinates":[-3.330440,50.996594]},"properties":{"type":"tree"}} {"type":"Feature","geometry":{"type":"Point","coordinates":[-3.330896,50.996337]},"properties":{"type":"tree"}} {"type":"Feature","geometry":{"type":"Point","coordinates":[-3.331937,50.996821]},"properties":{"type":"tree"}} ``` ## Creating a GeoJSON Stream Alright! So how do we actually build this? DuckDuckGo is a little light on examples of this, so I had to forge my own path. Protect Earth runs on [Laravel PHP](https://laravel.com/?ref=apisyouwonthate.com) so we’re going to use that as an example for producing GeoJSON streams, but the idea is the same in any language. Here’s a working example of a route (which for Laravel fans is actually inside a [Nova component](https://nova.laravel.com/docs/v5/customization/tools?ref=apisyouwonthate.com) but shouldn’t make too much difference). ``` units()->cursor(); // Wrap this in a callback so it can run inside stream() $callback = function () use ($parcel, $cursor) { $count = 0; foreach ($cursor as $unit) { yield chr(0x1E).json_encode([ 'type' => 'Feature', 'geometry' => $unit->coordinates->toArray(), 'properties' => [ 'id' => $unit->id, 'type' => $unit->unit_type->value, 'species' => $unit->specie?->name, ], ]).char(0x0A); if (0 === ++$count % 100) { flush(); } } }; return response()->stream($callback, status: 200, headers: [ 'Content-Type' => 'application/geo+json-seq', ]); }); ``` A few things here might feel a little weird, but it makes sense when we go through one bit at a time. ``` yield chr(0x1E).json_encode( ``` The `yield` is passing strings to `response()->stream()` which is using generators to stream the responses. This is all handled via `Symfony\Component\HttpFoundation\StreamedResponse` directly, instead of trying to use any of the Laravel streamed JSON stuff which has a few too many of its own conventions to give us the freedom to work with JSON Text Sequence properly. The `char()` PHP function helps output ASCII characters and `char(0x1E)` is the hexadecimal representation of the ASCII Record Separator character (callback to [RFC 7464: JSON Text Sequence](https://datatracker.ietf.org/doc/html/rfc7464?ref=apisyouwonthate.com)). The `chr(0x0A)` is another handy ASCII control character which pops a “Line Feed” character onto the end. That’s a bit more specific than the usual `\n` and/or `\r` sort of thing which can have various discrepancies across operating systems. Together they delimit a start and end line, so you could even output fancy formatted JSON if you wanted, so long as the RS and LF characters were where they should be. Then finally, in PHP it can help to flush the output buffer now and then to avoid memory leaking. This says to flush the output buffer every 100 records, because who cares, it’s already output. ``` if (0 === ++$count % 100) { flush(); } ``` The only other slightly different bit is the `Content-Type` header. ``` return response()->stream($callback, status: 200, headers: [ 'Content-Type' => 'application/geo+json-seq', ]); ``` Previously the content type header was `application/geo+json` which is basically `json` with a subtype of `geo` to let clients know it’s GeoJSON. Now it’s `json-seq` with the same subtype of `geo` to explain the response is GeoJSON Text Sequence. Together that all looks like this. ![Streaming GeoJSON](https://bump.sh/website/images/blog/geojson-streaming/streaming-geojson.gif) *The response is actually a lot quicker than it looks from the gif, because the HTTPie is having a rough time making it all look pretty, but you get the idea.* Any tooling that can handle JSON Text Sequence can handle this GeoJSON Text Sequence response. Sadly not any tooling which can handle GeoJSON can handle GeoJSON Text Sequence. ## Visualizing Streamed GeoJSON with Mapbox GL [Mapbox GL JS](https://docs.mapbox.com/mapbox-gl-js/?ref=apisyouwonthate.com) is perfectly happy to be pointed at GeoJSON. You give it a URL and it will go fetch the GeoJSON and pop it onto the map for you. ``` map.addSource('places', { type: 'geojson', data: 'http://localhost/maps/parcels/1', }); ``` To work with streamed data we sadly need to handle the stream ourselves. This boils down to creating an object to collect all features, then keep appending more data to that collection each time more streamed data comes in. ```
``` A big chunk of code yes, but as always we can break it down to make it make sense. ``` map.on('load', () => { const featureCollection = { type: 'FeatureCollection', features: [] }; // Initial empty source map.addSource('streamSource', { type: 'geojson', data: featureCollection }); ``` Firstly seeing as Mapbox wants a `FeatureCollection` but the stream is returning only features, we can just emulate a non-streamed GeoJSON response by building our own `FeatureCollection`, and passing it straight in. This is empty, so we go `fetch()`. ``` fetch('http://localhost/parcel/1') .then(res => { const reader = res.body.getReader(); const decoder = new TextDecoder(); ``` Built-in [HTTP client Fetch](https://developer.mozilla.org/en-US/docs/Web/API/Fetch%5FAPI/Using%5FFetch?ref=apisyouwonthate.com) is handy, and as it returns a [Promise](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global%5FObjects/Promise?ref=apisyouwonthate.com) we can use `.then()` (easier than using `await` in this example but either way.) Normally you would use `response.json()` to get the JSON body, and guess what? That’s asynchronous too, meaning you’d need to use `.then()` or `await`, because it’s waiting for *the whole response*, but we want direct access to the stream. Using `res.body.getReader()` and getting a TextDecoder ready will help make this happen. ``` let buffer = ''; function read() { return reader.read().then(({done, value}) => { if (done) return; buffer += decoder.decode(value, {stream:true}); let lines = buffer.split('\n'); buffer = lines.pop(); // keep partial line ``` Start up a buffer string as empty, then we can start reading the stream from the reader. If the reader is done, then give up, that’s the end of the response. If not, carry on! Decode the value, point out it’s a stream, and see how many lines end up coming back in the “chunk”. Splitting by `\n` gives us 1 to N lines from the server, and any trailing characters after the `\n` stick around for the next iteration of this self-recursing `read()` function to deal with. ``` for (const line of lines) { if (!line.trim()) continue; // Strip RS (0x1E) off the beginning const clean = line.replace(/^\u001E/, '').trim(); try { const feature = JSON.parse(clean); if (feature.type === 'Feature') { featureCollection.features.push(feature); map.getSource('streamSource').setData(featureCollection); } } catch (e) { console.error('Bad GeoJSON feature:', e, line); } } return read(); ``` Here’s the real crux of it all. For each line in the chunk, we can skip empty lines, and strip off RS characters, then remove whitespace. Finally with all that done we should have a clean JSON object to work with, so let’s parse that into a JS object and look at what we’ve got. Every new Feature can get pushed straight into `featureCollection.features`, and then `map.getSource('streamSource').setData(featureCollection);` does all the magic of updating the map view. If you put that altogether, you are rewarded with an amazing smooth visual loading experience as each tree is popped onto the map one at a time. ![A whole load of pins representing trees appear one at a time on a map showing a field which has recently been planted with new trees.](https://bump.sh/website/images/blog/geojson-streaming/streaming-pins.gif) ## Testing No work is complete until some tests have been put in place, and if Laravel PHP is the API backend then let’s test that. ``` get('/parcel/' . $parcel->id); // Will return 200 regardless of how many units $response->assertStatus(200); // Loop through each unit and assert its presence in the streamed content $streamedItems = $parcel->units()->get()->map(function ($unit) { return json_seq_encode([ 'type' => 'Feature', 'geometry' => $unit->coordinates->toArray(), 'properties' => [ 'id' => $unit->id, 'type' => $unit->unit_type->value, 'species' => $unit->specie->name, ], ]); }); return $response->assertStreamedContent(implode('', $streamedItems->toArray())); })->with([ '0 units' => fn () => Parcel::factory()->create(), '5 units' => function () { return Parcel::factory() ->has(Unit::factory(5)->tree()) ->create(); }, ]); ``` Not too shabby! This will make sure the endpoint works if theres only one parcel or if there’s multiple, with the help of `assertStreamedContent()` in [Laravel’s HTTP Testing framework](https://laravel.com/docs/12.x/http-tests?ref=apisyouwonthate.com#assert-streamed-content). ## Performance & Timing One last thing we should do now that it’s all working nicely is take a look at the performance of it all. Is this quicker, slower, or some third thing? There are a few different measurements that are relevant to this the response time here. 1. Waiting - The server has acknowledged the request and is working on a reply, but the response has not started. 2. Sending - The response has begun, so the timer has started, and it will end when the response is finished. Let’s go through the before, the after, and the after-after when I added `Cache-Control` headers. **Single GeoJSON Response (Not Streamed)** ![](https://bump.sh/website/images/blog/geojson-streaming/timings-no-stream.png) 458ms of waiting! That’s a lot of waiting. This is not even the largest site, only 3,000 trees, so if it was any bigger… ooh heck. The majority of the time spent here is on the waiting. Nothing for the client do but wait. As soon as the server is done producing all that JSON it only takes 4ms to return it all, and the client then suddenly needs to start throwing all the pins onto the map all at once. **Streamed GeoJSON Sequence** ![](https://bump.sh/website/images/blog/geojson-streaming/timings-streamed-first.png) Immediately we see the waiting time has shot right down from 458ms to 121ms, what a win! The receiving has actually gone up a little, but that’s to be expected as it’s looping through a bunch of things and doing a JSON encode each line. The received time could tweaked and benchmarked with send 5-10 lines at a time to see if that helps, and there are usually faster JSON encoders around than the core language one, but a little bit of extra time is fine with me. The important thing here is that it started getting results quickly, and helped the client start drawing information earlier, instead of nothing then **smack**. **Streamed GeoJSON Sequence with Cache Headers** Streams can be cached! Remember, a stream is just a response that the client can optionally accept one chunk at a time, but if the client already has the response then it can reuse that. The quickest response is always for the request you didn’t make! ![](https://bump.sh/website/images/blog/geojson-streaming/timings-streamed-second.png) There we go. 0ms. Always favourite response time. ## Summary A lot of ground has been covered in this post, and not just with trees! We’ve learned how that JSON Streaming exists, how it works, how it can be used for GeoJSON conceptually, and looked through some code examples in PHP (server-side) and JavaScript (client-side), but plenty of other languages exist. You could see if there are libraries in your language(s) of choice, to see if you can skip getting too hands on with the control characters, because I was certainly a little sloppy in respecting every possible thing the RFC wants me to handle. Streaming is a lot of fun, and GIS is a lot of fun, so combining the two has been a brilliant ride. Share a comment with us to let us know what you’re using GeoJSON for, and if streaming might help. ## Further Reading - [Streaming Data with REST APIs](https://apisyouwonthate.com/blog/streaming-data-with-rest-apis/), APIs You Won't Hate. - [JSON Streaming in OpenAPI](https://bump.sh/blog/json-streaming-openapi-3-2/?ref=apisyouwonthate.com), Bump.sh. - [Sequential Media Types](https://learn.openapis.org/specification/media-types.html?ref=apisyouwonthate.com), Learn OpenAPI. ### OpenAPI 3.2... Finally! URL: https://apisyouwonthate.com/newsletter/openapi-3-2-finally/ Last updated: 2025-10-01T12:35:26.000Z It's finally here, [OpenAPI 3.2](https://github.com/OAI/OpenAPI-Specification/releases/tag/3.2.0?ref=apisyouwonthate.com). Honestly, this is a loaded release. From multipurpose tags that support nesting to the new HTTP query method, there is something for everyone here. I have linked the release notes from GitHub in the link above if you're interested in a complete breakdown. Hats off to everyone involved - congrats on the launch! \-- Alexander ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2025/08/Pasted_Image_8_4_25__7_14---AM.png) ## The fastest way to build production-ready MCP servers Gram by Speakeasy: turn your API platform into an AI platform. Create tools from OpenAPI, curate into custom toolsets, and deploy hosted MCP servers [Get started today](https://app.getgram.ai/?utm%5Fmedium=email&utm%5Fsource=apisywh) ## The API Roundup *API News, links, and tools from around the web* ### [Paris Meetup: Scaling API workflows with OpenAPI](https://bump.sh/blog/meetup-scaling-api-workflows-with-openapi/?ref=apisyouwonthate.com) Bump.sh is hosting an API meeting in Paris on Wednesday, October 8th, featuring a stellar lineup. Zach Winters from MongoDB is discussing building API documentation, and Marco Spinello is sharing AI insights from documentation pipelines. Our own Phil Sturgeon will also be making an appearance, discussing how to create faster and more efficient APIs, with a closing talk from the Algolia team. ### [Cap'n Web: a new RPC system for browsers and web servers](https://blog.cloudflare.com/capnweb-javascript-rpc-library/?ref=apisyouwonthate.com) A new RPC protocol implemented in pure TypeScript from the Cloudflare team. It works over HTTP, WebSocket, and postMessage() and can be extended to other transports. While it is helpful for any JavaScript applications that communicate with each other over a network, it is particularly well-suited to interactive web applications with real-time collaborative features. ### [First-Class API Versioning in Spring Boot 4](https://www.danvega.dev/blog/spring-boot-4-api-versioning?ref=apisyouwonthate.com) Spring Boot 4 is here, bringing support for API versioning via one of the four options: URL, Media Type, query parameter, and path parameter. For best practices and guidance on testing versioning, check out this killer blog post from Dan Vega. ### [GraphQL Conf 2025 Recap](https://the-guild.dev/graphql/hive/blog/graphql-conf-2025-recap?ref=apisyouwonthate.com#graphql-conf-2025-recap) The GraphQL conference took place in Amsterdam this year, marking ten years since the GraphQL specification was introduced to the world. If you missed the conference, I have a blog post for you, covering all the biggest announcements, key takeaways, and some of the improvements coming to GraphQL. ### [Mastering API Testing with Quarkus](https://www.the-main-thread.com/p/quarkus-api-testing-restassured-pact-jqwik?ref=apisyouwonthate.com) Using Java and Quarkus to build APIs, and want to improve your testing setup? Well, we got you covered with multiple test types in this blog post. Example-based tests with RestAssured for happy path verification. Contract testing with Pact to ensure service compatibility, and property-based testing with Jqwik that generates random inputs to discover unexpected edge cases. ### [The Protocol is the API](https://overreacted.io/open-social/?ref=apisyouwonthate.com) A deep dive into the AT Protocol powering Bluesky and the broader vision of open social. Dan Abramov explains how open social could do for social media data what open source did for code. He breaks down how the AT Protocol works, similar to the traditional web, where you own your data and can move between platforms without losing your social graph. --- ## APIs You Won't Hate The latest from the team at API's You Won't Hate. ### [JSON Streaming in OpenAPI v3.2.0](https://bump.sh/blog/json-streaming-openapi-3-2/?ref=apisyouwonthate.com) APIs You Won't Hate's very own Phil Sturgeon wrote a guest post for our pals over at Bump.sh going over JSON Streaming in the new OpenAPI spec version. Streaming data back and forth is becoming more and more common with realtime apps and LLM-driven use cases. This tutorial will get you off to a flying start. --- ## From our Community Articles written and shared in our free [Slack community](https://slack.apisyouwonthate.com/?ref=apisyouwonthate.com). ### [JSON Schema Faker](https://github.com/json-schema-faker/json-schema-faker/issues/289?ref=apisyouwonthate.com#issuecomment-3276769330) Phil posted in our Slack community that JSON Schema Faker is looking for contributors. Currently, the project is only maintained by one person. Love API's and want to contribute to the ecosystem? Check out the open issues on GitHub. ## 🫶 Support APIs You Won't Hate When you become an member, you'll get access to members-only content while directly supporting our work. Your support helps us to keep making resources for the API community. [Become a member today](https://apisyouwonthate.com/#/portal) *Special thanks to* [*Jeremy Hynes*](https://unsplash.com/@hynesight?utm%5Fcontent=creditCopyText&utm%5Fmedium=referral&utm%5Fsource=unsplash) *on* [*Unsplash*](https://unsplash.com/photos/brown-and-white-owl-in-close-up-photography-2VsPnDt2SQs?utm%5Fcontent=creditCopyText&utm%5Fmedium=referral&utm%5Fsource=unsplash) *for the🦉 used in our cover image this week!* *Thanks so much to our members:* **Kin L, Juxt, Alex R, Nolan S, Frank, James D, Rich, Ryan T, and Umair**. Your support means the world to us! ✌️ Until next time, Alexander, Phil & Mike ### Is Your API Secure? URL: https://apisyouwonthate.com/newsletter/is-your-api-secure/ Last updated: 2025-09-20T12:18:37.000Z In case you missed it, [BobDaHacker](https://bobdahacker.com/?ref=apisyouwonthate.com) shared how they hacked Burger King and found serious flaws in their API. These flaws allowed them to create new store locations, orders, and access audio recordings from drive-thrus. The lack of security and good practices here was frightening. Unfortunately, the blog has been taken down, and even the Wayback Machine entry is gone. The last time I checked, [this backup](https://archive.is/zIteR?ref=apisyouwonthate.com#selection-1735.0-1735.30) still worked. The hack revealed an absolute dystopian nightmare. If you're going to store a recording of every drive-through, you should at very least make sure it's secure 🤪. \-- Alexander ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2025/08/Pasted_Image_8_4_25__7_14---AM.png) ## The fastest way to build production-ready MCP servers Gram by Speakeasy: turn your API platform into an AI platform. Create tools from OpenAPI, curate into custom toolsets, and deploy hosted MCP servers [Get started today](https://app.getgram.ai/?utm%5Fmedium=email&utm%5Fsource=apisywh) ## The API Roundup *API News, links, and tools from around the web* ### [HTTP Golden Girls](https://www.goldengirls.codes/?ref=apisyouwonthate.com) There are numerous HTTP status codes, making it challenging to recall them all without reference. Instead of reading from a boring list, you can now enjoy a Golden Girls themed status code list. My favourite has to be the [multi-status](https://www.goldengirls.codes/?code=207&ref=apisyouwonthate.com). ### [Terminal-Based OpenAPI Spec Viewer](https://github.com/plutov/oq?ref=apisyouwonthate.com) If you're one of those developers who love doing everything in their terminal, then you'll love this terminal OpenAPI spec viewer. Works with OpenAPI 3.0 and 3.1, supporting both YAML and JSON formats. Definitely worth a look. ### [Surf - Advanced HTTP Client for Go](https://github.com/enetx/surf?ref=apisyouwonthate.com) A new HTTP client library whose logo is a gopher wearing a Batman-style costume. If that is not a reason to rewrite your whole project with it, I don't know what is. Surg comes with HTTP/3 support, middleware setup, API automation and more. ### [Express OpenAPI Validator](https://github.com/cdimascio/express-openapi-validator?ref=apisyouwonthate.com) A new version of the Express OpenAPI Validator was released last week. Drop in your OpenAIP spec and validate all requests and responses. Supports Express 4 and 5, NextJS, Koa and Fastify. Check out their release docs for all the updates. ### [Bringing Node.js HTTP servers to Cloudflare Workers](https://blog.cloudflare.com/bringing-node-js-http-servers-to-cloudflare-workers/?ref=apisyouwonthate.com) Cloudflare has been working diligently to bring the Node HTTP server to Cloudflare Workers. You can now run Express, Koa and other Node applications right on workers without cold starts. Cloudflare runs through the challenges of bringing over Node HTTP Server APIs and how to get your Node applications up and running. --- ## APIs You Won't Hate The latest from the team at API's You Won't Hate. ### [JSON Streaming in OpenAPI v3.2](https://apisyouwonthate.com/blog/json-streaming-in-openapi-v3-2/) JSON Streaming is coming to OpenAPI v3.2\. Phil walks you through how streaming works in the OpenAPI spec. If you're unsure what JSON Streaming actually is, don't worry, Phil has got you covered with a handy intro. ### [Streaming Data with REST APIs](https://apisyouwonthate.com/blog/streaming-data-with-rest-apis/) Want to know how actually to stream JSON? Now that the OpenAPI spec can describe JSON Streaming, Phil shows you how to do it, as well as giving details on why it is RESTful, along with design considerations. --- ## From our Community Articles written and shared in our free [Slack community](https://slack.apisyouwonthate.com/?ref=apisyouwonthate.com). ### [API Days London](https://www.apidays.global/events/london?ref=apisyouwonthate.com) The amazing Lorna Mitchell dropped a message in our Slack to remind us that API Days is taking place in London from September 22nd to 24th. If you're interested in socialising with people from our group outside the conference, please visit our [Slack community](https://apisyouwonthate.com/community) for more information. ## Support APIs You Won't Hate When you become an member, you'll get access to members-only content while directly supporting our work. Your support helps us to keep making resources for the API community. [Become a member today](https://apisyouwonthate.com/#/portal) *Special thanks to* [*Jeremy Hynes*](https://unsplash.com/@hynesight?utm%5Fcontent=creditCopyText&utm%5Fmedium=referral&utm%5Fsource=unsplash) *on* [*Unsplash*](https://unsplash.com/photos/brown-and-white-owl-in-close-up-photography-2VsPnDt2SQs?utm%5Fcontent=creditCopyText&utm%5Fmedium=referral&utm%5Fsource=unsplash) *for the🦉 used in our cover image this week!* *Thanks so much to our members:* **Kin L, Juxt, Alex R, Nolan S, Frank, James D, Rich, Ryan T, and Umair**. Your support means the world to us! ✌️ Until next time, Alexander, Phil & Mike ### Streaming Data with REST APIs URL: https://apisyouwonthate.com/blog/streaming-data-with-rest-apis/ Last updated: 2025-09-23T16:09:00.000Z It's exciting times in the world of API design, with [OpenAPI v3.2](https://spec.openapis.org/oas/v3.2.0.html?ref=apisyouwonthate.com) finally showing that a version after 3.1 was possible. Amongst various improvements it's bringing "JSON Streaming" as a new feature, which begs the question: WTF is JSON Streaming? ## HTTP Streaming Let's look at streaming in HTTP and the browser, because whenever we're talking about HTTP/REST APIs, there's usually not much difference. Images can be streamed, and that's what has that top-to-bottom effect with large images on slow connections. ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2025/09/Screen-Recording-2025-09-02-at-6.49.46---PM.gif) A massive image of a massive fascist, slowly loading his shameful walk away from being pegged about the chest with a milkshake. Streaming is not just for images, the browser isn't waiting for 100% of HTML to load before it decides to render anything either. > Once the browser receives the first chunk of data, it can begin parsing the information received. Parsing is the step the browser takes to turn the data it receives over the network into the DOM and CSSOM, which is used by the renderer to paint a page to the screen. – **Source:** [MDN, How Browsers Work](https://developer.mozilla.org/en-US/docs/Web/Performance/Guides/How%5Fbrowsers%5Fwork?ref=apisyouwonthate.com#parsing). The server sends chunks of data which are more manageable, and as soon as a client knows what to do with however many chunks it has so far it can do whatever with them. In the case of anything visual that'll be rendering as much as it can as it gets more chunks in, but this also works nicely for data. ## Can HTTP/REST API responses be streamed? Absolutely! Imagine a HTTP API was responding with a CSV file. We can stream that CSV content record by record. ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2025/09/stream-csv.gif) When a client knows how to handle chunks of a stream, it will. Curl does a great job of showing how a client could handle it by putting one line out to the terminal as each line comes in, but you could imagine each line being processed and saved into a database instead. How does that stream get sent though? What magic am I doing on the server-side of things?! Nothing as exciting as you might think. ``` import express from "express"; const app = express(); app.get("/tickets.csv", async (req, res) => { res.setHeader("Content-Type", "text/csv; charset=utf-8"); res.setHeader("Transfer-Encoding", "chunked"); // Write CSV header immediately res.write("TrainNumber,Departure,Destination,Price\n"); const tickets = [ { train: "ICE 123", from: "Berlin", to: "Munich", price: 79.90 }, { train: "TGV 456", from: "Paris", to: "Lyon", price: 49.50 }, { train: "EC 789", from: "Zurich", to: "Milan", price: 59.00 } ]; let i = 0; const interval = setInterval(() => { if (i < tickets.length) { const t = tickets[i]; res.write(`${t.train},${t.from},${t.to},${t.price}\n`); i++; } else { clearInterval(interval); res.end(); } }, 1000); // 1 row per second for demo }); app.listen(port, () => { console.log(`Server running at http://localhost:3000`); }); ``` All its doing is writing lines as it goes, instead of writing everything all at once. Also this is not some special streaming endpoint that is massively different from the normal CSV response. In this example I've simply upgraded the existing CSV endpoint with special sauce added via a few extra headers. The key bits in this Node/Express.js example are `res.setHeader("Transfer-Encoding", "chunked");` and the loop outputting `res.write(...)` calls instead of instead of building up a big array and converting it all to JSON all at once. In any language that has the concept of "generators" or "yielding" instead of working with big arrays and returning it all, this can be a huge memory reduction. We simply don't need to think about N things at once, only ever one thing at a time. This removes a lot of strain on memory of both the server and client. If a client does not understand the stream or doesn't want to mess with that, it will simply wait until it has received the entire response before continuing offering graceful fallbacks for this functionality. Instead of using the CLI you might use something like `fetch()` and the built-in reader, which you can [learn more about here](https://developer.mozilla.org/en-US/docs/Web/API/Streams%5FAPI/Using%5Freadable%5Fstreams?ref=apisyouwonthate.com). ## What about streaming JSON You can imagine a problem here. If you tried to stream JSON line by line you'd get a lot of invalid syntax errors, because there would be an opening `{` or `[` then a bunch of other bits and bobs, and no ending `}` or `]`. This syntax error would throw out any existing JSON parsers or validators so people would be stuck coding up awkward homegrown solutions. Some people have tried to build and popularize their [homegrown solutions](https://github.com/egyjs/progressive-json-php?ref=apisyouwonthate.com), but it's pretty awkward, and as usual we have been rescued by standards. Of course there cannot be just one standard, there need to be a few slightly different ones. You can use [JSONL](https://jsonlines.org/?ref=apisyouwonthate.com), [NDJSON](https://github.com/ndjson/ndjson-spec?ref=apisyouwonthate.com), or [RFC 7464: JSON Text Sequences](https://www.rfc-editor.org/rfc/rfc7464.html?ref=apisyouwonthate.com) and they're basically the same thing. All these approaches are simply known as "JSON Streaming", and the goal is to modify JSON just a little bit to bring those streaming benefits to a HTTP API. This is how a JSONL or NDJSON response might look in an API. ``` {"train":"ICE 123","from":"Berlin","to":"Munich","price":79.9} {"train":"TGV 456","from":"Paris","to":"Lyon","price":49.5} {"train":"EC 789","from":"Zurich","to":"Milan","price":59} ``` At first you might think that this is just JSON but... enhance. It's not an array of objects. This is a series of JSON objects, each on their own new line. This solves the streaming syntax issue, by making each and every line a completely valid object. ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2025/09/stream-jsonl.gif) Any request to `/tickets.jsonl` will respond with this stream of JSON objects on new lines. Any programming language can handle this easily, and core/standard JSON tooling can be used to handle each line one at a time. Streams can serve a set collection of data, or they can run for as long as the server or client keeps the connection open. This is handy when the response is thousands of things, or potentially infinite things, but definitely super handy when collections are massive. ## When to Stream? When would you use JSON streaming in a REST versus something more common like pagination? Let's look at a few use-cases: - Some folks are using it for AI agents to help people buy eggs for 10x the usual price for some reason. - The Twitter team are using it so the Nazi community can filter through the firehose of fascist shitposts to recruit new members in real-time. - I'm using it to help reforest the U.K. by helping [Protect Earth](https://www.protect.earth/?ref=apisyouwonthate.com) handle survival surveys and data collection, showing tens of thousands of pins for each tree we've planted onto our various site maps around the country. Doing this without crashing out the server or the browser was important as we consistently work on larger and larger projects. Some of these use-cases are more helpful to the world than others, but they all fit with streaming better than pagination. ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2025/09/streaming-pins.gif) Several thousand red dots appearing one at a time once Protect Earth switched the "show all our trees for this site on a map" from GeoJSON to [GeoJSON Text Sequence](https://www.rfc-editor.org/rfc/rfc8142.html?ref=apisyouwonthate.com), based on [JSON Text Sequence](https://www.rfc-editor.org/rfc/rfc7464.html?ref=apisyouwonthate.com). Pagination is more about showing some content to a client which is going to decide if it wants to grab more data or not. Perhaps the client (or end-user) found what it wanted on the first page and doesn't need to go through everything else. It would have been a misuse to use pagination for "do you want to see some more trees on this map" even if I could get the UI to do that. Streaming is more helpful when all the data is needed but it's not "all or nothing", allowing people to do what they can but with a reasonable exception they will need a whole bunch more. It's also helpful when you want to keep sending infinite updates without polling, WebHooks, or WebSockets, because they might be overkill for something like getting some updates to a single payment attempt in the moment. Once again there's a standard for that, called [Server Sent Events](https://developer.mozilla.org/en-US/docs/Web/API/Server-sent%5Fevents/Using%5Fserver-sent%5Fevents?ref=apisyouwonthate.com), and it works really well inside HTTP/REST APIs for sending updates. This might sound a bit like the [Subscription](https://graphql.org/learn/subscriptions/?ref=apisyouwonthate.com) feature in GraphQL, but as with everything in GraphQL it's inspired by a substantially more useful HTTP standard/convention that came out at least a decade before. [Using server-sent eventsDeveloping a web application that uses server-sent events is straightforward. You’ll need a bit of code on the server to stream events to the front-end, but the client side code works almost identically to websockets in part of handling incoming events. This is a one-way connection, so you can’t send events from a client to a server.![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/icon/favicon-3.svg)MDN](https://developer.mozilla.org/en-US/docs/Web/API/Server-sent%5Fevents/Using%5Fserver-sent%5Fevents?ref=apisyouwonthate.com) That's two pretty different use cases for streaming, but it's not a single feature that has a single use, it's a fundamental rethink to assumptions about API design that assume we always have to respond once, and only once. ## API Design Considerations In the example of the CSV response we automatically streamed the response and let clients decide if they wanted to read it as a stream or not. With the JSON streaming thats a little different, because this is not simply JSON in "Stream mode". For example, if you started sending JSONL (multiple JSON objects over multiple lines) to a client which expected actual JSON, you would have a whole lot of syntax errors and grumpy customers. To avoid anyone making this mistake copying sample code I popped everything onto their own endpoints like `GET /tickets.jsonl`, but there's no need to work like that. You could amend an existing `GET /tickets` to respond differently to a client sending `Accept: application/json` or `Accept: application/jsonl`, giving this choice to the client and using the `Accept` header for exactly what it's for. ## Is it RESTful to stream data? According to Nordic APIs [you can't](https://nordicapis.com/rest-vs-streaming-apis-how-they-differ/?ref=apisyouwonthate.com) stream data or events in REST, because "Streaming APIs are almost the exact opposite of the REST ethos." Thankfully they're wrong about that. There is nothing at all in REST that says an API can only have one response to a request, or that the response has to be sent all at once. Some people have an outdated understanding on REST APIs constraint about using the **uniform interface** of the web, which is really about not inventing proprietary nonsense. That doesn't mean we're forced to ignore new HTTP functionality as it rolls out. By this logic REST APIs would be stuck in some state of Amish-like time-based technology cut-off, where only HTTP functionality from circa 2000 allowed and everything newer was strictly forbidden. Some of this is a semantic issue, and Nordic APIs are incorrectly setting their definition of "Streaming APIs" == "[Event-driven APIs](https://tyk.io/learning-center/what-is-event-driven-architecture/?ref=apisyouwonthate.com)". At least I presume, as they're mentioning "RabbitMQ, ActiveMQ, or Azure Event Hub, ... \[and\] event streaming platforms include Apache Kafka, Apache Flink, and Apache Beam." That is misleading. Those are types of Asynchronous APIs, and async APIs are a subset of event-driven APIs. These may co-exist on the same architecture as REST APIs handling different pieces of the puzzle. Sometimes a REST API will be there to answer questions about the current state because a REST API is a [state-machine over HTTP](https://apisyouwonthate.com/blog/representing-state-in-rest-and-graphql/), and the event-driven APIs will be running around [updating or commanding](https://www.youtube.com/watch?v=STKCRSUsyP0&ref=apisyouwonthate.com) each other, pushing those REST resources through various states in various workflows as they go. But a REST API can also stream, that's definitely not something only Event-driven or Async APIs can do. Let's reuse the diagram from Nordic API as its a perfect diagram for explaining how REST APIs can stream despite it saying that they can't. ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2025/09/REST-vs-Streaming-APIs-How-They-Differ-DIAGRAM.webp) If the REST API is streaming events then this diagram is already just fine. If it's streaming data would could rename "Event" to "Chunk" or "Item", then once again this diagram is now perfectly describing how REST APIs can handle streaming responses to a request. Event-driven and message-driven APIs are fundamentally rather different, and nobody should be trying to crowbar JSON streaming into REST APIs when what they really needed was a full blown event-based architecture, but by the same token we need to stop pretending people need to roll out complex systems like Kafka or even WebSockets when the only thing that was needed was an extra HTTP header and a different content type. ## Describing JSON Streaming with OpenAPI Nothing about JSON Streaming is particularly new. Data scientists and GIS communities have been usig it for ages, with AI hype types getting a use out of it too, but it's being brought to the API community especially thanks to the upcomming OpenAPI v3.2 release adding the JSON streaming support. If you'd like to learn more about how JSON streaming looks in OpenAPI then check out this handy guide written for Bump.sh. [JSON Streaming in OpenAPI v3.2.0 · Bump.shFrom API contract to documentation portal: streamline your workflow and deliver the best API experience. Built for engineers and tech writers.![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/icon/apple-touch-icon-9.png)Bump.shPhil Sturgeon![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/thumbnail/json-streaming.png)](https://bump.sh/blog/json-streaming-openapi-3-2/?ref=apisyouwonthate.com) ## Keeping up with HTTP It always feels like API designers ignore a lot of lessons from the wider Internet. We are still designing bloated API responses with all loosely related secondary and tertiary data all thwacked in together to reduce number of HTTP calls. This was a concept akin to [Image Sprites](https://css-tricks.com/snippets/css/perfect-css-sprite-sliding-doors-button/?ref=apisyouwonthate.com) which fell out of popularity once HTTP/2 took over in \~2015, but API developers will just not let the concept go. [Let’s Stop Building APIs Around a Network HackJSON-API has been one of the most popular standards for API development for a while now. It was conceived in 2013, battled through some rather different RC versions (changing drastically as it went), finally stabilizing with v1.0 back in mid-2015\. My interest in JSON-API has changed substantially over the![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/icon/apis-logo-square-12.png)APIs You Won't HatePhil Sturgeon![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/thumbnail/cover-1.png)](https://apisyouwonthate.com/blog/lets-stop-building-apis-around-a-network-hack/) With Similarly getting people to enable HTTP caching on their APIs (generally just by headers and probably reusing CDNs they already pass through!) has become a full time job for years. People instead focus on the response times of requests that didn't need to be made, instead of learning to skip making requests that didn't need to be made. [API Design Basics: CacheabilityDesigning an API with cacheability in mind produces a more sensible and better separated set of resources, and it just so happens to be more performant, cheaper, and better for the environment.![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/icon/apis-logo-square-13.png)APIs You Won't HatePhil Sturgeon![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/thumbnail/cacheability.jpg)](https://apisyouwonthate.com/blog/api-design-basics-cacheability/) If we take these more learnings from the larger HTTP world, and combine them with streaming certain datasets instead of defaulting to pagination (a.k.a polling with a cursor), or pushing events instead of forcing polling, then APIs can become a whole lot more efficient. Efficiency reduces costs, reduces hardware requirements, lowers carbon emissions of the software, and helps companies do less carbon accounting. Learn more about how this all links up with the Green Software Practioner course. It's free. [Welcome | Learn Green SoftwareIf you want to make your software applications more sustainable and build a career in green software, then this training is for you![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/icon/favicon-14.ico)Green Software Practitioner![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/thumbnail/og-image.png)](https://learn.greensoftware.foundation/?ref=apisyouwonthate.com) --- *Cover Photo for this article by* [*Jachan DeVol*](https://unsplash.com/@jachan%5Fdevol?utm%5Fcontent=creditCopyText&utm%5Fmedium=referral&utm%5Fsource=unsplash) *on* [*Unsplash*](https://unsplash.com/photos/landscape-photography-of-river-with-trees-ssEQdOiKd8U?utm%5Fcontent=creditCopyText&utm%5Fmedium=referral&utm%5Fsource=unsplash) ### JSON Streaming in OpenAPI v3.2 URL: https://apisyouwonthate.com/blog/json-streaming-in-openapi-v3-2/ Last updated: 2025-09-23T16:07:57.000Z Streaming data allows API servers to send and receive data in real-time or in chunks, rather than waiting for the entire response to be ready. This is already how browsers handle HTML, images, and other media, and now it can be done for APIs working with JSON. This can improve responses with lots of data, or be used to send events from server to client in realtime without polling or adding the complexity of Webhooks or WebSockets. Streaming works by sending “chunks”, which clients can then work with individually instead of waiting for the entire response to be ready. Streaming JSON in particular is increasingly useful as expectations around big data, data science, and AI continue to grow. JSON on its own does not stream very well, but a few standards and conventions have popped up to expand JSON into a streamable format, and OpenAPI v3.2 introduces keywords to describe data in these stream formats. ## JSON Streaming Streaming JSON is a bit tricky because JSON is not designed to be streamed. A naive approach might look like this: ``` { {"timestamp": "1985-04-12T23:20:50.52Z", "level": 1, "message": "Hi!"}, ``` This would trip up most tooling (because the closing bracket is not present in the payload), but we can use something like [JSON Lines](https://jsonlines.org/?ref=apisyouwonthate.com) (a.k.a JSONL) to send one JSON instance per line. ``` {"timestamp": "1985-04-12T23:20:50.52Z", "level": 1, "message": "Hi!"} {"timestamp": "1985-04-12T23:20:51.37Z", "level": 1, "message": "Hows it hangin?"} {"timestamp": "1985-04-12T23:20:53.29Z", "level": 1, "message": "Bye!"} ``` This format allows each line to be a valid JSON object, making it easy to parse with standard native tooling and a `for` loop. There are a bunch of other streaming formats you might want to work with in your API like [Newline Delimited JSON](https://github.com/ndjson/ndjson-spec?ref=apisyouwonthate.com) (NDJSON), [JSON Text Sequence](https://www.rfc-editor.org/rfc/rfc7464.html?ref=apisyouwonthate.com), [GeoJSON Text Sequence](https://datatracker.ietf.org/doc/html/rfc8142?ref=apisyouwonthate.com). Thankfully they are all quite similar and working with them in OpenAPI is almost identical. ## Streaming with OpenAPI OpenAPI v3.0 & v3.1 were able to stream binary data, but struggled to support JSON streaming formats as there was no standard way to define the **schema of individual events** in a stream. People would try to describe things as an array: ``` content: application/jsonl: schema: type: array items: type: object properties: timestamp: type: string format: date-time level: type: integer message: type: string ``` You might see this sort of thing around, but it’s not valid, and will confuse tooling. A stream cannot be described as a single array, and it is a sequence of multiple objects on new lines which is rather different. Some tools could spot the `application/jsonl` content type and figure that out, but we don’t need awkward hacks anymore because the OpenAPI team have solved the problem. OpenAPI v3.2 introduces two new keywords to describe streamed data and events: - `itemSchema` \- define the structure of each item in a stream. - `itemEncoding` \- define how those items are encoded (or serialized), as text, JSON, binary, etc. ### itemSchema Describing a stream with `itemSchema` works just like `schema` with one difference: it will be applied to each item in the stream, instead of the entire response. Consider an example like the [train travel API](https://github.com/bump-sh-examples/train-travel-api/?ref=apisyouwonthate.com) running a stream of tickets: ``` HTTP/1.1 200 OK X-Powered-By: Express Content-Type: application/jsonl; charset=utf-8 Transfer-Encoding: chunked Date: Tue, 19 Aug 2025 18:36:10 GMT Connection: keep-alive Keep-Alive: timeout=5 {"train":"ICE 123","from":"Berlin","to":"Munich","price":79.9} {"train":"TGV 456","from":"Paris","to":"Lyon","price":49.5} {"train":"EC 789","from":"Zurich","to":"Milan","price":59} ``` To describe this stream of items, we can use the `itemSchema` keyword: ``` content: application/jsonl: itemSchema: type: object properties: train: type: string from: type: string to: type: string price: type: number format: float ``` Tooling now has two important switches it can use to figure out how to handle the response. The `itemSchema` makes it clear the response is a stream, and the `application/jsonl` content type lets tooling decide how to present that. For streaming formats that just handle streams of JSON, the `itemSchema` is often sufficient to describe the structure of each item in the stream. For more complicated formats, additional encoding information may be needed. ### itemEncoding The `itemEncoding` keyword allows you to specify how each item in the stream should be encoded, with the same encoding object as the `encoding` keyword. Using `itemEncoding` is only possible for `multipart/*` responses, so it is not very useful for an API that’s streaming JSON, unless you were streaming a mixture of JSON and assets/images on a single response. ``` content: multipart/mixed: itemSchema: $comment: A single data image from the device itemEncoding: contentType: image/jpg ``` Let’s ignore `itemEncoding` for now and focus on the major use case of streams for APIs: streaming data and events. ## Popular Streaming Formats - [JSON Lines](https://jsonlines.org/?ref=apisyouwonthate.com) - [NDJSON](https://github.com/ndjson/ndjson-spec?ref=apisyouwonthate.com) - [JSON Text Sequences](https://datatracker.ietf.org/doc/html/rfc7464?ref=apisyouwonthate.com) - [Server-Sent Events (SSE)](https://developer.mozilla.org/en-US/docs/Web/API/Server-sent%5Fevents/Using%5Fserver-sent%5Fevents?ref=apisyouwonthate.com) They all work a little different, but they share the common goal of allowing data to be sent in a continuous stream rather than as a single, complete response. ### JSON Lines & NDJSON Working with JSON Lines or NDJSON is basically identical in OpenAPI, and feels very much like working with plain JSON responses just with a different header and a bit of `itemSchema` usage. If using JSONL use content type `application/jsonl`, and if using NDJSON use content type `application/x-ndjson`. ``` paths: /logs: get: summary: Stream of logs as JSON Lines responses: '200': description: | A stream of JSON-format log messages that can be read for as long as the application is running, and is available in any of the sequential JSON media types. content: application/jsonl: itemSchema: type: object properties: timestamp: type: string format: date-time level: type: integer minimum: 0 message: type: string examples: JSONL: summary: Log entries description: JSONL examples are just a string where each line is a valid JSON object. value: | {"timestamp": "1985-04-12T23:20:50.52Z", "level": 1, "message": "Hi!"} {"timestamp": "1985-04-12T23:20:51.37Z", "level": 1, "message": "Hows it hangin?"} {"timestamp": "1985-04-12T23:20:53.29Z", "level": 1, "message": "Bye!"} ``` The example once again shows JSONL as a series of JSON objects with a newline character `\n` (0x0A) between them. This can only be described as a [YAML multiline string](https://yaml-multiline.info/?ref=apisyouwonthate.com), because JSONL/NDJSON cannot be described as plain JSON/YAML due to the newline characters. > Remember to use `value: |` to write multi-line strings in YAML, because the pipe will allow newlines to be passed through. Using `value: >` would remove newlines and put each JSON instance onto the same line. The sample code for either of these formats could look a bit like this: ``` app.get("/tickets", async (_, res) => { res.setHeader("Content-Type", "application/jsonl; charset=utf-8"); res.setHeader("Transfer-Encoding", "chunked"); for (const ticket of tickets) { res.write(JSON.stringify(ticket) + "\n"); } res.end(); }); ``` ### JSON Text Sequence A third JSON streaming format which would be identical other than a weird little complication. The other two formats are just a newline character `\n` (0x0A) at the end of the line, but [RFC 7464: JSON Text Sequence](https://www.rfc-editor.org/rfc/rfc7464.html?ref=apisyouwonthate.com) requires a control character at the start ASCII Record Separator (0x1E). This is not a visible character in most contexts, but it will be in there like this: ``` 0x1E{"timestamp": "1985-04-12T23:20:50.52Z", "level": 1, "message": "Hi!"} 0x1E{"timestamp": "1985-04-12T23:20:51.37Z", "level": 1, "message": "Hows it hangin?"} 0x1E{"timestamp": "1985-04-12T23:20:53.29Z", "level": 1, "message": "Bye!"} ``` The `0x1E` (ASCII Record Separator) indicates the start of a new JSON object in the stream. Control characters are a bit magical and invisible to most text editors so it can be a little confusing. Working with JSON Text Sequence tooling for both producing the stream and reading the stream can solve this problem, letting the tooling insert and read out the control characters without you needing to worry. ``` import { Generator } from "json-text-sequence"; // ... snip express setup ... app.get("/tickets", async (_, res) => { res.setHeader("Content-Type", "application/json-seq"); const g = new Generator(); g.pipe(res); for (const ticket of tickets) { g.write(ticket); } res.end(); }); ``` The [json-text-sequence](https://www.npmjs.com/package/json-text-sequence?ref=apisyouwonthate.com) package makes this easier and provides a simple method for generating and consuming JSON Text Sequences. ### Server-Sent Events (SSE) Streaming JSON as chunks of data is only one way that JSON gets streamed. What about sending events, with some JSON being passed along as attributes? [Server-Sent Events](https://developer.mozilla.org/en-US/docs/Web/API/Server-sent%5Fevents/Using%5Fserver-sent%5Fevents?ref=apisyouwonthate.com) (SSE) can handle this, as a standard for sending real-time updates from a server to a client over HTTP. In OpenAPI, you can define SSE streams using the `text/event-stream` content type and the `itemSchema` keyword to describe the structure of the events being sent. ``` content: description: A request body to add a stream of typed data. required: true content: text/event-stream: itemSchema: type: object properties: event: type: string data: type: string retry: type: integer required: [event] # Define event types and specific schemas for the corresponding data oneOf: - properties: event: const: addString - properties: event: const: addInt64 data: format: int64 - properties: event: const: addJson data: contentMediaType: application/json contentSchema: type: object required: [foo] properties: foo: type: integer ``` The `oneOf` is optional, but a handy use of [polymorphism](https://bump.sh/blog/json-streaming-openapi-3-2/%5Fguides/openapi/specification/v3.2/data-models/schema-composition.md?ref=apisyouwonthate.com) to describe different schemas for each event - which can really help with documentation and validation. Valid events to come through this stream might look like: ``` event: addString data: This data is formatted data: across two lines retry: 5 event: addInt64 data: 1234.5678 unknownField: this is ignored event: addJSON data: {"foo": 42} ``` ## Sentinel Events Some streaming systems do not always send all data or events in the exact same way. The items in a stream could be polymorphic objects, or there could be some special events that come through to say the stream is closed (also known as sentinel events). Instead of trying to handle all of these edge cases with special new keywords, OpenAPI allows you to use the standard JSON Schema keywords to model these variations. ``` text/event-stream: itemSchema: oneOf: - - const: { data: "[DONE]" } ``` Whatever the schema is, it can be defined using the standard JSON Schema keywords like `oneOf`, `anyOf`, or `allOf` to handle variations in the event structure. This allows you to define a flexible schema that can accommodate different types of events in the stream. ## Conclusion Not too many tools and frameworks in the world of API are outright offering support for JSON Streaming, and with [OpenAPI v3.2 only just being released](https://github.com/OAI/OpenAPI-Specification/releases/tag/3.2.0?ref=apisyouwonthate.com) it'll will be a while until some of the "wait and see" OpenAPI tooling vendors are supporting it. Many have been working on it for months already though, so who knows! That said I've been diving into [getting JSON streaming to work](https://apisyouwonthate.com/blog/streaming-data-with-rest-apis/) in a handful of languages and it's been smooth as a whale in JavaScript, PHP, Ruby, and Go, so read this if you'd like to learn more about the concept in general away from OpenAPI specifics. [Streaming Data with REST APIsAre you forcing API clients to wait for every single byte of massive JSON collections to be sent from the server before letting them render data that’s ready already?![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/icon/apis-logo-square-14.png)APIs You Won't HatePhil Sturgeon![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/thumbnail/streaming-in-depth.jpg)](https://apisyouwonthate.com/blog/streaming-data-with-rest-apis/) --- This post was [originally written](https://bump.sh/blog/json-streaming-openapi-3-2/?ref=apisyouwonthate.com) for [Bump.sh](https://bump.sh/?ref=apisyouwonthate.com) who make amazing API documentation/exploration/catalogs, and who are good enough to sponsor my time writing, otherwise I couldn't do this sort of thing. ### A Love Letter to OpenAPI URL: https://apisyouwonthate.com/newsletter/love-letter-to-openapi/ Last updated: 2025-09-04T11:24:07.000Z With so many exciting OpenAPI stories in this issue, I find myself reflecting on what I love most about OpenAPI: planning. Using OpenAPI correctly and effectively promotes API Design First practices, which in turn leads to better APIs. A lack of design first principles always reminds me of why my creative writing essays failed in high school: a lack of planning. My English teacher always told me to stop shooting from the hip and write some outlines instead, then build upon them. Personally, teenage Alexander thought she was an idiot, but she was right. Better planning leads to better software and more effective APIs. I am sorry I never listened Ms Robbinson. If you need a refresher on [API design first principles](https://apisyouwonthate.com/blog/a-developers-guide-to-api-design-first/), Phil has got you covered. \-- Alexander ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2025/08/Pasted_Image_8_4_25__7_14---AM.png) ## The fastest way to build production-ready MCP servers Gram by Speakeasy: turn your API platform into an AI platform. Create tools from OpenAPI, curate into custom toolsets, and deploy hosted MCP servers [Get started today](https://app.getgram.ai/?utm%5Fmedium=email&utm%5Fsource=apisywh) ## The API Roundup *API News, links, and tools from around the web* ### [JSON Streaming in OpenAPI v3.2.0](https://bump.sh/blog/json-streaming-openapi-3-2/?utm%5Fsource=bluesky&utm%5Fmedium=social&utm%5Fcampaign=json%5Fstreaming&utm%5Fcontent=bump) Just like the browser handles HTML and images by streaming, you can also stream JSON APIs. A few standards have arisen to make this possible, and now the OpenAPI Spec allows you to define them. Phil is back writing another article for Bump.sh on documenting your streaming endpoints in OpenAPI. ### [Run Express.js on Cloudflare Workers](https://jross.me/run-express-js-on-cloudflare-workers/?ref=apisyouwonthate.com) When building APIs on Cloudflare, my go-to stack has always been HonoJS. Cloudflare is continually working on its Node.js compatibility and now supports one of JavaScript's oldest and most popular web application frameworks, Express.js. Now you have more options for your Cloudflare-powered APIs. ### [Speakeasy: OpenAPI Parser](https://www.speakeasy.com/blog/release-oss-openapi-library?ref=apisyouwonthate.com) Speakeast has open-sourced their OpenAPI parser, something they use for their own OpenAPI work, SDK generation and more. Super interesting and not just because it is written in GO, but also supports Arazzo workflows and OpenAPI Overlays. Definitely worth checking out. ### [API World](https://apiworld.co/?ref=apisyouwonthate.com) One of the largest API conferences, API World, is taking place from September 3rd to 5th. If you don't have tickets and are interested in the talks, you can purchase tickets to the online version, which will take place on September 10th-12th. ### [OpenAPI Specification in Fabric](https://blog.fabric.microsoft.com/en-US/blog/openapi-specification-code-generation-now-available-in-fabric-user-data-functions/?ref=apisyouwonthate.com) Microsoft now allows you to generate an OpenAPI spec from your Data Functions in Fabric. If you don't know what Fabric is, it's basically a suite of tools for working with data, and think of User Functions like Lambdas for taking actions on that data. With the ability to generate an OpenAPI spec, you easily generate client applications to trigger your functions via a REST API call. --- ## From our Community Articles written and shared in our free [Slack community](https://slack.apisyouwonthate.com/?ref=apisyouwonthate.com). ### [Congress API](https://api.congress.gov/?ref=apisyouwonthate.com) For our American readers, did you know that there is a REST API for information on Congress? Can not believe I missed this one for our Civic-themed newsletter a while back. The API provides details on Bills, members of Congress, and more. Mike Bifuclo also shared an [MCP server version](https://congressmcp.lawgiver.ai/?ref=apisyouwonthate.com) with me this week. ## Support APIs You Won't Hate When you become an member, you'll get access to members-only content while directly supporting our work. Your support helps us to keep making resources for the API community. [Become a member today](https://apisyouwonthate.com/#/portal) Thanks so much to our members: **Kin L, Juxt, Alex R, Nolan S, Frank, James D, Rich, Ryan T, and Umair**. Your support means the world to us! ✌️ Until next time, Alexander, Phil & Mike ### Playwright Does API Testing Now? URL: https://apisyouwonthate.com/newsletter/playwright-does-api-testing-now/ Last updated: 2025-09-01T11:25:47.000Z Playwright is a popular tool from Microsoft for writing component, integration and end-to-end tests for frontend. Did you know you can also use it for API testing? Playwright allows you to validate API requests in a variety of ways using [APIRequestContext](https://playwright.dev/docs/api-testing?ref=apisyouwonthate.com). While this tool enables you to set up a state via an API in UI tests, it can also be used for end-to-end tests of your API. In the TypeScript/JavaScript world, we can take this further and combine the OpenAPI Spec to validate these tests—a perfect set-up for JavaScript full-stack projects with your API and frontend in the same repo. One final thought: how did Microsoft end up owning so many of the tools and libraries for JS development? \-- Alexander ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2025/08/Pasted_Image_8_4_25__7_14---AM.png) ## The fastest way to build production-ready MCP servers Gram by Speakeasy: turn your API platform into an AI platform. Create tools from OpenAPI, curate into custom toolsets, and deploy hosted MCP servers [Get started today](https://app.getgram.ai/?utm%5Fmedium=email&utm%5Fsource=apisywh) ## The API Roundup *API News, links, and tools from around the web* ### [JSON Stringify Now Twice as Fast](https://v8.dev/blog/json-stringify?ref=apisyouwonthate.com) I was a little reluctant to add this as it's not API specific, but considering this is one of the main functions used in every JavaScript-powered API, I think it's worth including. The V8 team shared how a complete rethinking of `JSON.stringify` even down to its core memory and character handling has led to a 2x performance increase. Get ready for faster APIs. ### [Introducing Bump.sh Embed](https://bump.sh/blog/introducing-bump-sh-embed/?ref=apisyouwonthate.com) Bump has released embeddable API docs, giving you complete control over design, how and where they appear. Totaly feels like a step towards headless API docs, which Bump even mentioned themselves. It's exciting to see Bump consistently deliver incredible API tools. ### [API Snip](https://github.com/Tuurlijk/apisnip?tab=readme-ov-file&ref=apisyouwonthate.com) Fun little tool that crossed my BlueSky feed this week, API Snip. An interactive terminal app allowing you to select which endpoints to keep in a spec. Makes it easier to generate small, focused OpenAPI Docs. ### [Slow Express](https://github.com/express-rate-limit/express-slow-down?ref=apisyouwonthate.com) Middleware and Express go hand in hand, and another little plugin that caught my eye: Express Slow Down. This nifty library allows you to slow down API requests and is customisable via milliseconds, after X requests, and window size. It also has options for connecting to an external store. --- ## APIs You Won't Hate The latest from the team at API's You Won't Hate. ### [Goodbye Apiary.io, You'll Be Missed](https://apisyouwonthate.com/blog/goodbye-apiary-io/) Phil gives a heartfelt goodbye to the exceptional API design first documentation tool, Apiary. Unfortunately, after being acquired by Oracle back in 2017, it has been slowly killed off in favour of other tools. If you're not excited about using Oracle and still have some Apiary spec (API Blueprint) lying around, Phil walks you through moving to OpenAPI. --- ## From our Community Articles written and shared in our free [Slack community](https://slack.apisyouwonthate.com/?ref=apisyouwonthate.com). ### [The Indie API Forecasting Your Future](https://netapinotes.com/the-indie-api-forecasting-your-future/?ref=apisyouwonthate.com) Dark Sky API died once it had been acquired by Apple, and left a gaping hole in our hearts. Matthew walks us through why Dark Sky was popular and an innovative new public good indie project that has arisen to fill your weather needs. Mathew talks us through the challenges the project has faced, especially as getting hold of data has become harder, thanks to the cost-cutting of critical services by the current US government. ## Support APIs You Won't Hate When you become an member, you'll get access to members-only content while directly supporting our work. Your support helps us to keep making resources for the API community. [Become a member today](https://apisyouwonthate.com/#/portal) Thanks so much to our members: **Kin L, Juxt, Alex R, Nolan S, Frank, James D, Rich, Ryan T, and Umair**. Your support means the world to us! ✌️ Until next time, Alexander, Phil & Mike ### About Slack's new rate limits... URL: https://apisyouwonthate.com/newsletter/about-slacks-new-rate-limits/ Last updated: 2025-08-04T11:21:59.000Z [Mike](https://mikebifulco.com/newsletter?ref=apisyouwonthate.com) gave me a nudge this week that [Slack changed its rate limits](https://api.slack.com/changelog/2025-05-terms-rate-limit-update-and-faq?ref=apisyouwonthate.com) for their conversation history and replies API back in May to **one request per minute** and a maximum of 15 objects. These API changes will affect all external apps (not in the Slack marketplace), which could make sense for a few reasons. However, what is interesting is that this also affects any internal customer-built app. From the outside, it appears to me that this is an attempt to restrict Slack's data from LLMs and to prevent customers from using their own Slack data to power an LLM. The changes force you to use Slack's AI offerings. There will be other reasons for this change, but I'm pretty sure LLMs are the driving force.. Unfortunately, just as the web itself is becoming more walled off due to AI, I think this trend will extend to APIs as well. \-- Alexander ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2025/08/Pasted_Image_8_4_25__7_14---AM.png) ## The fastest way to build production-ready MCP servers Gram by Speakeasy: turn your API platform into an AI platform. Create tools from OpenAPI, curate into custom toolsets, and deploy hosted MCP servers [Get started today](https://app.getgram.ai/?utm%5Fmedium=email&utm%5Fsource=apisywh) ## The API Roundup *API News, links, and tools from around the web* ### [Ktor OpenAPI Generation](https://github.com/ktorio/ktor-klip/pull/3?ref=apisyouwonthate.com) Kotlin has become one of my all-time favourite languages, and I often find myself staring longingly at Ktor while I am stuck using Spring Boot. Now I have even more reasons to be jealous, as they are adding the ability to generate an OpenAPI spec from your application code. ### [Generate Go Boilerplate from OpenAPI 3 Specs](https://golangweekly.com/link/171859/746b4b8391?ref=apisyouwonthate.com) Ever wanted to convert OpenAPI into GO code and avoid writing boring boilerplate? Well, now you can do both server and client-side. The [v2.5 release notes](https://golangweekly.com/link/171860/746b4b8391?ref=apisyouwonthate.com) tease some new tweaks and eventual OpenAPI 3.1 support, or you can [use some tricks to achieve](https://golangweekly.com/link/171861/746b4b8391?ref=apisyouwonthate.com) it for now. ### [HyperFetch](https://hyperfetch.bettertyped.com/?ref=apisyouwonthate.com) Another JS framework crossed my feed this week. Initially, I was drawn in by the prospect of generating client SDKs from your OpenAPI spec, but I realised there was a lot more to it. HyperFetch works in any TypeScript environment, so server, browser, Electron and ReactNative, offering a unified integration for your API. Worth a look. ### [Pocketbase](https://pocketbase.io/?ref=apisyouwonthate.com) An Open Source backend ready to go. Instantly get a Go REST API with this wonderful little project. Great for spinning up an API for any frontend framework. Nice to see something so feature-rich, simple and open-sourced. ### [APIs and Agents: What Developers Need to Know](https://www.oreilly.com/radar/apis-and-agents-what-developers-need-to-know/?ref=apisyouwonthate.com) Another post on API's and AI. Honestly, I'm getting a little reluctant to share this type of content because everyone keeps telling me I am not needed anymore. Despite my impending obsolescence, I am now buried in slop that is also on fire got sidetracked there, sorry. A nice short article from O'Reilly, APIs are not going anywhere, but with a few tweaks, you can support users and agents. --- ## APIs You Won't Hate The latest from the team at API's You Won't Hate. ### [The 5 Best API Docs Tools in 2025](https://apisyouwonthate.com/blog/top-5-best-api-docs-tools/) Phil is back, and this time he is digging into the best API Doc Tools. Which one is the best? Well, that depends on what you need. Dive in deep and look at the pros and cons of the best tools in 2025. ### [Generating OpenAPI Docs for Java with SpringBoot](https://apisyouwonthate.com/blog/generating-openapi-spring-boot/) This one's for all of our `public static void main()` pals - learn how to export OpenAPI from your Spring Boot application with Springdoc. --- ## From our Community Articles written and shared in our free [Slack community](https://slack.apisyouwonthate.com/?ref=apisyouwonthate.com). ### [Train Travel OpenAPI Level Up](https://portal.wiremock.cloud/km9l9/?ref=apisyouwonthate.com#/) We all love the OpenAPI Train Travel example, right? Well, Tom Akehurst made an advanced mock of the latest version in WireMock Cloud. ## Support APIs You Won't Hate APIs You Won't Hate is a member-driven community. Show your support with a paid membership, and everything will be `200 OK` [Become a member today](https://apisyouwonthate.com/#/portal) Thanks so much to our members: **Kin L, Juxt, Alex R, Nolan S, Frank, James D, Rich, Ryan T, and Umair**. Your support means the world to us! ✌️ Until next time, Alexander, Phil & Mike ### Goodbye Apiary.io, You'll Be Missed URL: https://apisyouwonthate.com/blog/goodbye-apiary-io/ Last updated: 2025-08-04T11:12:50.000Z [Apiary.io](https://apiary.io/?ref=apisyouwonthate.com) was the first API documentation tool I ever used, and helped pioneer the API-design first workflow as we know it today, and now that it's finally being shuttered after being absorbed into an amorphous mega-corporation purely as a user aquisition exercise I wanted to say a few words. Apiary.io was founded in 2011, bringing modern looking “Stripe-like” three column API documentation to teams which would otherwise have been documenting their API with a horrendous CMS or Wiki. OpenAPI didn’t exist back then, and Swagger was still incredibly difficult to work with with minimal tooling outside of the SmartBear offering, so they created their own description format: [API Blueprint](https://apiblueprint.org/?ref=apisyouwonthate.com). Unlike Swagger/OpenAPI which is based on JSON/YAML, the team pioneered a brand new approach, making API Blueprint based on Markdown. This made it very easy to write, and that reduced friction for teams adopting and modifying API Blueprint. It was a lot easier for me to say "Hey you're already writng Markdown for your manual docs, but if you do it like this you'll have an actual API description which you can use programatically too, for docs, mocks, testing, and SDK generation!" Apiary was well respected for offering a solid SaaS product, and was well known for their open-source projects: [Dredd](hhttps://github.com/apiaryio/dredd) and [Gavel](https://github.com/apiaryio/gavel-spec?ref=apisyouwonthate.com). These tools offered powerful testing functionality which helped teams ensure their API implementation matched the API description, which was foundational to bringing the API Design-first workflow to REST APIs. This API description-based testing would help point out discrepancies in a test suite instead of having complaints from users. [Dredd: Do Your HTTP API JusticeUpdate 2021-02-08: Old post is old! I really don’t recommend messing with Dredd anymore, it was a handy stepping stone on the way to contract testing with API descriptions but these days using OpenAPI v3.x and using those descriptions to power contract testing in your existing test suite is![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/icon/favicon-13.ico)Phil SturgeonPhil Sturgeon![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/thumbnail/dredd.jpg)](https://philsturgeon.com/dredd-api-testing-documentation/?ref=apisyouwonthate.com) Sadly what is easier for humans is rarely easier for computers, and whilst API Blueprint's ecosystem of open-source tooling maintainers were doing their best it was legitimitely hard to work with, and the official drafter C++ parser was overwhelmed with issues. Third party parsers in other languages would not support everything. This is in my opinion one of the major contributing factors to OpenAPI winning the API description format wars, but Apiary cleverly hopped on the OpenAPI bandwagon adding support for v3.0 too. Abstraction was always the goal for tooling companies working with any of these formats, and this inspired others to diversify their support too. Sadly all of this is history now, because Dredd and Gavel have been archived along with all the other open-source code from the Apiary team, and the hosted product will be shuttered in September. ## Why is Apiary shutting down? As is so often the case in API industry, they have been assimilated by a giant corporation that wanted to grab the cool new thing and its users. [Oracle bought Apiary](https://techcrunch.com/2017/01/19/oracle-boosts-cloud-offering-with-apiary-api-management-tool-acquisition/?ref=apisyouwonthate.com) back in 2017, and have been migrating users over to their “Oracle Cloud” offering for years. Paid Apiary plans had been deprecated since 2018, and now finally users are seeing banners on their dashboards telling them Apiary is finally turning off the lights September 9th 2025. Why did Oracle do this? The same reason SmartBear bought Stoplight, Postman bought Akita, and Atlassian bought Optic. - Money - Users - Relevance - Laziness Capitalism is a history of big companies eating up small companies, whether that's acquiring to absorb talent, or acqui-firing to get code/data that's interesting or purely to put people out of business, sometimes it's a little of all three. Oracle has a big nebulous confusing product that nobody wants to use and they bought the cool kid alternative of the time, then seemed to want to do nothing with it and just force people to move over to Oracle Cloud, very much like SmartBear are doing forcing people away from Stoplight and over to API Hub. ## Do you really want to use Oracle Cloud? Oracle Cloud is a large offering similar to Amazon Web Services: there’s a whole lot of stuff to sift through to get to the bit that you want, and all you wanted was API documentation that looked nice with almost no work. This shouldn’t be so hard, especially as Apiary always focused on simplicity. Now it’s incredibly confusing to figure out where you’re meant to go or what you’re meant to do, and that pisses me off. It’s impossible to even find out how much it might cost, and you know what they say: if you have to ask, you can’t afford it. ## Converting from API Blueprint to OpenAPI I've been helping API documentation products like [Bump.sh](https://bump.sh/?ref=apisyouwonthate.com) prepare for the incoming wave of users finally getting out of Apiary.io before they're forced into Oracle Cloud, and I wanted to make sure there was a good open-source solution to converting API Blueprint to OpenAPI. There are lots of tools out and about, but trust me they will only do part of the job and leave you very confused about whats wrong. It's been an issue for almost a decade but I've spent most of that decade sorting stuff like this out, so I've made you this script which glues together a few options and corrects the most common mistakes as it goes. Go take a look at [apiblueprint2openapi](https://github.com/bump-sh-examples/apiblueprint2openapi?ref=apisyouwonthate.com). ``` git clone git@github.com:bump-sh-examples/apiblueprint2openapi.git cd apiblueprint2openapi cp ~/src/my-project/example-api/latest.apib ./original.apib ``` Wherever your `.apib` is, probably in your GitHub repo, grab it and pop it in as `orignal.apib` because I didn't have time to make a proper CLI wrapper for this. Then run NPM because all of these tools I've cellotaped together are NPM packages. ``` npm install npm run convert ``` The conversion script will fill up the `generated/` directory with output from two alternative conversion scripts so you can see which you prefer, and both will get upgrades and corrections. - `apib2openapi.yaml` \- Using NPM module `apib2openapi` to produce OpenAPI v3.0. - `apib2openapi.31.yaml` \- Upgraded `apib2openapi` output with `openapi-format` to produce OpenAPI v3.1 and generally tidied up. - `apispecconverter.yaml` \- Using NPM module `apib2openapi` to produce OpenAPI v3.0. - `apispecconverter.31.yaml` \- Upgraded `apib2openapi` output with `openapi-format` to produce OpenAPI v3.1 and generally tidied up.0. This is probably a lot to think about so to keep it simple, **you probably want to use `generated/apispecconverter.31.yaml`**. ``` cp generated/apispecconverter.31.yaml ~/src/my-project/example-api/openapi.yaml ``` Want to see how they look with zero effort involved? ``` $ npx bump-cli preview generated/apispecconverter.31.yaml > Your preview is visible at: https://bump.sh/preview/9b563ce6-e1a5-4da6-8f5d-d51ae0b26c1d (Expires at 2025-06-27T17:49:34+02:00) > * Let's render a preview on Bump.sh... done ``` ## How did I upgrade to OpenAPI v3.1? You gotta learn about [OpenAPI Format](https://www.npmjs.com/package/openapi-format?ref=apisyouwonthate.com) if you work with OpenAPI. Not only can it upgrade from v3.0 to v3.1, but it can tidy up and refactor your OpenAPI document, improving readability and fixing mistakes. ``` npm install -g openapi-format openapi-format openapi.legacy.yaml -o openapi.yaml --convertTo '3.1' ``` This will output a new OpenAPI document called `openapi.yaml` which is formatted nicely and upgraded to the latest version. ## Tidying up the mess Anyone who has ever managed a developer portal trying to handle a deluge of disparate quality API descriptions coming in will know that API teams demand the best results despite generally not wanting to do any work to help that happen. Even if they want to play ball, god knows what their pipeline is and how possible that would even be. Could be [creating OpenAPI from code annotations](https://docs.bump.sh/guides/openapi/code-first/?ref=apisyouwonthate.com). Could be [generating from HTTP traffic](https://apisyouwonthate.com/blog/creating-openapi-from-http-traffic/). Could be [AI slop](https://create-api.dev/?ref=apisyouwonthate.com). Could just be a well meaning team inexperienced with OpenAPI. Learning to programatically uplift OpenAPI is incredibly helpful regardless, and OpenAPI Format does a great job of shaving the edges off. ``` openapi-format generated/apispecconverter.yaml -o generated/apispecconverter.31.yaml --convertTo '3.1' -c config/openapi-format.json ``` There is a **lot** you can do with [the config file](https://github.com/thim81/openapi-format?tab=readme-ov-file&ref=apisyouwonthate.com#openapi-formatting-configuration-options), but the bare minimum to get the converted API Blueprint output into usable OpenAPI for me meant passing vacuum linting rules, and this did the job. ``` { "sort": true, "casingSet": { "operationId": "kebab-case", "properties": "snake_case" }, "generateSet": { "operationIdTemplate": "-", "overwriteExisting": true } } ``` This removed the nasty and invalid `operationId` entries, and gave calm consistent ones which would match conventions in other APIs nicely. It also sorted things alphabetically and set naming conventions for properties. What an absolute treat. ## Iteratively improve OpenAPI The conversion process is not always perfect, but a larger problem is that API Blueprint didn’t have a lot of the features that OpenAPI has, so you may find that your OpenAPI document is missing some information. For example, API Blueprint [didn’t have a way to describe authentication](https://help.apiary.io/api%5F101/apib-authentication/?ref=apisyouwonthate.com). That felt like a huge admission at the time, and folks were working on it with [API Blueprint RFC 002: Authentication](https://github.com/apiaryio/api-blueprint-rfcs/blob/master/rfcs/0002-authentication.md?ref=apisyouwonthate.com), but seeing as that never got done you’ll likely have to add this all in manually. OpenAPI v3.1 can describe all sorts of authentication and authorization via [Security Schemes](https://docs.bump.sh/guides/openapi/specification/v3.1/advanced/security/?ref=apisyouwonthate.com), so with a little bit of work you can add this in to your OpenAPI document and get it where it needs to be. ## So long Apiary To the team who made Apiary awesome, thanks for everything you did, and it was a pleasure to meet you all that one time in Prague. I hope you got paid handsomely for all your hard work and they didn't force you to stick around too long. To Oracle, pffffft. --- If you're looking for a new API documentation solution to get away from Oracle, check out our comparison of the [Top 5 Best API Documentation Tools](https://apisyouwonthate.com/p/6514776d-55f9-4760-85ce-4484b3d1c4d3/). [The 5 Best API Docs Tools in 2025Which API documentation tool is the best? It Depends™! Let’s go through the best modern tooling and look at when you might want to pick one over another.![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/icon/apis-logo-square-9.png)APIs You Won't HatePhil Sturgeon![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/thumbnail/twitter-img-1-3.png)](https://apisyouwonthate.com/p/6514776d-55f9-4760-85ce-4484b3d1c4d3/) ## ### Generating OpenAPI docs for Java with Spring Boot URL: https://apisyouwonthate.com/blog/generating-openapi-spring-boot/ Last updated: 2025-08-04T11:12:42.000Z API Code-first is the art of building an API, and then popping some annotations or metadata in there to output API documentation in an API description format like [OpenAPI](https://spec.openapis.org/oas/latest.html?ref=apisyouwonthate.com). ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2025/07/image-4.png) Preview of OpenAPI-based API reference documentation in [Bump.sh](https://bump.sh/?ref=apisyouwonthate.com). The most popular API Code-first approach in Spring uses a tool called [Springdoc](https://springdoc.org/?ref=apisyouwonthate.com), which can generate OpenAPI v3.1 from any Spring-based application (Boot, Web MVC, Webflux, etc). A basic skeleton will be generated with absolutely no work whatsoever, then adding annotations will keep improving the usefulness of the generated OpenAPI. ## Prerequisites To follow this guide, you will need: - [Java 21](https://www.oracle.com/java/technologies/downloads/?ref=apisyouwonthate.com) or later - [Maven 3.5+](https://maven.apache.org/download.cgi?ref=apisyouwonthate.com) or [Gradle 7.5+](https://gradle.org/install/?ref=apisyouwonthate.com) - A Spring Boot application, which you can create by following the [Spring Boot REST Service guide](https://spring.io/guides/gs/rest-service?ref=apisyouwonthate.com) or cloning down the [sample code](https://github.com/bump-sh-examples/spring-code-first?ref=apisyouwonthate.com). Here’s a few commands to help you out. ``` # check for java. v21 or above is fine java --version # check if maven or gradle are there which mvn || which gradle ``` If you are new to working with Java applications it’s recommended to use an IDE like [IntelliJ IDEA](https://www.jetbrains.com/idea/?ref=apisyouwonthate.com) or [VS Code](https://code.visualstudio.com/?ref=apisyouwonthate.com) (using this great guide to running [Spring in VS Code](https://spring.io/guides/gs/guides-with-vscode?ref=apisyouwonthate.com)). You can use the command line if you prefer. ## Creating OpenAPI with Springdoc ### Step 1: Install the Springdoc dependency If you are using Maven, add the following dependency to your `pom.xml` file: ``` org.springdoc springdoc-openapi-starter-webmvc-api 2.8.8 ``` If you are using Gradle, add this to your `build.gradle` file: ``` dependencies { implementation 'org.springdoc:springdoc-openapi-starter-webmvc-api:2.8.8' } ``` The `springdoc-openapi-starter-webmvc-api` package supports Spring MVC and Spring Boot, but for Spring Webflux support use the `springdoc-openapi-starter-webflux-api` package instead. ### Step 2: Configure Springdoc You can customize the OpenAPI generation by adding properties to your `application.properties` or `application.yml` file. ``` # application.properties springdoc.api-docs.path=/openapi ``` Or if you are using YAML: ``` # application.yml springdoc: api-docs: path: /openapi title: My API version: 1.0.0 ``` The path setting defines where the OpenAPI description will be available. Springdoc makes it available on `http://localhost:8080/v3/api-docs` by default, with the `v3` being a reference to OpenAPI v3.x, but you can change it to something more suitable for your application. Why not a classy `/openapi` path so the OpenAPI description can be hosted on a nice clean `http://localhost:8080/openapi` once the application is running. ### Step 3: Create/find some REST controllers This step is less of a step, and more “have some controllers and models” which you probably already do. Whether you are starting from scratch or looking at an existing application, let’s use the example of a `EmployeeController`, using Hibernate and Spring Data JPA to make life a little easier. ``` # src/main/java/com/bumpsh/demo/EmployeeController.java import java.util.List; import org.springframework.web.bind.annotation.GetMapping; import org.springframework.web.bind.annotation.PathVariable; import org.springframework.web.bind.annotation.PostMapping; import org.springframework.web.bind.annotation.RequestBody; import org.springframework.web.bind.annotation.RestController; @RestController class EmployeeController { private final EmployeeRepository repository; EmployeeController(EmployeeRepository repository) { this.repository = repository; } @GetMapping("/employees") List all() { return repository.findAll(); } @PostMapping("/employees") Employee newEmployee(@RequestBody Employee newEmployee) { return repository.save(newEmployee); } @GetMapping("/employees/{id}") Employee one(@PathVariable Long id) { return repository.findById(id) .orElseThrow(() -> new EmployeeNotFoundException(id)); } // snipped PUT and DELETE for now } ``` The repository is wrapping around the model to handle some tedious database interaction, and the model looks like this: ``` # src/main/java/com/bumpsh/demo/Employee.java import java.util.Objects; import jakarta.persistence.Entity; import jakarta.persistence.GeneratedValue; import jakarta.persistence.Id; @Entity class Employee { private @Id @GeneratedValue Long id; private String name; private String role; // snip constructor, getters and setters } ``` So far we’ve done absolutely nothing to this application code specific to Springdoc or OpenAPI, but this is already enough for Springdoc to generate a basic OpenAPI description. ### Step 4: Run the application Boot up the Spring Boot application to install all the dependencies and start the HTTP server. If you are using an IDE, you can run the application directly from there. For example, in IntelliJ IDEA, you can right-click on the main class (the one annotated with `@SpringBootApplication`) and select “Run”. If you are using Maven, run: ``` mvn spring-boot:run ``` If you are using Gradle, run: ``` ./gradlew bootRun ``` Once the application is running and the HTTP server is available, the OpenAPI description will be available at [http://localhost:8080/openapi](http://localhost:8080/openapi?ref=apisyouwonthate.com) for JSON, or [http://localhost:8080/openapi.yaml](http://localhost:8080/openapi.yaml?ref=apisyouwonthate.com) for YAML. ![](https://bump.sh/website/images/blog/springdoc-initial.png) Very basic, and it’s only saying “what” which misses out on the critical “why” and “how” aspect of API documentation, but it will be a start. ### Step 5: Publish API Documentation Before we get stuck into making this OpenAPI perfect, lets get used to deploying it to Bump.sh so we can see how things look. Iterative improvements are easy with Bump.sh, as it can be hooked up to deploy a new version of the documentation every time a pull request is merged in. If you’re new to Bump.sh, [create your first API](https://docs.bump.sh/help/getting-started/upload-your-first-definition/?ref=apisyouwonthate.com). Then retrieve the name and token of this documentation from the *CI deployment* settings page. We’ll use this to deploy the OpenAPI description with the Bump.sh CLI. ``` npm install -g bump-cli ``` Now we can deploy the OpenAPI description to Bump.sh using the `bump deploy` command. ``` $ bump deploy http://localhost:8080/openapi \ --doc spring-code-first \ --token my-documentation-token * Your new documentation version will soon be ready at https://bump.sh/bump-examples/hub/code-samples/doc/spring-code-first ``` This can then be automated in your CI/CD pipeline, so every time you merge a pull request, the OpenAPI description is automatically deployed to Bump.sh. We’ll show you how to do that in the end, but for now, let’s focus on improving the OpenAPI description. ### Step 6: Improve OpenAPI with Java annotations The skeleton OpenAPI description generated by Springdoc is a start, but it lacks context and explanations. To make it more useful, we can add annotations to the controllers and models providing more context to how and why this API works the way it does. This is done with annotations from the `io.swagger.v3.oas.annotations` package, which allows you to add metadata around the Java source code. The Spring framework is already making clear which HTTP methods and URL paths are being used, so the job of these new annotations is to focus on the rest of it: parameters, request body, responses, status codes, and adding useful descriptions and examples wherever possible. Let’s look at some examples of how to do this. **Describing Operation** Adding a summary and description to the operations in the controller is an important first move, because the summary is used to generate the navigation menu, and the description is used to explain what the operation does in more detail. ``` # src/main/java/com/bumpsh/demo/EmployeeController.java import io.swagger.v3.oas.annotations.Operation; @RestController class EmployeeController { // Add the @Operation annotation to the method @GetMapping("/employees/{id}") @Operation(summary = "Get an employee by ID", description = "Returns the details of an employee based on the provided ID") List all() { return repository.findAll(); } // Do this for each HTTP method in the controller @PostMapping("/employees") @Operation(summary = "Create a new employee", description = "Adds a new employee to the system") Employee newEmployee(@RequestBody Employee newEmployee) { return repository.save(newEmployee); } } ``` Just adding summary and description to the operation is a solid first step to making your OpenAPI documentation more useful and interesting to look at. See the OpenAPI YAML description by Springdoc after adding these annotations. ``` openapi: 3.1.0 info: title: OpenAPI definition version: v0 servers: - url: http://localhost:8080 description: Generated server url paths: /employees/{id}: get: tags: - Employee Management summary: Get an employee by ID description: Returns the details of an employee based on the provided ID operationId: one parameters: - name: id in: path required: true schema: type: integer format: int64 responses: "200": description: OK content: '*/*': schema: $ref: "#/components/schemas/Employee" ``` Of course these example operations are very basic, but this is a good place to get stuck into letting users know what to expect when they call your API. Is the response paginated? Is there a default sort order? ![](https://bump.sh/website/images/blog/operation-description.png) **Tagging Controllers** ``` # src/main/java/com/bumpsh/demo/EmployeeController.java import io.swagger.v3.oas.annotations.tags.Tag; // Add the @Tag annotation to the class @RestController @Tag(name = "Employee Management", description = "Operations related to employee management") class EmployeeController { // ... existing code ... } ``` **Adding Parameters** Parameters can be added to the method using the `@Parameter` annotation. This is useful for describing path parameters, query parameters, and request body parameters. These parameter annotations are applied to the method parameters, like this: ``` # src/main/java/com/bumpsh/demo/EmployeeController.java import io.swagger.v3.oas.annotations.Parameter; @RestController class EmployeeController { @GetMapping("/employees/{id}") @Operation(summary = "Find an employee", description = "Returns the details of an employee based on the provided ID") public Employee one( @Parameter(in = ParameterIn.PATH, description = "ID of the employee to retrieve", required = true, schema = @Schema(type = "integer")) @PathVariable Long id) { // ... existing code ... } } ``` This will add a parameter to the OpenAPI description for the `id` path variable, which looks like this in the generated OpenAPI: ``` paths: '/employees/{id}': get: parameters: - name: id in: path required: true description: ID of the employee to retrieve schema: type: integer format: int64 ``` Parameter descriptions can contain all sorts of handy information, like letting people know that a date parameter is in ISO 8601 format, or that a string parameter is a UUID. This is useful for consumers of the API to understand how to use the API correctly without having to look at the code or ask questions. ![](https://bump.sh/website/images/blog/parameter-description.png) **Adding Responses** How about adding a response schema to the `@ApiResponse` annotation to describe what the response body will look like? By default, Springdoc will use the return type of the method to generate the response schema, but you can override this by using the `@ApiResponse` annotation. ``` # src/main/java/com/bumpsh/demo/EmployeeController.java import io.swagger.v3.oas.annotations.responses.ApiResponse; import io.swagger.v3.oas.annotations.media.Content; import io.swagger.v3.oas.annotations.media.Schema; @RestController class EmployeeController { @GetMapping("/employees/{id}") @Operation(summary = "Find an employee", description = "Returns the details of an employee based on the provided ID") @ApiResponse( responseCode = "200", description = "Successful operation", content = @Content(mediaType = "application/json", schema = @Schema(implementation = Employee.class)) ) public Employee one(@PathVariable Long id) { // ... existing code ... } ``` **Annotating the Model** To make the OpenAPI description even more useful, you can annotate the model class to describe its properties. This is done using the `@Schema` annotation. ``` # src/main/java/com/bumpsh/demo/Employee.java import jakarta.persistence.Entity; import jakarta.persistence.GeneratedValue; import jakarta.persistence.Id; import io.swagger.v3.oas.annotations.media.Schema; @Entity @Schema(description = "Employees are individuals who work for an organization, " + "contributing their skills and expertise to achieve the organization's goals." + "Roles are the specific functions or positions that employees occupy within the organization, " + "defining their duties, responsibilities, and the scope of their work." ) class Employee { private @Id @Schema(accessMode = Schema.AccessMode.READ_ONLY, description = "Unique identifier which should not be shared publicly", example = "1234") @GeneratedValue Long id; @Schema(description = "Employee name", example = "Bilbo Baggins") private String name; @Schema(description = "Employee role", example = "Software Engineer") private String role; Employee() {} Employee(String name, String role) { this.name = name; this.role = role; } // Getters and Setters } ``` This will add a description to the `Employee` model, and also describe each property with its type, description, and example value. The generated OpenAPI will look like this: ``` components: schemas: Employee: type: object description: "Employees are individuals who work for an organization, contributing\ \ their skills and expertise to achieve the organization's goals.Roles are\ \ the specific functions or positions that employees occupy within the organization,\ \ defining their duties, responsibilities, and the scope of their work." properties: id: type: integer format: int64 description: Unique identifier which should not be shared publicly example: 1234 readOnly: true name: type: string description: Employee name example: Bilbo Baggins role: type: string description: Employee role example: Software Engineer ``` This will make the OpenAPI description much more useful, as it provides a clear understanding of the model and its properties, including their types, descriptions, and example values. Over time as properties change, the OpenAPI description will also change, allowing consumers of the API to always have an up-to-date understanding of the API and its models. ### Step 7: Deploy the improved OpenAPI description Now that we’ve added some annotations to the controllers and models, we can generate a new OpenAPI description and deploy it to Bump.sh. Run the application again to generate the new OpenAPI description, and then deploy it to Bump.sh using the `bump deploy` command. ``` $ bump deploy http://localhost:8080/openapi \ --doc spring-code-first \ --token my-documentation-token * Your new documentation version will soon be ready at https://bump.sh/bump-examples/hub/code-samples/doc/spring-code-first ``` This will update the OpenAPI description on Bump.sh with the new version, which will now include the additional context and explanations we added with the annotations. ### Step 8: Automate the deployment To automate the deployment of the OpenAPI description to Bump.sh, you can add a step in your CI/CD pipeline to run the `bump deploy` command every time a pull request is merged. This way, the OpenAPI description will always be up-to-date and available on Bump.sh. You can use a CI/CD tool like GitHub Actions, GitLab CI, or Jenkins to run the deployment command automatically. For example, if you are using GitHub Actions, you can create a workflow file `.github/workflows/bump.yml` with the following content: ``` name: Check & deploy API documentation permissions: contents: read pull-requests: write on: push: branches: - main pull_request: branches: - main jobs: deploy-doc: if: ${{ github.event_name == 'push' }} name: Deploy API documentation on Bump.sh runs-on: ubuntu-latest steps: - name: Checkout uses: actions/checkout@v4 - name: Set up JDK uses: actions/setup-java@v2 with: distribution: 'temurin' java-version: '21' - name: Build and run the application run: | mvn clean spring-boot:run & - name: Wait for the application to start run: sleep 30 - name: Deploy API documentation uses: bump-sh/github-action@v1 with: doc: spring-code-first token: ${{secrets.BUMP_TOKEN}} file: http://localhost:8080/openapi ``` This workflow will run every time a commit is pushed to the `main` branch, build the application, wait for it to start, and then deploy the OpenAPI description to Bump.sh using the `bump deploy` command. Make sure to set the `BUMP_TOKEN` secret in your GitHub repository settings with the token you retrieved from the Bump.sh CI deployment settings page. ## Sample Code The sample code for this guide is published on GitHub so you can try that if you’re having trouble adding it to your application: [spring-code-first](https://github.com/bump-sh-examples/spring-code-first?ref=apisyouwonthate.com), and the [deployed documentation](https://bump.sh/bump-examples/hub/code-samples/doc/spring-code-first?ref=apisyouwonthate.com) is over here. --- *Thanks to* [*Bump.sh*](https://bump.sh/?ref=apisyouwonthate.com) *for sponsoring my time to work on this guide. You can use the same approach to publish off to other API documentation providers if you like but Bump.sh is a solid choice.* ### The 5 Best API Docs Tools in 2025 URL: https://apisyouwonthate.com/blog/top-5-best-api-docs-tools/ Last updated: 2025-07-30T10:43:23.000Z Without documentation nobody knows an API exists or how to use it, so it’s worth investing the time in creating clear, useful, understandable documentation. This will help customers integrate with their applications quicker, cut down on support calls, and allow coworkers to onboard quicker as they join the company or move teams. Fully documenting an API can be a long and difficult process, but new tools are always popping aiming to solve the problem in different ways. Some are open-source and some are Software-as-a-Service, some focus on beautiful interfaces, some focus on powerful functionality, and some focus on jamming AI all over the place. There’s no one tool to rule them all, and different tools may be chosen depending on who is in charge of setting up the documentation (API developers, technical writers, governance teams) and the intended audience (public APIs, internal APIs, partner APIs). There’s also the question of if this is for a single API or multiple APIs, whether or not other types of guides are supported (often via Markdown), and whether they support API discoverability through API catalogs to help customers pick between those various APIs or if you need to build that yourself. To learn which documentation tools can be helpful for different scenarios, let’s compare the most popular OpenAPI documentation tools, in no particular order: - [Redoc](#redoc) - [Scalar](#scalar) - [Stoplight Elements](#stoplight-elements) - [Bump.sh](#bumpsh) - [ReadMe.io](#readmeio) ## **Redoc** Redoc is an old champion in the OpenAPI documentation world, built by Redocly to offer a beautiful “Stripe-like” two or three panel experience back when the only real choice was the less appealing Swagger UI. ![Redoc UI](https://bump.sh/website/images/blog/top-5-redoc.png) Redoc is also available as a self-hosted/open-source option, and a hosted version is available as part of a larger SaaS platform [Realm](https://redocly.com/docs/realm?ref=apisyouwonthate.com). Self hosted Redoc can only do one API at a time, but the hosted Redoc on the $10/month Pro level can handle 1 project, which can have up to 100 pages. OpenAPI documentation and other guides all count as pages, so you can get quite a few APIs into that. The $24/month Enterprise plan comes with SSO for managing who can edit APIs, and “guest SSO” for hiding the resulting API documentation which is handy for partner APIs. **Pros:** - ✅ Supports AsyncAPI, and OpenAPI 3.1, 3.0, and 2.0. - ✅ Highly customizable as React components. - ✅ Can be self-hosted or used as a cloud service. - ✅ Includes “Try It” functionality on the cloud version. - ✅ Provides API governance through the hosted version, and a handy linting and bundling CLI tool. - ✅ Supports developer portal through Realm and Reunite SaaS products. **Cons:** - ❌ Requires developer effort for customization. - ❌ No Try It on the self-hosted version. - ❌ Configuration is all done with YAML. - ❌ Builds interface on-the-fly by reading an OpenAPI document so large APIs load slowly. - ❌ Hosted version suffers very slow server-side rendering too. - ❌ Only add one Guest SSO identity provider per organization making partner APIs for multiple organizations difficult. **Best for:** - Teams looking for highly customizable and branded API documentation. - Publishing high quality reference documentation and guides for a single partner. ## **Scalar** Scalar is made by developers, for developers, and you can tell. From the heavy focus on open-source tooling, to the dark-mode default with small text, and use of JetBrains Mono font, a quick glance at Scalar screams “programmers were here”. ![Scalar UI](https://bump.sh/website/images/blog/top-5-scalar.png) The toolsuite is very new, but they are focused on laying a groundwork of solid open-source tooling to build on, and are integrating with countless open-source projects from Gitbook to Nitro to Rust. Scalar is taking an interesting approach of blurring the lines between API documentation and API client, offering the most powerful Try It of any of the documentation tools out there, and even offering a desktop version of the API client as a dedicated application which reads and even modifies the OpenAPI document instead of creating something new like a Postman collection, pushing that API client into OpenAPI editor territory. The docs have the client, the client have the docs, and the hosted version has the client too. **Pros:** - ✅ A drop-in replacement for SwaggerHub, replacing Swagger UI & Swagger Editor. - ✅ Supports OpenAPI 3.1, 3.0, and 2.0. - ✅ Can pick from [multiple CSS themes](https://github.com/scalar/scalar/blob/main/documentation/themes.md?ref=apisyouwonthate.com) and present layouts. - ✅ All versions of Scalar support a powerful Try It / API client. - ✅ Cloud version has a built in text-based OpenAPI editor. - ✅ Supports guides as well as reference documentation. **Cons:** - ❌ Builds interface on-the-fly by reading an OpenAPI document so large APIs load slowly. - ❌ Doesn’t offer advanced API governance tools. - ❌ Free SaaS only gets one user, then its $12/seat, with SSO behind a “Talk to the CEO” button. - ❌ GitHub Sync only available on Pro, with no other way to deploy via CLI or CI/CD. - ❌ Supports multiple APIs but no API Catalog / Dev Portal functionality. **Best for:** - Solo API developers who want a desktop HTTP client which happens to also produce API documentation. - Open-source tools (e.g. web application frameworks, content management systems) which could themselves be self-hosted with an API and generate API endpoints / OpenAPI. Scalar would help turn that users generated OpenAPI into API documentation for their end-users without any SaaS involved. - Technical Writers who would rather have a text-editor included with the SaaS and don’t need figuring out Git integration or CI/CD deploys. ## **Stoplight Elements** [Stoplight Elements](https://stoplight.io/open-source/elements?ref=apisyouwonthate.com) is a Web/React component that drops into existing documentation or open-source, allowing anyone to enjoy Stripe-like API documentation. It focuses on being as beautiful as possible, whilst still delivering the same functionality and covering the same use cases as Swagger UI. ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2025/07/image-2.png) The tool was aiming to knock Swagger UI off the “top spot” years back, but active development has slowed to a crawl in a plume of irony after Swagger UI owners SmartBear bought Stoplight. It’s getting bug fixes and a handful of tweaky features, and development may well pick up again in the future, so it’s not out of the game just yet. The self-hosted tool can take any OpenAPI document by URL or path, but can only handle one API at a time like most of the self-hosted tools. There is an extension called Elements DevPortal, which can handle multiple APIs and Markdown guides, but this assumes all the APIs are in the same Stoplight Platform “project”. It’s essentially a mini API Catalog, but the catalog cannot pull OpenAPI/Markdown content out of multiple projects or repos making the catalog functionality a little limited. **Pros:** - ✅ Beautiful, interactive documentation with Try-It functionality. - ✅ Supports OpenAPI 3.1, 3.0, and 2.0. - ✅ Easy integration into existing documentation or apps via Web/React components. - ✅ Integrates with Stoplight Platform, a SaaS ecosystem which includes a GUI for API design, and API governance. **Cons:** - ❌ Limited out-of-the-box customization compared to Bump, Scalar, or Redoc. - ❌ Confusing to get started for non-technical users as it needs a web server and code to load the component. - ❌ Stoplight’s full suite can be expensive, with unlimited APIs/projects but paying per user. - ❌ All "API Catalog" functionality requires both the SaaS and React/JS components. - ❌ Builds interface on-the-fly by reading an OpenAPI document so large APIs load slowly. **Best for:** - Teams already using Stoplight for API design. - Companies that prefer embedding documentation within existing apps. ## Bump.sh Bump.sh is a SaaS solution focused on building “Stripe-like” three column API reference documentation from OpenAPI and AsyncAPI documents from any source, using any workflow. ![Bump.sh UI](https://bump.sh/website/images/blog/top-5-bump.png) Bump.sh focuses on getting out of your way, staying clear of [the “Walled Garden” approach](https://bump.sh/blog/from-swaggerhub-to-bump-git?ref=apisyouwonthate.com) others take. Instead of forcing everything to be done through a very specific way through a user interface, it [integrates with existing Git/CI workflows](https://docs.bump.sh/guides/openapi/specification/v3.1/the-perfect-modern-openapi-workflow/?ref=apisyouwonthate.com), providing useful insight through their automatic changelog and breaking change detection in pull requests. It’s also one of the first few tools to support AsyncAPI as well as OpenAPI, allowing for event-driven APIs to be documented along side the usual REST/HTTP APIs. API Catalogs are a strong new feature known as Hubs, which allow for multiple APIs to be deployed from a variety of Git repos, or you can get creative with CI/CLI/API integrations to bring in API descriptions for anywhere. This is brilliant for large organizations trying to bring all their APIs into one place, even if teams have wildly different workflows, directory structures, design-first/code-first, spread around different GitHub/GitLab/Azure workspaces, and even allows for some older groups still using Subversion for some reason. This is the only tool compared which allows for this flexible approach to API catalogs. With different “Guests” able to view each Hub, this is brilliant for Partner APIs, especially as they can subscribe to API changes to be alerted of anything they need to know about directly. The still-in-beta API Explorer has brought powerful Try It functionality to documentation which people are coming to expect in all API docs. Multiple branches and versioning also make [Bump.sh](http://bump.sh/?ref=apisyouwonthate.com) a great option for teams working on Public APIs where multiple versions of API documentation need to be maintained. The team pride themselves on availability, and zero chatbots or automation between you and getting an answer from a friendly dedicated technical support staff member. They’re real people working on making real software and not trying to game anything with AI. **Pros:** - ✅ Version control-focused API documentation. - ✅ Scans OpenAPI / AsyncAPI on the server instead of doing it on the fly every time. - ✅ SEO-friendly thanks to the pre-rendered crawlable documentation. - ✅ Automatically detects and highlights API changes. - ✅ Supports AsyncAPI, OpenAPI 3.1, 3.0 & 2.0, and Webhooks. **Cons:** - ❌ Cloud hosted only, no self-hosted or open-source options. - ❌ Fewer customization options than Redocly. - ❌ Embedding as a component is still work-in-progress but coming soon. **Best for:** - CI/CD-focused teams integrating API documentation into their DevOps workflows. - Anyone trying to avoid being stuck in a walled garden. Your Git/CI is the source of truth. - Teams heavily focused on API versioning and change management. - Tech reviewers eager for breaking change detection and API diffs to save them staring at a wall of potentially irrelevant YAML changes. ## **ReadMe** ReadMe is a hosted developer portal which allows for API documentation in all forms, not just API reference documentation, but supports Markdown guides and even [Recipes](https://docs.readme.com/main/docs/recipes?ref=apisyouwonthate.com) for documenting workflows and breaking down code samples. ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2025/07/image-3.png) Instead of deploying changes from a Git repository or providing an editor to make changes directly, there is a [two-way sync](https://docs.readme.com/main/docs/rdme?ref=apisyouwonthate.com) to merge in changes. ReadMe is a strong pick for DevRel teams as it has built-in support for user feedback, a discussion forum, and tracks analytics. This can allow for data-driven improvements to the developer experience. The pricing starts off ok for startups at $99/month, but ramps up quickly to $399/month for custom CSS/HTML. Extras send the price even higher, with $100/month for developer dashboards. and another $150/month to enable a ChatGPT-enabled “Owlbot” taking a punt at answering questions with what might even occasionally be correct answers sometimes. **Pros:** - ✅ Provides interactive API docs with an API explorer. - ✅ Includes a developer portal with guides and tutorials. - ✅ User feedback and discussions. - ✅ Great for SaaS companies offering public APIs. **Cons:** - ❌ Pricing can be high for startups. - ❌ Cloud hosted only, no self-hosted or open-source options. - ❌ Less control over the doc styling compared to self-hosted solutions. **Best for:** - Teams that need API documentation and developer engagement tools. - Businesses that want built-in analytics and API usage tracking. --- ### **Recommendations** So which is the best tool? As always “it depends”, so here are some quick pointers to help find the right tool for you. - **For customization & governance:** [Redocly](https://redocly.com/?ref=apisyouwonthate.com) - **For distributing preinstalled in CMS or API/web frameworks:** [Scalar](https://scalar.com/?ref=apisyouwonthate.com) - **For a focus on breaking changes and versioning:** [Bump.sh](http://bump.sh/?ref=apisyouwonthate.com) - **For performance & SEO:** [Bump.sh](http://bump.sh/?ref=apisyouwonthate.com) - **For interactive docs within a design workflow:** [Stoplight Elements](https://stoplight.io/open-source/elements?ref=apisyouwonthate.com) - **For DevRel teams who want user feedback:** [ReadMe](https://readme.com/?ref=apisyouwonthate.com) ### OpenAPI Spec 3.2 is coming URL: https://apisyouwonthate.com/newsletter/openapi-3-2-is-here/ Last updated: 2025-07-15T13:26:03.000Z The OpenAPI team is looking to bring new features to a close for the upcoming version 3.2\. There are a bunch of new features coming in the next version. Support for the new QUERY HTTP method, enhanced multipart support, support and guidance for event streams and a new Tag Object format. The Tag Object is an interesting one as it comes with the ability to create hierarchical structures through categories and nested tags. Ready for more? You can check the [release milestone](https://github.com/OAI/OpenAPI-Specification/milestone/12?ref=apisyouwonthate.com) on GitHub for more info. \-- Alexander ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2025/04/State-of-API-DevEx-Report.png) ## Help shape the future of APIs If you build, manage, or use APIs, your insights can help highlight key trends and challenges in the ecosystem. Take the 10 minute survey, and you'll get early access to the report — plus a chance to win a PS5, NuPhy mechanical keyboard, and more! [Take the Survey](https://www.speakeasy.com/state-of-api-devex/2025?utm%5Fsource=newsletter&utm%5Fmedium=email&utm%5Fcampaign=apisyouwonthate) ## The API Roundup *API News, links, and tools from around the web* ### [Secure Express API with Cedar](https://aws.amazon.com/blogs/opensource/secure-your-application-apis-in-5-minutes-with-cedar/?ref=apisyouwonthate.com) Cedar is an open-source language for defining permissions as policies and then evaluating them. Recently, they released a plugin for ExpressJS, allowing you to add policy-based authorisation in your Express API within minutes. The Cedar approach to authorisation decouples the logic from your application, making it easier to maintain and audit. ### [Stop Converting Your REST APIs to MCP](https://www.jlowin.dev/blog/stop-converting-rest-apis-to-mcp?ref=apisyouwonthate.com) An interesting article arguing against turning your OpenAPI Spec into an MCP server. The article claims an OpenAPI spec is rich in context for humans, but an OpenAPI spec for MCP should be "ruthlessly curated and minimalist". An interesting read, but I find myself disagreeing. For example, the issue of the LLM calling the delete everything endpoint. Why is there a delete everything endpoint? Sounds like design first might have been thrown out the window. ### [Generating Client Libraries with OpenAPI](https://amf3.github.io/articles/api/hello%5Fopenapi/?ref=apisyouwonthate.com) Good OpenAPI specs can be used for more than just describing APIs; they can also be used to generate code. In this article, learn how to create a working Go HTTP server and a Python HTTP client with an OpenAPI spec. Dig into generating the client interfaces and some gotchas to watch out for. ### [How to Automate Your API Governance](https://nordicapis.com/how-to-automate-your-api-governance/?ref=apisyouwonthate.com) A five-step process for automating API governance, from adopting OpenAPI specifications to gateways, rules, and automating documentation generation. Stop development bottlenecks through API consistency and by integrating these tools into your CI/CD. --- ## APIs You Won't Hate The latest from the team at API's You Won't Hate. ### [Powerful HTTP/API Clients: Alternatives to Postman](https://apisyouwonthate.com/blog/http-clients-alternatives-to-postman/) This isn't a new article, but it has been growing in popularity the last few weeks. We created a comparison of quick and easy graphical HTTP clients you can use if you're considering alternatives to Postman. --- ## From our Community Articles written and shared in our free [Slack community](https://slack.apisyouwonthate.com/?ref=apisyouwonthate.com). ### [OpenAPI Community Calendar](https://www.openapis.org/get-involved/about?ref=apisyouwonthate.com) Although I've worked with OpenAPI for years, today I learned that they share a calendar with all their working group meetings, which you're welcome to attend. ## Support APIs You Won't Hate When you become an member, you'll get access to members-only content while directly supporting our work. Your support helps us to keep making resources for the API community. We ❤️ our community - thanks for being here! [Become a paid supporter](https://apisyouwonthate.com/#/portal) Thanks so much to our members: **Kin L, Juxt, Alex R, Nolan S, Frank, James D, Rich, Ryan T, and Umair**. Your support means the world to us! ✌️ Until next time, Alexander, Phil & Mike ### Caching, up-front URL: https://apisyouwonthate.com/newsletter/design-for-caching-up-front/ Last updated: 2025-07-01T12:45:17.000Z Caching is essential in REST APIs, and for many, it is often an afterthought. Including caching in the design process requires your team to think more carefully about your models, and how to structure the resource layer. Ask me how I know... I've spend the last week attempting to speed up a data-intensive analytics portal. *Pre-fetching* data is far more performant and provides a better end-user experience in a web application, especially if the underlying API utilises caching most efficiently. If you need a little refresher, check out this great [article from Speakeasy](https://www.speakeasy.com/api-design/caching?ref=apisyouwonthate.com) and, of course, our article [Design Basics: Cacheability](https://apisyouwonthate.com/blog/api-design-basics-cacheability/). \-- Alexander ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2025/04/State-of-API-DevEx-Report.png) ## Help shape the future of APIs If you build, manage, or use APIs, your insights can help highlight key trends and challenges in the ecosystem. Take the 10 minute survey, and you'll get early access to the report — plus a chance to win a PS5, NuPhy mechanical keyboard, and more! [Take the Survey](https://www.speakeasy.com/state-of-api-devex/2025?utm%5Fsource=newsletter&utm%5Fmedium=email&utm%5Fcampaign=apisyouwonthate) ## The API Roundup *API News, links, and tools from around the web* ### [OpenAPI Fundamentals](https://training.linuxfoundation.org/express-learning/openapi-fundamentals-lfel1011/?utm%5Fcampaign=Promote%20OpenAPI%20Foundations%20Course%202024&utm%5Fsource=home-page&utm%5Fmedium=top-banner&utm%5Fcontent=2024-banner) The Linux Foundation offers some fantastic courses; however, I missed that they have a free 90-minute course to Learn the basics of the OpenAPI Specification. You'll explore how to write, use, and extend OpenAPI documents to streamline API development and integration. Worth checking out if you're new to OpenAPI. ### [GraphQLConf 2025](https://graphql.org/conf/2025/schedule/?ref=apisyouwonthate.com) GraphQL Conf has dropped the schedule for their conference in Amsterdam in September. The lineup is quite interesting, featuring GraphQL at Meta, GraphQL performance issues at Netflix, and addressing GraphQL's biggest mistakes, such as not being REST 😉. ### [Hono Comes to NodeJS](https://www.youtube.com/watch?v=4ks1RvEM99Y&ref=apisyouwonthate.com) Hone creator Yusuke Wada talks us through using only Web Standards to build the web framework Hono and running it on Cloudflare Workers, Deno, and Bun. Yusuke details the challenges of bringing Hono to Node and how they were overcome. I love Hono and am super excited to see it come to Node. ### [Ktor 3.2.0 Is Now Available](https://blog.jetbrains.com/kotlin/2025/06/ktor-3-2-0-is-now-available/?ref=apisyouwonthate.com) If you haven't heard of Ktor before, it's a modern web framework in Kotlin that focuses on asynchronous programming features and intuitive, imperative flow. Honestly, I am not doing it justice. You should be checking it out. They recently dropped a new release with support for [HTMX](https://htmx.org/?ref=apisyouwonthate.com). ### [JSON evolution in Go](https://antonz.org/go-json-v2/?ref=apisyouwonthate.com) Go's `encoding/json v2` introduces a more robust and flexible API for JSON handling. It offers enhanced performance, streaming-friendly functions like `MarshalWrite`, `UnmarshalRead` and customisable options for formatting and field behaviour, enabling developers to create more efficient and maintainable web APIs. --- ## APIs You Won't Hate The latest from the team at API's You Won't Hate. ### [All about Mocking, with Wiremock's Tom Akehurst](https://apisyouwonthate.com/podcast/all-about-mocking-with-wiremocks-tom-akehurst) Mocking is one of those things that started out as a simple idea, and has evolved to support an incredible amount of developer use cases over the years. In this episode of the Podcast, Tom from Wiremock chats with Phil about *what* mocking is, strategies for not littering OpenAPI doc with mocking details, and ways to keep api dev teams empowered by mocking, instead of hampered by it. --- ## From our Community Articles written and shared in our free [Slack community](https://slack.apisyouwonthate.com/?ref=apisyouwonthate.com). ### [MCP Week Roundup](https://zuplo.com/blog/2025/06/16/mcp-week-roundup?ref=apisyouwonthate.com) Adrian Machado shares the highlights from MCP week at Zuplo. Covering some interesting topics from how MCP won't be an API killer, how to build MCP servers with Zuplo and how AI Agents are going to be using your APIs. Worth checking out, thanks for sharing, Adrian. ## Support APIs You Won't Hate When you become an member, you'll get access to members-only content while directly supporting our work. Your support helps us to keep making resources for the API community. [Become a member today](https://apisyouwonthate.com/#/portal) Thanks so much to our members: **Kin L, Juxt, Alex R, Nolan S, Frank, James D, Rich, Ryan T, and Umair**. Your support means the world to us! ✌️ Until next time, Alexander, Phil & Mike ### All about Mocking, with Wiremock's Tom Akehurst URL: https://apisyouwonthate.com/podcast/all-about-mocking-with-wiremocks-tom-akehurst/ Last updated: 2025-07-01T15:05:50.000Z ## Show notes [Wiremock](https://wiremock.org/?ref=apisyouwonthate.com) - on BlueSky [@wiremock.bsky.social](https://bsky.app/profile/wiremock.bsky.social?ref=apisyouwonthate.com) - on Mastodon [@fosstodon/wiremock](https://fosstodon.org/@wiremock?ref=apisyouwonthate.com) - on GitHub [@wiremock/wiremock](https://bsky.app/profile/wiremock.bsky.social?ref=apisyouwonthate.com) #### Transcript Interview with Tom Akehurst from Wiremock ****Phil Sturgeon:** \[00:00:00\] Hello everybody. Welcome to another episode of APIs. You Won't Hate the podcast version. Today I am joined by Tom Hurst and we are gonna talk all about mocking and not taking the piss outta people nowhere to talk about API mocking which is a whole different kettle of fish. So, Tom, hello. How you doing? ****Tom Akehurst:** Good, thanks. Yeah. I'm really pleased to, to finally, finally manage to get this organized and and get to speak to you on the show. ****Phil Sturgeon:** Straight in with the mocking. Brilliant. Yeah, we have, we have been trying to record this podcast for about six months, and I feel like you know more about what's going on in my life than many of my friends, just by the wide variety of excuses I've used for why we've had to cancel and reschedule this podcast. ****Tom Akehurst:** A never dull moment. ****Phil Sturgeon:** Yeah. Yeah. I hope, I hope your life's going smoothly. What have you been up to lately? ****Tom Akehurst:** I mean, running a startup is a bit of, a bit of an all consuming endeavor. So sadly, I I've probably got, you know, more to say about that than anything else. There's \[00:01:00\] a, a bit of background. I'm the CTO and co-founder of why I'm not the company, which is based on why I mock the open source project. It's been around for ish years now. And when I'm not dealing with childcare issues. That's kind of, you know, the other thing that I'm doing. ****Phil Sturgeon:** Yeah, brilliant. Running startups is a whole lot of fun. Have the reason for this taking so bloody long and the fact that most of our listeners might be incredibly surprised to see this podcast popping up in their feed. 'cause we've had quite a few long gaps between episodes is just trying to, trying to run. Trying to do, trying to work on APAs You Won't Hate, and Green Turtle, which is like a software consultancy and protect Earth, which is like Reforesting, the uk. It is a bit much, and all three of them are going well. It's nice, but oh my God, can I not handle that? ****Tom Akehurst:** startups rather than one. ****Phil Sturgeon:** Basically, yeah. And, and protect Earth is nuts. But we have just hired a new project manager, which I'm really excited about. 'cause he basically is taking over the, the day-to-day of like finding, you know, finding. Farmers to plant with \[00:02:00\] and get in the right trees in the right place and taking over the day-to-day operations of that more project based stuff. And it leaves me to run around being like, Hey, we should buy that chunk of land over there and hey, that you come partner with us, give us some money, and, and all that sort of other important business work without the whole, we need 2000 of these trees over there by Tuesday. It's ****Tom Akehurst:** So you can be the CEO delegating all the grunt work to someone. ****Phil Sturgeon:** Yeah, exactly. ****Tom Akehurst:** It? ****Phil Sturgeon:** Yeah, although we, we all just spend my, my brain is slightly wrecked from spending Tuesday, two days ago we were doing a, a first aid course with a forestry module, and it's just all of the creative ways you can like maim and destroy yourself using a wood chipper or getting ticks and Lyme disease, which I bloody out. That was the reason for the last delay. I'm like, I'm sorry I can't do a podcast. I've just found eight ticks in me and I'm ill now. Life is madness. But a talking about computers is fun. Let's, I, I am, I am going mad, but I do love to hear what's going on in the API community. \[00:03:00\] And so ****Tom Akehurst:** So. ****Phil Sturgeon:** glad to have you on talking about wire mock. Wire mock is cool. So in the world of mocking, mocking is, is, is a really important topic, I think. But I think it's one of those words that's become so completely overloaded that no one's ever talking about the same thing. And so. Just wondering if you can give us a bit of an intro to kind of yeah, where, where mocking fits in. Like mocking appears in testing, but it also appears in like API design. And then some people talk about like sandboxing and, and what's that, or prototyping and what's that. So that's quite a, a big topic to just dive straight into, but could you give us a bit of a lay of the land on that? ****Tom Akehurst:** Yeah, sure. The terminology is pretty tricky and I think I've probably played my part in in muddying the waters further over the years, I'll have to admit. So with apologies to kind of Martin Fowler in the, the XP community generally, who originally came up with a very specific definition of mocking, which was a, you know, essentially a particular type of activity within. Sort of the creation of test doubles more \[00:04:00\] broadly. I guess test doubles were the overall category, and then mocking was a, a technique for doing, you know, behavioral verification. I think over time the word mocking has sort of broadened and become this, this this sort of umbrella term in itself describe test doubles further muddying the waters. Originally it was applying to kind of mocking in code, so substituting interfaces in code, whereas. in the context I'm talking about, and in the context of wire markets, it's about APIs over the wire, you know, so so, so networks APIs rather than kind of in, you know, in code or in memory. And I, I think there's a, I think there's further, further sort of water muddying aspects to this as well, and that there's an older generation of tools referred to as service virtualization. Which is obviously incredibly confusing because when, but when you say virtualization, you tend to think of kind of hypervisors and that kind of stuff. so there's an older generation of tools that do that. And then, and that now also there's this term that sort of entered the mix of API simulation, which sort of is used by, by newer tools. And, you know, we use to describe our \[00:05:00\] product as well. I, I suppose the. What I've kind of settled on as my, the, the way I describe this is a, is kind of a continuum where mocking tends to be the, the closest to the kind of marketing failure definition. The stuff you do, you're probably driving it in code. You are, you're using mocks as essentially as, as sort of close test doubles in. Unit or unit like tests. And you know that it's like code driven and bundled together, whereas, and also they, it is probably worth pointing out. They, they're, they're quite simple. They tend, they, they're kind of, know, canned responses, canned stubs, you know, to, to, ****Phil Sturgeon:** Hmm. ****Tom Akehurst:** to sort of match requests. Whereas at the, sort of, at the other end of this continuum, you have simulation, which is probably just a, you know, warming over of, of service virtualization as a concept where the thing you've built is the simulation is much more sophisticated and realistic. So you have. You know you know, complex kind of behavioral variations. You maybe have the ability to generate responses dynamically rather than just sending static responses. You have the ability to store \[00:06:00\] and retrieve states pull things from data sources, that kind of thing. So, so I, I find that sort of a use, useful, useful continuum to, you know, to think of these things. And I think there are where you sort of choose to be along with continuum. Kind of depends on the, you know, what you're trying to do, what your, your use case. ****Phil Sturgeon:** Yeah, that makes sense. I mean, yeah, the the way, the way that I tried to talk about it. When it was working on Prism at stoplight, what, which is another mocking tool that's no longer really being actively developed, I don't think the, the, the terminology we'd used was like, yeah, mocking is kind of a simple, you make a request and you're gonna get a response of something. It may or may not be useful, and there are ways to like increase the usefulness and relevance of the generated thing. I think we, we had like static and dynamic responses, so static would just be. This mock is based off the open API it said there's gonna be a name, which is a string, and there's gonna be an email, which is format email. And looking at that, it can generate a random name. Oh no. The static would just be like, whatever the example was in \[00:07:00\] the schema. And the dynamic would be, alright, let's try and generate a random name using faker and generate a random email using faker. So that was at least different, not the same thing every time. But there was no like logic there. And, and, and no persistence there. And so I think that for me, the definition of mocking was kind of a bit dim without mean to offend anyone. That was the way I, I classed it. And then, and then sandboxing was like, oh, and we can, we can put a database, spin that up in the background so when you save stuff, we'll put it out and we'll try and match up the fields if possible and, and return that back to you. But that was really blooming hard when people would do things like, they would, you know, send data as a raw JSON object with name and email, and then they'd return it wrapped in like a data envelope and then it wouldn't know, oh, do you have to like do a lot of weird stuff to try and figure out. Matching request to responses, even with a database in the background. So that was always a bit weird. And then kind of prototyping was another term that was like, you are just building an application before \[00:08:00\] you build your application. And you might build a, a, a rubbishly little application, just fire out in, in no JS or something with the, the grand scheme of it being done in in Java or something later on. You might just work on something really, really quickly and then make it later. But that was always a big faf 'cause people would just. Start writing enough code that they'd get attached and then they just wanna put that to production, even though they skipped infinite, there's like massive gaping holes everywhere and you can just bend the roof, the future, like, you know, just was it mass assignment issues all over the place and all that sort of rubbish. So it's always been really interesting trying to make any tools in this space because whatever you make, somebody will want the other stuff. And there are so many different verticals that this mocking. This mocking functionality could be used in so many different parts of the lifecycle, like with wire mock. Where, where do you feel like your tooling, where, where does your tooling aim to be in terms of like stage of the lifecycle and, and what sort of verticals are you going after? I.\[00:09:00\] ****Tom Akehurst:** So, so the, the genesis of the tool is definitely in the of. You know, the, the mocking for in the, in the developer's toolkit. So, so, so the mocking to support, kind of test driven development, you know, acceptance testing, test automation, that kind of thing. ****Phil Sturgeon:** Hmm. ****Tom Akehurst:** and, you know, so it, it fits in that kind of you developer in a loop. You know, they, they, I mean, the open source tool is very much that sort of developer in a loop where you are, you are building functionality and you are, you are, you know, you are supporting the construction of your code with, you know, with mock APIs. The, you know, our commercial products is, is kind of deliberately about broadening that base. And so it's in, one sense, it's broadening it in, in terms of the, you know, an individual organization software development lifecycle. So where you want to start deploying mocks into. These kind of mixed, integrated environments, you know, where you want to be able to do of isolate sets of related services and test \[00:10:00\] them, or where you want to be able to kind of deploy, say a microservice into a real environment. But you need to, to mock most or all of its dependencies to, you know, to be able to, to isolate it and test it ****Phil Sturgeon:** Hmm. ****Tom Akehurst:** When you, you know, particularly, you know, I mean, microservice environments are probably the best example of this. You know, places where. You have lots of different teams who are all building APIs and they're all talking to each other's APIs and everything's changing you know, rapidly and somewhat unpredictably and, and you know, we're sort of about kind of trying to help tame some of that, that kind of chaos. But we do, but we do what we are also sort of playing in the, I, I suppose at the top level. I kind of think of that being, you know, sort of two, two kind of top level, I guess. Types of use case with sort of maybe even different sort of buyers or different sets of personas, different user personas associated with them. And I think one of them is the one I just described. So developers and, and test engineers. And then the other one is, is kind of I suppose API people probably more, you know, the, the maybe main listeners of this podcast. \[00:11:00\] So people who are building APIs for consumption by the public, by customers, you know, who are you know, thinking about how to, to make APIs you know, accessible, well designed. You know, where their APIs a kind of competitive asset for their company. ****Phil Sturgeon:** Mm. ****Tom Akehurst:** And I, and I think that, you know, mocking gets used in both of those, those contexts. But what I've observed so far actually is that the, the, the tools get built for one or the other. And I think, ****Phil Sturgeon:** Right. ****Tom Akehurst:** Prism. Prism earlier, I think is an example of a tool that was optimized for the, the sort of API or you know, API building audience. ****Phil Sturgeon:** Yeah, for sure. ****Tom Akehurst:** and I, I think Postman probably falls into that. Yeah, postman's mocking functionality falls into that category as well, ****Phil Sturgeon:** Yeah. ****Tom Akehurst:** it's about supporting the design process and supporting the supporting sort of production of onboarding materials and documentation for, for end users and making it easier for them to, to discover an API. But like I say, then there's this whole other sort of category of users who are doing, who are trying to do serious kind of qa, you know, dev and QA activities with, with \[00:12:00\] mocks. And, and I think, yeah, this, these two sort of parallel kind of families of products I think have, have grown up around those, those two, you know, quite different sets of use cases. And I think we're, you know, we're, we, we're, we're trying to of, you know, plain in, in both of these, you know, we're trying to ensure that they, the, the product gives you the, the affordances to, yeah. To, to, to be able to use mocking to support API documentation to support the design process, but also be then deployable into a, a really demanding QA process. ****Phil Sturgeon:** That's, it's really hard to, to walk that line, isn't it? 'cause yeah, certainly when working on Prism I, I was around for the rewrite from it, turning from a, a go library into kind of a a, a. Type script based CLI, and it just, it did everything. It did it tried to do every single verse call you could imagine. And a lot of mock tools get into this. They, they're like, because they understand the schema of what's allowable and they are trying to cover a few like instances. And they also have like traffic coming through it. You get this like \[00:13:00\] mocking brilliant. Let's do that. We've got the, we've got the open API, let's. Fire out some some potential endpoints and requests and responses, but then actual traffic starts to come through and you can get into like validation of that. Like, Hey, are the requests coming through to this matching the open API? And so then you, then you actually get like validation responses, but then you go, Hey, what? We can have a proxy where you pass real traffic through. And then we could start to say, well do those, match the, the requests and responses in open API. And then. ****Tom Akehurst:** then. ****Phil Sturgeon:** You can use that for QA and you can use that for maybe in production, if you don't mind having the, you know, the actual critical path. Go through some random library that may or may not be particularly performant. And then you start going into like learning of well actually there's not, there's not an implementation here that we're proxying to, but people are sending lots of traffic through this and so let's, let's make an API out of that. Or maybe there is an implementation that doesn't have any open API, so let's. Make funny. You open the traffic and it starts to become so confusing that whenever you have a tool that can do \[00:14:00\] anything, no one knows what the hell to do with it. Like, you've got that read me and you look at it and go, blah, what, what is all this Then? And so that was something I did with Prism was just like, delete a lot of that functionality from the roadmap with the intention of bringing it back later. Simpler. But it just becomes really complicated to try and deal with that many different personas that many different use cases. Well, and so really open-ended questions here, but just like, how do you approach that? 'cause you've got, you've got open A-P-I-G-R-P-C, all this stuff, and you've got like testing and, and, and using it in the design process. Like, how the heck do you manage any sort of roadmap and balance of personas and functionality there? ****Tom Akehurst:** Yeah, it's, it's difficult. I mean, we, we experience all of those problems. I mean, I, I've, I've been working on y mark open source for 14 years now, more or less. And you know, so there's, there's a lot in there and, and there's a lot in there for sort of, lots of, lots of different people. And it, and I, I think I probably only recently have start really seriously started thinking about, kind of segmenting the audience up \[00:15:00\] into, you know, in, into you know, different user needs and and, and trying to work out how to focus on those. But it's, it's a tough balance. I mean, when, you know, when you're selling a product in a way that's, that can be a good thing because it gives you these, these signals about you know, people will tell you what they want to buy and what they don't want to buy. So you can kind of use, use that as a, as a sort of as an easy filtering ****Phil Sturgeon:** Just turn that off. No one's using that. Don't worry about that. ****Tom Akehurst:** Yeah, yeah, yeah. Or likewise, you, you know, there'll be one thing where you do some, you know, almost you, you chuck in some sort of sort of token wave towards it, and then everyone goes, oh, that's great. We want more of that. And suddenly you find that you've pivoted all your engineering effort to kind of making this thing really you know, the, the, the kind of a central feature. So yeah, having, having customers that are willing to talk to you is kind of a, is a really, really useful thing there. The funny thing about doing open source is that people although you get, you know, if you have a healthy project, you have contributors coming along, contributing things. It's sort of, very hard to, you know, \[00:16:00\] there's a lot of noise in the, in the, the signal in terms of what people really need, you know, because people will show up often with really. obscure things. And, but push really hard to, to get it into the code ****Phil Sturgeon:** Mm, ****Tom Akehurst:** and you don't really know whether, you know, is this a, is this a strong signal that this is something people really need or is this just kind of one person kind of really making a, making a big deal out of it? ****Phil Sturgeon:** I can't believe. ****Tom Akehurst:** you get people kind of giving thumbs up on, on GitHub and so on like that. So I guess it's, you know, there's, there's some degree of signal in those things. But I, I, I certainly, I, I found running the commercial products has been a lot more. Informative in terms of getting you know, understanding what people really need it for than, than running the open source community for years and years was, I guess, ****Phil Sturgeon:** Yeah, that's interesting. Yeah. 'cause I mean, yeah, the number of times I've heard like, oh, I can't believe you haven't got random feature you've never heard of in there. And me and these other three thumbs agree that it should all work. But we all work at the same company and we're the only people that care. It can be quite hard to dedicate your time. ****Tom Akehurst:** risk, isn't it? Is it they they'll go and get all their mates to go and stick a tick on it and you go, oh, this is really popular, isn't it? And then. Yeah, \[00:17:00\] it's, it's it's hard to know really. It's the, you know, the, the, the perils of small numbers. ****Phil Sturgeon:** Yeah, that's funny. Long pause, which I'll hopefully remember to edit out. 'Cause I had a really good question and a really great segue and then I focused on doing a stupid bit instead. Are we talking about here? So with open API with open API it's like that's the main focus for me a lot of the time. And so I'm super excited about how you handle that. 'cause I feel like. ****Tom Akehurst:** I. ****Phil Sturgeon:** Talking about those different personas where you've got kind of the API design part and the testing part that really affects the expectations of what open API is and how it's used. And so. When you are trying to do basic mocking, and we talked about the ability to get kind of validation and, and other behavior and stuff into there, I've seen quite a lot of people requesting the functionality or suggesting the functionality or working on building the functionality into their tools. Where the open API document starts to get all of these. X extensions where they kind of litter the \[00:18:00\] open API with starting off with littering it with like Xen faker to let it know that this should be a randomly generated IP address that seems pretty harmless. But then it starts getting into when these three fields come in, they should be multiplied together and a tax rate should be put on top. Depending on the country and like, and all this really complicated behavior that you can't, you can't really infer just from looking at a standard open API, it means someone's gotta start. Building that somewhere. And either you start wedging all of that into the open APIs extensions, or you start littering the GitHub repo with like random JavaScript files that, that handle some of it. And you kind of reference those somehow. And whatever it is you end up doing, there's the assumption that you can, and, and I'll. Interested in changing the open API and obviously you support GRPC and everything else. My experience is more open api. So I ask like how, how do you handle mocking behavior in wire mock without just literal an open \[00:19:00\] api? Or is that what you do? ****Tom Akehurst:** So so yeah, so we do support a little bit of the, the littering through vendor extensions, but it's, it's not something we've kind of leaned into particularly heavily. I mean, yeah, this is a, this is genuinely a hard problem. This is something I, I, I end up talking about quite a lot. The, you know, open API, you know, in its, you know, unextended form anyway, is, is describing you know, a, a certain aspects of an API. It's not a complete description of an API's you know, constraints or business rules or, you know what I mean? There are all sorts dimensions about a, a way an a p will will operate that that you, you can't capture with, with open a necessarily when you, when you take. The fidelity of the mock is gonna be limited by the you know, what can be expressed in, in open API in the, in the first place. ****Phil Sturgeon:** Hmm. ****Tom Akehurst:** actually this kind of relates back to this use case, this thing in the first place because I think there, there are a few tools out there that, that make it very easy to get from open API to sort of something re, you know, a credible mock of some \[00:20:00\] kind. But for the and like I say, if you're, if you're doing this sort of bringing documentation to life thing, then, then that's probably all you need. And that's great, you know? ****Phil Sturgeon:** You just wanna have a little for, for that? Bringing documentation to life. I feel like the use case is just, we've got documentation, which is a list of endpoints with some sample requests, some sample responses, and if you click the try it button, then you can see the sample response in a different format. And you could even potentially copy this URL or copy this curl command and do it in your terminal so that instead of looking at it through the docs, you're looking at it in your favorite. Editor, but that's about as far as it goes. Right? That's just a like, poke, poke this and get something back. And that's, that's it. ****Tom Akehurst:** exactly. Exactly. And the something you get back is, is, is, is is very limited, you know? So yeah, know there are all sorts of fa you know, fairly trivial things that you can't really express very well. It's like, yeah. Lots of APIs will say, you know, if you give one, one combination of request parameters, then you'll get a, a response of one shake back. Whereas if you, if you give a different combination, you'll, you'll get a sort of slightly different response ****Phil Sturgeon:** the. ****Tom Akehurst:** and you, you can't describe the \[00:21:00\] relationship between that pattern of parameters and, and that type of response in. In open API for instance, you know, you can't express, you know, even trivial business rules. I mean, the, the, the, the demo I give actually a lot is, is the the, the train travel API that the, the open API that you ****Phil Sturgeon:** Yeah. ****Tom Akehurst:** I think. There in the so you can, you can load that into wire mock and it will generate a mock API. And then we've got a little demo app that's like a, yeah. Simple train booking app that uses it and it, and it kind of works, but you can see very quickly that, for instance, you can put in. A search date that's in the past and it will happily give you back results, whereas ****Phil Sturgeon:** Right. ****Tom Akehurst:** do that in a real system. So you, you want to be able to kind of express this rule that this date has got to be in the future, ****Phil Sturgeon:** Hmm. ****Tom Akehurst:** but that's not something you can say in open API. so, the, you know, the approach that we take and it's, it's a, a, it's an imperfect one, but it's one that you know, one that works better than, than just doing the sort of vanilla, you know, one way translation is to, to kind of say that you, the Open API and the mock are, are both partial descriptions of the API \[00:22:00\] and they both describe sort of slightly different aspects. And so between them, you know, you get more than either one of them can provide on their own. So we. We maintain both side by side. You can generate in both directions. So you can as I've already said, you can, you can upload an, an open API or you can, you can, you know, incrementally edit one and we'll, we'll generate and amend your stubs for you. You can also go the other way so you can, I mean, and this is probably another, a whole other conversation, but you can prototype by kind of usings as examples and generating the open API from those. ****Phil Sturgeon:** Oh, cool. Yeah. ****Tom Akehurst:** But the, you know, the important thing is a common pattern, and this is you, this is the one I demonstrate sometimes with the train travel app, is to think of the, the o the thing you've, you've built from an open API point as import as a, as a starting point. And then you're kind of layering on, you know, the, the sort of behavioral richness, you know, to the, to the extent that you need for your use case. And you know, it might be that you just want to, you know, just for your. Your API portal users, you want to be able to give them several variations of requests that they can make. You know, you want the you know, the paginated, non \[00:23:00\] paginated versions, you know, maybe you want the yeah. To be able to retrieve an invoice in several different states, you know, with, with you know, different combinations of data rather than just the, the default one, you know, all of these kind of things. So, so you might wanna just do that and layer on a little bit of richness in a way that's tricky to do, purely from open API. Or you might want to go quite a lot further and, and model you know unhappy cases, false stateful, all this kind of thing in the way that you need, if you're doing sort of serious QA activities, and particularly if you're using it in a like I say, the, the, the, what I, what I've tended to find is that the, the sort of broader the test that you're running. So the more the, the more of the system that it, it encompasses. and thus, you know, the sort of , the broader the scope of a, of an individual test case, the more realistic the mocks need to be. ****Phil Sturgeon:** Yeah, I gotcha. ****Tom Akehurst:** you end up being down. Yeah. If you've got a, if you've got say, you know, collection five microservices that are collaborating, and you kind of built an outer shell of mocks around that so that you are, you know, you, you don't have to spin up your, your other 500 microservices just to make it work.\[00:24:00\] ****Phil Sturgeon:** Yeah. Yeah. Gotta wait for Kubernetes to load just so I can test this one endpoint. ****Tom Akehurst:** And you, I mean, it's amazing the extent to which you see organizations trying to do that. ****Phil Sturgeon:** yeah. Being ****Tom Akehurst:** do you know, coming up with ****Phil Sturgeon:** there, ****Tom Akehurst:** really like co complex kind of routing systems that, that let you combine. I ****Phil Sturgeon:** I. ****Tom Akehurst:** these are sort of remote systems that have, you know, combined local and and remote services for testing. And I'm sure some people make it work very well, but it strikes me as being you know, being a sort of maddening endeavor. But anyway, , the point I was trying to make is that if you're, if you're doing that, then. You, you know, you might sort of be testing a single operation from sort of an end user's point of view or a caller of one API that actually ****Phil Sturgeon:** Hmm. ****Tom Akehurst:** out into, you know, this sort of multi-step kind of stateful interaction. And generally, if you want to, if you want to be able to do that kind of test then. You know, the mocks need to be you know, they often need to be stateful, or at least they need to be sort of pseudo stateful and, and model that behavior a bit more realistically, um, than the, the sort of sort of stateless, canned response you can get away with if you're doing like a narrow unit test. ****Phil Sturgeon:** \[00:25:00\] Talking about that kind of behavioral functionality for the different personas, I think might kind of break it down for folks that haven't thought about this too much. So if you are working on that kind of API design or kind of documentation documentation like Dev X. Side of things, then it might be really helpful for Yeah, it, it's helpful for you to have some behavior in the mock where basically if you think of the, like stripe example where they used to say, if you post post a payment with a credit card of like 4, 4, 4, 4, 4, then it will succeed. If it's like 4, 2, 4, 2, 4, 2, then it'll fail, whatever that was. You can, you can get different responses, come back. From, from the request. And that can be just a little bit of looking at that and, and doing a rule to give you a response. Now, it doesn't necessarily have to kind of persist that data to a point where you are making some other requests to a list of previous payment attempts, and that one will show up. That would also be useful, but in the kind of API \[00:26:00\] design. Use case that's just literally, we've got, we, we are trying to build the API client and the API itself in tandem. And we would like the kind of iOS developer, web developer to verify that this will work at all. They can make requests in the right sort of datas in the right place. It might be the wrong person's bloody name when they get the response, but like, fundamentally, does it at all kick in the tires help at all? And once they say yes, you can then quickly rush off and write some of that. Code and then kind of build the prototype and they can get that back and go, oh, that works lovely. Thank you very much. And you can go and tidy it up and turn it up and use the open API as the contract test to verify that you haven't changed it since then. So that's, that's really helpful for mocking in the kind of API design world, but it absolutely useless for most sort of qa. Functionality where you're like you know, do, does a webhook fire off if I send a failed? You know, does, does the failed webhook send off if I send a this credit card or if something else go off, if I do that one where you need it to be more realistic and have more behavior in \[00:27:00\] there, and maybe talking about webhooks as a step too far, but like, you know what I mean? ****Tom Akehurst:** Yeah. No, we do WebBook as well. They're very popular feature. Yeah, I think more people use them than, than admit to it, it seems. ****Phil Sturgeon:** But could you, could you mock a open API and have it sending off like mock webhooks in certain situations? ****Tom Akehurst:** so actually we don, we, the converter doesn't convert to our web hooks automatically. I'm ashamed to say at the moment. I mean, we, it is one of those things that's on, been on the roadmap for a while, but it's, you know, it's just been less important than other things. So we haven't done it. There's no ****Phil Sturgeon:** Mm-hmm. ****Tom Akehurst:** I mean, , the web hook definitions you can define in WireMock are not identical syntactically, they're capable of accommodating what you can describe in in open API. So ****Phil Sturgeon:** That's cool. No, what whenever anyone mentions anything, I'm like, oh, what about this? Like you, I, I will always have an idea on top of what it is you currently do. I'll be like, but that's great. But what about that? Oh, sorry. We haven't done that yet. No worries. I think. ****Tom Akehurst:** Just to, pull on a thread. You started a moment ago. The prototyping these really is, is really interesting and, and actually may, it's may be a slightly sort of controversial type, but I \[00:28:00\] dunno, maybe one you'd be sympathetic towards. I so we, we, we sort of promote the idea of using mox as prototypes. ****Phil Sturgeon:** Hmm. ****Tom Akehurst:** I, I think, again, depending on which of these two personas you're in, you know, that it, it's a kind of similar activity, but sort of with different. Different terminology and , different emphasis. So some of our first customers actually, the reason they, they, they they bought the product was because they had, you know, backend teams building APIs and then they had a bunch of different front end teams like, you know, mobile and web and so on like that. And they were frustrated at having to, you know, work in this very serialized way because, ****Phil Sturgeon:** Hmm. ****Tom Akehurst:** know, they would. Come up with, yeah, a new feature would need a new, new bit of API and they would sit and whiteboard it and figure out what they, they thought it should do. ****Phil Sturgeon:** Six months later. ****Tom Akehurst:** would go off and build something and ****Phil Sturgeon:** Yeah. ****Tom Akehurst:** a test environment a few weeks later. And only then could the, the, the teams calling it really kind of get going. Building stuff, and then there'd be this sort of slightly long, painful feedback loop where, oh, actually we forgot all of this stuff in the design session. And you know, now the backend team have to go and remediate it. And \[00:29:00\] the, know, use, using Mark kind of solves two problems there. You know, one, one, it means that you can, you can very quickly have something that the front end teams can start building against. So they're not waiting for, for the backend teams to, To ship something, all but by, by doing that, it gives, it gives the opportunity for, for early feedback ****Phil Sturgeon:** Yeah. ****Tom Akehurst:** It means that they, you know, by Yeah, because this is the thing you, you don't really know. Until you've actually built something against an API, whether it, whether, I mean, I know, I know there's lots of other, you know, there, there's governance tooling and there are lots of sort of heuristics you can apply to APIs to, to, to say whether they're good or not. But it's like, you know, sort of a, you know, like a, an ounce of practice is worth a ton of theory in, in, in this case. You know, you try and if you try and build something against an API, then, then you'll really know. it's fit for purpose, if this design is any good. And so the earlier you can do that, and, and, and also, and, and this is the crucial point I was gonna make. The, the, the, the more sort of breadth of behavior of that API you can cover in that exercise. The, the quicker you are gonna converge on \[00:30:00\] an API design, which is which is really suitable and, ****Phil Sturgeon:** Hmm. ****Tom Akehurst:** You know, the, the, I I think what, you know, one, one of the, the, the downsides of using a the tools that sort of do, do you like a one shot open API to mock generation is that you, you lose the ability to go and exercise all those kind of secondary parts. So I'd say particularly if you have a a very kinda state heavy API, you know, one where the response you get back, you know, the, the, the structure of the payload for instance, varies quite a lot depending on the state of the resource. That you've retrieved then yeah. That's the sort of thing that, that doesn't, you know, it, it doesn't really, it kind of gets lost in all of that. You know, you get the one representative response back, which is probably just gonna be like one state outta the set, you know? And, and if you're using one of those tools and if you can actually, yeah. If you have a, a tooling that lets you, you model that in a slightly more sophisticated way. So that you can get through all of that and you can cr you know, you can raise the level of realism, of the experience that the consumer gets, then you're, you're, you're much more likely to, to produce a a better design more quickly that you know, that \[00:31:00\] you can then ship something with, rather than having to fi Yeah. Again, getting into that kind of, you know, shipping stuff into a production like environment and testing it, and only then realizing that, that actually, you know, when, when we, when we get into the real staple version of this a p, it's not what we. ****Phil Sturgeon:** Yeah, absolutely. I feel like the, the, the key thing is to make sure that it's not. Somebody asked for functionality, they get told to go and look at their shoes for six months until we finished building it all. We had to rush because people were complaining at us. So we built some rubbish and we told ourselves it would be a prototype, but actually we've ended up just shipping that live and it doesn't really do what they said. 'cause the whole six month wait with no feedback. And that's just a mess like that. That's basically like default in API. That's what everyone does. That's just what they do. And it's why I've spent so much time being like API design first is good, actually you should try this. 'Cause it sounds like just being the nerd that was like, you have the decade that everyone was running around, like the religion of TDD or the religion of anything else, right. Of just, Hey, we've got this wacky idea that doesn't \[00:32:00\] really make a lot of sense and it massively slows you down. Or like yeah, like domain driven design for everything all the time and just. It. It kind of sounds like the sort of thing that similar, those similar types of people would say, but it genuinely saves everyone a whole crap load of time if you just actually sit down, talk about what you think you might want, and then make a mock for it. And that's step one is good for everyone. Whatever mocking tool you use, probably you can do that step one where you're like, Hey, here's, here's a thing. The Prism classic example was you using the pet dog 'cause it was pre-trained travel API, and it's burned into my brain. But the example of you post a cat and you get a dog as a response, right? Tech technically fine, technically valid. Like you, you send it a request and you get something back and you can do the most basic. I have built an interface that when I send this off, I will then get a response and I can build something. But if you want to get a a, a cat back instead of a dog. You have to start adding in these weird little switches. And I think we did something like \[00:33:00\] using the prefer header where you kind of add a HTP you start messing with what's real and what's the, what's the thing? ****Tom Akehurst:** that's, that's always been one of my kind of cardinal design principles actually is ne never do that. You know, the, ****Phil Sturgeon:** Yeah, yeah. ****Tom Akehurst:** caller should never have to change its behavior in order to to interact with a mock versus a real thing. ****Phil Sturgeon:** Yeah. ****Tom Akehurst:** you know, the, you, you should always try and try and find ways of using the. Yeah, the signals that are sort of intrinsic to the, the call that's being made to you know, to, to affect behavior. Even if that's kind of harder to do in, ****Phil Sturgeon:** For sure. Yeah. 'cause I think all, all of our like all of prism's validation responses, if you send a request and it's invalid, you would get like RFC 7, 8, 0 7 or whatever. There's a new one now. But you would get that kind of standard error response. And then under the hood, I think that was, I, I did suggest that to all, and I was thinking, well, with my mole on of, of course, everyone's using the standard for error responses or. And, and e even if I didn't truly believe that, I'm like, but they should. So, you know, fuck 'em, it's a \[00:34:00\] problem. But I had, we had to do something of just like, there has been an issue, the response, the validation response wasn't defined. Let's take a guess at this. And then they actually go ahead and implement the error format. And it's just like. A simple string or a text or HDML or something. And so again, you, you kind of end up blurring the lines and all of the line blurring just always felt weird, but it's, it's hard to get functionality out of a mock. And so I think we, we skipped from basic dumb mocking with that little bit of faker to come up with some, some, well, whatever's an email address shut up. We skipped straight from mocking through the proxy. We had functionality that was. You could proxy to an actual implementation, which would if you returned a 5 0 1 not implemented, would return the mock. Whereas if you returned anything else, it would return the real one. And so that kind of blurred the line between mock and prototype and beta really. 'cause what we were basically saying was. Instead of launching into a \[00:35:00\] prototype where you're trying to get that fast feedback, but actually it will still take a while to make this prototype. And actually you'll probably end up getting attached to it and trying to put it into production instead of that awkward jack of all trades. Not very helpful. Middle of the road prototype first approach, we were saying step one, kick the tires with this kind of shy mock and then you can use that feedback to quickly go and implement endpoint at a time over here. And it can be a slightly better kind of. Build, which you can start to put live piece at a time as and, and so instead of the entire API as waterfall, bits of it were slightly more agile. And they could, they could use the mock for something else. ****Tom Akehurst:** Thing. I mean, it's slightly different approach to it, but you know, with wok you can, you can set up a proxy stub that says, when these criteria are matched, you go and proxy off to this other address ****Phil Sturgeon:** I ****Tom Akehurst:** than ****Phil Sturgeon:** Nice. ****Tom Akehurst:** aub. So typically what you do is you, you have one of those that's very loose, matching as a, as at a low priority, and then you stack a load of specific stubs on top of that for the things that you. And essentially there's two use \[00:36:00\] cases, either like a load of things you haven't built yet, so they only exist in stub form. Or there's because the other, the other funding you do with wok is really low level faults. You know, you can like send reset packets and drop the connection and all that kind of thing. ****Phil Sturgeon:** Oh, cool. ****Tom Akehurst:** and, and you know, based on matching criteria. So, so if you want to use it as a kind of fault injection engine or two. To inject unusual responses that you can't make the real test environment make, then you can just kind of layer those on top. So, so what WMA will do is that, you know, request comes in, it'll run down the list and say, does this request match anything? If yes, oh, okay, this is the go and send a reset packet response. So I'll do that. If not, it runs all the way down the list, gets to the proxy and says, okay, I'll go to the test environment and serve a real test, ****Phil Sturgeon:** That's good. So you can kind of skip needing to set up proxy, proxy or similar as a, as a separate tool and just say like, if, if this request comes from Dave, give him a really bad day and it'll be a really slow response. Or, or like anyone, or you could literally say like, if this if, if this is coming from you could pretend like certain country is getting a \[00:37:00\] very slow response. Does it, does it work? Realistically well with the client if the responses are that slow or do we need to go deploy another server over there? That's really helpful. ****Tom Akehurst:** Yeah. No, it's very. Powerful way of using it. And as I say, whe whether another kind of sub use case of the QA thing, you know, is the kind of non-functional you know, the, the possibility for non-functional testing that mocking opens up. You know, you can you can go and you can run ****Phil Sturgeon:** I. ****Tom Akehurst:** tests and soap tests and you know, things that are, that are about performance and stability, but you can create a set of environmental conditions that you are, are really hard to kind of make real systems create on demand. ****Phil Sturgeon:** Yeah. Yeah. ****Tom Akehurst:** Quite a few years ago I was working at a, a big retailer in, in the uk and, and they had the sort of the, you know, the same kind of Black Friday problems that every, all you all retailers did, you know, where all of their customers would show up on one day and they would, you know, hammer the hell out of their site and all the payment gateways would be running, you know, taking four times as long to respond ****Phil Sturgeon:** Mm, ****Tom Akehurst:** they're all getting hammered as well. And Immensely stressful for ****Phil Sturgeon:** Yeah. ****Tom Akehurst:** involved if you're a techie, but the, the nice thing about obviously Yeah, the, \[00:38:00\] yeah. The problem you have when you are testing in the run up to Black Friday is, it's not currently Black Friday, you ****Phil Sturgeon:** Yeah. ****Tom Akehurst:** your, your systems are, are not under load in the same way. And you know, the, the third party system you depend on aren't under load and they're not ****Phil Sturgeon:** Mm. ****Tom Akehurst:** weirdly and all that kind of thing. You can very easily deceive yourself. And I, I mean, I've lost count at the number of times I've seen this happen. You know, where organizations have done what they think is really extensive performance testing in ideal environmental conditions. And then, and then the big day comes up when, when everything's going to hell and. Something that they didn't test for or predict happens because, you know, because some crucial aspects of the ****Phil Sturgeon:** Yeah. ****Tom Akehurst:** The great thing about, about all this about using mocking tools that have these kind of non-functional features is that you can deliberately degrade aspects in a very controlled way of the, the environment and ****Phil Sturgeon:** Mm. ****Tom Akehurst:** out ahead of time what's gonna happen. ****Phil Sturgeon:** That's brilliant. Yeah, I mean, that was for my, my two years at WeWork of just dealing with the most insane problems of systems architecture. Where there was a favorite example of all is \[00:39:00\] that one where there was a third party, API, that everyone forgot about that handled like. Adding new key cards for new members of the coworking space. And that was handled by some random company who had a SLA promising it would be always less than a second, never more to get a response. And we had, I think there was like, I. West coast, east coast and other, which was fine when other was like Australia and London. And then eventually it was like China and Russia and Brazil and every single other major country you can think of all on this one TV little like EC2 instance floating about somewhere. And yeah, every time Australia had a busy day on the first of the month when all their customers turned up to, to check in for their new contract, no one in Los Angeles could go to the bathroom because the key cards wouldn't work. And things like that of just like, well, we've tested all of our APIs. It's like, right, but you did forget to test that other dependency that you had. And so yeah, you could just like save a couple of requests and responses. You don't actually care what the actual thing coming back is. Exactly. You just want it to cut, you know, when it does come back, it's a valid response. That's fine. But \[00:40:00\] you want to test what happens if that takes two minutes to respond. Like it was it was like two minutes. That's mad. That's so slow that we've actually kind of rejected it from our logging system because they, they said it would be less than a second. So two minutes must be the logging system's broken. So you need to be able to, you need to be able to test things like that. ****Tom Akehurst:** I've spent a lot of time doing this kind of performance and resilience engineering, those runtime characteristics , the systems you're building and, and in particular with respect to APIs is is really important. You know, the difference between something taking, you know, half a second to respond versus two seconds in the context of a busy system, ****Phil Sturgeon:** Yeah, ****Tom Akehurst:** is, is absolutely enormous. And and you know, it can mean the difference between stability and instability fundamentally. ****Phil Sturgeon:** for sure. ****Tom Akehurst:** There are techniques like sort of chaos engineering that you can sort of bring, bring to bear in order to try and make sense of these things. But I, I think, suppose with that in particular, no, no criticism of it at all. I think there are probably you know, if you're trying to build really highly scale, highly available systems, then. Then it's an essential technique, but the, the problem is, it is a, there's quite a big barrier \[00:41:00\] to even, you know I, I think starting doing that kind of thing. so if you can deploy these little tactics where, I mean, I, I often wonder how many, how many organizations have run kale? Experiments, ****Phil Sturgeon:** Yeah. ****Tom Akehurst:** know, these big expensive things where you have whole teams of people involved and you have to have done all of this kind of DevOps maturity work to even get to the point where it's safe to, to do it. And then all you find is like someone's set a time out too high on, on some API ****Phil Sturgeon:** Yeah. Yeah. ****Tom Akehurst:** kind of think, you know, fine, you know, it's good that you found it, but maybe you could have found that in a. Much earlier in the process and much more cheekly and, you know, with much, much less kind of labor involved than, than this way. And, ****Phil Sturgeon:** Yeah. ****Tom Akehurst:** so I, I kind of like to think about how, you know, how you can, how you can kind of surface those things as, as early as possible. You know, it's kind of, it's, it's like with the prototyping thing really, I think you, the, a lot of this is about, about sort of, I mean, I, you know, I say I, I, I admit, I, I was a, a sort of fairly fanatical XP adopter back in the day. I do still have this fairly, sort of strong belief and bias towards the idea that if you \[00:42:00\] can make feedback easy and early, you'll end up with a, a better quality product being produced more quickly. Both of these examples, the prototyping example and the, the fault injection. If you make something that's sort of trivially easy for someone to, to discover the sort of correctness and properties of, and so like that, then they'll do it and they'll, you know, it'll get built into people's routines. Whereas if you make it happen late and, and it's expensive and requires lots of inve investment and upskilling, then it, it probably won't happen. There's a sort of , an example I heard from a friend about this recently where, i, you know, prototyping API prototyping wasn't being used. So he works in one of the large banks, you know, and they have obviously, you know, huge sprawling kind of API integrated systems of, of various generations. And they, they were, there was a, a team he was involved with that were working on an API that sort of fronted, you know, this sort of layer upon layer of, of legacy technology, you know, all the way back to kind of bits of mainframe, I think. And they. They, you know, they've done this. Yeah. The, what I described earlier, this kind of whiteboard exercise of sketching out what this \[00:43:00\] API needed to do in order to build this new product they wanted to launch. And then, you know, they did all of that months and months of work happened. You know, a first version of this, a p was shipped into a test environment and then someone immediately went we're missing that ID field that we need in order to be able to do any of this ****Phil Sturgeon:** Yeah. Yeah. ****Tom Akehurst:** and, and, and this resulted in a three month project ****Phil Sturgeon:** What. ****Tom Akehurst:** five teams or something like that in order to. You know, to be able to go and wire everything up to, to surface this, this one additional I Id field. ****Phil Sturgeon:** Oof. ****Tom Akehurst:** and I, you know, people, people doing like greenfield microservices kind of go, oh, it's easy to change things, it's no problem. But, ****Phil Sturgeon:** Yeah. ****Tom Akehurst:** Yeah, I suppose the other thing I always like to point out is that a lot of us are not in that situation. A lot of it's build APIs as modernizing facades over very hard to change technology. And so the, the, the cost of getting it wrong is enormous. ****Phil Sturgeon:** Yeah, that's rough. Good job from pulling us back from the brink from me, about to launch into a tirade about how at this point in late stage capitalism with the finance bros taking over the world of tech, we are currently actively \[00:44:00\] firing. Every single engineer that can't be replaced with copilot instead of investing in things like a group of 10 people whose whole job it is, is to sit around and fuck things up. On the off chance there's an issue they can discover. Well, that's probably another topic for another day. But on the topic of ai, the last thing I wanted to, the only thing I really wanna mention about that is that I've seen a few companies starting to do like AI mocking and I feel like. How do you feel about that? Is that something you have on your roadmap? I know you have MCP, that's a topic for another podcast and it makes sense for you to support all the protocols that you possibly can. But for AI responses we've talked about, like static and dynamic, we've talked about faker, we've talked about kind of persisting data and trying to wire it back up. We've talked about trying to glue together kind of behavioral stuff in a, in a different mock, in a different format. How do you feel like AI can help that, or do you feel like it's a silly distraction, hype, bandwagon, nonsense thing? ****Tom Akehurst:** So, so I, I think it \[00:45:00\] can actually be helpful. I agree that the, you know, with ai, the, the, the hype often runs, you know, a bit ahead of, of you know, the real, real ****Phil Sturgeon:** I. ****Tom Akehurst:** of it. But but nonetheless, actually, I think this is a, a domain where AI can you know, can be genuinely helpful. So we one, one of the things we see a lot, you know, organizations that, that, that use mocks kind of really extensively. they, you know, traditionally, you know, you know, particularly kind of, obviously the last three years trying to build a, a product, which, which makes lot, lots of aspects of this easier, but sort of prior to that where people, we, we were kind of cobbling together mocking solutions out of open source tools. just a lot of toil involved. You know, that if you're, you know, if you're in a, a microservices environment and you've got 500 or a thousand services in your organization and they're constantly changing and everybody, everybody who calls 'em has built their own set of mocks for everyone else's service. You know, you've got ****Phil Sturgeon:** Hmm. ****Tom Akehurst:** Sort of combinatorial explosion of mocks being created and maintained, and and it's just a lot of toil, you know, to, to maintain them, to make sure that you are, you are testing against mocks that are. \[00:46:00\] Are true to, the real thing that you're gonna call. PE people, , quite rightly don't really wanna do it. They wanna focus on their actual job and they, they want the, the amount of effort they're devoting to that once they've got over the initial sort of novelty of like, Hey, this is this fun, new thing to play with. ****Phil Sturgeon:** Hmm. ****Tom Akehurst:** And it just becomes, becomes a, a, you know, another maintenance headache. So I, I think it can really genuinely reduce you know, reduce that toil. So I'll give you a few specific examples of things that we're, we're working on and that we're experimenting with at the moment. So, so one of them is so like I say this, this, this big problem of drift, you know, where real APIs change mocks don't get updated. Tests are being run against you know, out date mocks and, and they become untrustworthy. You get false positives and then failures happening further down the line, which people don't like. So ideally you want to be able to, to automate the, you know, the sort of detection and, and remediation of those mocks so that you know, those, you know, everything stays aligned and, and you can, you know, the, the, your, your level of trust is maintained in your, your tests. So the, you know, one thing a, a AI is good at is, is is is kind \[00:47:00\] of, you know, poking around and experimenting with things. And and I I, I find it particularly good when you pair it with, sort of non-AI techniques for, for, you know, validating things, for instance. ****Phil Sturgeon:** Okay. ****Tom Akehurst:** AI plus Open API actually is a, is a, a really, a really powerful combination. So we, we have a, we also have a, a validator built into the product. So what you can do is, can, you can via our MCP tool, you can say, say to an ai, go and pull the latest open API for this, you know, this API that we're mocking. ****Phil Sturgeon:** Hmm. ****Tom Akehurst:** and now I want you to, you know, so, so import it into yox so that we, we've got the latest, the latest open API now far off a bunch of test requests to, to everything in there. And if there are any validation errors, go and update the stubs in a way that that corrects for them, and then give me a report on what you've changed. ****Phil Sturgeon:** Okay. Yeah, so kinda using it for fuzz testing on somebody else's API based on their, their open API definition You fuzz, test the heck out of that and then see what's, what's broke. Yeah. ****Tom Akehurst:** well, it's, it's kind of, it's kind of this closed feedback loop. I mean, obviously the, you know, \[00:48:00\] AI's hallucinate a lot and if you just say to them, here's a, here's a new open a p, I go, go and fix my stuff. Sometimes it'll get it right and sometimes it'll get it massively wrong. So ****Phil Sturgeon:** Yeah. ****Tom Akehurst:** losing a lot of the, the productivity benefit ****Phil Sturgeon:** you are asking it to hallucinate though, in that instance you're saying like, Hey, can you go and hallucinate a bunch of requests that may or may not match at the API fire those all off. That's literally what QA is. That's like creatively coming up with sometimes right, sometimes wrong, sometimes absolute bullshit requests. And if you're getting the AI to be creatively stupid, that is is a perfect use case for it, in my opinion. ****Tom Akehurst:** Yeah. Well this, I mean, the other, the other thing I'm playing around with is this idea of an API crawler. I mean it, yeah, it pretty much exactly what you described at the moment. When, I mean maybe it's QA people or devs or anybody that who needs to interact with an API. There's this process they often end up following, which is sort of. They'll get a little bit of information about it. You know, it might be, you know, just some kind of, some folk knowledge from somebody else in the organization. It might be you know, a really sketchy out of date swagger document. It might be you know, sort \[00:49:00\] of a half-baked postman collection or a ****Phil Sturgeon:** Mm. ****Tom Akehurst:** doc or whatever, you know, do you know what I mean? You've got some, ****Phil Sturgeon:** Yeah. ****Tom Akehurst:** so something to start with, but, but it doesn't really tell you the full story. So you fire up an HDDP client and you. You get yourself an API key and you go in there and you start kind of poking around and making requests and ****Phil Sturgeon:** Nice. ****Tom Akehurst:** you know, via trial and error, you build up much more full picture of how that API works until you can do whatever it is you need to do with it. You know, whether it's testing it or whether it's an integrating something with it as a developer, and yeah, that, that seems to me to be a very, very automatable process as well. You ****Phil Sturgeon:** Hmm. ****Tom Akehurst:** and, and, and this is what I'm playing around with saying, Hey, here's a, here's a doc with it. You know, it's got a bunch of like URLs in it that you can go and call and I'll give you an an API key. Try and work your way through this. API follow any links you find, you know, try posting payloads that sort of look similar to the ones you've read back. ****Phil Sturgeon:** Yeah. Yeah. ****Tom Akehurst:** picture of what's, you know, what operations are available, how you post them, and so on. And I. We've sort of rigged that up, you know, essentially uses our prototyping mechanism. So, so you can, as it's doing it, it will feed things into a recording that then become a, \[00:50:00\] a, a, a mock from, you know, that's generated from that exploratory session that you've asked the AI to do. And then that in turn produces an open API, so ****Phil Sturgeon:** Okay. I like that. ****Tom Akehurst:** you know, do you know what I mean? It's like, like sort of that, that again, this, this kind of grunt work of of just going around and poking everything. I'm trying to make sense of it ****Phil Sturgeon:** Yeah, for sure. ****Tom Akehurst:** A useful live relevant piece of documentation and knowledge about this API, you know, you're, you're kind of asking the AI to do that, and, but I say because it's ultimately making requests that are, they will either succeed or fail. You know, there's no, it's not like the the AI hallucinating is going to degrade the quality of the output in that instance because the, you know, you're talking to a real API that will, you know, will either accept your request and return you something real, or it will just say, no, that's wrong. I'm not accepting that. ****Phil Sturgeon:** Yeah. Okay. ****Tom Akehurst:** you know, ****Phil Sturgeon:** I. ****Tom Akehurst:** of those are sort of you know, like VA validating information that, that that sort of avoids, avoids things being hallucinated. So I think, I ****Phil Sturgeon:** That's really smart. ****Tom Akehurst:** sort of thing's really useful. ****Phil Sturgeon:** It's a lot more useful than like having a shared mock that you give to all of your users. And then \[00:51:00\] like the main user profile that comes back is like D'S nuts or like Hitler, none of which is very helpful. Anyway, we have gone long and that is my fault for bringing up the topic of AI at minute 47 of the recording. But I would just like to thank you for your time and for having a really interesting chat with me about mocking. Before you go, how can people find you and your awesome stuff that you're working on? ****Tom Akehurst:** So I'm on LinkedIn, Tom Hurst. The company is yoc and you also, there's yoc.org is the, the open source website. And there's like links to Slack community and everything else on there. ****Phil Sturgeon:** Lovely. We'll sprinkle all those in on the footnotes. Once again, thank you very much and cheers for listening in everybody. ### OpenAPI 🤝 OpenMCP URL: https://apisyouwonthate.com/newsletter/openapi-openmcp/ Last updated: 2025-06-16T15:34:17.000Z For the last two months, my feeds have been awash with API and MCP content. The relentless tidal wave became a little boring until something caught my eye. Say hello to [OpenMCP](https://www.open-mcp.org/about?ref=apisyouwonthate.com). First, their example makes converting any OpenAPI documents into an MCP server easy. However, what excites me is the push for best practices and focusing on the challenges that MCP servers can face, such as scaling and authorization. \-- Alexander ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2025/04/State-of-API-DevEx-Report.png) ## Help shape the future of APIs If you build, manage, or use APIs, your insights can help highlight key trends and challenges in the ecosystem. Take the 10 minute survey, and you'll get early access to the report — plus a chance to win a PS5, NuPhy mechanical keyboard, and more! [Take the Survey](https://www.speakeasy.com/state-of-api-devex/2025?utm%5Fsource=newsletter&utm%5Fmedium=email&utm%5Fcampaign=apisyouwonthate) ## The API Round-Up *API News, links, and tools from around the web* ### [HTTP 204 Is the Best DELETE Response](https://apichangelog.substack.com/p/http-204-is-the-best-delete-response) The best `DELETE` response is not 200\. Bruno Pedro talks about why he thinks 204 is the best response type for a `DELETE` endpoint. Bruno also shared some of the feedback he got from his socials on the idea. Funny, my go-to is to send back a 200 with the body containing the resource you deleted. ### [H3](https://h3.dev/?ref=apisyouwonthate.com) H3 came up in my feed this week, and I immediately found myself diving into their docs. It is a server framework built on web standards, focusing on performance and composability. H3 is runtime agnostic, so it will work the same across Deno, Bun, and Node. They recently had a new release with some interesting inclusions. Check out the [release blog](https://h3.dev/blog/v2-beta?ref=apisyouwonthate.com) for more info. ### [Node and PHP Together at Last](https://blog.platformatic.dev/seamlessly-blend-php-with-nodejs?ref=apisyouwonthate.com) A new package called `php-node` allows you to run PHP inside a Node.js environment. Instantly route requests that need PHP processing to a PHP worker, which is then executed and delivered back to the Node.js application. Check out the blog for the full breakdown, as some interesting things are happening to make this work. ### [Building a gRPC Node.js API](https://www.honeybadger.io/blog/building-apis-with-node-js-and-grpc/?ref=apisyouwonthate.com) A fantastic guide on how to build a gRPC API with Node. While also covering the basics of setting up a project and API, Salem explains how gRPC works and some of the tradeoffs. However, I disagree with the statement that REST APIs are less efficient due to HTTP overhead. Once again, I feel I should hit you with the statement that you should choose the right tool for the job, and your REST API is only as good as the level of planning/design you put in. --- ## APIs You Won't Hate The latest from the team at API's You Won't Hate. ### [API Basics: Cache-ability](https://apisyouwonthate.com/blog/api-design-basics-cacheability/) Designing an API with cache-ability in mind produces a more sensible and better separated set of resources, and it just so happens to be more performant, cheaper, and better for the environment. --- ## From our Community Articles written and shared in our free [Slack community](https://slack.apisyouwonthate.com/?ref=apisyouwonthate.com). ### [JSON Schema Tooling](https://json-schema.org/tools?query=&sortBy=name&sortOrder=ascending&groupBy=toolingTypes&licenses=&languages=&drafts=&toolingTypes=&environments=&showObsolete=false&supportsBowtie=false&ref=apisyouwonthate.com) An interesting thread on defining JSON Schema through a DSL popped up in our Slack community last week. Jeremy shared a link to the wonderful JSON Schema Tooling page. I often forget about these lists of resources, which usually save me from building something myself. It's worth remembering they exist. ## 🫣 Support APIs You Won't Hate Become a paid member to get access to members-only content while directly supporting our work. Your support helps us to keep making resources for the API community. [Become a member today](https://apisyouwonthate.com/#/portal) Thanks so much to our members: **Kin L, Juxt, Alex R, Nolan S, Frank, James D, Rich, Ryan T, and Umair**. Your support means the world to us! ✌️ Until next time, Alexander, Phil & Mike ### JSON Schema: Good, Bad and Ugly URL: https://apisyouwonthate.com/newsletter/json-schema-good-bad-and-ugly/ Last updated: 2025-06-02T12:20:37.000Z An interesting thread on BlueSky last week pointed out limitations with OpenAPI, from non-optional parameters to [JSON Schema](https://json-schema.org/understanding-json-schema/reference/schema?ref=apisyouwonthate.com) being a validation spec, not a schema. It's a good read, and gets juicier when Sylvain Wallez jumps into not using OpenAPI for the Elasticsearch API [because of JSON Schema](https://bsky.app/profile/swallez.com/post/3lq674vmg3c2d?ref=apisyouwonthate.com). Instead, they use TypeScript definitions to generate an OpenAPI spec for specific tools. Sylvain points out that this is why Microsoft created [TypeSpec](https://typespec.io/?ref=apisyouwonthate.com) and AWS created [Smithy](https://smithy.io/?ref=apisyouwonthate.com). As we all know, there is never one golden perfect solution in tech; everything is tradeoffs, except JavaScript, which is fantastic and has no flaws. I think there are a few issues here: Optional parameters suggest either a broken approach to API design or that a REST API is not the right fit here. However, the problems with JSON Schema ring a little closer to home, as I have come up against these myself. However, I always push for improving the tools where possible over introducing more competing standards. These conversations are great, though, and help push specs and guidelines forward most of the time. \-- Alexander --- ## The API Round-Up *API News, links, and tools from around the web* ### [The HTTP QUERY Method](https://www.ietf.org/archive/id/draft-ietf-httpbis-safe-method-w-body-11.html?ref=apisyouwonthate.com) A renewed push for an HTTP method called Query, allowing you to include a body in GET requests. Designed to help with complex quries or filtering, moving them out of the URL into the body. It is not a new idea, and there have been versions of it in the past. It will be interesting to see if it makes it into the spec. ### [Which Go router should I use?](https://www.alexedwards.net/blog/which-go-router-should-i-use?ref=apisyouwonthate.com) When building an API in Go, which router do you use? The standard library includes one, but there are many third-party frameworks. Taking some of the most popular third-party options, Alex Edwards walks you through the standard library and shows you when to reach for a third party. ### [The Future of MCP is GraphQL](https://www.apollographql.com/blog/the-future-of-mcp-is-graphql/?ref=apisyouwonthate.com) Everyone is adopting MCP quickly. Apollo has released an MCP server, allowing AI systems like Claud and ChatGPT to interact with GraphQL APIs. Apollo walks you through their setup and explains why GraphQL fits MCP and AI well. ### [API Versioning in Spring ](https://docs.spring.io/spring-framework/reference/7.0-SNAPSHOT/web/webmvc-versioning.html?ref=apisyouwonthate.com) Spring is rolling out support for built-in API versioning, allowing you to map versions to annotated controller methods. WebClient and RestClient also support the version. It's great to see Spring making API versioning easier. ### [Migrating from API keys to OAuth 2.1](https://www.scalekit.com/blog/migrating-from-api-keys-to-oauth-mcp-servers?utm%5Fsource=node-weekly&utm%5Fmedium=developers-guide&utm%5Fcampaign=paid-inclusion) Using API keys for server-to-server calls? OAuth 2.1 has become mandatory for MCP servers. ScaleKit explains how to migrate to OAuth and how it can make this process easier. ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2025/04/State-of-API-DevEx-Report.png) ## Help shape the future of APIs If you build, manage, or use APIs, your insights can help highlight key trends and challenges in the ecosystem. Take the 10 minute survey, and you'll get early access to the report — plus a chance to win a PS5, NuPhy mechanical keyboard, and more! [Take the Survey](https://www.speakeasy.com/state-of-api-devex/2025?utm%5Fsource=newsletter&utm%5Fmedium=email&utm%5Fcampaign=apisyouwonthate) ## APIs You Won't Hate The latest from the team at API's You Won't Hate. ### [API Design Reviews Don't Have to be Hard](https://apisyouwonthate.com/blog/api-design-reviews-dont-have-to-be-hard/) Do you love staring at YAML files? No? Phil shows you can handle API design reviews in PRs using Bump.sh instead of comparing large diffs in YAML files. ## Support APIs You Won't Hate Paid members get access to exclusive content while directly supporting our work. Your support helps us to keep making resources for the API community. [Become a member today](https://apisyouwonthate.com/#/portal) Thanks so much to our members: **Kin L, Juxt, Alex R, Nolan S, Frank, James D, Bill D, Rich, Ryan T, and Umair**. Your support means the world to us! Until next time, Alexander, Phil & Mike ### API Design Reviews Don't Have to be Hard URL: https://apisyouwonthate.com/blog/api-design-reviews-dont-have-to-be-hard/ Last updated: 2025-05-23T23:57:19.000Z An API Design Review is one of a few parts of a larger API Governance program, where various people review changes to an API to make sure its a good choice for the architecture and wider ecosystem. This often involves a wide variety of persona, including API designers, API developers, technical writers, system architects, and maybe even a dedicated [governance team](https://docs.bump.sh/guides/openapi/leveraging-the-openapi-specification-for-api-governance/?ref=apisyouwonthate.com). Design reviews are generally done for an API overall when its first form is created, then with following iterations as new endpoints and other changes are made. Beyond the basic tire-kicking, a design review is important to ensure the API is well-designed, compliant with standards and style guides, meets the needs of its consumers, and does not introduce problems to the wider ecosystem. ## What happens during a design review - **Consistency and Standards Compliance** \- Ensure the API follows established design standards and conventions, covering naming conventions, Hypermedia formats, authentication strategies, etc. - **Clarity** \- Verify that the API endpoints, parameters, and responses are clearly documented. Ensure descriptions are meaningful and useful for developers of both the API and its consumers. - **Functionality and Usability** \- Make sure the API endpoints provide the necessary functionality, evaluating the ease of use for developers integrating with the API. - **Performance and Scalability** \- Evaluate the API design for potential performance bottlenecks. Are consumers going to need to load 1,000 endpoints to solve the main use-case, or is data being computed on load which might slow things down? - **Backward Compatibility** \- Identify any breaking changes that could affect existing consumers of the API. Ensure that changes are backward compatible or properly versioned. - **Forwards Compatibility** \- Is this change going to work properly and fit in with future changes that are either planned to happen, or likely to happen within the wider ecosystem, to avoid conflicts or duplication. - **Security and Privacy** \- Check that authentication and authorization mechanisms are present and appropriate for the task at hand, and personal/private information is not being leaked by URL fiddling. - **Error Handling and Responses** \- Ensure that error responses are sensible, following the right standards, and are both meaningful and insightful for developers and potentially any end-users who end up seeing the error messages directly. - **Impact on Documentation** \- Determine how changes will affect existing documentation (reference docs, guiders, tutorials, etc). Ensure that documentation is updated to reflect any changes in the API before the change is made to avoid anything getting out of sync. It seems like a whole lot to think about but that is why design reviews pull in multiple people from various departments or walks of life. Getting all these aspects nailed down during a design review means you can make sure the API is useful, well-documented, consistent and standards compliant, hopefully not riddled with security holes, and generally reduce the chances of problematic API changes getting into production. ## Where do design reviews happen? There are countless ways people try to handle this. One of the most complicated came from a famous API thought leader who suggested folks email Word documents full of feedback around with spreadsheets to track them... A far more common approach is to put the review process through the same Pull Request workflow that developers are already using to review code changes on GitHub, GitLab, Azure Devops, etc. To know the best way forward it's helpful to know the history of the pull request because it hasn't been around all that long. For decades developers would email around a diff file (short for difference) called `new-function.diff` and people would chat about that technical change on their IRC channels or around the water cooler. It was a massive faff and multiple diffs could conflict, so when Git gained popularity and added the [git-request-pull](https://git-scm.com/docs/git-request-pull?ref=apisyouwonthate.com) command, that eventually ended up in GitHub as a Pull Request and the whole code review process matured. ### Reviews done right with Pull Requests As the pull request matured it became the home for everything, allowing everyone can see a summary of changes, view source code if they want, see previews, comment on specific lines, suggest changes, and people could checkout the changes if they felt like playing around. All of these abilities gave far more people the option to interact with changes in their own preferred way, to see what is changing, and if it matters to them. This is exactly how code changes should be managed, but API design reviews are not code reviews. The design review is more interested in the interface being built by the API than the source code on the inside, and forcing people to read code to work out what that might mean in terms of HTTP and JSON is not helpful. Only the developers should be worrying about the code, the system architects, governance teams, and technical writers want to have a higher level view instead of having to learn every single teams' favourite web application frameworks conventions and syntax. ### The role of API specifications This is where API descriptions like OpenAPI and AsyncAPI come in. When stored together with the source code (and used for this like contract testing to ensure the code and description match perfectly) it means these discussions can be had around the YAML/JSON that describes the API instead of getting everyone pouring over the source code. A huge step in the right direction, but reviewing hundreds of lines of YAML changes is not easy or fun. Some tooling has appeared trying to move the whole API design review process into a rough approximation of Git that's all trapped in their walled garden and requires constant syncing. [Bump.sh](https://bump.sh/?ref=apisyouwonthate.com) instead focuses on allowing design reviews happen in the pull request, and we'll see how to handle that below. ## Diffs Aren't Design Reviews Technical writers often report difficulty staring directly into a pull request that's making loads of changes to massive OpenAPI documents. A single pull request might involve anything from reordering some parameters, complete reorganization of the file structure of infinite `$ref`'ed YAML, to a breaking change that could the vast majority of integrations API consumers. It can all be lost in a sea of red and green changes in the "diff" view. Technical writers shouldn’t need to learn YAML and stare into the void like this, nor should they need to use complex CLI tools or anything else just to see what has changed in an API. Nobody should need to do that. Bump.sh helps simplify this process by analyzing OpenAPI documents and identifying meaningful changes so tech writers can focus on relevant updates without getting lost in unnecessary details. ## Why Not All Changes Matter Some changes in OpenAPI documents have no impact on API consumers, such as: - [Splitting larger documents](https://docs.bump.sh/guides/openapi/specification/v3.1/advanced/splitting-documents-with-ref/?ref=apisyouwonthate.com) into smaller pieces with `$ref`. - Renaming referenced files. - Renaming component names (e.g. renaming `components.schemas.Something` which does not change the actual API) - Reformatting YAML for consistency because some documents were an awkward mixture of 2 spaces and 4 spaces. 🤢 These modifications might make the OpenAPI documents cleaner, but do not alter the API’s behavior. The huge wall of changes would look like loads of change is happening, and maybe some changes have been sprinkled into what was otherwise just a formatting change. Here's a change which looks like it's probably not going to break anything, it's just moving an inline definition of a parameter into a reference, and adding a new optional parameter. ![](https://storage.googleapis.com/bump-blog-resources/2025-05-15-api-design-reviews/replaced-with-ref.png) Unfortunately when copying and pasting some of the reusable parameter definitions elsewhere something was changed, and that would be hard to spot without having both definitions open on two different monitors and comparing the two closely. Thankfully Bump.sh can handle that tedious work for you. ![](https://storage.googleapis.com/bump-blog-resources/2025-05-15-api-design-reviews/breaking-change-detected.png) Bump.sh automatically filters out inconsequential YAML/JSON changes and automates all of the following: - **See only relevant changes** – Filtering out noise from YAML/JSON restructuring and internal updates. - **Get automatic pull request comments** – Highlighting meaningful updates directly in pull requests. - **Be alerted to breaking changes** – Clearly identifying changes that impact API consumers. - **Track API history over time** – Keep a clear record of modifications without manually digging into commits. ## Handling API Design Reviews Ok so we've got the theory down, how do you actually do this? ### 1\. Get Bump.sh into Pull Requests First of all lets get Bump.sh running on pull requests. Bump.sh users working with GitHub may already have a `.github/workflows/bump.yml` workflow, but if not add one. Other [continuous integration](https://docs.bump.sh/help/continuous-integration/?ref=apisyouwonthate.com) providers are supported but we'll stick to [GitHub Actions](https://docs.bump.sh/help/continuous-integration/github-actions/?ref=apisyouwonthate.com) for this guide. ``` name: Check & deploy API documentation on: push: branches: - main pull_request: branches: - main permissions: contents: read pull-requests: write jobs: deploy-doc: #optional if: ${{ github.event_name == 'push' }} name: Deploy API documentation on Bump.sh runs-on: ubuntu-latest steps: - name: Checkout uses: actions/checkout@v4 - name: Deploy API documentation uses: bump-sh/github-action@v1 with: doc: token: ${{secrets.BUMP_TOKEN}} file: openapi.yaml api-diff: if: ${{ github.event_name == 'pull_request' }} name: Check API diff on Bump.sh runs-on: ubuntu-latest steps: - name: Checkout uses: actions/checkout@v4 - name: Comment pull request with API diff uses: bump-sh/github-action@v1 with: doc: token: ${{secrets.BUMP_TOKEN}} file: openapi.yaml command: diff env: GITHUB_TOKEN: ${{secrets.GITHUB_TOKEN}} ``` The `deploy-docs` job is doing what it suggests, deploying API documentation when merged to `main`. That's the standard workflow and can be removed if you don't need that right now. The important part here is the `api-diff` job. Once you've updated `` to the ID found in the Bump.sh API settings page, popped in the `token` hopefully via the Secrets interface on GitHub, and pointed the `file: openapi.yaml` to where the OpenAPI or AsyncAPI document lives, Bump.sh will start adding summaries of changes to pull requests as comments. ### 2\. Reviewing Changes Whenever a pull request is made, or changes are pushed to that pull request, Bump.sh will automatically scan the changes being made to the OpenAPI/AsyncAPI documents. Any consequential changes will be highlighted in the summary of changes. - New endpoints or properties. - New query string parameters added. Breaking changes will be flagged so they're hard to miss. - Removing or renaming API endpoints. - Making an optional property required. - Altering response structures. Raising breaking change warnings like this can help system architects and other reviewers decide on a level of panic. It could well be that property has been deprecated for a long time and everyone knows no consumers are actually using it any more, so Panic Level 0, carry on. If the summary flags up something that's not immediately clear, adds a new endpoint that needs more review, etc. then the Preview feature can be used. ### 3\. Preview changes When Bump.sh comments on a pull request with the summary of changes, there is also a little link on there marked Preview and this is pretty magical. ![](https://storage.googleapis.com/bump-blog-resources/2025-05-15-api-design-reviews/preview.png) Instead of just reviewing the YAML changes and hoping the docs look good when you merge, you can go and review the docs to make sure all is good before you merge. Technical writers can use this to make sure descriptions are good enough, or if developers won't do it tech writers can make sure their [overlays are improving descriptions](https://docs.bump.sh/guides/openapi/augmenting-generated-openapi/?ref=apisyouwonthate.com) correctly. System architects can make sure that new endpoints look solid, with appropriate data being accepted and returned to solve the needs for clients. Everyone can all review the parts of the changes they are interested in, and comment back with concerns, or use GitHub Suggestions to make improvements. ### 4\. Approve Changes If everything is fine then reviewers can hit Approve on the pull request. When API code and API descriptions are in the same repository can be a little confusing as it feels like a technical writer is approving a code change, but this can be solved with [Code Owners](https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/customizing-your-repository/about-code-owners?ref=apisyouwonthate.com) in GitHub. By assigning the API descriptions to the appropriate review teams, and the code is assigned to the code, you can make sure both relevant groups have approved the changes to stop problems squeaking by. Create a new file called `CODEOWNERS` in `.github/` or the root of the repository, and use the following syntax. ``` # CODEOWNERS docs/* @org/review-team src/* @org/dev-team ``` This says that changes made to `docs/openapi.yaml` or any of the other referenced bits of description should be approved by the review team, and any code changes will need to be made by the dev team. You can then go into branch settings and make sure that pull reviews are approved by code owners before progressing. You can get as creative as you like with this, and have all stakeholders involved so there system architects and the governance teams all get their say before things progress. Make sure its possible to bypass this in emergencies, like pushing a hotfix, but generally speaking this should help to make sure things always get better and things don't sneak through. ### 5\. Linting to reduce repetition These reviews can end up with a fair amount of repetition, with folks arguing over things like which naming conventions to use or whether there should be full stops at the end of descriptions or not. Everything is far more efficient when rules are written down, and that tedious stuff can be automated away with [API linting](https://docs.bump.sh/guides/bump-sh-tutorials/api-linting-with-vacuum?ref=apisyouwonthate.com) using tools like vacuum so that its already handled before even getting humans involved with the design review. ![Screenshot from GitHub.com, showing the "GitHub Actions / API Lint Results" check result outputting annotations.](https://docs.bump.sh/images/guides/api-linting-with-vacuum/vacuum-annotations.png) With Bump.sh spotting changes and offering previews, vacuum highlighting concerns automatically, API design reviews should be a pretty simple affair. API design reviews don’t have to be a struggle for tech writers. With Bump.sh, you can automate the process of detecting relevant changes, stay on top of breaking updates, and manage API documentation efficiently. All of this can be done without needing to deep dive into YAML or complex IDEs. Let machines do the heavy lifting so you can focus on delivering great documentation, and steer the evolution of your APIs to make them useful and profitable. ## Conclusion API design reviews are a critical part of ensuring that APIs are consistent, usable, and aligned with organizational standards. By leveraging tools like Bump.sh, you can streamline the review process, reduce manual effort, and focus on meaningful changes that impact API consumers. Automating repetitive tasks, integrating reviews into pull requests, and using API linting tools can help teams collaborate effectively and maintain high-quality APIs. With the right processes and tools in place, API design reviews don’t have to be hard—they can be an opportunity to build better APIs and foster cross-functional collaboration. --- *Article originally written for* [*Bump.sh*](https://bump.sh/?ref=apisyouwonthate.com) *who keep me in food, cider, and canal boat parts. If you'd like to literally keep me afloat I'm available for all sorts of* [*API/tech consulting*](https://philsturgeon.com/consulting/?ref=apisyouwonthate.com)*. – Phil* ### Does Every Framework Need Its Own Cloud? URL: https://apisyouwonthate.com/newsletter/does-every-framework-need-a-cloud-hosting-service/ Last updated: 2025-05-19T11:36:43.000Z The team behind FastAPI, the popular high-performance web framework for Python, recently released FastAPI Cloud, a streamlined deployment solution explicitly designed for FastAPI applications. This release follows a familiar pattern across other ecosystems: framework creators eventually offer managed deployment options. I find myself often in two minds when this happens. On the one hand, best practices are baked in, and there is ease of deployment and usually amazing CI/CD integrations. However, many of these services just wrap AWS, adding extra costs you could avoid if you learn to do things yourself. When investors also get involved, initially accelerating development, the frameworks involved sometimes turn in a direction that is not always best for the community. Hopefully, this leads to great things for the FastAPI team and the FastAPI community. \-- Alexander ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2025/04/State-of-API-DevEx-Report.png) ## Help shape the future of APIs If you build, manage, or use APIs, your insights can help highlight key trends and challenges in the ecosystem. Take the 10 minute survey, and you'll get early access to the report — plus a chance to win a PS5, NuPhy mechanical keyboard, and more! [Take the Survey](https://www.speakeasy.com/state-of-api-devex/2025?utm%5Fsource=newsletter&utm%5Fmedium=email&utm%5Fcampaign=apisyouwonthate) ## The API Round-Up *API News, links, and tools from around the web* ### [Documenting Multiple API Versions](https://bump.sh/blog/multiple-api-versions-documentation?utm%5Fsource=bluesky&utm%5Fmedium=social&utm%5Fcampaign=multiple%5Fversions%5Fguide&utm%5Fcontent=bump) Phil is writing more delicious API content over at Bump.sh again this time covering managing multiple API versions. Maintaining old API versions when releasing new ones can be a hassle; thankfully, Bump.sh makes this a breeze. ### [FastAPI Cloud](https://fastapicloud.com/blog/fastapi-cloud-by-the-same-team-behind-fastapi?ref=apisyouwonthate.com) If you want to deploy your FastAPI in a second, look no further than FastAPI Cloud. It's an opinionated way to build and deploy web applications, but it ensures your APIs will be production-ready when using FastAPI. There is currently a waitlist, but check out the release blog for more info. ### [Build a REST API with Java and Quarkus](https://myfear.substack.com/p/quarkus-rest-api-openshift-postgresql-guide) Looking to build a modern, cloud-native Java application? This hands-on guide walks you through creating a RESTful API using Quarkus, integrating Postgresql for data persistence, and deploying seamlessly on OpenShift. Quarkus has caught my attention recently as a modern version of Spring with less hidden magic that I usually encounter when building a Spring Boot Application. ### [TypeSpec 1.0 GA: API First](https://typespec.io/blog/typespec-1-0-GA-release/?ref=apisyouwonthate.com) TypeSpec 1.0 is now generally available. If you have not encountered TypeSpec before, it allows you to generate OpenAPI specs and more by designing your data and types up front. I have mixed feelings about TypeSpec, as it feels like extra steps, but I love tools that encourage planning. ### [Routing & Middleware in GO](https://www.alexedwards.net/blog/organize-your-go-middleware-without-dependencies?ref=apisyouwonthate.com) Middleware is a typical pattern for building APIs in many frameworks and languages. In GO, people often turned to different third-party packages to manage these; however, this doesn't need to be the first option. Go introduced pattern matching in version 1.22, making building your own routing logic much easier. ### [Node HTTP Performance ](https://github.com/nodejs/node/pull/58288?ref=apisyouwonthate.com) Node has been growing rapidly over the last year. Now, the amazing Yagiz Nizipli is turning his attention to increasing HTTP performance. Check out the linked GitHub issue for more details. --- ## APIs You Won't Hate The latest from the team at API's You Won't Hate. ### [Openapi.tools](https://openapi.tools/?ref=apisyouwonthate.com) OpenAPI.tools is a community-driven list of high-quality, modern tools for OpenAPI, managed by the APIs You Won't Hate team. Submissions to the list are welcome via github as well at [apisyouwonthate/openapi.tools](http://github.com/apisyouwonthate/openapi.tools/?ref=apisyouwonthate.com). --- ## From our Community Articles written and shared in our free [Slack community](https://slack.apisyouwonthate.com/?ref=apisyouwonthate.com). ### [Speedrun API Governance](https://netapinotes.com/speedrun-api-governance-a-ttrpg-thought-experiment/?ref=apisyouwonthate.com) Have you ever wished API Governance were a tabletop board game? Well, look no further. Matthew Reinbold has put some real creative juices into this one, designing a fantastic game that lets you roleplay your way through five chaotic rounds of corporate transformation. ## Support the community When you become an APIs You Won't Hate member, you'll get access to members-only content while directly supporting our work. Your support helps us to keep making resources for the API community. [Become a member](https://apisyouwonthate.com/#/portal) Thanks so much to our members: **Kin L, Juxt, Alex R, Nolan S, Frank, James D, Bill D, Rich, and Umair**. Your support means the world to us! Until next time, Alexander, Phil & Mike ### Evolving REST is a Pain URL: https://apisyouwonthate.com/newsletter/evolving-rest-is-a-pain/ Last updated: 2025-05-01T14:17:17.000Z Dan Abramov's [JSX Over The Wire](https://overreacted.io/jsx-over-the-wire/?ref=apisyouwonthate.com) post has gained traction lately. While I respect Dan and his solution addresses specific use cases, I disagree with claims like `Evolving REST endpoints is a pain` and `REST Resources lack firm grounding in reality`. These arguments advocating for JSX over wire and criticising REST-view alignment indicate you might not need a REST API or any API. It sounds like Dan misses the old AJAX endpoints we all used to build before *misusing* APIs became popular. In software, everything involves trade-offs. My team built a REST API because we support our frontend and other teams' services/reports. API Design-first principles would resolve most of Dan's issues, as would choosing appropriate tools. Alternative approaches like database-over-the-wire or local-first would be a better solution here. No shade to Dan, but REST isn't inherently problematic - the real issues stem from inadequate planning or mismatched tool selection. \-- Alexander ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2025/04/State-of-API-DevEx-Report.png) ## Help shape the future of APIs If you build, manage, or use APIs, your insights can help highlight key trends and challenges in the ecosystem. Take the 10 minute survey, and you'll get early access to the report — plus a chance to win a PS5, NuPhy mechanical keyboard, and more! [Take the Survey](https://www.speakeasy.com/state-of-api-devex/2025?utm%5Fsource=newsletter&utm%5Fmedium=email&utm%5Fcampaign=apisyouwonthate) ## The API Round-Up *API News, links, and tools from around the web* ### [Hurl - Run and Test HTTP Requests](https://hurl.dev/?ref=apisyouwonthate.com) Hurl fell across my desk last week, and, dam, is it fun. Hurl is a command-line tool that turns complex HTTP requests into plain text you can actually read! Perfect for REST, SOAP, GraphQL, or any API that speaks JSON or XML. I had a lot of fun with this one. ### [Koa 3.0](https://koajs.com/?ref=apisyouwonthate.com) Koa dropped a brand new version. Koa appeared in the Node ecosystem as an alternative to Express, focusing on modern JavaScript alternatives. With Express returning from the dead with a major release, it's great to see Koa is still going. ### [Assets Author API](https://developer.adobe.com/experience-cloud/experience-manager-apis/api/stable/assets/author/?ref=apisyouwonthate.com) Adobe just dropped an OpenAPI spec for their AEM Assets Author. What is AEM? I honestly did not know either, but "Adobe Experience Manager" is a content management system for websites, web, and mobile apps. Great to see effort like this put into documentation. ### [How to write error messages](https://piccalil.li/blog/how-to-write-error-messages-that-actually-help-users-rather-than-frustrate-them/?ref=apisyouwonthate.com) One of my biggest frustrations is API errors that return terrible error messages. While this blog post is not API specific, its guide on writing content for errors is on point. Want more API error-specific guides? Check out [Creating Good API Errors in REST, Graphql, and grp](https://apisyouwonthate.com/blog/useful-api-errors-for-rest-graphql-and-grpc/)c and [Why Show Users Garbage API Errors](https://apisyouwonthate.com/blog/why-show-users-garbage-api-errors/), which our very own Phil Sturgeon put together. --- ## APIs You Won't Hate The latest from the team at API's You Won't Hate. ### [API Design Basics: Cacheability](https://apisyouwonthate.com/blog/api-design-basics-cacheability/) Stop thinking of caching as something to come and add in later for performance. Designing an API with cachability in mind produces a more sensible and better separated set of resources, and it just so happens to be more performant, cheaper, and better for the environment. ### [HTTP Caching APIs with Laravel and Vapor](https://apisyouwonthate.com/blog/http-caching-for-laravel-vapor/) Stop wasting server(less) resources answering the same questions repeatedly by enabling CloudFront for your Laravel REST/HTTP API. When paired with the article above, your better-designed APIs will now cost a lot less to run, and your clients will be much happier. --- ## From our Community Articles written and shared in our free [Slack community](https://slack.apisyouwonthate.com/?ref=apisyouwonthate.com). ### [Green Software Practitioner](https://learn.greensoftware.foundation/?ref=apisyouwonthate.com) Have you taken the Green Software foundations practitioner course? Phil is feeling super generous. For everyone who completes the course, he will send over a certificate for five trees planted by [Protect Earth](https://protect.earth/?ref=apisyouwonthate.com). Reach out to him in our [green-tech channel](https://apisyouwonthate.slack.com/archives/C08Q2N2GZED/p1745679423338799?ref=apisyouwonthate.com). ## Support APIs You Won't Hate When you become an member, you'll get access to members-only content while directly supporting our work. Your support helps us to keep making resources for the API community. [Become a member today](https://apisyouwonthate.com/#/portal) Thanks so much to our members: **Kin L, Juxt, Alex R, Nolan S, Frank, James D, Bill D, Rich, and Umair**. Your support means the world to us! Until next time, Alexander, Phil & Mike ### HTTP Caching APIs with Laravel and Vapor URL: https://apisyouwonthate.com/blog/http-caching-for-laravel-vapor/ Last updated: 2025-04-27T18:57:14.000Z [Laravel Vapor](https://vapor.laravel.com/?ref=apisyouwonthate.com) is a serverless hosting solution tailored to [Laravel PHP](https://laravel.com/?ref=apisyouwonthate.com) applications, which sits on top of Amazon Web Service. Severless can be a great solution for HTTP/REST APIs which do not want to waste server resources when nobody is asking any questions, but when people are asking the same questions the serverless approach is still wasting resources rewarming the Lambda, flailing around in a mess of PHP and SQL calls serving up those answers over and over again for no reason. HTTP network caching is the solution to this problem, and it's simply a case of turning it on. There are many solutions for HTTP network caching, both self-hosted solutions like [Varnish](https://www.varnish-software.com/products/varnish-cache/?ref=apisyouwonthate.com) and [Squid](https://www.squid-cache.org/?ref=apisyouwonthate.com), or SaaS solutions like [Fastly](https://www.fastly.com/?ref=apisyouwonthate.com) and [Cloudflare Cache](https://developers.cloudflare.com/cache/get-started/?ref=apisyouwonthate.com). We don't need to pull in any of these solutions because Laravel Vapor already utilizes [AWS API Gateway](https://aws.amazon.com/api-gateway/?ref=apisyouwonthate.com), which like most API gateways has it's own network caching solutions built in. Despite Laravel Vapor making most everything incredibly easy, nowhere in the web UI or YAML-based config is there an option to turn HTTP network on, so we're going to have to roll our sleeves up and do it ourselves. It's worth the work, because it cuts down on costs, and even helps [reduce the carbon impact of your software](https://philsturgeon.com/what-is-green-software/?ref=apisyouwonthate.com), so it's basically rude not to turn it on. ## What is HTTP caching? HTTP caching tells API clients like browsers, mobile apps, or other backend systems if they need to ask for the same data over and over again, or if they can use data they already have. This is done with HTTP headers on responses that tell the client how long they can "hold onto" that response, or how to check if it's still valid. When HTTP cache servers (reverse proxies) are involved, it goes up a level, making sure that even if clients are not bothering to do the caching, that the API's application server is not being pestered if a cached response could be used. There are infinite ways to handle this, but with the proliferation of Content Delivery Networks (CDNs), it's now easier than ever to pop a cache proxy between the client and the server, storing responses for reuse whenever possible. ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2025/04/image-1.png) ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2025/04/image.png) This works very differently from server-side caching tools like Redis or Memcached, which always involve the HTTP connection coming into the web server, but then help the application short circuit querying the database or something else that was too slow to do every time. This is still web traffic to the application servers, but they will answer those repeatitive requests a bit quicker. HTTP caching instead helps negative the request ever being made (client caching), or avoids it reaching the web server (network caching). In a serverless environment it can really help to have a CDN caching at the network level, because the serverless approach pays per request. Utilizing HTTP caching means reducing traffic, reducing how many times that Lambda needs to spin up, and by reducing the number of responses that need to be generated by the server which is also going to reduce bandwidth costs. *There's a more to learn about how exactly HTTP caching works for APIs and how to design your APIs to be more cacheable, so check out our recent guide for* [*API Design Basics: Caching*](https://apisyouwonthate.com/blog/api-design-basics-cacheability/) *to learn more about that. This guide is going to focus on enabling it for Laravel applications hosted on Laravel Vapor.* ## Network Caching with AWS Gateway + CloudFront AWS has a CDN called CloudFront, and it works nicely with AWS API Gateway, both of which are already spun up by Vapor so let's just get them talking to each other. Using CloudFront might sound a little odd at first, as CDNs like CloudFront are generally more associated with caching images, CSS, and JS for frontend applications, but there is absolutely no difference in caching JS, CSS, and images, as there is to caching a REST/HTTP API. Everything in a REST API is considered a resource, much like all the images and other assets are considered resources. Any of them could change over time, and then those multiple resource versions are floating around for a while, with the server defines rules about how stale is acceptable based on the particulars for that resource. Far too many API developers are scared of utilizing HTTP caching, and this fear of the unknown is wasting money and natural resources for no reason. There are plenty of ways to handle validation (fetching latest information only if it changed) and invalidation (purging the CDN in extreme scenarios) so there's no reason to fear it. Soldier on. Once caching is enabled at the network level, we can add caching logic to the Laravel Framework with the caching middleware, and this will automatically cover caching `GET` and `HEAD` requests using the standard `Cache-Control` and `ETag` headers so we don't have to think about it. ### Step 1\. Upgrade to AWS API Gateway v2 First of all the best thing to do is upgrade to the latest version of the AWS API Gateway. AWS API Gateway v2 is [cheaper and faster](https://blog.laravel.com/vapor-api-gateway-vs-load-balancers?ref=apisyouwonthate.com), so it's a good idea to upgrade even if you get bored and forget to continue with the rest of this guide. Edit the `vapor.yml` file to add the `gateway-version: 2` line. ``` name: tree-tracker-api environments: production: domain: api.protect.earth gateway-version: 2 ``` If Vapor is set up to handle DNS automatically this is as a simple case of pushing/deploying changes, and everything will take care of itself. A new API will appear and your Vapor application will soon be utilizing it. ![Screenshot of the AWS admin panel, on the API Gateway section, with APIs selected on the navigation. There is one API called protect-earth-api-production which was made by Vapor.](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2025/04/aws-new-api.png) Those of you managing DNS manually will need to deploy this change to see the new CNAME details in the Laravel Vapor UI, but you might as well hold off until the next step or things will need to be changed again. ## Step 2: Slide CloudFront in front of API Gateway Pop over to the CloudFront section of the AWS panel, and create a new distribution pointing to the AWS Gateway for the application. ![Screenshot of the AWS interface, CloudFront section, under Distributions on the nav, titled Create distribution. Origin section is asking for an AWS origin or original domain name, with the "protect-earth-api-production" API gateway selected as the origin.](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2025/04/pick-api-origin.png) Set the DNS alias as whichever domain/subdomain you intend on using. For me its `api.protect.earth` as the main `protect.earth` site is, for now, using SquareSpace (🤮). Under **Cache key and origin requests** pick the recommended option. This will show a dropdown, and there the **Cache Policy** can be set to "CachingOptimized". ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2025/04/pick-cache-policy.png) Enable "Origin Cache Headers" to respect `Cache-Control` settings from the API (more on this shortly). When this distribution is created it will spend a little time spinning up, and as soon as its ready the distribution will display a subdomain like `d1tpxxxxxxxx.cloudfront.net`. Use this to update the DNS for your API subdomain like `api.protect.earth`, or get more creative to proxy it off to a subdirectory like `protect.earth/api`. However you do it, the goal is to send all API traffic through CloudFront so that any caching headers the API emits will now be respected by the CDN at all various edge locations selected. ### Step 3: Adding caching headers to the API There are several ways to add cache headers. One approach is to just shove out some headers directly onto the response in each controller, like so: ```php return response([]) ->header('Cache-Control', 'public,max-age=86400'); ``` This will work for the most basic of things, but there is some automatic functionality being missed that will come in handy when we start using `ETags` or `Last-Modified`, so I would recommend invoking the built-in `Illuminate\Http\Middleware\SetCacheHeaders` middleware. Make sure the following `cache.headers` alias is enabled in `app/Http/Kernel.php`. ``` # app/Http/Kernel.php protected $middlewareAliases = [ 'cache.headers' => \Illuminate\Http\Middleware\SetCacheHeaders::class, ]; ``` This can then be applied to the entire "api" group, or added to each route indivudually, something a bit like this. ``` # routes/api.php Route::get('/sites', App\Http\Controllers\Maps\SiteController::class) ->middleware('cache.headers:public;max_age=86400'); ``` For data that's not likely to change often at all, you can really crank the expiry time up. For example, Protect Earth has tree planting [projects](https://www.protect.earth/projects?ref=apisyouwonthate.com) at various "sites" and we're unlikely to rename a site too often if ever, trees are planted once ever and maybe a few more each winter or maybe none more are needed, and we're never going to move a field somewhere else so the coordinates aren't going to change. We really doesn't need to pretend like this is real-time data. Starting with a day long cache might be a reasonable first step, and it's easy to crank it up over time. For other that might change more often, you can learn more about validation using `ETags` to increase the cachablity of changeable data. I could not find any documentation for how this Laravel HTTP Caching middleware works, but seeing as it is a wrapper around [Symfony HTTP Cache](https://symfony.com/doc/current/http%5Fcache.html?ref=apisyouwonthate.com) we can glean what we need from there. Here are the options. ``` 'must_revalidate' => false, 'no_cache' => false, 'no_store' => false, 'no_transform' => false, 'public' => true, 'private' => false, 'proxy_revalidate' => false, 'max_age' => 600, 's_maxage' => 600, 'immutable' => true, 'last_modified' => new \DateTime(), 'etag' => 'abcdef' ``` All of these terms and keywords derive their meaning from [RFC 9111: Caching](https://www.rfc-editor.org/rfc/rfc9111?ref=apisyouwonthate.com), so we can use combinations of these for specific situations. **Do not store response in any cache** If information is sensitive or could contain PII you want to avoid being saved in any proxy caches, browser caches, or anywhere at all, stick a `no_store` in there. ``` ->middleware('cache.headers:no_store'); ``` **Cache for five minutes to this user** Data doesn't need to be public, it could be constrained to the user as defined in the `Authorization` header. Another handy reason to use this for APIs instead of inventing contentions like `My-Special-API-Key`. ``` ->middleware('cache.headers:private;max_age=300'); ``` **This is never going to change** Some information simply will not change ever. If a document that is already versioned, like `/files/abc123/versions/123456` and if it was edited then a new version would be created. In this case you can give it a huge max-age and pop an immutable on there to say "don't even bother trying to revalidate this, it wont be different". ``` ->middleware('cache.headers:max_age=31536000,immutable'); ``` **This might actually change** Sometimes you have a pretty sizeable response, and you know there's clients polling it for updates. For Protect Earth it's a list of orders for organizations, being polled every day to see if they've got any new trees planted. I ask them not to do this because sometimes we go months without planting any trees (planting season is October through April in UK) but still, they do, and the responses are huge. We can use the following cache headers to set a weekly cache, because a week is soon enough, and then we pop the etag on there to make sure clients revalidate. ``` ->middleware('cache.headers:max_age=6048000;etag'); ``` The built-in Laravel middleware will response with an ETag on each request, and when that ETag is sent back through the `If-None-Match` header by any cache-aware HTTP client, CloudFront will know exactly what to do with it. If there is an entry in the cache that matches this ETag value, it will respond with a 304 and no body. ``` http GET https://api.protect.earth/orgs/some-uuid -h 'If-None-Match: "9e9736203e9f15f11a4b263350561ea6"' HTTP/1.1 304 Not Modified Cache-Control: max-age=300, public ETag: "9e9736203e9f15f11a4b263350561ea6" X-Cache: Hit from cloudfront ``` This is a hit, but the CDN knows the client application already has a response which it can reuse, instead of the CDN even bothering to emit that same massive JSON and waste resources. The API server did nothing, the CDN didn't do much, and the client is happy. There's loads more we could go through, but that's quickly becoming a whole other article, so let's save that for another day. ## Sign up for APIs You Won't Hate The largest community for API Devs on the web. Subscribe Email sent! Check your inbox to complete your signup. No spam. Unsubscribe anytime. ### Step 4: Add some tests It's usually a good idea to pop some tests in to make sure the cache headers are coming out as expected, because there are so many different ways to apply those headers somebody might accidentally undo your work and nobody will notice until the server costs spike up over the month. ``` hasUnit() ->create(); $this->get('/certificates/'.$certificate->uuid) ->assertHeader('cache-control', 'max-age=86400, public'); }); ``` If these tests look a little alien, check out our guide on [contract testing with Laravel and OpenAPI](https://apisyouwonthate.com/blog/contract-testing-apis-laravel-php-openapi/), then you can pop this test right in there next to those. [Contract Testing a Laravel API with OpenAPIYour API does a bunch of great stuff, and your OpenAPI document tells everyone about all the great stuff that your API can do, but making sure those two sources of truth agree can be a bit of a struggle at first. Whether you followed the API design-first workflow and![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/icon/apis-logo-square-6.png)APIs You Won't HatePhil Sturgeon![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/thumbnail/spiderman-pointing.jpg)](https://apisyouwonthate.com/blog/contract-testing-apis-laravel-php-openapi/) ### Step 5: Prod the API to see if it worked First thing is to make sure our API is still up. ``` $ time http HEAD https://api.protect.earth/sites > Cache-Control: max-age=86400, public > X-Cache: Miss from cloudfront > ...snip... > 0.49s user 0.14s system 16% cpu 3.839 total ``` Great, 0.49 seconds, and the `X-Cache` shows it was a "miss". A miss means there was nothing in the cache able to serve the request, which is to be expected as we only just turned caching on a moment ago. Trying to again should really speed things up. ``` $ time http GET https://api.protect.earth/sites > Cache-Control: max-age=300, public > X-Cache: Hit from cloudfront > .. snip ... > 0.29s user 0.10s system 44% cpu 0.891 total ``` Cache hit, which means something was found in the cache that met the requirements, and because this is a lot faster than pestering the origin web server the response is down from 0.49s to 0.29s. A decent improvement, especially as I'm doing this from an Alp with avalanches actively coming down around me and the whole village being on emergency power. So with that working, let's learn how we can monitor how the API is doing beyond a quick poke. ### Step 6: Monitor Hit Rates in AWS All of the client applications now using the API are going through the network cache, so you can monitor how this is going through the AWS UI. Go to the API CloudFront distribution and click **Cache statistics**. There you will be a breakdown of hits and misses over time. The number of requests that hit or miss is known as the hit rate, and for every hit that's one less thing the API had to bother doing. ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2025/04/hit-rate.png) The size of the responses for a hit or miss is also interesting, because you can see how much data is *not* being generated slowly by your API but is instead being served quickly from the CloudFront cache. ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2025/04/bytes-transferred.png) CloudFront being in front of our API traffic does a few other useful things, like allowing us to see where our traffic is coming from. ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2025/04/bots-crawlers.png) 50% bots and crawlers seems like a lot. With our users optimized and loading information far more quickly, the next step in keeping our servers from stressing out for no reason will be to block crawlers and bots from messing with us, but that's yet another article for another day. ## Take this and run Dipping your toe into the world of API caching with a little bit of `Cache-Control` is a solid start, and learning more about ETags and the whole validation flow will help open the door to caching more documents which may change over time. People often underestimate how much of their API could be cached, but it's worth digging into because reducing redundant requests will: - Cut down on server load (lowering hosting costs). - Reduce network traffic (lowering bandwidth fees). - Minimize energy consumption (lowering environment impact). Imagine millions of users no longer making unnecessary requests for unchanged data. [Designing APIs to be cache-friendly](https://apisyouwonthate.com/blog/api-design-basics-cacheability/) from the start not only benefits the environment but also leads to faster, more efficient, and user-friendly APIs. It's a win-win: better performance for users, lower operational costs for providers, and a positive impact on the planet. Also, now that you've added the cache headers there is another added benefit in traffic reduction: Browsers, and [cache-enabled HTTP applications](https://apisyouwonthate.com/blog/http-client-response-caching/), will not even bother to make a HTTP request for a resource they already have in their local cache. [Speeding Up APIs/Apps/Smart Toasters with HTTP Response CachingCaching is a huge topic, and there’s a lot of different types of caching. No one type of cache is going to suite all needs and cover…![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/icon/10fd5c419ac61637245384e7099e131627900034828f4f386bdaa47a74eae156)APIs You Won't HatePhil Sturgeon![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/thumbnail/1-tYNUaltPqEB9dlIdMQSY6Q.jpeg)](https://medium.com/apis-you-wont-hate/speeding-up-apis-apps-smart-toasters-with-http-response-caching-a67becf829c6?ref=apisyouwonthate.com) At first much of this can seem a bit scary, and worries of "how do we invalidate data" start to creep into your mind, but all of this is a solved problem and its simply a case of learning a bit more about the amazing caching mechanics of HTTP. Start with these resources: - [MDN: HTTP Caching](https://developer.mozilla.org/en-US/docs/Web/HTTP/Caching?ref=apisyouwonthate.com) - [What is Cache Control?](https://www.fastly.com/blog/cache-control-wild?ref=apisyouwonthate.com) \- Fastly - [ETags: What are they and how to use them?](https://www.fastly.com/blog/etags-what-they-are-and-how-to-use-them?ref=apisyouwonthate.com) \- Fastly Then pop into the comments with questions, or ideas for what to cover in future articles. --- ***If this has peaked your interest but you need a bit more help, Phil Sturgeon is*** [***available for consulting***](https://cal.com/philsturgeon?ref=philsturgeon.com) ***(avalanches dependent) and you can schedule a call to get stuck into all this and more.*** ### API Design Basics: Cacheability URL: https://apisyouwonthate.com/blog/api-design-basics-cacheability/ Last updated: 2025-06-16T12:29:56.000Z API caching can save servers some serious work, cut down on costs, and even help reduce the carbon impact of an API. However, it is often considered an optimization rather than what it truly is: an integral part of API design. A fundamental part of REST is APIs declaring the “cacheability” of resources. When working with HTTP there are many amazing caching options available through HTTP Caching; a series of standards that power how the entire internet functions. This can be used to design more useful APIs, as well as being faster, cheaper, and more sustainable. ## What is HTTP caching? HTTP caching tells API clients (like browsers, mobile apps, or other backend systems) if they need to ask for the same data over and over again, or if they can use data they already have. This is done with HTTP headers on responses that tell the client how long they can “hold onto” that response, or how to check if it’s still valid. This works very differently from server-side caching tools like Redis or Memcached, which cache data on the server. HTTP caching happens on client-side or on intermediary proxies like Content Delivery Networks (CDNs), acting as a proxy between the client and the server and storing responses for reuse whenever possible. Think of server-side caching as a way to skip application work like database calls or outgoing HTTP requests, by fetching precomputed results from Redis or Memcached. HTTP caching reduces traffic and computational load further, by reducing the number of requests that even reach the server, and by reducing the number of responses that need to be generated. ## How does it work? HTTP caching is driven by cache headers. In its most simple form, when an API sends a response, it includes instructions that tell the client and other network components like CDNs if they are allowed to cache the response, and if so for how long. The guide on [API responses](https://www.speakeasy.com/api-design/responses?ref=apisyouwonthate.com) briefly introduced the `Cache-Control` header: ``` HTTP/2 200 OK Content-Type: application/json Cache-Control: public, max-age=18000 { "message": "I am cached for five minutes!" } ``` Here the server is telling the client (and any cache proxies) that they can cache this response for 5 minutes, and they can share it with other clients too. This means that a client can use this data for up to 5 minutes without checking back with the server, and when that time has expired it will make a new request. Fetching data, processing it, and sending it back to the client takes time and resources. Even when all of those processes are as optimized as possible, if the data hasn’t changed, why bother repeating these requests? Instead of wasting resources answering the same requests over and over again, the server could be processing more useful requests, saving energy, and save money by scaling down unnecessary server capacity. ### Cache-Control Header Defined in [RFC 9111: HTTP Caching](https://www.rfc-editor.org/rfc/rfc9111?ref=apisyouwonthate.com), this header sets out the rules. It tells clients what to do with the response: - `Cache-Control: max-age=3600` — The client can use this data for up to an hour (3600 seconds) without checking with the server. - `Cache-Control: no-cache` — The client must check with the server before using the cached copy. - `Cache-Control: public` or `private` — Defines whether just the client or everyone (like proxies) can cache it. These directives can be combined in various combinations for more control, with handy advanced options like `s-maxage` for setting how long data should live on shared caches like CDNs. Some simple APIs will only use `Cache-Control` to manage caching, but there’s another powerful tool in the cache toolbox: `ETag`. ### ETag Header ETags (short for “Entity Tags”) are like a fingerprint for a particular version or instance of a resource. When the resource changes, the ETag will change. No two versions of a resource should have the same ETag, and the ETag is unique to the URL of the resource. When a server sends a response, it can include an ETag header to identify that version of the resource: ``` HTTP/2 200 OK Content-Type: application/json ETag: "abc123" { "message": "Hello, world!" } ``` Then when a request is reattempted for whatever reason, the client sends a request with the ETag in the `If-None-Match` header. Doing this basically says “Only download the response if the ETag is different to this”. ``` GET /api/resource HTTP/2 If-None-Match: "abc123" ``` - If the server responds with `304 Not Modified`, it tells the client, “That response is still good. Nothing has changed since then, so no need to download it again.” - If the data has changed, the server returns the new data with a new ETag. This is especially helpful for large responses that don’t change often, especially when combined with `Cache-Control`. Sending `Cache-Control` and `ETag` lets the client confidently reuse the data for a while without even needing to send a HTTP request to the server, then after that time it can switch to doing a check for changes instead of downloading the whole response again. All of this is done without the client needing to know anything about the data, or how it’s stored, or how it’s generated. The server will handle it all, and the client will just keep requesting the data, allowing the cache-aware HTTP client to do the heavy lifting. ## Using Cache-Control and ETags in code Let’s add these headers to a basic Express.js API to see how it might look on the server-side. ``` const express = require('express'); const app = express(); app.get('/api/resource', (req, res) => { const data = { message: "Hello, world!" }; // Simulated data const eTag = `"${Buffer.from(JSON.stringify(data)).toString('base64')}"`; if (req.headers['if-none-match'] === eTag) { // Client has the latest version res.status(304).end(); } else { // Serve the resource with cache headers res.set({ 'Cache-Control': 'max-age=3600', // Cache for 1 hour 'ETag': eTag }); res.json(data); } }); app.listen(3000, () => console.log('API running on http://localhost:3000')); ``` The ETag is generated by hashing the data, then the server checks if the client has the latest version. If it does, it sends a `304 Not Modified` response, otherwise it sends the data with the `ETag` and `Cache-Control` headers. In a real codebase, would be doing something like fetching from a datasource, or computing something that takes a while, so waiting for all of that to happen just to make an ETag is not ideal. Yes, it avoids turning that data in JSON and sending it over the wire, but if the API is going to ignore it and send an `304 Not Modified` header with no response, the data was loaded and hashed for no reason. Instead, an ETag can be made from metadata, like the last updated timestamp of a database record. ``` const crypto = require('crypto'); function sha1(data) { const crypto.createHash('sha1').update(data).digest('hex'); } const trip = Trips.get(1234); const eTag = `"${sha1(trip.updated_at)}"`; ``` This example creates a SHA1 hash of the updated time, which will automatically change each time the record is updated. No need to specify the name of the Trip resource, or even mention the trip ID, because an ETag is unique to the URL and that is already a unique identifier. When working with resources that have their own concept of versioning, why not use that version number as an ETag instead of creating one from something else. ``` const trip = Trips.get(1234); const eTag = `"${trip.version}"`;HTTP/2 200 OK Content-Type: application/json ETag: "v45.129" ``` Regardless, ETags are brilliant and easy to reconcile. If clients don’t use them, it doesn’t have any effect, but if they do use a HTTP client with [cache middleware](https://apisyouwonthate.com/blog/http-client-response-caching/) enabled then both the client and the server can save a lot of time and resources. ## Public, private, and shared caches Using `Cache-Control` headers its possible to specify whether the response can be cached by everyone, just the client, or just shared caches. This is important for security and privacy reasons, as well as cache efficiency. - `public` — The response can be cached by everyone, including CDNs. - `private` — The response can only be cached by the client. - `no-store` — The response can’t be cached at all. When a response contains an `Authorization` header, it’s automatically marked as `private` to prevent sensitive data from being cached by shared caches. This is another reason to use standard auth headers instead of using custom headers like `X-API-Key`. ## Which resources should be cached? Some people think none of the data in their API data is cacheable because “things might change.” It’s rare that all data is so prone to change that HTTP caching cannot help. All data is inherently out of date before the server has even finished sending it, but the question is how out of date is acceptable? For example, a user profile is not likely to change particularly often, and how up to date does it really need to be? Just because one user changes their biography once in a year doesn’t mean that all user profiles need to be fetched fresh on every single request. It could be cached for several hours, or even every day. When talking about more real-time systems, one common example is a stock trading platform. In reality, most trading platforms publish a new public price every 15 minutes. A request to `/quotes/ICLN` might return a header like `Cache-Control: max-age=900`, indicating the data is valid for 900 seconds. Even when clients are “polling” every 30 seconds, the network cache will still be able to serve the response for 15 minutes, and the server will only need to respond to 1 in 30 requests. Some resources might genuinely change every second, and depending on the traffic patterns network caching could still be helpful. If 1,000 users are accessing it simultaneously then network caching will help significantly reduce the load. Instead of responding to 1,000 individual requests per second, the system can reuse a single response per second. This would be a 99.9% reduction in server load, and a 99.9% reduction in bandwidth usage. A safe default for most data is to apply some level of `max-age` caching (such as 5 minutes, an hour, a day, or a week, before it needs to be refreshed) paired with an ETag to check for fresh data past that time if the response is large or slow to generate. The introduction of ETags to an API can increase confidence in using longer cache expiry times. ## Designing cacheable resources All new APIs should be designed with cachability in mind, which means thinking about how to structure resources to make them more cacheable. The changes needed to make an API more cacheable are often the same changes that make an API more efficient and easier to work with. ### Resource composition One of the largest problems API designers face is how to sensibly group data into resources. There’s a temptation to make fewer resources so that there are fewer endpoints, with less to document. However, this means larger resources, which become incredibly inefficient to work with (especially when some of the data is more prone to change than the rest). ``` GET /invoices/645E79D9E14 ``` ``` { "id": "645E79D9E14", "invoiceNumber": "INV-2024-001", "customer": "Acme Corporation", "amountDue": 500.00, "amountPaid": 250.00, "dateDue": "2024-08-15", "dateIssued": "2024-08-01", "datePaid": "2024-08-10", "items": [ { "description": "Consulting Services", "quantity": 10, "unitPrice": 50.00, "total": 500.00 } ], "customer": { "name": "Acme Corporation", "address": "123 Main St", "city": "Springfield", "state": "IL", "zip": "62701", "email": "acme@example.org", "phone": "555-123-4567" }, "payments": [ { "date": "2024-08-10", "amount": 250.00, "method": "Credit Card", "reference": "CC-1234" } ] } ``` This is a very common pattern, but it’s not very cacheable. If the invoice is updated, the whole invoice is updated, and the whole invoice needs to be refreshed. If the customer is updated, the whole invoice is updated, and the whole invoice needs to be refreshed. If the payments are updated, the whole invoice is updated, and the whole invoice needs to be refreshed. We can increase the cachability of most of this information by breaking it down into smaller resources: ``` GET /invoices/645E79D9E14 ``` ``` { "id": "645E79D9E14", "invoiceNumber": "INV-2024-001", "customer": "Acme Corporation", "amountDue": 500.00, "dateDue": "2024-08-15", "dateIssued": "2024-08-01", "items": [ { "description": "Consulting Services", "quantity": 10, "unitPrice": 50.00, "total": 500.00 } ], "links": { "self": "/invoices/645E79D9E14", "customer": "/customers/acme-corporation", "payments": "/invoices/645E79D9E14/payments" } } ``` Instead of mixing in payment information with the invoice, this example moves the fields related to payment into the payments sub-collection. This is not only makes the invoice infinitely more cacheable, but it also makes space for features that are often used in an invoice system like payment attempts (track failed payments) or partial payments. All of that can be done in the Payments sub-collection, and each of those collections can be cached separately. The customer data is also moved out of the invoice resource, because the `/customers/acme-corporation` resource already exists and reusing it avoids code duplication and maintenance burden. Considering the user flow of the application, the resource is likely already in the browser/client cache, which reduces load times for the invoice. This API structure works regardless of what the data structure looks like. Perhaps all of the payment data are in an `invoices` SQL table, but still have `/invoices` and `/invoices/{id}/payments` endpoints. Over time as common extra functionality like partial payments is requested, these endpoints can remain the same, but the underlying database structure can be migrated to move payment-specific fields over to a `payments` database table. Many would argue this is a better separation of concerns, it’s easier to control permissions for who is allowed to see invoices and/or payments, and the API has drastically improved cachability by splitting out frequently changing information from rarely changing information. ### Avoid mixing public and private data Breaking things down into smaller, more manageable resources can separate frequently changing information from more stable data, but there are other design issues that can effect cachability: mixing public and private data. Take the example of a train travel booking API. There could be a Booking resource, specific to a single user with private data nobody else should see. ``` GET /bookings/1234 ``` ``` { "id": 1234, "departure": "2025-08-15T08:00:00", "arrival": "2025-08-15T12:00:00", "provider": "ACME Express", "seat": "A12" } ``` In order for a user to pick their seat on the train, there could be a sub-resource for seating: ``` GET /bookings/:my_booking_ref/seating ``` ``` { "my_seat": "A12", "available_seats": [ "A1", "A2", "A3", "A4", "A5", "A6", ... ] } ``` Creating the seating sub-resource like this will make a unique seating chart for every single user, because “all the seats” and “this users seat” have been mixed together. These responses could still be cached, but it would have to be a `private` cache because the generic information has been “tainted” with data unique to each user. 10,000 users would have 10,000 cache entries, and the chance/impact of them being reused would be rather small, so there isn’t much benefit to filling the entire cache with all this. Consider breaking this down into two resources: - `GET /bookings/:my_booking_ref` \- See booking details, including current seat. - `GET /trips/:trip_id/seats` \- List seat availability on the train. - `PUT /bookings/:my_booking_ref` \- Update booking (eg to reserve a seat). By moving the seat information to the booking resource, the seating availability becomes generic. With nothing personalized about it at all, the resource can be cached for everyone who is trying to book a seat on this train. There is no downside to caching this data, because it is the same for everyone. Even if it changes, it’s easy to grab the latest data from the server and suggest the user select another seat if it’s no longer available. This allows the seat availability to be cached for a long time, and only worry about refreshing the plan if the `PUT` request fails because a seat is no longer available. ## Content Delivery Networks (CDNs) HTTP caching works well when clients use it, and many do automatically, like web browsers or systems with caching middleware. But it becomes even more powerful when combined with tools like [Fastly](https://www.fastly.com/?ref=apisyouwonthate.com) or [Varnish](https://www.varnish-software.com/products/varnish-cache/?ref=apisyouwonthate.com). These tools sit between the server and the client, acting like intelligent gatekeepers: ![A sequence diagram showing a Client, Cache Proxy, and Server. A web request travels from client to proxy, then is sent on to the server, showing a "cache miss". The response then travels back from the server to the cache proxy, and then is sent to the client](https://www.speakeasy.com/_next/image?url=%2F_next%2Fstatic%2Fmedia%2Fhttpcachemiss.c37035ae.png&w=1080&q=75) ![A sequence diagram showing a Client, Cache Proxy, and Server. A web request travels from client to proxy, but does not go to the server, showing show a "cache hit". The response is served from the cache proxy to the client without involving the server](https://www.speakeasy.com/_next/image?url=%2F_next%2Fstatic%2Fmedia%2Fhttpcachehit.fc5f5094.png&w=1080&q=75) Client-caching like this is certainly useful, but the real power of caching comes when API web traffic is routed through a caching proxy. Using hosted solutions like Fastly or AWS CloudFront, this could be a case of changing DNS settings. For self-hosted options like Varnish, instead of pointing DNS settings to a hosted solution somebody will need to spin up a server to act as the cache proxy. Many API gateway tools like Tyk and Zuplo have caching built in, so this functionality may already be available in the ecosystem and just need enabling. ## Save emissions (and money) with HTTP caching The Internet (and it’s infrastructure) is responsible for [4% of global CO2 emissions](https://www.bbc.com/future/article/20200305-why-your-internet-habits-are-not-as-clean-as-you-think?ref=apisyouwonthate.com), and with [83% of web traffic coming from APIs](https://www.akamai.com/newsroom/press-release/state-of-the-internet-security-retail-attacks-and-api-traffic?ref=apisyouwonthate.com), it becomes critical to consider the carbon impact of new APIs. Each unnecessary API request costs server resources, bandwidth, and energy. That energy comes with a carbon footprint, whether it’s from a data center powered by renewable energy or not. ## Summary By reducing redundant requests, HTTP caching can: - Cut down on server load (lowering hosting costs). - Reduce network traffic (lowering bandwidth fees). - Minimize energy consumption (a win for the environment). Imagine millions of users no longer making unnecessary requests for unchanged data. Designing APIs to be cache-friendly from the start not only benefits the environment but also leads to faster, more efficient, and user-friendly APIs. It’s a win-win: better performance for users, lower operational costs for providers, and a positive impact on the planet. Next time a new API is being designed, ask the question: How much of this data do I really need to serve fresh each time, and how much of this can be cached with a combination of `Cache-Control` and `ETag` headers? ## Further Reading - [MDN: HTTP Caching](https://developer.mozilla.org/en-US/docs/Web/HTTP/Caching?ref=apisyouwonthate.com) - [ETags: What are they and how to use them?](https://www.fastly.com/blog/etags-what-they-are-and-how-to-use-them?ref=apisyouwonthate.com) - [What is Cache Control?](https://www.fastly.com/blog/cache-control-wild?ref=apisyouwonthate.com) - [Speeding up Client Applications with HTTP Caching](https://apisyouwonthate.com/blog/http-client-response-caching/) [Speeding Up APIs/Apps/Smart Toasters with HTTP Response CachingCaching is a huge topic, and there’s a lot of different types of caching. No one type of cache is going to suite all needs and cover everything needed to make a performant application, but one type that’s often overlooked is client caching at the HTTP level. Building![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/icon/apis-logo-square-5.png)APIs You Won't HatePhil Sturgeon![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/thumbnail/toaster.jpeg)](https://apisyouwonthate.com/blog/http-client-response-caching/) ### Schrödinger's Protocol URL: https://apisyouwonthate.com/newsletter/schrodingers-protocol/ Last updated: 2025-04-15T12:42:14.000Z Posts about Model Context Protocol (MCP) have dominated my API socials. Either the end of OpenAPI is now, or we don't need MCP as we have OpenAPI. Both of these takes miss the mark; check out our previous newsletter ([MCP ❤️ APIs](https://apisyouwonthate.com/newsletter/mcp-loves-apis/)) for what MCP is and how OpenAPI fits it. We have included a fantastic article from our Slack community on MCP in this newsletter. Easter is fast approaching, and I have decided to take a holiday this year. I will be spending my time eating chocolate surrounded by beautiful Karri trees. Happy Easter to everyone. \-- Alexander ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2025/04/State-of-API-DevEx-Report.png) ## Help shape the future of APIs If you build, manage, or use APIs, your insights can help highlight key trends and challenges in the ecosystem. Take the 10 minute survey, and you'll get early access to the report — plus a chance to win a PS5, NuPhy mechanical keyboard, and more! [Take the Survey](https://www.speakeasy.com/state-of-api-devex/2025?utm%5Fsource=newsletter&utm%5Fmedium=email&utm%5Fcampaign=apisyouwonthate) ## The API Round-Up *API News, links, and tools from around the web* ### [Hot Chocolate](https://chillicream.com/docs/hotchocolate/v15?ref=apisyouwonthate.com) Trying to keep up with the Easter theme, I remembered that .Net has a great GraphQL server called Hot Chocolate. It is fully compliant with the latest specs, which makes it compliant with all clients, such as Strawberry-Shake, Relay, Apollo and more. We do love naming things, aye 😅. ### [OpenAPI Org is now on BlueSky](https://bsky.app/profile/openapis.org?ref=apisyouwonthate.com) Over the last six to eight months, the tech sector has found a prominent home on BlueSky. While there are still people on Mastodon and Twitter, BlueSky is growing. It's great to finally have the OpenAPI Org on there. ### [OpenAPI Community Hero – Phil Sturgeon](https://www.openapis.org/blog/2025/04/01/openapi-community-hero-phil-sturgeon?ref=apisyouwonthate.com) Why does Phil spend so much time driving development in the OpenAPI space? What is he excited about, and what does he see as the future of OpenAPI? Phil shares some insights on the OpenAPI blog. ### [Choosing Your OpenAPI Strategy with Quarkus vs. Spring](https://myfear.substack.com/p/spec-first-or-code-first-choosing) Working in enterprise for the last two years has forced me to write a lot of Java/Kotlin and Spring Boot. I have never enjoyed how Spring favours the code-first approach to building APIs. It leads to more work, more code, and a bastardised version of trying to do design first. Quarkus, an alternative to Spring, is growing in popularity. It has fixed these issues and offers a spec-first approach. ### [Mobile GraphQL at Meta in 2025](https://play.pocketcasts.com/podcasts/c4ede3e0-1fbf-0136-c266-7d73a919276a/278f0fac-e429-48ad-b71f-94b33aadda71?ref=apisyouwonthate.com) Sabrina from Meta's Mobile GraphQL Platform Team talks to Pascal Hartig about Meta's evolution of GraphQL. They discuss data fetching on mobile devices and how they use GraphQL to improve user experiences. It is an enjoyable listen that covers how they build on the GraphQL system. ### [OpenAPI: How to Handle File Management](https://thenewstack.io/openapi-how-to-handle-file-management/?ref=apisyouwonthate.com) Lorna Mitchell is back with another killer article focusing on managing large OpenAPI specs by splitting them into separate files. This is a great approach using the $ref syntax, which allows you to choose the ideal level of file granularity based on your needs. --- ## APIs You Won't Hate The latest from the team at API's You Won't Hate. ### [A Developer's Guide to API Design-First](https://apisyouwonthate.com/blog/a-developers-guide-to-api-design-first/) API Design gets floated around a lot, but what does it mean, and how do you implement it? Put down your code first approaches, learn the benefits, and start building and designing first when it comes to APIs. ### [Enforcing API Consistency for Large Teams](https://apisyouwonthate.com/blog/enforcing-api-consistency-with-a-large-team/) Keeping APls consistent in large organizations to speed up integrations, improve code-reuse, and avoid wasting time and money. --- ## From our Community Articles written and shared in our free [Slack community](https://slack.apisyouwonthate.com/?ref=apisyouwonthate.com). ### [Breaking Down MCP: An Architectural Analysis](https://www.layered.dev/breaking-down-mcp-an-architectural-analysis?ref=apisyouwonthate.com) Kevin's excellent post in our Slack group breaks down the Model Context Protocol. He answers some great questions, from why create another protocol to whether REST would have been more web-friendly. Thanks for sharing, Kevin. ## Got a buck to spare? When you become an APIs You Won't Hate member, you'll get access to members-only content while directly supporting our work. Your support helps us to keep making resources for the API community. [Become a paid member](https://apisyouwonthate.com/#/portal) Thanks so much to our members: **Kin L, Juxt, Alex R, Nolan S, Frank, James D, Bill D, Rich, and Umair**. Your support means the world to us! Until next time, Alexander, Phil & Mike ### Enforcing API consistency with a large team URL: https://apisyouwonthate.com/blog/enforcing-api-consistency-with-a-large-team/ Last updated: 2025-04-15T12:29:11.000Z In large organizations where multiple teams contribute to an API ecosystem, maintaining consistency can be a challenge. Without some sort of defined strategy, APIs may diverge in their style, error handling, naming conventions, security practices, or even call the same thing different names. All of this causes confusion for users of the API, and this confusion costs time and money as support staff need to answer questions, profitable integrations are delayed, or breakages occur due to inconsistent behavior. To ensure uniformity, organizations need a combination of structured guidelines, automated enforcement, centralized API gateways, and thorough review processes. ## Define an API style guide As 100 developers how to do something and you’ll get 101 answers on the best way to do it, and this can become a real headache when building APIs. Instead of picking unique approaches to everything, and API style guide can be written as a foundation for standardization, ensuring that all teams follow a common approach when designing and building APIs. A good guide should clearly define naming conventions for endpoints, query parameters, and request/response properties, and require establish standards for structured things like [pagination](https://www.speakeasy.com/api-design/pagination?ref=apisyouwonthate.com), [error handling](https://www.speakeasy.com/api-design/errors?ref=apisyouwonthate.com), [collections](https://www.speakeasy.com/api-design/collections?ref=apisyouwonthate.com), and [HTTP status codes](https://www.speakeasy.com/api-design/status-codes?ref=apisyouwonthate.com). Authentication and security guidelines must be outlined, covering mechanisms like OAuth, API keys, and rate limiting, all of which should work the same across APIs wherever possible to allow code reuse and reduce the cognitive load. Additionally, the guide should include a versioning strategy to manage changes effectively and set documentation standards for OpenAPI descriptions to ensure completeness and clarity. These style guides are living documents that should be updated regularly to reflect changes in best practices and organizational requirements, and can be published internally or publicly to help other organizations who might like your style. Lots of companies have done this, including [Google](https://cloud.google.com/apis/design?ref=apisyouwonthate.com), and many more which can be found [here](https://apistylebook.com/design/guidelines/?ref=apisyouwonthate.com). ## Automated style guides Writing all these decisions down is a good start, but it’s not enough. Humans make mistakes, misread things, and misremember things. As guides evolve and new advice is added, people are unlikely to come back and read the guide again and might not spot the new advice. API linting tools exist to help, with two popular tools being [Spectral](https://github.com/stoplightio/spectral/?ref=apisyouwonthate.com), [vacuum](https://github.com/daveshanley/vacuum?ref=apisyouwonthate.com), and the [Speakeasy CLI](https://www.speakeasy.com/docs/prep-openapi/linting?ref=apisyouwonthate.com). This will not only validate OpenAPI documents to make sure they’re syntactically correct, but can also be programmed to enforce the advice set out in the style guide. [Automated Style Guides for REST, GraphQL and gRPCAsk 100 developers where a semicolon should go, and you’ll either get 100 answers, or a all-on-all fist fight. To save this from happening at work, most folks implement a style guide, which beyond helping with consistent style to avoid new developers getting shouted at for “doing it wrong”. Linters![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/icon/apis-logo-square-3.png)APIs You Won't HatePhil Sturgeon![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/thumbnail/rules-regulations.jpeg)](https://apisyouwonthate.com/blog/automated-style-guides-for-rest-graphql-grpc/) This means that teams who have built an API can check it’s ok before deploying it to production, and teams who are following the API design-first workflow can get real-time feedback on the API as it is still being defined, further saving time and money from being wasted coding something problematic. Spectral, vacuum and Speakeasy all automate a big chunk of the API style guide, making API design reviews considerably easier, letting the review focus on more complex issues like “is this the right way to solve the problem?” rather than “is this the right way to capitalize a property?”. Having all APIs following the same automated style guide, all integrated into OpenAPI editors, code editors, and run again in CI/CD pipelines, teams can be certain that APIs are going to be consistent, and only get better over time. ## Leveraging API gateways for centralized functionality One way to remove discrepancies between APIs is to not have multiple APIs doing the same thing in the first place. API gateways like [Kong](https://konghq.com/?ref=apisyouwonthate.com), [Tyk](https://tyk.io/?ref=apisyouwonthate.com), [Express Gateway](https://www.express-gateway.io/?ref=apisyouwonthate.com), and [AWS API Gateway](https://aws.amazon.com/api-gateway/?ref=apisyouwonthate.com) play a crucial role in standardizing authentication and authorization policies, rate-limiting, traffic filtering, and network caching. Leaving these features to be implemented in multiple APIs can be tricky even when installing the same libraries. For example, while OAuth 2 is a standard and should be implemented the same way regardless of the software, two different APIs might be using two different versions of the Ruby on Rails OAuth 2 server Doorkeeper, and one might have fixed a bug that the other hasn’t. Those two will vary again from another implementation written in another framework or another language. They also help maintain uniform logging and monitoring practices by centralizing API usage tracking and performance metrics. Additionally, gateways can facilitate request and response transformations, ensuring backward compatibility without requiring changes across multiple services. This is a helpful way to remove inconsistencies from APIs which are already in production without complicating the codebase. ## API design reviews The API Design Review is a core component of a broader API governance program, where stakeholders evaluate proposed API changes to ensure they align with the organization’s architecture and ecosystem. This process typically involves a diverse group, including API designers, developers, technical writers, system architects, and governance teams, all working together to maintain consistency and quality across the API ecosystem Just like code reviews are now common on pull requests, design reviews allow API designers and developers to submit OpenAPI-based proposals for review either before any code is written, or at the same time as code is written. A dedicated review committee, including API architects and experienced developers, should evaluate proposals based on adherence to guidelines and best practices. The automated checks can be run with linting on these pull requests before manual reviews to catch common issues efficiently, then design review meetings provide a platform to further discuss key API decisions, trade-offs, and potential improvements. This catches all the things a linter cannot, like “is this the right name for this concept” or “this was just added to another API, can we reuse that?” ## Summary Achieving consistency across an API ecosystem in a large organization requires a combination of well-documented style guides, automated enforcement through linting, centralized functionality via API gateways, and a structured API design review process. Implementing all or some of these approaches should ensure that APIs remain scalable, maintainable, and high quality, providing a surprise-free experience for API consumers and hopefully making life easier for API producers as well. ### MCP ❤️ APIs URL: https://apisyouwonthate.com/newsletter/mcp-loves-apis/ Last updated: 2025-04-01T17:09:22.000Z Scrolling through the feeds and newsletters this week, I have seen far too many posts saying OpenAPI is dead and MCP is the future. If you are unaware, MCP (Model Context Protocol) is a standardised way for AI to interact with tools. I don't know what it is about AI that has chunks of the industry claiming something is dead every few weeks. My favourite one recently was about not needing any doctors. Yep, that's right, we don't need to learn and explore any more, the computer tells us all. ...anyway, the interesting part is that MCP needs a well-written API to work. Mike Amundsen shared a great article on this, so I made it the first link to clear up this nonsense. \-- Alexander ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2025/03/generate-mcp-server-NDS-2.png) ## Connect your APIs To AI with Speakeasy Speakeasy can now bundle a runnable MCP server in TypeScript SDKs that works with Claude, Cursor and many other apps. [Learn more](https://www.speakeasy.com/post/release-model-context-protocol?ref=apisyouwonthate.com) ## The API Round-Up *API News, links, and tools from around the web* ### [MCP: The Ultimate API Consumer (Not the API Killer)](https://www.layered.dev/mcp-the-ultimate-api-consumer-not-the-api-killer?ref=apisyouwonthate.com) This excellent article walks you through what MCP is, why it's not the API an API killer, and why OpenAPI is more critical than ever. I love articles like this, cutting through the middle of the hype vs fear of disruption. ### [New URLPattern API Available in Node.js and Cloudflare Workers](https://blog.cloudflare.com/improving-web-standards-urlpattern/?ref=apisyouwonthate.com) Some fantastic work has been happening in NodeJS and Cloudflare. Support for standardised URL pattern matching is now available in NodeJS. This is huge progress. Cloudflare has supported this for a few years but has worked with the NodeJS team to bring this latest update. ### [Building APIs with Next.js](https://nextjs.org/blog/building-apis-with-nextjs?ref=apisyouwonthate.com) Most people think of NextJS for building websites or full-stack web apps. However, you get a flexible, modern way to build APIs using its App Router and Handlers. Check this in-depth article for a step-by-step guide. ### [**APIful Extensible, Typed API Tooling**](https://apiful.byjohann.dev/?ref=apisyouwonthate.com) I came across a new tool this week, APIful. It allows you to manage all your API interactions in one place, with full-type safety support and more. Load your OpenAPI spec, and off you go. ## APIs You Won't Hate The latest from the team at API's You Won't Hate. ### [Enforcing API consistency with a large team](https://apisyouwonthate.com/p/75e05d1e-ea14-40cd-9adc-39289b3dea0b/) Phil has been writing more API goodness from his canal boat. In his latest post, he walks us through maintaining consistency across APIs in large organizations. Focusing on consistency helps speed up integrations, improve code reuse, and avoid wasting time and money. ### [API Basics: File Uploads](https://apisyouwonthate.com/blog/api-design-basics-file-uploads/) For this one, we go back to basics: How does a REST API handle uploading images or spreadsheets when it normally only handles JSON or XML? --- ## From our Community Articles written and shared in our free [Slack community](https://slack.apisyouwonthate.com/?ref=apisyouwonthate.com). ### [The OpenAPI Advantage: Breaking Free from Vendor Lock-in](https://www.layered.dev/the-openapi-advantage-breaking-free-from-vendor-lock-in?ref=apisyouwonthate.com) Another article from Kevin Swiber was shared directly by the man himself on our Slack channels this month. Kevin talks us through the journey to OpenAPI and how this has broken us free from vendor lock-in. ## Support APIs You Won't Hate When you become an member, you'll get access to members-only content while directly supporting our work. Your support helps us to keep making resources for the API community. [Become a member today](https://apisyouwonthate.com/#/portal) Thanks so much to our members: **Kin L, Juxt, Alex R, Nolan S, Frank, James D, Bill D, Rich, and Umair**. Your support means the world to us! Until next time, Alexander, Phil & Mike ### API Design Basics: File Uploads URL: https://apisyouwonthate.com/blog/api-design-basics-file-uploads/ Last updated: 2025-04-01T10:54:19.000Z File uploads can be confusing to work with at first because it takes a bit of a mental shift to think about. Firstly, a file is usually not just a file, it also has metadata needs to go with it and that can be hard to keep track of. Secondly, it is not really a file upload, simply a resource or collection of resources with a `Content-Type` of something other than the usual JSON or XML. ## URL design To visualize how file uploads could be designed into an API, let’s see how images could be added for two different use-cases. A user could have an avatar sub-resource, which might look like this: ``` /users//avatar ``` This can then be uploaded and retrieved on the same URL, making it a consistent API experience with any other type of resource. Multiple images could be needed for product thumbnails, and that can be a sub-collection of the product. ``` /product//thumbnails ``` A collection of resources could be available, and a particular thumbnail could be retrieved or deleted using regular semantics like `GET` and `DELETE` on the particular resource URL. ``` /product//thumbnails/ ``` ## POST or PUT There is no particular [HTTP method](https://www.speakeasy.com/api-design/http-methods?ref=apisyouwonthate.com) specific to file uploads, instead we use the appropriate hTTP method for the resource or collection being worked with. For the example of a single avatar for each user, the URL is already known, and it does not make any difference whether this is the first avatar they have uploaded, or they have remade the same request 10 times in a row after an intermitted internet connection messed up the first few. This should use a `PUT`, because that means “The end result should be this, regardless of what is there right now.” ``` PUT /users//avatar ``` When working with a collection, the URL of the resource is not known until it has been created. For this reason a `POST` would be more appropriate. ``` POST /product//thumbnails ``` How these uploads work could vary depending on the use case, so let’s look at the most popular methods. ## Different methods of file upload There are a few popular approaches to file uploads in APIs: 1. Uploading a file by itself, like adding an avatar for an existing user. 2. Uploading a file with metadata in the same request, like a video file with a title, description, and geodata. 3. Importing a file from a URL, like a user’s avatar from Facebook. It’s not entirely unreasonable to consider an API using all of these approaches for different use cases throughout the API depending on the specifics. Lets learn how these things work, and talk about when to use one over the other. ### Method A: Direct file uploads When no metadata is needed to be uploaded with a request, a direct file upload is beautifully simple. - Uploading a CSV of emails being imported to send a tree sponsorship email to. - A new logo for a funding partner. - A replacement avatar for a user profile. In all of these situations, the file is the only thing that needs to be uploaded and they also have a handy content type that can go right into the HTTP request to let the API know what’s coming. ``` PUT /users/philsturgeon/image HTTP/2 Authentication: Bearer Content-Type: image/jpeg Content-Length: 284 ``` Any file can be uploaded this way, and the API can infer the content type from the `Content-Type` header. The API can also infer the user from the token, so the request does not need to include any user information. The API will then save the file, and return a response with a URL to the file that was uploaded. This URL can be used to access the file in the future, and can be used to link the file to the user that uploaded it. The response here will have a simple body: ``` { "url": "https://cdn.example.org/users/philsturgeon.jpg", "links": { "self": "https://example.org/api/images/c19568b4-77b3-4442-8278-4f93c0dd078", "user": "https://example.org/api/users/philsturgeon" } } ``` That `user` was inferred from the token, and the `url` is the resulting URL to the avatar that has been uploaded. Normally this would be some sort of Content Delivery Network (CDN) URL, but it could be a direct-to-S3 URL, or a URL to a Go service that handles file uploads. It’s up to you, but its good to split off file uploads to a separate service to keep your API servers free to do more impactful work than serving files. ### Method B: Upload from URL Depending on how the client application works, uploading from a file might not be the preferred approach. A common pattern is mobile clients uploading user images directly from the photo libraries on the mobile device, and the web teams were pulling avatars from Facebook or Twitter profiles after they have done a “social login” flow. This is common because its harder for the web application to access the raw content of a file using just browser-based JavaScript. At some point a server needs to be involved to read that, so whether they have uploaded via cloudinary or some other upload service, the API server is going to need to take a URL and download the file. The same endpoint that handled the direct upload can serve this same logic, with the `Content-Type` header changed to `application/json` and the body of the request containing a URL to the file. ``` PUT /users/philsturgeon/image HTTP/2 Authentication: Bearer Content-Type: application/json { "url" : "https://facebook.com/images/dfidsyfsudf.png" } ``` The API will then download the file from the URL, save it, and return a response with a URL to the file that was uploaded. This URL can be used to access the file in the future, and can be used to link the file to the user that uploaded it. ``` { "url": "https://cdn.example.org/users/philsturgeon.jpg", "links": { "self": "https://example.org/api/images/c19568b4-77b3-4442-8278-4f93c0dd078", "user": "https://example.org/api/users/philsturgeon" } } ``` Supporting both might not be necessary, but if they are, just support both the image types you need and the JSON alternative of that. HTTP makes that incredibly easy to do thanks to being able to switch `Content-Type`. ### Method 3: Separate metadata resource The above examples are great for simple file uploads, but what if you need to upload metadata with the file? This is where things get a bit more complex. One approach would be multipart forms, but they’re pretty complex to work with and not ideal for large files. If sending a massive video file, you don’t want to have to send the title, description, and tags in the same request as the video file. If the video file upload fails, you’ll have to re-upload the video file and all of the metadata again. The way YouTube handles uploads via API are an interesting examples of splitting out metadata and a video file. They use a two-step process which focuses on metadata first, which allows for the metadata to be saved and the video can then be retried and uploaded without losing the metadata. The YouTube Data API (v3) approach to [Resumable Uploads](https://developers.google.com/youtube/v3/guides/using%5Fresumable%5Fupload%5Fprotocol?ref=apisyouwonthate.com) works like this. First, they make a POST request to the video upload endpoint with the metadata in the body of the request: ``` POST /upload/youtube/v3/videos?uploadType=resumable&part=snippet,status HTTP/1.1 Host: www.googleapis.com Authorization: Bearer Content-Length: 278 Content-Type: application/json; charset=UTF-8 { "snippet": { "title": "My video title", "description": "This is a description of my video", "tags": ["cool", "video", "more keywords"], "categoryId": 22 }, "status": { "privacyStatus": "public", "embeddable": true, "license": "youtube" } } ``` The response then contains a `Location` header with a URL to the video upload endpoint: ``` HTTP/1.1 200 OK Location: https://www.googleapis.com/upload/youtube/v3/videos?uploadType=resumable&upload_id=xa298sd_f&part=snippet,status,contentDetails Content-Length: 0 ``` Then to upload the video it’s back to direct file uploads. The video file can be uploaded to the URL provided in the `Location` header, with the content type set to `video/*`: ```PUT PUT https://www.googleapis.com/upload/youtube/v3/videos?uploadType=resumable&upload_id=xa298sd_f&part=snippet,status,contentDetails HTTP/1.1 Authorization: Bearer AUTH_TOKEN Content-Length: Content-Type: video/mp4 ``` What’s cool about this approach, is that URL *could* be part of your main API, or it *could* be a totally different service. It could be a direct-to-S3 URL, Cloudinary, or some other service that handles file uploads. Larger companies will be more prone to building a service to handle such files coming in, whilst smaller teams might want to keep things simple and let their API do the heavy lifting. The larger the file, the more likely you’ll want to split that off, as having your API handle these huge files - even if the uploads are chunked - will keep the HTTP workers busy. Maintaining those connections might slow down a Rails-based API for a long time, for example, so having another service would help there. ## Best practices ### Check Content-Type and Content-Length It is worth noting that the `Content-Type` header is not always reliable, and you should not trust it. If you’re expecting an image, you should check the first few bytes of the file to see if it is a valid image format. If you’re expecting a CSV, you should check the first few lines to see if it is a valid CSV. **Never trust input.** The only thing worth mentioning on that request is the addition of `Content-Length`, which is basically the size of the image being uploaded. A quick check of `headers['Content-Length'].to_i > 3.megabytes` will let us quickly reply saying “This image is too large”, which is better than waiting forever to say that. Sure, malicious folks could lie here, so your backend code will need to check the image size too. **Never trust input.** Protecting against large files is important, as it can be a denial of service attack. If you allow users to upload files, they could upload a 10GB file and fill up your disk space. This is why it’s important to check the size of the file before writing it to disk. To make sure it seems to be the right type, and to make sure it’s not too large, you can read the file in chunks. This can be done with a simple `File.open` and `File.read` in Ruby, or similar in other languages. The file is read in chunks, and then written to a file on disk. This is a good way to handle large files, as you’re not trying to load the whole file into memory at once. ```ruby def update if headers['Content-Type'] != 'image/jpeg' render json: { error: 'Invalid content type' }, status: 400 return end if headers['Content-Length'].to_i > 3.megabytes render json: { error: 'File is too large' }, status: 400 return end file = File.open("tmp/#{SecureRandom.uuid}.jpg", 'wb') do |f| f.write(request.body.read) end # Do something with the file end ``` ### Securing File Uploads Allowing file uploads can introduce all sorts of new attack vectors, so it’s worth being very careful about the whole thing. One of the main issues with file uploads is directory traversal attacks. If you allow users to upload files, they could upload a file with a name like `../../etc/passwd`, which could allow them to read sensitive files on your server. Uploading from a URL could allow for [Server-Side Request Forgery (SSRF)](https://owasp.org/API-Security/editions/2023/en/0xa7-server-side-request-forgery/?ref=apisyouwonthate.com) attacks, where an attacker could upload a file from a URL that points to a sensitive internal resource, like an AWS metadata URL, or something like `localhost:8080` which allows them to scan for ports on the server. The [OWASP File Upload Cheat Sheet](https://cheatsheetseries.owasp.org/cheatsheets/File%5FUpload%5FCheat%5FSheet.html?ref=apisyouwonthate.com) has a lot of good advice on how to secure file uploads, including: - Limiting the types of files that can be uploaded. - Limiting the size of files that can be uploaded. - Storing files in a location that is not accessible via the web server. - Renaming files to prevent directory traversal attacks. - Checking the file type by reading the first few bytes of the file. - Checking the file size before writing it to disk. - Checking the file for viruses using a virus scanner. ## Summary Think about what sort of file uploads are needed, how big the files are, where they’re going, and what sort of clients will be using the API. The YouTube approach is a bit complex, but a combination of 1 and 2 usually take care of the job, and help avoid complicated multipart uploads. As always, build defensively, and never trust any user input at any point. --- *Thank you to* [*Speakeasy*](https://www.speakeasy.com/?ref=apisyouwonthate.com) *for sponsoring the time to write this article, as part of their new* [*API Design Guide*](https://www.speakeasy.com/api-design?ref=apisyouwonthate.com) *which you should absolutely go and check out.* ### Civic APIs URL: https://apisyouwonthate.com/newsletter/civic-apis/ Last updated: 2025-03-17T11:40:20.000Z Mike suggested a theme last week of APIs for civic good. While researching this topic, I stumbled across Google's Civic API, which provides information on elected officials. Unfortunately, it is being shut down later this year. While trying to find a replacement for the API, I fell down a tunnel of how other government bodies handle civic data and how important access to it is. I have shared links from the EU, the US, Australia, and the UK, and you can see the different approaches each country takes to this sort of data. You can skip the first four links if you're just after the weekly round up. \-- Alexander ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2025/03/generate-mcp-server-NDS-2.png) ## Connect your APIs To AI with Speakeasy Speakeasy can now bundle a runnable MCP server in TypeScript SDKs that works with Claude, Cursor and many other apps. [Learn more](https://www.speakeasy.com/post/release-model-context-protocol?ref=apisyouwonthate.com) ## The API Round-Up *API News, links, and tools from around the web* ### [Google Civic Information API](https://developers.google.com/civic-information/docs/v2/representatives?ref=apisyouwonthate.com) Enter someone's state and get their elected official. Like I said in the intro, Google is shutting it down in April this year. The data seems to have come from Tech and Civic Life, so someone could easily spin up another. Interestingly, the US appears to rely on private companies to get this data via an API. ### [Developer Hub UK](https://developer.parliament.uk/?ref=apisyouwonthate.com) I fell out of my chair when I discovered what the UK has available: APIs on elected officials, what they vote on, bills, strategies, and more. Using Open API spec 3.0.1, the APIs are presented with Swagger Docs. It's amazing to see this level of detail from the government itself. ### [OpenAustralia Foundation](https://www.oaf.org.au/?ref=apisyouwonthate.com) OpenAustralia is a charity foundation that presents data on elected officials, votes, and planning alerts. After seeing what the UK offers, I am very disappointed that the government does not do this. Looking at the [official government list of APIs](https://api.gov.au/apis?ref=apisyouwonthate.com), we have five listed, and one is an example API. Come on, Australia. ### [EU Developers Corner](https://data.europarl.europa.eu/en/developer-corner/opendata-api?ref=apisyouwonthate.com) The EU is leaps and bounds ahead of Australia, offering a complete list of APIs. One area where the EU goes further is offering ready-to-download datasets of the data that powers its APIs. It is open, transparent, and accessible, which is terrific. ### [Migrating from OpenAPI 3.0 to 3.1](https://apichangelog.substack.com/p/migrating-from-openapi-30-to-31) It's 2025 and some tools still don't support OpenAPI3.1\. Bruno Pedro explains why this is so and showcases some differences between the specs. ### [Create a Server Driven CLI from your REST API](https://zuplo.com/blog/2025/02/02/generate-cli-from-api-with-climate?ref=apisyouwonthate.com) After Node, Go is my favourite language. What could be better than REST APIs and Go? Learn how to turn a REST API into a CLI. This will be a super fun weekend coding project. ### [Fast HTTP V1.59](https://github.com/valyala/fasthttp?ref=apisyouwonthate.com) Fast is an HTTP package for Go that focuses on performance. It is a lot faster than the default Go package. Fast released a new version recently. Check out the release note for the full details. ### [Jira Rest API](https://github.com/MrRefactoring/jira.js?ref=apisyouwonthate.com) We all know that everyone loves Jira 😰. Well, now you can interact with it purely through a TypeScript API with nearly complete coverage of Jira Cloud, Service Desk, and Jira Agile. ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2025/03/apis-you-won-t-hate--2-.png) ## Instantly offer API Docs and SDKS Companies like [Square](https://link.buildwithfern.com/square-sdks-openapitools?ref=apisyouwonthate.com), [Intercom](https://link.buildwithfern.com/intercom-sdks-openapitools?ref=apisyouwonthate.com), and [ElevenLabs](https://link.buildwithfern.com/elevenlabs-docs-openapitools?ref=apisyouwonthate.com) trust Fern to power their developer experience. Start with an API spec. Generate SDKs in multiple languages and API documentation tailored to your brand. [Book a Demo](https://link.buildwithfern.com/demo-openapitools?ref=apisyouwonthate.com) ## APIs You Won't Hate The latest from the team at API's You Won't Hate. ### [API Design Basics: Designing for Security](https://apisyouwonthate.com/blog/api-design-basics-security/) Phil walks you through everything you must consider when planning and designing an API. These principles should be part of your development flow to sidestep major security issues before anyone starts coding. --- ## From our Community Articles written and shared in our free [Slack community](https://slack.apisyouwonthate.com/?ref=apisyouwonthate.com). ### [How Arazzo Is Defining the Future of API Workflows](https://thenewstack.io/the-rise-of-ai-agents-how-arazzo-is-defining-the-future-of-api-workflows/?ref=apisyouwonthate.com) Frank from Smartbear recently shared this great article in our Slack on how Arazzo spec is helping build AI Agents. Interesting article which once again reminds people how vital well documented APIs are. ## Become a member Like what we do? Got a buck to spare? Your support helps us to keep making resources for the API community. 🫶 [Become a paid member today](https://apisyouwonthate.com/#/portal) Thanks so much to our members: **Kin L, Juxt, Alex R, Nolan S, Frank, James D, Bill D, Rich, and Umair**. Your support means the world to us! Until next time, Alexander, Phil & Mike ### A Developer's Guide to API Design-First URL: https://apisyouwonthate.com/blog/a-developers-guide-to-api-design-first/ Last updated: 2025-05-23T23:58:11.000Z API Design-First, also known as "schema-first" or "contract-first", is all about designing the interface of an API before writing any code. It's about planning the [API contract](https://bump.sh/blog/api-contracts-extended-introduction?ref=apisyouwonthate.com) first, and defining what the API does and how it works so everyone's on the same page before implementation starts. This approach has been around for a while, and over time, it's evolved to meet the needs of different technologies. These days, OpenAPI has become the defacto standard for designing REST APIs, and AsyncAPI has become the defacto standard for describing event-driven APIs. This workflow can make life easier for everyone the whole way through the lifecycle of an API. ## A Brief History To understand why design-first is such a big deal, it helps to know where it came from. - **The early days: WSDL and SOAP**: In the late 90s and early 2000s, SOAP web services were popular, and people used the XML-based WSDL (Web Services Description Language) to describe them. WSDL files outlined all the operations and payloads in a verbose XML format. It was powerful, but many felt it was overly complex. Developers creating, maintaining, and using the APIs felt the frustration of working with them. - **REST APIs take over**: REST, introduced in the early 2000s, focused on simplicity and scalability. Unlike SOAP, REST didn't have a standard way of describing APIs. Many developers relied on manual documentation, example cURL commands, or tools like Postman collections. It worked, but it was messy, inconsistent, and could easily diverge from the implementation of the API. - **The OpenAPI era**: Around 2011, Swagger (later renamed OpenAPI) arrived to help describe REST APIs. It introduced a standard machine-readable format for defining operations, parameters, payloads, and validation rules, all using JSON or YAML. There were a few other similar projects (mainly RAML, API Blueprint) but they all fell out of use, and OpenAPI became the champion, especially with v3.0 and v3.1 improved on this base. - **AsyncAPI**: In 2017, AsyncAPI released a v1.0 of a new specification to help event-driven architectures describe their APIs in a similar way to OpenAPI, in fact it was a fork. It includes support for common message brokers such as Apache Kafka and RabbitMQ amongst many others. ## What Is API Design-First? API Design-First means you define the API's contract before writing any application code. This contract includes things like: - The endpoints (URLs) and their HTTP methods (GET, POST, etc.) - The structure and validation rules of resources and collections. - Authentication rules (like API keys, OAuth 2.x, OpenID). - Errors that could be expected, and an example of their structure. At first this all seems like extra work, but much like writing tests for an application, it will eventually speed up the delivery of APIs, saving everyone time and money, and reduce costly rewrites onces not-quite-right APIs make it to beta, or even worse get into production. ## Why Design-First? Here's why design-first is worth the effort: 1. **Clear communication**: Everyone – from frontend developers to testers to external users – knows what the API does. 2. **Parallel work**: Frontend and backend teams can work at the same time. Mock APIs can be set up using the design, so development doesn't have to wait. 3. **Consistency**: It's easier to enforce standards when the contract is agreed upon first. 4. **Automation**: You can auto-generate documentation, code snippets, and even parts of the implementation using tools. 5. **Version control**: It's easier to track and manage changes to the API over time. ## API Description Formats and How They Help Describing an API is the most important part of the API design-first workflow after planning is done, and for anyone building REST/RESTish APIs, the API description format of choice is OpenAPI. For anyone working with event-driven architectures the format of choice is AsyncAPI. OpenAPI documents are written in JSON or YAML, making them machine-readable, and somewhat human-readable too. They contain all the information needed to describe the interface of an API: requests, responses, reusable components, etc. ``` openapi: 3.0.0 info: title: Example API version: 1.0.0 paths: /users: get: summary: Get a list of users responses: '200': description: A list of users content: application/json: schema: type: array items: type: object properties: id: type: integer name: type: string ``` This snippet describes an API endpoint `/users` that responds with a list of users when a client sends a GET request. It then describes the responses a client could expect to see, with the status codes (e.g.: 200), content types (e.g.: `application/json`), then gets stuck into the `schema` which will outline the shape of the JSON. If you're just getting started with OpenAPI, we're here to help you on your journey. We've put together a guide to help you [learn OpenAPI from scratch](https://docs.bump.sh/guides/openapi/specification/v3.1/understanding-structure/basic-structure/?ref=apisyouwonthate.com), starting from the basic structure and going through every part of the functionality. AsyncAPI works in a very similar way, but instead of describing endpoints you describe "publishers" and "consumers". ``` asyncapi: 3.0.0 channels: user/signedup: address: user/signedup messages: publishUserSignedUp.message: $ref: '#/components/messages/userSignedUp' operations: publishUserSignedUp: action: send channel: $ref: '#/channels/user~1signedup' messages: - $ref: '#/channels/user~1signedup/messages/publishUserSignedUp.message' ``` ## Comparing Design-first and Code-first For years the API Code-first approach was the way to build an API. You'd sketch out the API you want to build on a whiteboard, then before that was even done somebody would be generating controllers and views in their favourite programming language and firing JSON around. The goal was always to get coding as fast as possible, so that clients could start integrating with it as soon as a prototype was ready. The rush to get coding often meant the first version clients get to see is not really anything like what they want, so a lot of time gets lost and wasted recoding controllers and doing database migrations. At some point everyone runs out of time and they have to go to production with whatever they have, even if it's a mess for clients to work with, and everyone just agrees to fix it all later in v2.0... For example, when OpenAPI is utilized in this approach, it is usually as annotations or code comments, popped into the application somewhere near the code it's describing, with the hope being that a developer will remember to update both at the same time. These annotations can then be exported to an `openapi.yaml` document which can be displayed as documentation or generate SDKs. ``` class UserController { @OpenApi( path = "/users", method = HttpMethod.POST, // ... ) public static void createUser(Context ctx) { // ... } } ``` Sadly this approach relies entirely on conflating proximity with accuracy. The annotations and code just a few lines below would often tell two completely different stories. Anyone who has been building APIs for more than a few years has probably done this and felt the pain, which is why so many API teams are starting to leverage the API design-first workflow. Here's a quick look at the two workflows for comparison. ![](https://storage.googleapis.com/bump-blog-resources/2025-01-27-dev-guide-api-design-first/code-first-design-first.png) Whilst there are a few more steps, the time invested on agreeing a contract early on brings massive time benefits through the rest of the API lifecycle. Combining the API-Design-first workflow with OpenAPI/AsyncAPI specifically allows for amazing benefits: 1. **Readable by humans and machines**: The YAML/JSON format means it's clear for developers and allows for API design reviews / governance with teams that don't have to read multiple programming languages. 2. **Interactive docs**: API Documentation generators like Bump.sh turn OpenAPI/AsyncAPI documents into interactive documentation, showing off parameters and examples, so clients can quickly and easily work with the API. 3. **Mock servers**: Tools like Microcks and Wiretap can use the API descriptions to simulate the API, allowing parallel development of API and client applications, and allowing feedback to come in early and often. 4. **Server-side Validation**: Instead of rewriting all of your validation logic in docs and code, you can use the API descriptions to power your application, making absolute certain the the documentation matches the implementation and reducing time spent writing code. 5. **Contract Testing**: Use automated tools to probe your API implementation based off the API descriptions, and add assertions to existing test suites saying "does this response match what it says in the API description", further ensuring the two are in agreement and saving time writing complicated contract testing by hand. 6. **Code generation**: Many tools generate client libraries or server stubs directly from an OpenAPI/AsyncAPI document, saving loads of time. 7. **API Style Guides**: Style guides are hard to enforce against code, developers need to check them manually, but with OpenAPI/AsyncAPI you can enforce standards on the API from the very first endpoint that is described. Anyone who has written API documentation by hand knows that it takes forever and is usually bad and outdated very quickly, so the fact that you have entirely accurate documentation from the start is a huge benefit for most teams. These other benefits may not have ever been considered, they were just things that you spent infinite time doing by hand and had never even considered automating, but when you combine them altogether in a single workflow your team becomes unstoppable. Speed and accuracy both go through the roof, reducing time, cost and client frustration with your API. ## TypeSpec Making OpenAPI Easier If you're looking at this thinking "I want all of those benefits, but writing up a lot of YAML sounds annoying" then take a look at [TypeSpec](https://docs.bump.sh/guides/openapi/accelerating-youropenapi-spec-generation-with-typespec/?ref=apisyouwonthate.com). Released by Microsoft, TypeSpec is a TypeScript-based DSL (Domain-Specific Language) for designing HTTP APIs. The main goal of TypeSpec is to split the language used for "design" and "description" in two. The design phase is more about ideating and things change quicker, and the description is more of an artifact of that process, but OpenAPI was essentially one language for both. OpenAPI is more verbose than any DSL could be, because it's written in JSON/YAML and that has limitations. You end up with a lot of text files, and the more you split your API description into multiple documents, the trickier it can be to rename things and keep all references up to date. Having the design phase handled with TypeScript allows rapid change to the whole model, with autocomplete, bulk renaming, and type-strict modelling of all your data. Later when it comes time to deploy documentation, run mock servers, do security checks, lint with style guides, etc. then TypeSpec does not have anywhere near as much tooling as OpenAPI, so you can say "ok, that design looks good, export OpenAPI" and run it through all of those tools, getting the best of both worlds. ## Wrapping Up API Design-First is all about getting the API's design nailed down before jumping into coding. It helps teams work faster, stay consistent, and avoid costly mistakes later on. OpenAPI has become the standard for REST APIs, making it easy to design, document, and manage APIs. AsyncAPI brings this same power to the event-driven API world. Tooling has evolved massively in the last few years to support these standards, so you aren't constantly having to convert things into multiple formats or try to duct-tape infinite tools together with no common source of truth. ### A Golden Age for APIs URL: https://apisyouwonthate.com/newsletter/golden-age-for-apis/ Last updated: 2025-03-03T12:38:47.000Z What an exciting time to build APIs! The growing popularity of Model Context Protocol (MCP) has put APIs center-stage in the world of AI tooling, making agents more and more powerful for automation. API building tools are constantly getting better, and the newer tools are pushing the design-first approach more and more. To top things off, OpenAPI is growing with additions like the Arazzo spec. There has never been a better time to build APIs. \-- Alexander ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2025/03/generate-mcp-server-NDS-2.png) ## Connect your APIs To AI with Speakeasy Speakeasy can now bundle a runnable MCP server in TypeScript SDKs that works with Claude, Cursor and many other apps. [Learn more](https://www.speakeasy.com/post/release-model-context-protocol?ref=apisyouwonthate.com) ## The API Round-Up *API News, links, and tools from around the web* ### [OpenAPI in Design-First Development](https://semaphore.io/blog/lorna-mitchell?ref=apisyouwonthate.com) The excellent Lorna Mitchell is on another podcast about OpenAPI and how software teams can fully harness its potential. Lorna's content is always top-notch, and this one is worth listening to. ### [Building world-class APIs with API Platform for Laravel](https://www.youtube.com/watch?v=DJJt%5F2V26V8&list=PLMdXHJK-lGoBWLQkMSyUORJkGk3ou27YI&index=18&ref=apisyouwonthate.com) Laracon EU had some fantastic talks this year, but if you missed them, this talk on API Platform is a must-see. API Platform is the framework for building world-class APIs with Laravel. Get started with this deep-dive session from the conference. ### [What's New in GraphOS: Apollo Winter 25 Release](https://www.apollographql.com/blog/whats-new-in-graphos-apollo-winter-25-release-apollo-connectors-native-query-planner-improved-tools-and-more?ref=apisyouwonthate.com) Apollo provides fantastic tools for building and querying GraphQL APIs and recently released connectors for REST APIs. You can now easily integrate a REST API into your GraphQL schema. Read the release for the full details. ### [TanStack Router and Query making REST easy](https://tanstack.com/router/latest?ref=apisyouwonthate.com) React is one of the go-to tools for building web applications nowadays. Recently, TanStack, responsible for excellent libraries such as React Query, released a router and an entire framework for building web applications. These libraries are worth mentioning as they shine with and encourage good REST API practices. Check them out if you have not already done so. ### [TypeScript SDKs for the Agentic AI ecosystem](https://www.speakeasy.com/post/release-model-context-protocol?ref=apisyouwonthate.com) Speakeasy now includes a Model Context Protocol (MCP) server in all TypeScript SDKs, enabling seamless AI agent integration with APIs. This lightweight wrapper helps AI agents interact more effectively with your API. This is a huge update, so read the release blog for all the details. ### [OpenAPI Arazzo Specification](https://www.youtube.com/watch?v=ivUyjJEYmKg&ref=apisyouwonthate.com) You will undoubtedly have heard about the Arazzo Specification by now but may still be a little lost about what it is or how to use it. Repetition is the key to learning, so check out this conversation with Frank Kilcommins of SmartBear on the specification. ## APIs You Won't Hate The latest from the team at API's You Won't Hate. ### [API Design Basics: Rate Limiting](https://apisyouwonthate.com/blog/api-design-basics-rate-limiting/) Phil is back at it again with some API basics on Rate Limiting. Phil talks you through why you should care about rate limiting, the different approaches and rounds of the posts, and some alternatives to Rate Limiting. ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2025/02/st-1.png) ## Open Source User Authentication Trusted by 2K+ production applications including HackerRank and ESPN [Learn More](https://supertokens.com/?utm%5Fsource=api1) ## From our Community Articles written and shared in our free [Slack community](https://slack.apisyouwonthate.com/?ref=apisyouwonthate.com). ### [Build, Document, and Secure a REST API](https://zuplo.com/blog/2025/02/03/laravel-api-tutorial?ref=apisyouwonthate.com) Adrian Machado and Steve McDougall published a guide to API development in Larvel. The guide goes beyond the simple Hello World examples and walks you through the whole process. Thank you for sharing, guys. ### [Find us on Bluesky](https://bsky.app/profile/apisyouwonthate.com?ref=apisyouwonthate.com) Here's your regular reminder that we're on Bluesky, and we'd love to follow you there! Chase us down at [@apisyouwonthate.com](https://bsky.app/profile/apisyouwonthate.com?ref=apisyouwonthate.com), and feel free to check out [the APIs You Won't Hate starter pack](https://bsky.app/starter-pack/mikebifulco.com/3l7l72ptyfj2t?ref=apisyouwonthate.com) to find some community members to follow. Looking forward to seeing you there! ## Support APIs You Won't Hate When you become an member, you'll get access to members-only content while directly supporting our work. Your support helps us to keep making resources for the API community. [Become a member today](https://apisyouwonthate.com/#/portal) Thanks so much to our members: **Kin L, Juxt, Alex R, Nolan S, Frank, James D, Bill D, Rich, and Umair**. Your support means the world to us! Until next time, Alexander, Phil & Mike ### API Design Basics: Rate Limiting URL: https://apisyouwonthate.com/blog/api-design-basics-rate-limiting/ Last updated: 2025-03-03T12:29:36.000Z Rate limiting is the art of trying to protect the API by telling "overactive" API consumers to calm down a bit, telling clients to reduce the frequency of theirrequests, or take a break entirely and come back later to avoid overwhelming the API. ## Why bother with rate limiting The main reason for rate limiting is to keep an API running smoothly and fairly. If all clients could fire off requests as fast as they like, it's only a matter of time before something breaks. A spike in traffic (whether accidental or malicious) can overwhelm servers, leading to slowdowns, crashes, or unexpected high infrastructure costs. Rate limiting is also about fairness. If there are loads of users accessing an API, it's important to make sure one consumers mistakes do not affect another. For public APIs, it's about making sure no one user can hog all the resources. For businesses, this could be different limits for free and various paid tiers to make sure profit margins are maintained. ## How does API rate limiting work? How can an API know when a client is making too many requests? That's where rate limiting comes in. Rate limiting is a system that tracks the number of requests made by a particular target (based on IP address, API key, user ID, or other headers), within a defined time window. The way this is implemented can vary, but the general process is the same: - **Request Received** \- A client makes a request to the API, asking for some data or to perform an action. - **Identify the client** \- The system identifies the client making the request, usually by looking at the IP address, API key, or other identifying information. - **Check usage history** \- The system checks how many requests the client has made in the current time window, and compares it to the limit. - **Allow or deny the request** \- If the client has made too many requests, the system denies the request with a `429 Too Many Requests` [status code](https://file+.vscode-resource.vscode-cdn.net/api-design/status-codes?ref=apisyouwonthate.com). If the client is within the limit, the request is processed as normal. ## Different rate limiting strategies There are a few different strategies for rate limiting, each with its own advantages and disadvantages. ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2025/02/rate-limiting-strategies.gif) - **Token bucket:** the system has a bucket of tokens, and each request consumes a token. Tokens are added to the bucket at regular intervals, 100 tokens a minute, or 1,000 tokens per hour. If there are no tokens left, the request is denied. Clients are rewarded for taking time out and accrue more tokens as they do. This can lead to a lot of sudden bursts of activity, but should generally keep an average amount of traffic going through the system. - **Fixed window:** the system sets a fixed limit for a specific time window. For example, "Make 100 requests per minute." This is the most common approach, but it can lead to very "lumpy" API traffic, where many clients are making the maximum number of requests at the start of a minute. This means an API can be stressed at the start of each minute and bored for the rest of it. - **Sliding log:** instead of using the same time windows for all clients, the system sets a maximum number of requests for any 60 second period. This avoids the lumpy traffic concerns of many clients all maxing out at the start of the window, then doing nothing for the rest of it, as they would all have their own windows starting and stopping at different times depending on their usage. - **Sliding window:** is a dynamic approach, adjusting limits based on real-time traffic patterns to optimize system performance and ensure fair access for all. This can be more complex to implement, but it can lead to a more efficient use of resources and a better experience for API consumers. ## Different limit targets There are a lot of choices to be made when it comes to rate limiting, and the first is: who or what are we trying to limit? Here are a few common targets for rate limiting: - **User-specific rate limits:** Identifying a user by their API key or user ID and setting a rate limit for that user. This is useful for ensuring that no single user can overwhelm the API and slow it down for others. - **Application-specific rate limits:** Identifying an application by its API key and setting a rate limit for that application. This is useful for ensuring that a misconfigured application cannot affect stability for other applications. - **Regional rate limits:** Manage traffic from different geographic regions, to make sure an API can continue to service critical regions, whilst still allowing other regions to access the API. ### Implementing rate limiting in HTTP Rate limiting can be implemented at various levels, from the network layer to the application layer. For HTTP APIs, the most common approach is to implement rate limiting at the application layer with HTTP "middlewares" that keep track of these things, or API gateways which handle rate limiting like Zuplo, Kong, Tyk, etc. Wherever the rate limiting is implemented, there are a few standards that can be leveraged to avoid reinventing the wheel. The first is to return a HTTP error with a [status code](https://file+.vscode-resource.vscode-cdn.net/api-design/status-code?ref=apisyouwonthate.com) of `429 Too Many Requests` (as defined in [RFC 6585](https://www.rfc-editor.org/rfc/rfc6585.html?ref=apisyouwonthate.com)). This tells the client that they've exceeded the rate limit and should back off for a while. ```http HTTP/2 429 Too Many Requests ``` Instead of leaving the client to guess when they should try again (likely leading to lots of poking and prodding adding more traffic to the API), the `Retry-After` header can be added to a response with a number of seconds, or a specific time and date of when the next request should be made. ```http HTTP/2 429 Too Many Requests Retry-After: 3600 ``` Why not also add some [proper error response](https://file+.vscode-resource.vscode-cdn.net/api-design/errors?ref=apisyouwonthate.com) to explain why the request was rejected, for any API consumer developers not familiar with these concepts. ```http HTTP/2 429 Too Many Requests Content-Type: application/json Retry-After: 3600 { "error": { "message": "Rate limit exceeded", "code": "rate_limit_exceeded", "details": "You have exceeded the rate limit for this API. Please try again in 1 hour." } } ``` Doing all of this makes it clear to the client that they have entered a rate limit, and give them the information they need to know when they can try again, but there is more that can be done to make this more user friendly. ### Rate limit headers Documenting the rate limit in the response headers can help API consumers to understand what's going on. There are various conventions for headers to help consumers understand more about what the rate limiting policy is, how much of the limit has been used, and what is remaining. GitHub for example uses the `X-RateLimit-Limit`, `X-RateLimit-Remaining`, and `X-RateLimit-Reset`. Twitter uses `X-Rate-Limit-Limit`, `X-Rate-Limit-Remaining`, and `X-Rate-Limit-Reset`. Similar but different, which causes all sorts of confusion. Designing an API to be the most user friendly means relying on standards instead of conventions, so it's worth looking at the [RateLimit header draft RFC](https://datatracker.ietf.org/doc/draft-ietf-httpapi-ratelimit-headers/?ref=apisyouwonthate.com) which outlines one new `RateLimit` header to cover all those use cases and a few more. The following example shows a `RateLimit` header with a policy named "default", which has another 50 requests allowed in the next 30 seconds. ``` RateLimit: "default";r=50;t=30 ``` The `RateLimit` header focuses on the current state of the various quotes, but it doesn't provide information about the policy itself. The same draft RFC also outlines a `RateLimit-Policy` header which can be used to provide information about how the policy works. This example shows two policies, "default" and "daily". The "default" policy has a quota of 100 requests and a window of 30 seconds, while the "daily" policy has a quota of 1000 requests and a window of 86400 seconds (24 hours). ```http RateLimit-Policy: "default";q=100;w=30,"daily";q=1000;w=86400 ``` Combining these two headers can provide a lot of information to API consumers to know what the rate limits are, how much they have used, and when they can make more requests. This can be a bit of work to set up, but it allows API consumers to interact with an API more effectively, with less frustration, and keep everything running smoothly. ### Alternatives to Rate Limiting Some people argue that rate limiting is a blunt tool. It can be frustrating for users who hit the limit when they're trying to get work done. Poorly configured rate limiting can be fairly arbitrary. Consider an API that could theoretically handle 1000 requests per second. If there are 1000 users, each with a rate limit of 1 request per second, the API would be maxed out. If that same API with 1000 users and only two of them are using up the their maximum quotas, then the API could absolutely handle the load, and the API is sitting their underutilized sitting around waiting for potential activity which wont come. Not only is that a waste of server resources (hardware, electricity, CO2 emissions), but it's also frustrating for those users who are constantly being told to calm down when they could be using the API to handle more activity; activity which could be profitable. One alternative approach is known as **backpressure**. This is a more dynamic system which tells clients to ease up when the system is under strain, with a `503 Service Unavailable` response with a `Retry-After` header. This could be applied to the entire API, to specific users, or even specific endpoints that are more resource intensive. Quota-based systems are another alternative. Instead of measuring requests per second or minute, users are assigned a monthly allowance. This works well for subscription-based APIs, where users pay for a certain amount of access. If they make a mistake and use up their quota too quickly, they can buy more, and other API consumers can still access the API. This lends itself better to auto-scaling up (and back down) based on number of active users and usage. ### Final Thoughts Rate limiting begins as a technical safeguard for an API (which makes managing it easier) but ensures nobody is hogging resources (which keeps users happily using the product). It's worth thinking about where and how to implement it, how to communicate it, and how to make it as user-friendly as possible. It's not always simple for junior developers to figure out how to work with rate limiting and they might not know all the HTTP status codes and headers. The more tooling you can provide to assist your users with responding to your rate limiting, the better. --- *Thank you to* [*Speakeasy*](https://www.speakeasy.com/?ref=apisyouwonthate.com) *for sponsoring the time to write this article, as part of their new* [*API Design Guide*](https://www.speakeasy.com/api-design?ref=apisyouwonthate.com) *which you should absolutely go and check out.* ### ...Surely you mean to call it OpenAPI? URL: https://apisyouwonthate.com/newsletter/surely-you-mean-to-call-it-openapi/ Last updated: 2025-02-17T10:37:41.000Z The OpenAPI specification was originally called Swagger before SmartBear transferred it to the OpenAPI Initiative. Despite the official name change, the term "Swagger" persists. For me, Swagger represents all the old, janky tools I have grown to dislike, while OpenAPI represents tools like Stoplight that feel more modern. The Swagger name sticks around primarily due to its deep roots in enterprise software and older frameworks like Spring Boot, whose ecosystem moves slower, and its tools are more challenging to update. While OpenAPI is now the correct term for the specification, the Swagger name likely won't fade away anytime soon. \-- Alexander ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2024/08/speakez-1.jpg) ## Speakeasy: Build APIs Your Users Love Your API deserves a great developer experience. Get one by using Speakeasy to generate idiomatic, type-safe SDKs from OpenAPI [Start Generating](https://www.speakeasy.com/?utm%5Fsource=apisyouwonthate&utm%5Fcampaign=240701) ## The API Round-Up *API News, links, and tools from around the web* ### [API Linting with Vacuum](https://bump.sh/blog/api-linting-with-vacuum?utm%5Fsource=linkedin&utm%5Fmedium=social&utm%5Fcampaign=vacuum&utm%5Fcontent=bump) I have mentioned Vacuum before, but Phil provides a step-by-step guide in this blog post. Vacuum is an open-source OpenAPI linter written in Go. It is fast and also supports the Spectral rule sets. ### [OpenAPI Spec for Rusts Crates API](https://blog.rust-lang.org/2025/02/05/crates-io-development-update.html?ref=apisyouwonthate.com) Crates.io is the place to download and publish Rust libraries. Recently, they published an OpenAPI spec. Unfortunately, due to security concerns, they did not ship a viewer, but they made good progress nonetheless. ### [OpenAPI Initiative: New Standards](https://thenewstack.io/openapi-initiative-new-standards-and-a-peek-at-the-roadmap/?ref=apisyouwonthate.com) The OpenAPI landscape is changing. Overlay and Arazzo specifications are enhancing API standardisation and orchestration. Lorna Mitchell walks us through the changing landscape and what's next for OpenAPI. ### [Node File Router](https://node-file-router.js.org/?ref=apisyouwonthate.com) A file-based router for Node inspired by Next.js and Nuxt works with Bun, Node, Express, and more. It has zero dependencies and supports TypeScript, Common JS, and ESM. It's time to build some file-based APIs. ### [Nock V14](https://github.com/nock/nock?ref=apisyouwonthate.com) Nock dropped version 14 over a week ago, and I could not be more excited. Nock is my go-to HTTP server mocking library for Node.js. For example, it is excellent for mocking HTTP requests to the AWS API. After listening to API calls, the recorder function prints out the code for you, which is fantastic. Not to mention, the new version brings support for native fetch. ### [Building Great APIs with Powertools](https://www.youtube.com/watch?v=bMH5634gpCM&ref=apisyouwonthate.com) Learn how to build REST APIs on AWS Lambda using Python and Lambda Powertools. Check out how Powertools can help you with routing, validation, and OpenAPI support. You will be up and running with serverless APIs in no time. ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2025/02/st-1.png) ## Open Source User Authentication Trusted by 2K+ production applications including HackerRank and ESPN [Learn More](https://supertokens.com/?utm%5Fsource=api1) ## APIs You Won't Hate The latest from the team at API's You Won't Hate. ### [OpenAPI Bundling Tools, compared](https://apisyouwonthate.com/blog/openapi-bundling-tools-compared-blog-post/) Compare Swagger CLI, JSON Schema Ref Parser, and Redocly CLI for OpenAPI bundling. Discover which tool best suits your API documentation needs. ### [APIs You Won't Hate on YouTube, too!](https://www.youtube.com/@APIsYouWontHate?ref=apisyouwonthate.com) A community member recently asked if we could repost podcast episodes on YouTube. Turns out - we've been doing exactly that for ages, but we don't talk about it much! We even publish a pod with video every now and again... [](https://apisyouwonthate.com/blog/openapi-bundling-tools-compared-blog-post/?utm%5Fsource=chatgpt.com) --- ## From our Community Articles written and shared in our free [Slack community](https://slack.apisyouwonthate.com/?ref=apisyouwonthate.com). ### [Share Virtual Keys, not API Keys](https://subkeys.io/?utm%5Fsource=apisyouwonthate&utm%5Fmedium=slack&utm%5Fcampaign=mission1&utm%5Fcontent=rcTMszq&dub%5Fid=2mRtyVcoMcvzPFpS) Burch shared this fantastic project he has been working on in our Slack community. Subkeys is a clever platform that lets developers track API usage with Virtual Keys, making it easy to catch and fix costly API issues in seconds. ## Support APIs You Won't Hate When you become an member, you'll get access to members-only content while directly supporting our work. Your support helps us to keep making resources for the API community. [Become a member today](https://apisyouwonthate.com/#/portal) Thanks so much to our members: **Kin L, Juxt, Alex R, Nolan S, Frank, James D, Bill D, Rich, and Umair**. Your support means the world to us! Until next time, Alexander, Phil & Mike ### HTTP Traffic vs OpenAPI URL: https://apisyouwonthate.com/newsletter/http-traffic-vs-openapi/ Last updated: 2025-02-03T12:58:15.000Z [Akita](https://www.akitasoftware.com/?ref=apisyouwonthate.com) will shut down in March after being acquired by Postman last year and moving to [Postman Insights](https://learning.postman.com/docs/insights/overview/?ref=apisyouwonthate.com). Optic, another HTTP traffic monitoring tool, was recently [acquired by Atlassian](https://www.atlassian.com/blog/announcements/optic-acquisition?ref=apisyouwonthate.com). These tools provide many features, from building API docs based on traffic to setting standards, detecting changes, etc. Generating docs is the feature I am most interested in. It is a fantastic tool for APIs released without documentation, but will this tool negatively impact the design-driven process? It's an interesting space to watch as these tools grow and more startups keep getting acquired. \-- Alexander --- ## The API Round-Up *API News, links, and tools from around the web* ### [Exploiting McDonald's APIs to Hijack Deliveries and Order Food for a Penny](https://eaton-works.com/2024/12/19/mcdelivery-india-hack/?ref=apisyouwonthate.com) Ever wanted to order a mountain of McDonald's hash browns for a penny? Vulnerabilities in McDonald's India's McDelivery system allowed for just that. Don't worry; the reporter earned a $240 voucher for reporting the cracks in the API layer. ### [Generating an API client to call any OpenAPI-described API](https://learn.microsoft.com/en-us/openapi/kiota/overview?ref=apisyouwonthate.com) Kiota is a generator from Microsoft that allows you to create an API client for any OpenAPI document in the command line. It supports many languages, including C#, CLI, Go, Java, PHP, Python, Ruby, and TypeScript. ### [Express.js: Triumphs of 2024 and an Ambitious 2025](https://expressjs.com/2025/01/09/rewind-2024-triumphs-and-2025-vision.html?ref=apisyouwonthate.com) Everything old is new again. Partially driven, I think, by a fresh injection of energy into the Node ecosystem and more modern frameworks like Hono, Express is working hard at improving itself. A reflection on what the changes in 2024 mean for this extremely popular web framework and where it plans to go. ### [Flexible and Open Source API Mocking](https://github.com/wiremock/wiremock?ref=apisyouwonthate.com) WireMock is a flexible API mocking tool designed for testing and development. It enables developers to simulate HTTP-based APIs, create stubs, and capture requests, making testing integrations and building reliable applications easier. ### [Bun 1.2: A Big Step Forward for the Fast JS/TS Runtime](https://javascriptweekly.com/link/164761/bbfce21b92?ref=apisyouwonthate.com) *Alex, why are you mentioning a JS runtime in an API newsletter? We don't care about JavaScript.* The new Bun release comes with some interesting improvements for building server-side applications, including implementing the S3 API. Its upload is much faster than Node's, so building APIs that deal with files just got faster. ### [Streamline Your Workflow with Swift OpenAPI Generator](https://www.youtube.com/watch?v=y%5F%5FRdURMZys&ref=apisyouwonthate.com) Swift OpenAPI Generator took centre stage at Swift Connection, showcasing how it eliminates tedious networking code through automated, type-safe implementations. Swift and OpenAPI write network code so you can focus on business logic and great UX. ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2024/08/speakez-1.jpg) ## Speakeasy: Build APIs Your Users Love Your API deserves a great developer experience. Get one by using Speakeasy to generate idiomatic, type-safe SDKs from OpenAPI [Start Generating](https://www.speakeasy.com/?utm%5Fsource=apisyouwonthate&utm%5Fcampaign=240701) ## APIs You Won't Hate The latest from the team at API's You Won't Hate. ### 🎙️ [Rails APIs & Clickfunnels, with Rich Stone](https://apisyouwonthate.com/podcast/clickfunnels-and-rails-apis-with-rich-steinmetz/) In the latest episode of the podcast, Phil & Mike sit down with community member Rich Steinmetz to talk about Clickfunnels, its API surfaces, and what it's like to build a developer product with Ruby on Rails and BulletTrain. ### [API Design Basics: Security](https://apisyouwonthate.com/blog/api-design-basics-security/) Phil walks us through some key principles for securing APIs, covering key topics such as secrets in URLs, validation, designing with the least privilege, and more. Phil also helps you get on top of this early by recommending rulesets for validating your OpenAPI. --- ## From our Community Articles written and shared in our free [Slack community](https://slack.apisyouwonthate.com/?ref=apisyouwonthate.com). ### [API Design on Reddit](https://www.reddit.com/r/apidesign/?ref=apisyouwonthate.com) Phil shared another API community in our Slack group this week, which focuses on API Design. It's another place to share your own content and get inspiration from others on all things API. ## Support APIs You Won't Hate When you become an member, you'll get access to members-only content while directly supporting our work. Your support helps us to keep making resources for the API community. [Become a member today](https://apisyouwonthate.com/#/portal) Thanks so much to our members: **Kin L, Juxt, Alex R, Nolan S, Frank, James D, Bill D, Rich, and Umair**. Your support means the world to us! Until next time, Alexander, Phil & Mike ### API Design Basics: Designing for Security URL: https://apisyouwonthate.com/blog/api-design-basics-security/ Last updated: 2025-02-03T12:35:10.000Z [](https://github.com/speakeasy-api/speakeasy-registry/blob/main/web/packages/marketing-site/src/pages/api-design/security.md?ref=apisyouwonthate.com#designing-for-api-security) When creating an API you're essentially opening a door to the outside world. Who is allowed through, what they can carry, and where they're allowed to go is incredibly important. In this guide we'll see how design choices made early on impact the security of an API once it's built. Many API security problems come down to coding errors or misconfigured infrastructure, but this guide focuses more on the foundational API design decisions that effect the security of your API from day one. ## Why care about API security [](https://github.com/speakeasy-api/speakeasy-registry/blob/main/web/packages/marketing-site/src/pages/api-design/security.md?ref=apisyouwonthate.com#why-care-about-api-security) APIs often protect sensitive data or critical functionality. Whether it's a payment gateway, a medical records system, or a social media app, your API needs to be designed with security in mind to protect both the organization and its users. API security breaches in 2022 caused losses worth [$12–$23 billion](https://www.darkreading.com/application-security/api-security-losses-billions-complicated?ref=apisyouwonthate.com) in the US and [$41–$75 billion globally](https://techwireasia.com/2022/06/api-vulnerabilities-costing-businesses-up-to-us75-billion-annually/?ref=apisyouwonthate.com). To pick just a few examples, since the introduction of General Data Protection Regulation (GDPR), Amazon Europe were fined €746m in 2021, Meta was fined €1.2bn in 2023, and - to show it's not just tech giants - Marriott International (a hotel chain) got stuck with a £20m fine in 2022. More countries and regions strengthening privacy laws along the lines of GDPR: California Consumer Privacy Act (CCPA), Canada's Personal Information Protection and Electronic Documents Act (PIPEDA), and Brazilian General Data Protection Law (LGPD). Even if data breaches and leaks don't result in hefty fines, the reputational damage that comes with leaking customers private information can be a big issue, so it's important to do everything possible to keep APIs secure. Let's walk through some key security concepts in API design and how your decisions can make or break your API's defenses before it's even built. ## Principle #1: Design with the least privilege [](https://github.com/speakeasy-api/speakeasy-registry/blob/main/web/packages/marketing-site/src/pages/api-design/security.md?ref=apisyouwonthate.com#principle-1-design-with-the-least-privilege) **Every API consumer should only have access to what they need and nothing more.** Imagine you're designing an API for an e-commerce platform. A customer should be able to view their order history, but not other customers' orders. Similarly, a "staff" user might need access to refund functionality but shouldn't necessarily see sensitive payment details. **What Could Go Wrong**: Failure to verify this could lead to Insecure Direct Object References (IDOR), a common flaw where attackers can manipulate identifiers to access data they shouldn't. **Design Decision**: The first issue to make sure you're using access control on the endpoints, restricting the the specific user, or to a user with the right role. ``` GET /orders/{orderId} Authorization: Bearer {access_token} ``` The application should verify that the `orderId` belongs to the authenticated user, unless the user has a role like `admin`. Refund logic and payment details can be split onto their own endpoints: ``` POST /orders/{orderId}/refund Authorization: Bearer {staff_access_token} ``` ``` GET /orders/{orderId}/payments Authorization: Bearer {admin_access_token} ``` This allows staff handle refunds, but does not leak sensitive credit card information to as many people within the company, whilst still making it possible to escalate customer problems to a higher access user. Better yet, **the payments collection is not even on the API**, it's something only viewable in an admin backend system thats protected with a firewall and VPN. ## Principle #2: Always validate input [](https://github.com/speakeasy-api/speakeasy-registry/blob/main/web/packages/marketing-site/src/pages/api-design/security.md?ref=apisyouwonthate.com#principle-2-always-validate-input) **Inputs should be treated as untrusted, even if the API is "internal" or "private".** Any incoming API traffic could be compromised in some way, even if it's considered to be a trusted source. An API could suddenly become public: either intentionally when infrastructure teams move things around, or accidentally when somebody de-compiles an iOS application or sniffs traffic to find an API that people thought was hidden. Even if an API is firewalled off from public traffic, another API or service could have been hacked giving them access to your API. It's best to treat everyone with suspicion, and validate all inputs as strictly as possible. **What Could Go Wrong**: Malicious data could be introduced, or private information leaked, leading to any number of issues. People could delete invoice payment records and updating payment details to trigger a second payment to the wrong person. They could change passwords for users so they can log in as them to access information and processes not even available in the API. **Design Decision**: Set strict rules for which properties are editable, which can be returned, and set strict validation rules for these properties. This can be described in OpenAPI early on utilizing `readOnly`, `writeOnly`, `required`, setting `additionalProperties: false`. [Learn more about additionalProperties](https://www.speakeasy.com/guides/openapi/additionalproperties?ref=apisyouwonthate.com). This means when the API is developed the OpenAPI can be used for integration testing to poke and prod to see if extra properties can sneak though. Comical examples of this was somebody hacking GitHub and Rails to update the `created_at` date to have the year 3012\. This attack is known as Bender from the Future (a reference to TV show Futurama) and made the concept of "Mass Assignment" popular. Whitelist which specific properties should be allowed to be written/read in your planning documents and OpenAPI, and either use that OpenAPI document for validation and serialization, or test against it once they've built the API. ## Principle #3: Keep secrets out of the URL [](https://github.com/speakeasy-api/speakeasy-registry/blob/main/web/packages/marketing-site/src/pages/api-design/security.md?ref=apisyouwonthate.com#principle-3-keep-secrets-out-of-the-url) Sensitive information like API keys or tokens should never appear in URLs. Let's say your API allows filtering resources: ``` GET /products?search=blue&apiKey=my-secret-key ``` **What Could Go Wrong**: Logs, browser history, and proxies often store URLs. If an API key or sensitive data is passed in the URL, it's at risk of exposure. **Design Decision**: Always pass sensitive data through headers or the body of the request, not the URL. The body will be encrypted when HTTPS is used, but the URL is not. ``` GET /products?search=blue Authorization: Bearer my-secret-key ``` Using `Authorization` has the added benefit over generic custom headers like `X-API-Key` because it will alert HTTP caching tools to not reuse this response for other users by default. This is not simply about authorization though, there are lots of other "sensitive" things which should not go into the URL. Email addresses, social security numbers, anything that should not be leaked to the public in general. Pop it in the body instead. A `GET` method generally should not have a HTTP request body (behavior is undocumented, support is inconsistent, generally unadvisable), but the [QUERY draft RFC](https://httptoolkit.com/blog/http-search-method/?ref=apisyouwonthate.com) could be solution we're all looking for. ## Principle #4: Limit one-time URLs [](https://github.com/speakeasy-api/speakeasy-registry/blob/main/web/packages/marketing-site/src/pages/api-design/security.md?ref=apisyouwonthate.com#principle-4-limit-one-time-urls) Logins and file uploads often involve allowing a user to pass in a URL, which will then be downloaded or redirected to. ``` POST /products/{productId}/images Authorization: Bearer {access_token} Content-Type: application/json { "import_url": "http://hopefully-innocent-website.com/something.jpg" } ``` **What Could Go Wrong**: THis can be a big source of problems for an API, even if the use case is something small and simple like importing an avatar for a user. THe URL could be: - A malicious file the API is being asked to download. - A very large file the API will run out of resources trying to download. - Intercepted by a malicious actor on an infected network to change the DNS of that URL to another server which is malicious. **Design Decision**: The API design for image uploads could be changed to take a HTTP request with the image directly. ``` POST /products/{productId}/images Authorization: Bearer {access_token} Content-Type: image/png ``` People could still try to upload malicious files directly, but its easier to scan the incoming request body for problems and reject it. This can be done on the API gateway or via other threat detection like Cloudflare. With an API gateway in place, if this request is problematically large the gateway will reject the request without consuming any resources at all on the API server. Malicious users on an infected network could still be messing with DNS settings, but they would have to mess with the API in question - which should have proper HTTPS setup and be much harder for them to do. Compared to their ability to mess with `http://hopefully-innocent-website.com/` which may not be set up so well. ## Principle #5: Don't help competitors steal data [](https://github.com/speakeasy-api/speakeasy-registry/blob/main/web/packages/marketing-site/src/pages/api-design/security.md?ref=apisyouwonthate.com#principle-5-don-t-help-competitors-steal-data) Using auto-incrementing IDs as identifiers in an API makes it incredibly easy for malicious actors to glean insights into potentially sensitive data a business might not want to expose, or allow outright theft of an entire dataset. A startup tracking street art around the world (think Banksy, Bragga, and smaller artists) built an amazing unique database of user-generated photographs and locations of all sorts of graffiti, sculptures, installations, etc. This data was not available anywhere else on the Internet, but their website relied on two API endpoints: ``` GET /artworks/234 GET /users/6138 ``` **What Could Go Wrong**: Looking at the URL `GET /users/6138`, its not too hard to assume I can look at `GET /users/1`. If that shows me data, I can reasonably assume they have at least 6138 users, but to find the total I can easily make a script that `id+1` and counts every HTTP status 200 to show me how many users are in the system. It can also counts things like 404 or 410, to give a accurate number of how many active users versus inactive users, leaking a "churn rate" which could be embarrassing in the press of scare off investors. Using the same approach a client can hit `GET /artworks/1` and loop through with \`id+1\` to grab a hold of all that data, which helped that company populate their own database, making a new competitor quite easily, and with a slightly better app as they didn't have to spend time or money building the dataset in the first place. This put the original startup out of business. **Design Decision**: There are non-incremental or "hard to guess" system of identifiers instead. Standards like [UUID](https://www.rfc-editor.org/rfc/rfc9562.html?ref=apisyouwonthate.com) or [Snowflake](https://en.wikipedia.org/wiki/Snowflake%5FID?ref=apisyouwonthate.com) instead. Instead of having `/artworks/1` and `/artworks/2`, design the API to use UUID: ``` GET /artworks/c1b07800-b001-4ba9-8372-e0260cf25242 GET /artworks/4e44cf4a-fbe0-4630-983f-ccd55b7e4870 ``` There is no way for anyone to glean from this how many resources the API has, or guess the next one, without brute forcing the API with infinite arbitrary requests... ## Principle #6: Rate limiting and throttling [](https://github.com/speakeasy-api/speakeasy-registry/blob/main/web/packages/marketing-site/src/pages/api-design/security.md?ref=apisyouwonthate.com#principle-6-rate-limiting-and-throttling) Prevent abuse by controlling how frequently clients can interact with your API. Consider a public API endpoint for retrieving weather data: ``` GET /weather?city=London ``` **What Could Go Wrong**: Without rate limiting, a single client could make thousands of requests per second, overloading your servers and possibly causing a denial of service (DoS). **Design Decision**: Implement rate limiting at the design level. Define thresholds for various user roles: - Free users: 100 requests per hour - Paid users: 1,000 requests per hour Communicate these limits clearly in your API documentation and return appropriate status codes like `429 Too Many Requests` when limits are exceeded. ## Principle #7: Security through obscurity is not enough [](https://github.com/speakeasy-api/speakeasy-registry/blob/main/web/packages/marketing-site/src/pages/api-design/security.md?ref=apisyouwonthate.com#principle-7-security-through-obscurity-is-not-enough) An e-commerce platform for online stores (shops) provides a listing page with the revenue charts for their hosted shops. Inspecting the browser requests, an attacker can identify the API endpoints used as a data source for those charts and their pattern: `/shops/{shopName}/revenue_data.json`. Using another API endpoint, the attacker can get the list of all hosted shop names. With a simple script to manipulate the names in the list, replacing {shopName} in the URL, the attacker gains access to the sales data of thousands of e-commerce stores. With `/shops/{shopName}/revenue_data.json` clients could access all the sales. Even if its a special UUID for the shop, that might be good until somebody shares that UUID or another developers exposes the uuids elsewhere not realizing they're being used as security... Use proper auth for things that need to be hidden or it will be exposed. ## Open Web Application Security Project (OWASP) [](https://github.com/speakeasy-api/speakeasy-registry/blob/main/web/packages/marketing-site/src/pages/api-design/security.md?ref=apisyouwonthate.com#open-web-application-security-project-owasp) OWASP is an online community that produces freely available content to help organizations avoid making costly security mistakes with their software. The [OWASP API Security Project](https://owasp.org/API-Security/?ref=apisyouwonthate.com) helps focus specific on risks and problems that can effect insecure APIs, and illustrating how these risks may be mitigated. If you'd like to make sure your APIs are as secure as possible, it's worth reading through the [OWASP API Security Top 10: 2023 Edition](https://owasp.org/API-Security/editions/2023/en/0x00-header/?ref=apisyouwonthate.com) and keeping up to date with new editions when they're released. ### Tooling [](https://github.com/speakeasy-api/speakeasy-registry/blob/main/web/packages/marketing-site/src/pages/api-design/security.md?ref=apisyouwonthate.com#tooling) Much of this advice and more can be applied to an OpenAPI automatically to help whole teams make good decisions early on in the API design process. - [Vacuum](https://quobix.com/vacuum/?ref=apisyouwonthate.com) via the built in [OWASP Ruleset](https://quobix.com/vacuum/rules/owasp/?ref=apisyouwonthate.com). - [Spectral](https://github.com/stoplightio/spectral?ref=apisyouwonthate.com) with the [Spectral OWASP Ruleset](https://github.com/stoplightio/spectral-owasp-ruleset?ref=apisyouwonthate.com). ## Summary [](https://github.com/speakeasy-api/speakeasy-registry/blob/main/web/packages/marketing-site/src/pages/api-design/security.md?ref=apisyouwonthate.com#summary) API security isn't a bolt-on, it's mindset. By making deliberate design choices around authentication, authorization, data handling, and rate limiting, many of the pitfalls outlined here and in the OWASP API Security Top 10 can be avoided. Remember, every design decision is a trade-off. Security measures often add complexity or impact usability. Your goal is to strike the right balance, keeping the needs of both your users and your business in mind. There's no need to go to massive massive and intrusive lengths to secure information that is fine out in the public, but it is important to establish good practices for limiting interactions for more sensitive data. Maybe this means creating more than one API. --- *This article is sponsored by* [*Speakeasy*](https://www.speakeasy.com/?ref=apisyouwonthate.com)*, who have been hiring me to work on a larger guide on* [*API Design*](https://www.speakeasy.com/api-design?ref=apisyouwonthate.com)*. Go check it out, and consider using Speakeasy for your OpenAPI-based SDK generation needs.* ### Clickfunnels and Rails APIs with Rich Steinmetz URL: https://apisyouwonthate.com/podcast/clickfunnels-and-rails-apis-with-rich-steinmetz/ Last updated: 2025-02-03T12:24:10.000Z ## Show Notes - Rich Steinmetz ([https://richstone.io/meet-rich](https://richstone.io/meet-rich?ref=apisyouwonthate.com)) - Clickfunnels - [https://clickfunnels.com/](https://clickfunnels.com/?ref=apisyouwonthate.com) - Developer portal [https://developers.myclickfunnels.com/](https://developers.myclickfunnels.com/?ref=apisyouwonthate.com) - Bullet Train for Ruby on Rails - [https://bullettrain.co/docs/api](https://bullettrain.co/docs/api?ref=apisyouwonthate.com) - SuperScaffolding - [https://bullettrain.co/docs/super-scaffolding ](https://bullettrain.co/docs/super-scaffolding ?ref=apisyouwonthate.com) - Hyrum's Law - [https://www.hyrumslaw.com/](https://www.hyrumslaw.com/?ref=apisyouwonthate.com) #### Transcript \[00:00:00\] ****Mike Bifulco:** Hello and welcome back to APIs you won't hate. My name is Mike Biko. I'm one of your APIs you won't hate co-hosts. I'm lucky to be hanging out today with my API's co-founder Phil, Phil Sturgeon. How are you doing today? \[00:00:12\] ****Phil:** Hello. I am doing all right. I'm in wet North Wales. Hiding out, hiding out from the weather. We got a \[00:00:18\] big old storm coming. \[00:00:20\] ****Mike Bifulco:** Yeah. Gosh, it must be the depths of winter there. I'm sure it's plenty, plenty chilly for you right now, huh? \[00:00:24\] ****Phil:** Yeah, absolutely. \[00:00:25\] ****Mike Bifulco:** I'm glad to have you here. I'm glad to be able to sit down and talk and you and I actually today are gonna be speaking with a, a guest. We have Rich Steinmetz on the call today. \[00:00:32\] Rich how are you doing? \[00:00:34\] ****Rich:** Yeah, hey, all good. From my side as well. I'm here in sunny Barcelona. It's been constantly sunny as always. \[00:00:42\] ****Phil:** it in. \[00:00:43\] ****Mike Bifulco:** my jealousy is infinite. Rich, you have been a community member for, with APIs you hate for, for as long as I can remember. And. And it feels like your name and your your voice have appeared quite a bit in our world. But I, I feel like I also wouldn't quite do justice to explain just to everything that you work \[00:01:00\] on \[00:01:00\] can you give a little background on yourself and kind of the things that you do in, in and out of APIs? \[00:01:05\] ****Rich:** Yeah, for sure. First of all, thanks a lot for having me on. I. I'm a listener of the podcast since probably four or five years, so it's basically one of the places that got me into the whole API thing. So it's, it always surprises me how it is when you listen to some, to a podcast or watch a show and then all of a sudden you are there. So thanks \[00:01:32\] ****Mike Bifulco:** think that officially makes you an OG too. If you've been around for four or five years, , you've been with us since, pretty early on. \[00:01:37\] ****Rich:** Yeah, I think, I think it is pretty much from the start. \[00:01:40\] ****Phil:** Nice. Just be, just be ready for how weird it gets when you kind of, you know, you, you, you do this recording, you pop off, you forget about it, and then however long it takes us to actually get this one out, you'll be minding your own business one day and you'll hear your voice coming in through your headphones. \[00:01:52\] You're like, I was just trying to listen to the podcast and it's me. Ah. \[00:01:55\] So just skip \[00:01:55\] ****Rich:** yeah, yeah. That, that's a, that's a scary one for sure, but \[00:02:00\] I'm looking forward to that. I, I'm not from a engineering background. I did get into engineering about like nine to 10 years ago, and I, obviously, I didn't know what I'm doing. I tried a bunch of different things, worked and different. Software teams from the beginning as well and different technologies from cake to cake. PHP, like from PHP \[00:02:25\] ****Mike Bifulco:** Oh yeah. \[00:02:26\] ****Rich:** Java to Python. And one day I had a contract in the Ruby Real space. And there, there I thought, yeah, that's a cool technology. That's a cool community. I want to stay here and. I was always a backend guy. \[00:02:48\] So although I've built full stack from like infrastructure to front end, I still in in the years before I touched and \[00:03:00\] created APIs a lot for one reason or another. But then. With this contract, this Rails contract, five years ago I, for the first time I worked in a team that they had an interesting setup. \[00:03:15\] Actually. They had like a, on the front end, they had a Rails app, and on the backend they had a, it's called Grape, API it's like a framework in Ruby to create APIs. And it was the first contract where I realized that. APIs are like a discipline, like it's something you can get into and be like, get good at and can learn more about. \[00:03:42\] And there I learned things like designing an API and they have like a swagger UI too. For the front end app to communicate with the backhand. So and this is where I match you guys in the, on the \[00:04:00\] podcast with my ears at least. And this is where I started to get deeper into APIs and like read books about it. \[00:04:07\] And yeah, follow the different shows and understanding like the business aspect of it, the technical aspect of it, the design aspect of it. And yeah, five years forward, I'm now, I, I had only rails Ruby Rails and API contracts ever since. And I'm now helping to build the business operating system, API for ClickFunnels. It's a company who basically. Provides the software for entrepreneurs, for solopreneurs, bootstrappers, but nowadays also for all kinds of companies to run their business on and basically have everything related to marketing and shops and \[00:05:00\] customer management running inside. \[00:05:02\] ****Mike Bifulco:** Thank you. That's super helpful. I , am the first to say that I understand marketing technology at like, let's call it a 100 level. I'm not super versed in marketing stuff but I'm also a victim of it fairly often, meaning I get to be an end user of it as a developer for my team at. \[00:05:22\] ****Rich:** Yeah. \[00:05:22\] ****Mike Bifulco:** something that I find really fascinating there is for, for the uninitiated, there is an incredible amount of technology that goes into making sure that money spent on ads, for example, is being spent well and being used to target fundamentally better audiences each day or week or cycle, however that works. \[00:05:39\] But then there's a lot of information that goes into tracking people's journey through a site. So how does someone reach your your website and then get from. Wherever they got to you to spending money presumably, or creating an account or whatever the goal is. And I think on some level that's kind of what ClickFunnels does, is that vaguely right. \[00:05:58\] ****Rich:** Yes, we do \[00:06:00\] have a good analytics. Analytics like module where you can follow your customer's journey through your funnels. Like the meat of the software is, it's in the name, it's the funnels. It's like a page builder for you to create your offer. Or Well, you create your offer in your mind, in the background on paper on the, on the whiteboard. \[00:06:29\] And then you make it visible for others in, in software. And it's basically there is a page pillar for that. And then you can have a bunch of different automations, for example. I think for craftwork you have, you have or will have a subscription, right? Mike is, \[00:06:48\] or is it one time payments. \[00:06:50\] ****Mike Bifulco:** yeah, it's one time payment. So we're a B2C company. We, we do home painting, so we're not a SaaS at all. Yeah. \[00:06:55\] ****Rich:** Yeah. I, I remember, yeah. You \[00:06:56\] do home paintings and then now imagine you have \[00:07:00\] your offer there. Hey, I have these amazing paintings in here. And then someone buys your offer. Let's say you have like a fixed price on it for this particular funnel, and then they're redirected to another page where you can provide them with more services that they need. Maybe, I don't know if they really need it, if you do like everything, but theoretically you could sell 'em the the tools to do the painting. Well, you'll be doing it of course, as a \[00:07:29\] ****Mike Bifulco:** Sure. Yeah, \[00:07:30\] ****Rich:** just as an example. \[00:07:31\] ****Mike Bifulco:** yeah, of course. And so ClickFunnel helps, helps you to understand that process and, and hopefully send people in the right direction. \[00:07:40\] ****Rich:** Exactly. And then you can have all kinds of automations around that. And of course going back to analytics, it's an important part to understand like where in your funnel and your funnel can consist of couple of steps can be very simple, but it can be also very complex with like \[00:08:00\] split testing and all kinds of things. \[00:08:04\] ****Mike Bifulco:** And imagine when it comes down to it. So I'm viewing, I'm sitting on ClickFunnel's homepage right now, and there is a massive grid of features here. Many, many different things that ClickFunnel seems to build and make available to customers. All of these things probably have to talk to each other on some level. \[00:08:21\] Is that where your work comes in? \[00:08:23\] ****Rich:** yeah, exactly. Like we have all these features and. It is actually a funny story. This software exists for 10 years now more or less, and they began mostly just being a funnel builder, like page builder, website builder. And at some point five years or so, for like five years ago, they decided to do like. A ClickFunnels 2.0, like a second version, and one of their \[00:09:00\] like huge factors was the technical aspect of how their app was built in terms of API ecosystem. Like everything around when I say API ecosystem, I'm thinking about like webhooks. The public, API, of course like the platform ability of your app, like other apps connecting to your app. \[00:09:28\] that was a huge factor because the, the infrastructure or the, the technical architecture there in the, in the first version wasn't. set up in a way that, that was easy. They had like the old version of the API. So basically to provide a much better experience for their users, give them even more power in terms of automations that they get with the web hooks and public API, and to also enable partnerships. They have like this big factor, Hey, let's, \[00:10:00\] let's build an. Ap ClickFunnels 2.0 and let's make the API ecosystem. Awesome. Like the, there was a, a lot of thought going in from the beginning into how to build, build it and make it in a way that it's like a platform. And it really shows fruit fruits now because, you were at their main page [probably@likeclickfunnels.com](mailto:probably@likeclickfunnels.com "mailto:probably@likeclickfunnels.com"). \[00:10:29\] If you would look for like developers.my clickfunnels.com, you would see the Our Developer hub. And there you can see the different resources that we kinda, that relate to the features, of course on clickfunnels.com. But then also. That are used by, by our partners or by our, like, power users and just also developers \[00:11:00\] or entrepreneurs who are very techy. \[00:11:02\] ****Mike Bifulco:** I feel like I've gone further and further down the rabbit hole here, and I'm finding more as I go. It's a big product. \[00:11:09\] ****Phil:** why I've been quiet for a while. 'cause I've just been clicking around the website going, oh man, this is gonna solve a lot of problems. Whenever I look at a product at the moment, I'm kind of thinking how it would work for Protect Earth. And I \[00:11:18\] think one of the \[00:11:18\] problems we've \[00:11:19\] got is, \[00:11:20\] Yeah, we're like a tiny little startup growing rapidly as a kind of reforestation charity. And there's so many different bits of software and we've got like a website on Squarespace and the you can sponsor Trees on via the Shopify store, which kind of we have to manually make, try and look a bit similar. And you can schedule appointments and CRM stuff with HubSpot and you could, you know, there's all we've got MailChimp for doing email marketing, but we've also got SendGrid for kind of \[00:11:47\] programmatically, \[00:11:47\] sending emails and there's a hundred different things in here and it looks like you guys do \[00:11:50\] basically. \[00:11:51\] all of it. So that's, that's pretty interesting. \[00:11:53\] ****Rich:** Yeah. So this is exactly the vision to have. Like the business operating \[00:12:00\] system, like to have everything in one place working together and in an automated way so you don't have to switch between the tools. You don't have to connect them because probably if you have those different tools fill, you probably also go from time to time and use something like Zapier maybe to, \[00:12:21\] ****Phil:** Yeah, and a \[00:12:23\] bunch of them charge, charge extra for the integration. I think like donor box, our donation platform. I've been looking to replace that for a while. 'cause Stripe already taken whatever reduced charity rate fee they take. But then donor box take another percentage of that and sometimes someone wants to bang in a thousand pounds and we lose like 50 or 60 or something. And tracking analytics is, is a massive pain in the backside. Fathom, who are amazing sponsors of the charity have given us like a free analytics thing but it's pretty basic because we've not managed to integrate all of the correct tracking links between Squarespace. going to donor box \[00:13:00\] and and joining it all up with that. \[00:13:02\] So yeah, I think the integrations when free require a little bit of work to make. Work properly and sometimes aren't free. so it's expensive and more work. So yeah, it's interesting to see someone trying to tackle all of this at once, which cannot be easy. \[00:13:22\] ****Rich:** Yeah, definitely. a lot of the features are also quite new and there is a lot in the pipeline as well, but we are now at a point that we zoom into making them awesome. on the UI side or on the API side. So, yeah. In terms of features, I, I don't think it will not grow like horizontally anymore a lot I think from what \[00:13:46\] ****Phil:** Right. \[00:13:47\] ****Rich:** From what I see. But we will just like double down on what we have now in the, in the coming time. \[00:13:54\] ****Phil:** Yeah. \[00:13:54\] How, how is it? What, what form does it take? Like is it basically a, a, a \[00:13:59\] Big \[00:14:00\] old cms \[00:14:00\] with lots of modules or is it something that is a \[00:14:03\] kind of a toolkit that you are meant to plug into. \[00:14:05\] some sort of site yourself? \[00:14:07\] ****Rich:** It's, it is just a UI with you, you start out basically when you do the free trial, for example, you start out and with a screen and a couple of apps. Predefined for you. Like let's say like funnels are native and then you can choose, pick and choose what you, what kind of an app. It's structured in an appy way and then you go to the app's menu and say, Hey, I want like, know to know more about analytics. So we pick the app and then it just. Part of your dashboard and you can just come back to it \[00:14:47\] ****Phil:** Oh, I gotcha. And there's like, there's a, a website editor where you are picking a theme, and then all of those themes will support all of the built in apps, I'm guessing, so you can just kind of add in whichever apps and it will have an interface. for that, for that, \[00:15:00\] that \[00:15:00\] ****Rich:** Exactly, yeah. There's an editor like the whole. Funnel thing, it's its own app. So there is an editor website. You connect a domain, you can connect a domain to it, and then you are provided with a domain, like with Click Funnels, domain first, and then you can just connect yours \[00:15:17\] ****Phil:** Do you have any integration with like random APIs? So could I dynamically fetch data into one of these pages from my own api, or would that be something else? \[00:15:27\] ****Rich:** You mean whether \[00:15:30\] ****Phil:** like Right, right now I've \[00:15:31\] ****Rich:** data from \[00:15:33\] ****Phil:** yeah, exactly. I've got I'm just turning this into free consulting now. Cheers. So for a common problem that I have with CMSs that, that try to do quite a lot or any sort of kind of SaaS that does quite a lot? Is that. the more stuff they do, the better. But you always find you know, the end of it somewhere. \[00:15:49\] And we've got we've got loads of projects, right? Like every site that we create a new woodland at is a, is another project. And I've just spent the last week or two hiring a guy to kind of \[00:16:00\] do manual data entry into our Squarespace website. 'cause there's no way to dynamic people in the information. \[00:16:05\] And I'm like, man, I really should, I should be doing better than this. I should not be hiring someone to manually do data entry on a website when all of this data already lives in an API. But also we're quite busy planting trees. And I think my point is yeah. Is there any way to get kind of custom data into it somehow? \[00:16:21\] ****Rich:** do you have like an example of what kind of data it would be like? \[00:16:25\] ****Phil:** If I've got an API with like name of project description, number of trees planted, and here are some photographs. \[00:16:30\] ****Rich:** So it is kind of custom data, right? It's not something that's part of ClickFunnels already. I, I know that like. can have , the first thing I I'm thinking about here now is like Salesforce, they have these and their APIs are very happy to work with, but they have this like maybe also HubSpot if I remember correctly. \[00:16:50\] But they have these objects you can create and you can name them whatever you want and things like that. Is that what, what you would look \[00:16:59\] ****Phil:** Yeah. \[00:16:59\] \[00:17:00\] I've worked with that sort of integration in the past that's quite clever. And I was quite impressed, even though it was cumbersome, like in Salesforce, you can define custom objects \[00:17:08\] in there that are like represented as o data \[00:17:12\] and therefore you can see that lives over here on my API and then it kind of does some syncing in the background or whatever because it, because it's o data, it knows what structure it is and what types are and everything else. \[00:17:23\] Yeah, I suppose that sort of thing. \[00:17:25\] ****Rich:** Hmm. No, we don't have that. We have the, the resources that we provide on, on the API, like, we basically hook in to what we offer to, to what we offer in the app. And that's basically. \[00:17:40\] ****Phil:** I gotcha. So I can integrate kind of products and courses and, and all of the various different app bits I could use. I could use an API to stick some stuff in there. \[00:17:48\] ****Rich:** Exactly like context products CRM you can do even fulfillments via the API. We have a amazing new feature, which is like creating orders via \[00:18:00\] the API. \[00:18:01\] ****Phil:** Oh, brilliant. Yeah, that's helpful. \[00:18:02\] ****Rich:** Yeah, so \[00:18:03\] ****Phil:** Excellent. I always like to find the limitations of a thing to help people get context of like where it sits, but that's, there's plenty on here, which will cover the vast majority of of needs. 'cause not everyone runs a reforestation charity, . \[00:18:15\] ****Mike Bifulco:** So Rich, I. \[00:18:16\] happen to stumble across the change log on the developer side of ClickFunnels. And I've noticed your name is under most of the change log entries of late for the past few months at least. One of the things I'm curious about is, especially like, as Phil's mentioned, there's so many things. \[00:18:30\] You're, you're serving so many different use cases here. How is your team identifying what to work on next and how do you balance, you know, the change of a feature with all the users who must be consuming the things that you've built? \[00:18:43\] ****Rich:** Yeah, that's a great question. This like our API launched almost two years ago and like end of. September, somewhere 2023\. I launched the second version of the API and \[00:19:00\] since then it has evolved. So we went from, Hey, let's give our users what we know they'll be using for sure. Let, let's give them like a basis. And we basically decided ourselves, like business decided together with the engineering team what it will be. They have always spoken a lot, a lot to customers. And so we were just shipping the, the thing that, that we knew that will have broad application. And at some point this shifted a little bit in terms of we went more into shipping for our partners. And as people started using the API more, they were requesting things more so we had direct feedback. By the different channels. We also encourage people to have feedback for us, so they come via support \[00:20:00\] or we start early on to foster this discussion, as you said, in the change log. Like that, that would be like the passive consumption of what's, what's going on. \[00:20:13\] But \[00:20:14\] we also early on went into having like a discussion forum so people have have a place where they have, can have questions or doubts. And then again, our like our API users, they aren't all necessarily like the hardcore developers who. Are hired to do the thing with Salesforce, for example. Our API users are a lot more, I think compared to, for example, Salesforce are a lot more just like solopreneurs who who are techie and they want to build that thing real quick. By fo fostering that discussion in d in different channels. We also have like a feature feature board, and \[00:21:00\] these are all the places basically like our partners who integrate with us, and they're, they're quite a lot of them. \[00:21:07\] Who basically with whom we have synergy, right? Like this is a big thing I think in the API space like the business aspect of the APIs. \[00:21:16\] If you provide like a good platform that can be easily or apt. Two that can be, that have exposes a lot of like API features. Then you have the synergy with other partners and you can kind of propel your, both businesses and through both customers in terms of like value. \[00:21:37\] ****Mike Bifulco:** I'm curious, can you tell us a little bit about how the product is built in the backend? Like , what tools are you using to manage and maintain the API? \[00:21:43\] ****Rich:** That's a good question as well. ClickFunnels is built on on Ruby Rails. \[00:21:50\] ****Mike Bifulco:** Okay. \[00:21:51\] ****Rich:** Additionally, it's also built on bullet train. So bullet train is an open source SAS building \[00:22:00\] framework on top of Ruben Rails. And it's basically like Ruben Rails on steroids for SaaS. The creator of this framework, Andrew Culver and the the open source team who's working on it, they have created this this pretty cool starter kit for, for SaaS and for APIs as well. \[00:22:22\] It is funny that . It was ahead of its time a little bit in, in the Ruby rail space because in the enterprise world where different technologies play a bigger role like Java and c and wherever and even Python, like in where microservices play a bigger role. It's more common to build something like API first. Would you agree? Like, because Ruby Rails is more like SaaS web applications. \[00:22:52\] ****Phil:** Yeah. I definitely feel like there, there's a lot more kind of, let's sit around and spend a year planning this thing in, in \[00:22:59\] those enterprises \[00:23:00\] just 'cause of the, the, the nature of the size of company and they wanna make sure they get it right, you know? They're, they're often a bit less agile, a bit more waterfall because. They are gonna spend, yeah, like sometimes, literally \[00:23:10\] six Months. \[00:23:11\] to a year, like planning what they need and involving stakeholders before, before they even kind of get around to writing any code. Whereas I am consistent, you know? In previous jobs where Rails has been the thing, you'd be in a meeting talking about what, what we might need from this API. \[00:23:28\] And someone's already started typing rails new into their command line. Like they will not stop coding because of the because of the kind of agile nature of it all. And the fact that there are like generators and Scaffolders people will immediately try and just be like, his hit. well, you've mentioned these five resources. I've. already made the controllers for them. The API's basically done, what are we talking about? So \[00:23:49\] ****Rich:** The funny thing, you, me mentioned scaffolds, like it's a console command for anyone who's not into like rails and, and this stuff where you have \[00:24:00\] you just type it in and it creates like controllers, models, like a bunch of code that you would write manually. In other language system frameworks and like Bullet train has super scaffold, so it provides you with extra functionality, a lot of UI things already built in. \[00:24:17\] So it is pretty cool and yeah, it is cool that you agree that there is like more ity on with like in Ruby and Rails teams, maybe also in PPHP teams or like some other, tech bubbles. Yeah. \[00:24:33\] ****Mike Bifulco:** That seems to be a pattern that's found some favor with people, especially with getting off to the races quickly. You can like Phil was saying, rails new, is a funny shortcut for just like, oh, I'm gonna build a thing. I've got the solution. It's coming. And scaffolding is kind of halfway in between where you may be growing your understanding of the thing and, you can use a scaffold to create all of the crud stuff very quickly. \[00:24:53\] My understanding of super scaffolding is that it's make the crud stuff so that your web app has all of the views for a list and a \[00:25:00\] show and an edit, but also that you now have an API that does those same things too. \[00:25:04\] ****Rich:** Oh yeah. Well, how do you know this? Have you worked with train? \[00:25:08\] ****Mike Bifulco:** I haven't worked with Bullet train but at Kraftwerk we're building our product on top of Rails. So I've done a bunch of research about Rails API, providers and patterns. And I think one of the interesting things about Ruby on Rails is that for a, a framework that is so, so hell bent on having like the rails way to do everything. \[00:25:27\] When it comes to APIs, there isn't really a fully formed opinion. And there's many types of ways of kind of getting to what feels like a similar end result enough to the point where it feels like maybe that should just be somewhat standardized at this point too. And super scaffolding seems, seems to come up every time I look up, you know, how do I make a nice rest API using my rails app? \[00:25:47\] ****Rich:** . This is exactly where, where I was saying that bullet train was a little bit ahead of its time in the ruby rails space because, because of its ity, because of how. \[00:26:00\] The rails main driver company, which is like Rails and so Source, right? And behind Rails, like the main core contributors are, , from a co from the rails creator basically. \[00:26:14\] Who, who, who's like DHH David Heiner Hansen. He's the founder or creator of Frails and he, like his company is basically the, the main driver of this technology. And if you look at their API, you might have, like Basecamp is one of their apps. If you look at the base cam play this a PIV four, you might get an idea why Rails. API standardization is in the state that it's currently in. Because it's funny, it's actually funny. Their APIs basically hosted on GitHub. It doesn't, it just examples with, with some text and I'm not sure they are auto generating anything. \[00:26:56\] Maybe they are. But in the end it's just like \[00:27:00\] example response, payload and text. So they haven't ever, I think I'm looking this up like a couple of times a year, probably every year for the last five years. And I don't think they ever with any of their products, they went into like generating an open API schema providing like a developer hub or anything. \[00:27:20\] So they never. Got into this stuff so it never became part of the Rails core framework and \[00:27:26\] ****Phil:** yeah. I think he's a bit busy writing fascist manifestos to get down writing any open API, but it would be, it would be nice to see a little bit more effort put into actually describing, describing any of the APIs involved properly. Because yeah, it has just been the, the classic, like worst possible example of here is a url. Here is some stuff you might throw. at it. We're not gonna, describe any of the, of the data types or validations or anything that you'll actually need and just go figure it out and see what happens. \[00:27:54\] Because I forget the name of the term. I think it's, Hirums Law. Or I'm just getting it wrong and \[00:28:00\] I'm googling frantically, but the, the law that any part of an interface. Can and will be used as an API. Yeah, yeah, yeah. There I actually got it right. Go me. I've said it to like three other people recently, so retroactively. That's good. With a sufficient number of users of an api. It does not matter what you promise in the contract All observable behaviors of your system will be depended on by somebody. And so with Rails, because \[00:28:23\] there are often a lot of different ways of doing something, and sometimes those things change if you are not documenting the way that things should be. \[00:28:31\] done and there are multiple ways of doing it, then you have no idea if you're breaking an API. Like the fact that Rails has from day one somewhat ruined the entire internet's Understanding of the difference between, \[00:28:43\] a put and a patch. \[00:28:44\] 'cause there isn't one, \[00:28:45\] it's just the same thing in Rails, right? And they've just never changed that. And the fact, and the fact that sometimes they, they support like putting a dot Jason on the end or, or a content type. Like there's a lot of these weird ways where they've just gone screw it both \[00:28:58\] and. \[00:29:00\] If that's not written down in a contract somewhere, you don't have a contract, you just have vibes that might shift and break everything. \[00:29:05\] So \[00:29:05\] I, I would like to see open API considered as important, not only to David Heil Hanson, but the companies that he works for and the, the core product itself, you know. \[00:29:19\] ****Rich:** Yeah, I'm on a mission, a actually to to get there and bullet train. Is, and I, I don't know how many years it'll take, but maybe, maybe someday. And bullet train. One more time back to the head of its time thing, it with the super scaffolding it provides you with with an API namespace, this is like the regular way. It's not classic API first development because. It's not like a separate service in the end Rails is still like a monolith but it has like, it gets like a touch of API first into the monolith and you get \[00:30:00\] the the specific API version namespace in there. You you have even the parameters. Are initially when you do the su super scaffold, like the parameters of the domain code in the controllers makes use of the parameters in the API control. \[00:30:20\] Control so that you kind of think from the beginning about, Hey, what's, what would be here? Like like something I can or should expose or shouldn't. And yeah, not to forget, there is the, the open API quote unquote problem, which, in the rails world, because it's an unsolved problem. I would consider it unsolved to. Generate your open API schema from code itself. I, I don't know you guys, do you agree that like in Python or like some type languages there are \[00:31:00\] extremely solid open API generators. And I'm always saying like, fast API is the most prominent example, but do you agree that there are like kind of bulletproof, more or less at least? \[00:31:11\] ****Phil:** No. I mean, every, every I. Every \[00:31:14\] Language has one framework that takes the \[00:31:16\] view that \[00:31:17\] open. \[00:31:18\] API is something that you should just vomit out after you've finished writing all of your code. \[00:31:22\] And, and \[00:31:23\] whether that's coming directly from the code because of types or whether \[00:31:26\] it's coming from \[00:31:27\] annotations that are \[00:31:28\] confusing, proximity with accuracy, \[00:31:30\] they kind of just, you know, throw something out 'cause they were told to and they couldn't be bothered to think about \[00:31:34\] doing it first. \[00:31:35\] And \[00:31:36\] that is. \[00:31:36\] you know, helpful in some regards. But I think most, most frameworks have. A way of ingesting open API, \[00:31:46\] Right? The \[00:31:47\] API design first workflow being, \[00:31:49\] I'm actually gonna do this open API first, and then whatever code I end up using \[00:31:53\] will simply \[00:31:53\] be \[00:31:54\] an \[00:31:54\] implementation of this. Without going the full, like I'm gonna generate the entire code \[00:31:58\] off of the open API, which is \[00:32:00\] usually a bit of a faf. \[00:32:01\] A lot of, \[00:32:01\] them have like. Server side request validation \[00:32:05\] so that you don't have to write that yourself \[00:32:06\] And \[00:32:07\] then make sure \[00:32:07\] that it matches. \[00:32:08\] because you can just say, Hey, this middleware, can you validate the incoming request based off of open API and reject anything that's \[00:32:15\] rubbish. And then they have like some other sort of like aspect, you know, test unit testing extension that just \[00:32:21\] Says \[00:32:21\] oh, and hey, can you make \[00:32:22\] Sure. \[00:32:22\] that this response coming back matches the open API that said it should and annoyingly most frameworks. Have those sort of options, but there are always some random little plugin main maintained by, you know, some guy. And I think that more and more API frameworks or any \[00:32:37\] Web application \[00:32:38\] framework that wants to \[00:32:39\] pretend that it's \[00:32:40\] you know, focusing on APIs \[00:32:41\] needs to bring those into the core, \[00:32:44\] and \[00:32:44\] make it incredibly easy and recommended \[00:32:46\] to, \[00:32:47\] instead of defining everything in some random DSL, you \[00:32:50\] could simply \[00:32:51\] point it \[00:32:52\] at your open API and save, wrangle that code. I, \[00:32:54\] I'd love to see \[00:32:55\] Ruby do more of that and Rails do more of \[00:32:57\] that. \[00:32:57\] 'cause there are really good extensions out there for \[00:32:59\] that sort of \[00:33:00\] thing. \[00:33:00\] ****Rich:** Yeah. it, that's exactly the thing. Like an R swag. It's one of the best options right now in the real space where you kind of write the tests first and then it also generates an O-P-A-P-I schema from the test. But it's still a little bit of A-D-D-S-L there, right? It's not really generating from code, \[00:33:20\] ****Phil:** oh yeah, sure. That's, that's something different. That's aspect is just the generic testing framework like PHP unit or J unit or just whatever test runner, \[00:33:28\] um, the. Ah, swag. Yeah. I think that doesn't even support the latest versions these days, and they can't be, they can't, can't be interested in up update updating it. \[00:33:36\] So mostly what I try and recommend to people is instead of. Some people don't like writing open API and instead of writing open API, they write some other sort of awkward DSL, which then gets out of date because they've tried to simplify it by taking shortcuts, which then don't represent the full reality of open API. So honestly, just like learn to write it or use, use copilot to help you or whatever. But \[00:33:58\] my, My. \[00:33:59\] preferred thing is still \[00:34:00\] like, just actually write the open API, it's portable. If you rewrite your version two of your API. And whatever go instead of Rails, then app days, it, it's still exactly the same. \[00:34:09\] And you have something there for breaking change detection because you, you know, that you haven't accidentally broken the API 'cause it's, you know, being tested and validated against the whole way. But that is probably a whole other episode and one that we \[00:34:21\] did recently. So I'll get off my soapbox. \[00:34:24\] ****Rich:** No, it, it is good stuff because we also thought about testing this as an alternative approach. Like what would be the cost of doing like this manually or design first versus having our own auto generation, which we have now. But it also comes with some things that need to be improved. Like , the final vision of this is that bullet, the bullet train open. API generation would have like the perfect generation generated from code and \[00:35:00\] just like clean, cool generation from code, but. Yeah, , that's the other thought experiment there. Like what would it cost actually in comparison to us getting it there, the technology there, what would it cost to actually doing the design first thing? \[00:35:14\] ****Mike Bifulco:** So Rich I, I know you've mentioned earlier in the chat that you have been doing some consulting work my understanding is maybe at least that you may be working on ClickFunnel as a consultant there too. Are you working with folks currently outside of ClickFunnel? \[00:35:28\] Are there projects you're working on? \[00:35:29\] ****Rich:** on? Yeah, sure. So yeah, my main consulting right now goes into the ClickFunnels business operating system, a BI, I'm super passionate about that project, but. I'm also working together with people to help them create their apps. Marketing First in the Ruby Rail space because we as developers tend to develop a lot and keep our heads in the code a lot, but then we don't really think about the Mark marketing aspect. \[00:35:57\] Unfortunately a lot of times \[00:36:00\] so. \[00:36:00\] Yeah. I'm working with developers and developers to help them build their app marketing first and maybe with maybe also some projects will come out with backed by the ClickFunnels, API, to just see how you can delegate your user management to, to the api, to an API or to this API. \[00:36:22\] So I'm working, that's one of my main. Projects right now, \[00:36:26\] ****Mike Bifulco:** I think that's something that I probably have shouted about on the internet infinitely for people building new products. Is that marketing your thing is every bit as important as making it work. Because if you build a product and nobody knows about it, ain't nobody gonna use it. And that's hugely problematic for a lot of founders who are super technically talented, but then never tell anyone what they're building. \[00:36:45\] And there is no longer, and if you build it, they will come. It's uh, you gotta do both now. \[00:36:50\] ****Rich:** them. Yes. How many apps died because someone shipped an app, it's online, they dropped the link somewhere and then no \[00:36:58\] one registered. \[00:36:59\] ****Mike Bifulco:** . \[00:37:00\] So Rich what's the best way for people to find you online if they're interested in working with you? \[00:37:04\] ****Rich:** Yeah, I'm also asking myself this sometimes because things change a lot. But I think one solid place is always the rich stone.io/meat dash rich, \[00:37:17\] That's the place where I kind of share where I'm at right now and what channels are best to find me on. I'm also an ex, I'm everywhere. I'm Rich Stone io basically. \[00:37:29\] ****Mike Bifulco:** Oh, that makes it easy. And I'll make sure to drop a link to your site in the show notes so people can find that as well. I've also jotted down a handful of things related to ClickFunnels and bullet train and, and the like. \[00:37:40\] We'd love to have you come back some time in wax philosophical little bit about Rails and open API I think that'd be a fun chat too. \[00:37:45\] ****Rich:** That'd be awesome. Yeah. \[00:37:47\] . And, and by the way, \[00:37:48\] I just wanted to mention that the Ruby Rails community being like the web app. Framework and, and language. They will really appreciate the things that you are sharing and \[00:38:00\] doing about like APIs and integrations. \[00:38:02\] So it's good to have you more in our world as well. \[00:38:07\] ****Mike Bifulco:** Well, thanks so much. Yeah, I appreciate it. It is a hugely, hugely responsive community, and they're definitely. Very opinionated , and nice to chat with too. People are pretty reasonable to talk through what they're working on and you know, why they prefer this over that, and so on. I'll find myself deeper and deeper in that as the the days where on certainly too. \[00:38:24\] Alright, fellas, well, let's call it a day from there. Thank you so much for hanging out. It's been really nice chatting with you, rich. Have a great day. Thank you for coming. \[00:38:30\] ****Rich:** Thank you very much. Bye. \[00:38:32\] ****Mike Bifulco:** Bye, Phil. Until next time \[00:38:33\] mate. \[00:38:33\] ****Phil:** yes. See it. ### New Year, New APIs URL: https://apisyouwonthate.com/newsletter/new-year-new-apis-2025/ Last updated: 2025-01-16T13:00:54.000Z APIs have never been more essential, thanks to the ever-increasing usage of LLMs and AI agents. A well-documented API is far easier for AI to interact with than an ever-changing UI and allows better interactions. It has never been more critical to build excellent API docs and push forward with API design first principles. Whether you need to integrate AI into everything is an entirely different question, but good API docs are always important, and now you have another reason to care about them. From the whole API's You Won't Hate team, we hope you have a great 2025\. \-- Alexander, Phil and Mike --- ## The API Round-Up *API News, links, and tools from around the web* ### [API Design](https://www.speakeasy.com/api-design?ref=apisyouwonthate.com) Phil has been working hard with Speakeasy to create a comprehensive guide to API Design. It covers everything from structuring URLs to request bodies, pagination, and more. The content is fantastic, with the main focus on principles and understanding rather than code. ### [Go Developer Survey 2024 H2 Results](https://go.dev/blog/survey2024-h2-results?ref=apisyouwonthate.com) The 2024 Go Developer Survey is out and contains some interesting results. I am sure your main question is, why share this in an API newsletter? 75% of Go developers reported using Go to build API/RPC services. I encounter Go in so many API systems these days, so maybe it's time to give Go a go 😜. ### [A visual interface for working with OpenAPI Overlays](https://overlay.speakeasy.com/?ref=apisyouwonthate.com) The team at Speakeasy put together this handy little browser-based tool for those of us working with OpenAPI Overlays. It's a great way to validate that your overlay's JSONPath is working correctly to update values in your schema. ### [OpenAPI Parser](https://github.com/scalar/scalar/tree/main/packages/openapi-parser?ref=apisyouwonthate.com) After getting extremely frustrated maintaining `swagger-parser` against his will, Phil reminds us that Scalar has a superb OpenAPI parser. Scalar's parser supports OpenAPI 3.1, 3.0 and Swagger 2.0\. The constant upgrades make it worth moving over if you're still using the old `swagger-parser`. ### [OpenAPI Arazzo in Go](https://github.com/speakeasy-api/openapi/tree/main/arazzo?ref=apisyouwonthate.com) Speakeasy has released a Go library for working with OpenAPI Arazzo documents. They use it in production for their end-to-end testing product, so it's definitely worth checking out. ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2024/08/speakez-1.jpg) ## Speakeasy: Build APIs Your Users Love Your API deserves a great developer experience. Get one by using Speakeasy to generate idiomatic, type-safe SDKs from OpenAPI [Start Generating](https://www.speakeasy.com/?utm%5Fsource=apisyouwonthate&utm%5Fcampaign=240701) ## APIs You Won't Hate The latest from the team at API's You Won't Hate. ### [Testing HTTP Middleware in Laravel](https://apisyouwonthate.com/blog/testing-http-middleware-in-laravel/) A short and to the point tutorial on adding tests for HTTP middleware in Laravel. Using Pest or PHPUni, see how easy it is to set up tests for your custom Middleware. ### [Generate SDKs with Speakeasy](https://docs.bump.sh/guides/bump-sh-tutorials/generate-sdks-with-speakeasy/?ref=apisyouwonthate.com) Phil has put another API guide together for Bump.sh. Phil walks you through using Speakeasy to generate SDKs from your OpenAPI docs. Phil covers configuring Github, publishing the packages and using overlays. --- ## From our Community Articles written and shared in our free [Slack community](https://slack.apisyouwonthate.com/?ref=apisyouwonthate.com). ### [Privacy-conscious API request logging](https://apitally.io/blog/request-log-release-announcement?ref=apisyouwonthate.com) Simon Gurcke from our community has an exciting indie project for monitoring REST APIs. They recently released a request log feature to bridge the gap between metrics and raw logs. Privacy features have also been rolled into the log process. Nice work, Simon. ## Support APIs You Won't Hate When you become an member, you'll get access to members-only content while directly supporting our work. Your support helps us to keep making resources for the API community. [Become a member today](https://apisyouwonthate.com/#/portal) Thanks so much to our members: **Kin L, Juxt, Alex R, Nolan S, Frank, James D, Bill D, Rich, and Umair**. Your support means the world to us! Catch you on the flip side! Alexander, Phil & Mike ### Testing HTTP Middleware in Laravel URL: https://apisyouwonthate.com/blog/testing-http-middleware-in-laravel/ Last updated: 2025-01-16T12:44:24.000Z HTTP middlewares allow APIs to have amazing and powerful functionality added to requests and responses without needing to mess with each controller. Laravel has its own [middleware](https://laravel.com/docs/11.x/middleware?ref=apisyouwonthate.com) implementation, with core middlewares handling CORS, caching, authentication, etc. Custom middlewares can be defined to standardize behavior across an API and to change the default way Laravel works. For example, the [Protect Earth API](https://bump.sh/protect-earth/doc/partner-api/?ref=apisyouwonthate.com) likes to default to `Accept: application/json` if no header has been sent, but should not override that if a different accept header has been sent. ```php # app/Http/Middleware/DefaultToAcceptJson.php header('Accept') === null) { $request->headers->set('Accept', 'application/json'); } return $next($request); } } ``` That'll probably work, but adding functionality like this without an accompanying test is reckless and will probably lead to an angry email, so let's pop a test on. ## Testing Laravel Middleware using Pest The idea is pretty much the same whether using Pest or PHPUnit: create an empty request, spin up an instance of the middleware, then add some assertions. ``` handle($request, function ($request) { expect($request->headers->get('Accept'))->toBe('application/json'); return response()->json(); }); }); it('will not overwrite the Accept header if it is already set', function () { $request = new Request(); $request->headers->set('Accept', 'text/html'); $middleware = new App\Http\Middleware\DefaultToAcceptJson(); $middleware->handle($request, function ($request) { expect($request->headers->get('Accept'))->toBe('text/html'); return response()->json(); }); }); }); ``` The `$middleware->handle(` part is confusing at first, but with a bit of thinking about it we can make it make sense. Middlewares can modify the request, and/or modify the response. ### Testing request middlewares When modifying the request, it's usually adding or changing some headers, then the request moves along the stack to the next thing, which could be another middleware or invoke the controller. The line `$middleware->handle($request, function($request)` middleware is "handling" the `$request` , doing whatever it's done, then moves onto the next request. At this point, the middleware should have done it's thing, so expectations can be set inside here. ```php it('will not overwrite the Accept header if it is already set', function () { $request = new Request(); $request->headers->set('Accept', 'text/html'); $middleware = new App\Http\Middleware\DefaultToAcceptJson(); $middleware->handle($request, function ($request) { expect($request->headers->get('Accept'))->toBe('text/html'); return response()->json(); }); }); ``` The `response()->json()` is just there to keep PHP signatures valid. ### Testing response middlewares Instead of hopping into the request chain to see if a request was modified, testing a response middleware means we need to grab the last result coming out of the whole request chain. ```php it('will shove out a Content-Type header', function () { $request = new Request(); $middleware = new App\Http\Middleware\SomeResponseMiddleware(); $response = $middleware->handle($request, fn(): string => 'response'); expect($request->headers->get('Content-Type')) ->toBe('application/json'); }); ``` There it is, middleware can add amazing functionality to a website but should not be left to change. Slap a test on it today and keep APIs working properly. ### Happy Holidays from APIs You Won't Hate URL: https://apisyouwonthate.com/newsletter/happy-holidays-2024/ Last updated: 2024-12-17T09:00:05.000Z It is time to close the laptop, reflect on your year, and spend time with family and friends. Hopefully, you will have a break over the holidays, but if you get bored, don't forget you can follow us on [BlueSky](https://bsky.app/profile/apisyouwonthate.com?ref=apisyouwonthate.com) for the latest API updates. If you crave even more API content, don't forget we assembled this excellent [API starter pack](https://bsky.app/starter-pack/mikebifulco.com/3l7l72ptyfj2t?ref=apisyouwonthate.com). We will take a break over the holidays, so there will be no newsletter on the 1st of January. Have a great break and happy holidays from the whole team at APIs You Won't Hate \-- Alexander, Phil and Mike --- ## The API Round-Up *API News, links, and tools from around the web* ### [Apideck raises $7.5M Series A for API platform](https://tech.eu/2024/11/20/apideck-raises-7-5m-series-a-for-api-platform/?ref=apisyouwonthate.com) It's great to see an API-focused startup raising money, especially as raising has become difficult over the last few years. Apideck plans on using some of the money from the raise to merge its unified APIs with LLM for real-time data processing. It will be interesting to see what that means when they implement it. ### [Undici v7 is here](https://blog.platformatic.dev/undici-v7-is-here?ref=apisyouwonthate.com) Undici, a modern HTTP client for NodeJS, dropped a major version recently, bringing some significant updates from better `fetch()` spec compliance, composing interceptors and client-side HTTP caching. Undici goes the extra mile to be spec-compliant, not to mention that the name is a Stranger Things reference. ### [Speakeasy x Scalar Docs](https://www.linkedin.com/feed/update/urn:li:activity:7270119690071388160/?ref=apisyouwonthate.com) Speakeasy and Scalar have been mentioned a few times in the newsletter, and now they're joining forces. Speakeasy allows you to generate SDKs for your APIs quickly, and Scalar enables you to create beautiful documentation. Now, your SDK libraries will have excellent documentation. ### [Microcks + Bump.sh: Testing, Mocking and Docs](https://bump.sh/blog/microcks-bump-sh-testing-mocking-docs?ref=apisyouwonthate.com) Learn how to use Microcks with Bump.sh to create a constant loop of documentation, simulation, and testing. Phil teams up with Laurent and Yacine to deliver this deep dive and explain the advantages of this flow. ### [Generate Server Code Using OpenAPI Generator](https://mydeveloperplanet.com/2022/02/08/generate-server-code-using-openapi-generator/?ref=apisyouwonthate.com) Finally, the SpringBoot blog post I have been waiting for has arrived. Check out this walkthrough on generating endpoints in SpringBoot from OpenAPI docs. I have been looking for a good blog post on this for a long time, and finally, someone has delivered. ### [API Client Language Popularity](https://radar.cloudflare.com/year-in-review/2024?ref=apisyouwonthate.com#api-client-language-popularity) An interesting study from Cloudflare. More than half of its traffic is related to APIs, and recently, Go has overtaken Node as the most popular API client—an interesting observation. ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2024/08/speakez-1.jpg) ## Speakeasy: Build APIs Your Users Love Your API deserves a great developer experience. Get one by using Speakeasy to generate idiomatic, type-safe SDKs from OpenAPI [Start Generating](https://www.speakeasy.com/?utm%5Fsource=apisyouwonthate&utm%5Fcampaign=240701) ## APIs You Won't Hate *The latest from the team at API's You Won't Hate.* ### Follow [@apisyouwonthate on Bluesky](https://bsky.app/profile/apisyouwonthate.com?ref=apisyouwonthate.com) The APIs You Won't Hate Community has been coming over to Bluesky in Droves, and we're thrilled to see it. We're starting to crosspost articles and discussions on Bluesky as they pop up. You may also want to check out the [APIs You Wont' Hate Starter Pack](https://bsky.app/starter-pack/mikebifulco.com/3l7l72ptyfj2t?ref=apisyouwonthate.com) to keep up with some of our community members. --- ## From our Community *Articles written and shared in our free* [*Slack community*](https://slack.apisyouwonthate.com/?ref=apisyouwonthate.com)*.* ### [OpenAPI Overlays](https://apievangelist.com/2024/12/05/api-evangelist-conversation-with-lorna-mitchell-about-openapi-overlays/?ref=apisyouwonthate.com) Lorna Mitchell talks to Kin Lane about OpenAPI Overlays, sharing how it will help bring more stakeholders into the API lifecycle and improve API experiences. ### [Generate OpenAPI with Django & DjangoREST Framework](https://www.speakeasy.com/openapi/frameworks/django?ref=apisyouwonthate.com) From community member Nolan over at Speakeasy, a guide to using DjangoREST to generate OpenAPI docs, for all our python pals out there. ### [Do you use HTTP Caching in your REST APIs](https://apisyouwonthate.slack.com/archives/C0HGJS734/p1734096690944559?ref=apisyouwonthate.com) Phil is conducting research on HTTP Caching in our Slack community. Please visit and let him know your thoughts. ## Support APIs You Won't Hate When you become an member, you'll get access to members-only content while directly supporting our work. Your support helps us to keep making resources for the API community. [Become a member today](https://apisyouwonthate.com/#/portal) Thanks so much to our members: Kin L, Juxt, Alex R, Nolan S, Frank, James D, Bill D, and Rich. Your support means the world to us! See you next year! Alexander, Phil & Mike *Cover Photo for this newsletter by* [*Keith Luke*](https://unsplash.com/@lukephotography?utm%5Fcontent=creditCopyText&utm%5Fmedium=referral&utm%5Fsource=unsplash) *on* [*Unsplash*](https://unsplash.com/photos/people-covered-by-confetti-GUAcpXPyFRc?utm%5Fcontent=creditCopyText&utm%5Fmedium=referral&utm%5Fsource=unsplash) ### Mr. Bluesky URL: https://apisyouwonthate.com/newsletter/mr-blue-sky/ Last updated: 2024-12-02T13:11:34.000Z The last two weeks have felt like the old Twitter is back. My feed is full of tech posts, interactions from other developers, not bots, and finding more people in my field to follow and learn from. I am, of course, talking about Bluesky. It feels great to finally not have to scroll through the cesspit that is Twitter's ForYou tab to find API content. Politics aside, filling a place with hate, conspiracies, and even more hate kills it. Hopefully, the Bluesky uptake continues, as it's nice to be learning from others over my morning coffee again. \-- Alexander --- ## The API Round-Up ### [The world's fastest OpenAPI linter](https://github.com/daveshanley/vacuum?ref=apisyouwonthate.com) I am always slightly suspicious of anything that says it's the fastest, but who can say no to the cute logo of Gopher with a hover (vacuum)? Vacuum is a super-fast OpenAPI linter written in Go that is compatible with existing Spectral rulesets. It's worth checking out. ### [Git Is How You Run in Bruno](https://apievangelist.com/2024/11/15/git-is-how-you-run-in-bruno/?ref=apisyouwonthate.com) Bruno has been mentioned a few times in the newsletter. In case you missed those, Bruno is a new open-source API client with a Git integration. Kin Lane talks about using it to manage his collections and environment variables. ### [Bluesky OpenAPI](https://bsky.app/profile/czechboy0.dev/post/3lb2rlsgfac23/?ref=apisyouwonthate.com) With large chunks of the tech community migrating to one place, we're seeing some fantastic content pop up on Bluesky. Honza Dvorsky walks us through his journey of generating a Swift OpenAPI client using the OpenAPI document from Bluesky. ### [Generate an OpenAPI Spec with Hono](https://www.speakeasy.com/openapi/frameworks/hono?ref=apisyouwonthate.com) If' you've been reading our newsletter, you know we love Hono around here. It's fast, it's configurable, and is a great alternative to Express. This guide walks you through generating an OpenAPI specification for a Hono API using Speakeasy, to create an SDK based on the generated specification. ### [HTTPie](https://httpie.io/?ref=apisyouwonthate.com) HTTPie is a testing client that runs in your terminal. Recently, they released a public beta of their new Web and Desktop versions, which I highly recommend checking out. ### [Bluesky, Mastodon, and Threads APIs](https://apievangelist.com/2024/11/20/bluesky-mastodon-and-threads-apis/?ref=apisyouwonthate.com) Here's more Bluesky fun with Kin Lane again. Kin walks us through the Bluesky, Mastodon, and Threads APIs, highlighting their distinct approaches to community engagement and technical implementation. It's a great read, especially given the surge of tech people moving away from Twitter. ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2024/08/speakez-1.jpg) ## Speakeasy: Build APIs Your Users Love Your API deserves a great developer experience. Get one by using Speakeasy to generate idiomatic, type-safe SDKs from OpenAPI [Start Generating](https://www.speakeasy.com/?utm%5Fsource=apisyouwonthate&utm%5Fcampaign=240701) ## APIs You Won't Hate The latest from the team at API's You Won't Hate. ### [Bluesky Starter Pack: APIs You Won't Hate Community](https://go.bsky.app/BNkzSpM?ref=apisyouwonthate.com) Starter Packs are one of Bluesky's best features - they let you aggregate and follow collections of accounts based on some common interest, and they're user-maintained. In addition to the newly minted [@apisyouwonthate](https://bsky.app/profile/apisyouwonthate.com?ref=apisyouwonthate.com) account, you can keep up with the community itself with one click from the [starter pack](https://bsky.app/starter-pack/mikebifulco.com/3l7l72ptyfj2t?ref=apisyouwonthate.com)! ### [The "Don't Have Time to Create API Documentation" Paradox](https://apisyouwonthate.com/blog/dont-have-time-for-api-docs-paradox/) Phil is back with another fantastic read on the common pitfalls of neglecting API documentation during development and how this can lead to confusion and inefficiencies. As always, Phil pushes for a design-first approach and highlights the benefits this brings. ### [Learn OpenAPI from these Realistic Examples](https://apisyouwonthate.com/blog/learn-openapi-from-these-realistic-examples/) If you prefer to learn by looking at examples, maybe a cheeky bit of copy-paste coding, these OpenAPI examples can get you started. --- ## From our Community Articles written and shared in our free [Slack community](https://slack.apisyouwonthate.com/?ref=apisyouwonthate.com). ### [Shadow APIs](https://danaepp.com/why-shadow-apis-provide-a-defenseless-path-for-threat-actors?ref=apisyouwonthate.com) Dana discusses the security risks associated with Shadow APIs – undocumented or unknown endpoints within an organization's infrastructure. In the last few years, endpoints like this have left some companies here in Australia open to "hacks," so it's definitely worth reading. ## Support APIs You Won't Hate When you become an member, you'll get access to members-only content while directly supporting our work. Your support helps us to keep making resources for the API community. [Become a member today](https://apisyouwonthate.com/#/portal) Thanks so much to our [members](https://apisyouwonthate.com/membership/): Kin L, Juxt, Alex R, Nolan S, Frank, James D, Bill D, and Rich. Your support means the world to us! Until next time, Alexander, Phil & Mike ### Learn OpenAPI from these Realistic Examples URL: https://apisyouwonthate.com/blog/learn-openapi-from-these-realistic-examples/ Last updated: 2024-12-02T12:31:51.000Z For what felt like forever everyone was using the [Pet Store](https://petstore.swagger.io/?ref=apisyouwonthate.com) as an example for OpenAPI and it never really made much sense. It was a bad API describing a weird problem badly, and these days it's even worse: it was based on OpenAPI v2.0 and vaguely updated, but in the world of OpenAPI v3.1 we need more realistic examples to use that leverage the amazing modern functionality and simplified syntax. Whether you're just trying to learn how OpenAPI works in a more realistic domain than a pet shop, or you want to run some tooling you've built through a test suite using these examples, or you want to demonstrate to your team how their APIs could look in a new [API Catalog](https://bump.sh/api-catalog?ref=apisyouwonthate.com)(ue) you just built, these demonostration OpenAPI's can help you out. ## Train Travel API, from Bump.sh The wonderful folks over at Bump.sh sponsored my time to [work on this OpenAPI description](https://bump.sh/blog/modern-openapi-petstore-replacement?ref=apisyouwonthate.com) of a fictional European train travel API. It covers finding stations, trips, making bookings, paying for the tickets, and more. We're even working on an [Arazzo workflow](https://apisyouwonthate.com/blog/introducing-arazzo-describe-api-workflows/) that will help demonstrate how to chain those requests and responses together. It's open-source and [available on GitHub](https://github.com/bump-sh-examples/train-travel-api?ref=apisyouwonthate.com), so use it for whatever (according to the Creative Commons license ofc) and send in any improvements you can think of. [Train Travel API: A Modern OpenAPI PetStore Replacement · Bump.shEveryone working with OpenAPI (formerly Swagger) will have come across the PetStore at some point. It’s a sample OpenAPI description for an imaginary Pet Store with an API, but the OpenAPI is old, and the API it describes is pretty far from best practices. We thought it was time for a refresh, so we’re bringing you the Train Travel API, a new sample OpenAPI you can use for your tooling and testing.![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/icon/favicon-1.svg)![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/thumbnail/train-travel-api-c45eac84362ac9752a7d.png)](https://bump.sh/blog/modern-openapi-petstore-replacement?ref=apisyouwonthate.com) Use the following URL, or download it. ``` https://raw.githubusercontent.com/bump-sh-examples/train-travel-api/refs/heads/main/openapi.yaml ``` ## Museum API, from Redocly This demo OpenAPI covers operation at a museum, with things like operational hours, managing events at the museum, and even returning QR codes. This helps showcase lots of OpenAPI v3.1 functionality, and also has an Arazzo document to help show off key workflows. [GitHub - Redocly/museum-openapi-example: An example OpenAPI description for an imaginary Museum API.An example OpenAPI description for an imaginary Museum API. - GitHub - Redocly/museum-openapi-example: An example OpenAPI description for an imaginary Museum API.![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/icon/pinned-octocat-093da3e6fa40-3.svg)GitHubRedocly![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/thumbnail/museum-openapi-example)](https://github.com/Redocly/museum-openapi-example/?ref=apisyouwonthate.com) Use the following URL, or download it. ``` https://raw.githubusercontent.com/Redocly/museum-openapi-example/main/openapi.yaml ``` ## Galaxy API, from Scalar The team over at Scalar have been hard at work on everything from JSON Schema reference parsers to amazing OpenAPI-based HTTP GUI Clients, and to test all that they've built an excellent sample API description that covers planets. Ok so unless you work at Nasa this one is not the most realistic domain, but it covers loads of authentication methods and awkward things like uploading images in the HTTP Request body, which others do not. [scalar/packages/galaxy/README.md at main · scalar/scalarScalar is an open-source API platform: 🌐 Modern Rest API Client 📖 Beautiful API References…![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/icon/pinned-octocat-093da3e6fa40-2.svg)GitHubscalar![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/thumbnail/scalar-1)](https://github.com/scalar/scalar/blob/main/packages/galaxy/README.md?ref=apisyouwonthate.com) Use the following URL, or download it. ``` https://cdn.jsdelivr.net/npm/@scalar/galaxy/dist/3.1.yaml ``` ## Find more OpenAPI examples If you need more examples of OpenAPI for any reason, there are loads over on the [APIs.guru Marketplace](https://apis.guru/?ref=apisyouwonthate.com). Most of these are real examples published by companies of all sizes, so you can see real world working OpenAPI descriptions. ### State of JS Survey Time URL: https://apisyouwonthate.com/newsletter/state-of-javascript-survey-2024/ Last updated: 2024-11-18T18:14:01.000Z Can you believe it is almost Christmas? Well, that can only mean one thing: time for the annual State of JavaScript survey. I know not all of you build APIs with JS, but plenty of JS APIs are out there. We need better representation of people from all walks of life and backgrounds in the results. So head to [the survey](https://survey.devographics.com/en-US/survey/state-of-js/2024?ref=apisyouwonthate.com) and start answering. Maybe it is time to create an annual API survey. \-- Alexander --- ## The API Round-Up ### [Accelerating your OpenAPI Spec Generation with TypeSpec](https://bump.sh/blog/accelerating-your-openapi-spec-generation-with-typespec?ref=apisyouwonthate.com) A deep dive into TypeSpec by Bump.sh on how to use it to generate OpenAPI spec quickly. TypeSpec is one of the new options that have popped up recently to detail your APIs. Keep in mind that TypeScpec is more developer-centric, so it is not as friendly for other stakeholders. ### [OpenAPI enhancements in ASP.NET Core](https://devblogs.microsoft.com/dotnet/announcing-dotnet-9/?ref=apisyouwonthate.com) .Net version nine dropped recently and comes with improvements to its OpenAPI support. You can now generate OpenAPI specs from your code and preview these specs in a Swagger-inspired UI. Check out the announcement for the full details ### [A powerful HTTP client that lives in your terminal.](https://github.com/darrenburns/posting?ref=apisyouwonthate.com) I'm not sure words will do this justice, so here is an image instead. [![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2024/11/home-image-ad-15aug24.svg)](https://github.com/darrenburns/posting?ref=apisyouwonthate.com) Posting, a powerful HTTP client that lives in your terminal ### [Hono OpenAPI](https://github.com/rhinobase/hono-openapi?ref=apisyouwonthate.com) Hono is a modern Express-inspired web framework that makes building and deploying JS APIs anywhere a breeze. With Hono OpenAPI, you can generate OpenAPI documents from your validation schemas. Currently, it only supports Zod, but support for TypeBox and Valibot is coming. ### [Introducing Express v5: A New Era for the Node.js Framework](https://expressjs.com/2024/10/15/v5-release.html?ref=apisyouwonthate.com) I have spoken extensively about the upcoming new version of Express in past issues. Well, they finally released a release with an official announcement. It covers breaking changes, new features, and details on future releases. ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2024/08/speakez-1.jpg) ## Speakeasy: Build APIs Your Users Love Your API deserves a great developer experience. Get one by using Speakeasy to generate idiomatic, type-safe SDKs from OpenAPI [Start Generating](https://www.speakeasy.com/?utm%5Fsource=apisyouwonthate&utm%5Fcampaign=240701) ## APIs You Won't Hate The latest from the team at API's You Won't Hate. ### [Powerful HTTP/API Clients: Alternatives to Postman](https://apisyouwonthate.com/blog/http-clients-alternatives-to-postman/) Phil went on a writing frenzy recently and smashed out a bunch of articles for our fantastic readers. In this post, Phil talks us through some great alternatives to Postman. While I am sure he did not get them all, he covers some strong contenders. ### [The OpenAPI Discriminator is Redundant & Confusing](https://apisyouwonthate.com/blog/openapi-discriminator-is-redundant/) Phil walks us through the Discriminator field in OpenAPI Spec version three and above, explaining what it is for and why it is confusing. Discriminator is primarily used to help code generators rather than validate, but check the article to ensure you're using it correctly. --- ## From our Community Articles written and shared in our free [Slack community](https://slack.apisyouwonthate.com/?ref=apisyouwonthate.com). ### [What are API Subaccounts?](https://zuplo.com/blog/2024/11/12/what-are-subaccount-api-keys?ref=apisyouwonthate.com) Adrian Machado gives us a brief but incisive overview of an API Subaccount and its benefits. He then wraps up by explaining who offers them and whether you should, too. ### [Is the latest book on "Pentesting APIs" any good?](https://danaepp.com/is-the-latest-book-on-pentesting-apis-any-good?ref=apisyouwonthate.com) Dana recently received a book on pen-testing from Packt Publishing and shared his review of it. I won't spoil it here, so follow the link for the full breakdown. Thank you, Dana. ## Support APIs You Won't Hate When you become an member, you'll get access to members-only content while directly supporting our work. Your support helps us to keep making resources for the API community. [Become a member today](https://apisyouwonthate.com/#/portal) Thanks so much to our [members](https://apisyouwonthate.com/membership/): Al N, Kin L, Juxt, Alex R, Nolan S, Frank, James D, and Bill D. Your support means the world to us! Until next time, Alexander, Phil & Mike ### The "Don't Have Time to Create API Documentation" Paradox URL: https://apisyouwonthate.com/blog/dont-have-time-for-api-docs-paradox/ Last updated: 2024-11-15T13:40:41.000Z You are building an API for a reason. You need to integrate with another business to make money. You need to get data to a new frontend application to get more users to make money. You have important business to do, so why should you waste time creating API documentation? As an API consultant who constantly works in rescuing companies from messes they've made for themselves, one of the most common problems being solved is trying to figure out how an API works after the developer/team who built it have handed it off, left the company, or just plain forgotten. ## Why do we even need API documentation? Let me tell you a story about a unicorn startup who had infinite developers and silly deadlines, who blew all their money on nonsense. This company was focused on agile development. Standard cliche Silicon Valley stuff; move fast and break things. They would need to rush out an API to hit incredibly tight deadlines, so they would get everyone sat around the same ping pong table to work on it. The backend and frontend teams would all hash it out, write a bunch of code one-handed eating cold pizza with the other, and with a few days a prototype API was built and the frontend team was integrating with it. Weeks go by, and more functionality is needed, so a few new endpoints or properties get popped in here and there. So far so good. Months go by, and new clients need to integrate with the API. Unfortunately those clients were not sat around that ping pong table at the time, the original developers have forgotten or moved on, and the Slack channel doesn't have the random JSON examples in history any more because its been full up with gifs. ## If nobody remembers how an API works do you even have an API? This company was building new APIs every few months because they'd forgotten how the old ones worked. Several of their 40+ API teams were building v3, v4, or v12 of their APIs because it seemed quicker and easier than working it out. These new API versions were being created regularly with no meaningful differences from the previous versions, it's just that people honestly could not remember how the API worked. There were no unit tests or contract tests to glean this information from, and some of the APIs did not even define the JSON output using serializers or types of any sort, because they were pulling in undeclared information from multiple other APIs and services, merging it together, and sending *whatever* to the client. Some of the APIs were acquired through other startups being bought and folder into the company, and were built in languages none of the team understood, let alone knew how to get running. Sniffing HTTP traffic going through production environments is another approach which can help to build a partial contract in these black box situations, but you never know if you've caught every endpoint, or just the popular ones. You also can never know if a particular field is ever nullable, because it just so happens to have had values set in all the messages observed lately... Companies who forget how their APIs work generally have two options. 1. Hire an expensive expert to come and figure it out. 2. Build a new API from scratch. If you pick option one, I'll see you soon. If you pick option two, are you going to make the same mistakes again, or are you going to write API documentation this time? ## OpenAPI saves API Developers time API developers being asked to produce API documentation at the end of the process is always going to lead to complaints unless it comes from the top. It's hard to convince business to do that when it very clearly feels like the API is "done" now, and they need to let that team spend an extra week or two writing things down that they say won't help anything. This whole dynamic is absolutely backwards, and it reminds me of the lost decades of software engineers pretending they didn't need to write unit tests because it "took more time". We now know that more time is saved by using unit tests to cover code you're about to write (Test-Driven Development) or code you wrote earlier (Spike and Stabilize). Much like testing, building good quality API reference documentation feels like a huge chore, but using OpenAPI means you can speed up both the API documentation process, and the API development process too. 1. Write OpenAPI before you write any code, so you can use mock servers ([with Microcks, for instance](https://docs.bump.sh/guides/bump-sh-tutorials/mocking-with-microcks/?ref=apisyouwonthate.com)) to see if stakeholders like the shape of the API before you waste time building something they don't want. 2. When confirmed you can use the OpenAPI to contract test the API responses instead of thinking about how to contract test it later (well, [also with Microcks](https://docs.bump.sh/guides/bump-sh-tutorials/testing-with-microcks/?ref=apisyouwonthate.com), by the way), meaning your API is more reliable and better tested incredibly early on, something else people often leave until bugs start happening to think about finding time to set up. 3. Use OpenAPI for server-side validation, pointing application framework middleware, to save you wasting loads of time explaining the shape of your payload data when its already written in the OpenAPI you created in step 1 and used in step 2. When you have completed all these steps guess what. You already have a pretty solid API contract, which is the core basis of API documentation. [With a single CLI command](https://docs.bump.sh/help/continuous-integration/cli/?ref=apisyouwonthate.com#bump-deploy-file) you can run `bump deploy openapi.yaml` and that OpenAPI will be turned into [beautiful hosted API documentation](https://bump.sh/api-documentation?ref=apisyouwonthate.com) that can be shared with your whole team. ### Handing over to Technical Writers Once you’ve built an amazing API accurately described with top quality OpenAPI you have a more stable and reliable API that does exactly what it says it does, and that’s a pretty good place to be in compared to the usual “Slack channel with some vague outdated notes” or a Excel spreadsheet with some endpoints in it. Documentation is about a lot more than just showing endpoints and properties, and writing long form descriptions of operations and workflows can take a huge amount of time. This is a good handing off point to work with Technical Writers, who can massage the OpenAPI you have created into shape and expand on your points. We have a couple of tips to [improve OpenAPI descriptions](https://docs.bump.sh/guides/technical-writing/?ref=apisyouwonthate.com), and put the accurate quantity of polish and useful info. The more context you can give them in the OpenAPI document the better, but it’s far easier for them to look at OpenAPI and ask questions than it is to try and look at 10 different programming languages and try to figure all that out. ## API Documentation Saves Everyone Time Getting your OpenAPI built early in the process means you can use it to speed up the whole API lifecycle, and then at any point you can pop that into [Bump.sh](http://bump.sh/?ref=apisyouwonthate.com) for beautiful documentation, and keep updating it every time you add new functionality or make any changes to the codebase, using a [git-centric workflow](https://bump.sh/blog/from-swaggerhub-to-bump-git?ref=apisyouwonthate.com). Your clients know how to onboard without having to pester you and your team either directly or through support. Your technical writers have more context so they can do more without you. Your developers remember how things are meant to work so they can make smaller changes easier. Nobody is going to have to waste time and money on rebuilding entire APIs because they forgot how they work. --- *Thanks to* [*Bump.sh*](https://bump.sh/?ref=apisyouwonthate.com) *for giving me the time to work on this article. Originally published* [*over here*](https://bump.sh/blog/the-dont-have-time-to-create-api-documentation-paradox?ref=apisyouwonthate.com)*.* ### The OpenAPI Discriminator is Redundant & Confusing URL: https://apisyouwonthate.com/blog/openapi-discriminator-is-redundant/ Last updated: 2024-11-15T13:36:46.000Z If you’ve worked with OpenAPI v3.x, you might have come across the `discriminator` field in schemas. It’s often used alongside `oneOf`, `anyOf`, or `allOf` when you’ve got different variations of a type—polymorphism, essentially. At first glance, it seems pretty handy; it’s meant to help you figure out which specific schema to use when certain values are present, but JSON Schema can handle this out of the box without using the poorly supported OpenAPI-only keyword `discriminator`. Let’s dive into why the `discriminator` is probably an outdated concept that you can skip struggling to learn, by looking through a couple of examples to show how things can be done without it. ### What’s the Discriminator All About, Anyway? In OpenAPI, the `discriminator` is a field that helps you figure out which schema you’re dealing with when you’ve got multiple possibilities. It’s kind of like a switch that tells you which specific object or type your payload is going to conform to, using a specific field in the input—often something like a `type` field. ``` components: schemas: Animal: oneOf: - $ref: '#/components/schemas/Dog' - $ref: '#/components/schemas/Cat' discriminator: propertyName: type mapping: dog: '#/components/schemas/Dog' cat: '#/components/schemas/Cat' Dog: type: object properties: type: type: string const: dog barkVolume: type: integer Cat: type: object properties: type: type: string const: cat whiskerLength: type: integer ``` Here, we’ve got an `Animal` schema that could either be a `Dog` or a `Cat`. The `discriminator` is saying “look at the `type` field in the payload to figure out whether this is a `Dog` or a `Cat`.” If the `type` field says `dog`, then OpenAPI knows to check against the `Dog` schema, and if it says `cat`, it’ll check against the `Cat` schema. Perhaps you send a payload like this: ``` { "type": "dog", "barkVolume": 10 } ``` It will use the `Dog` schema. ### But Here’s the Thing... The `discriminator` doesn’t actually do anything in terms of validation. It’s only there as a hint to speed things up for tooling like code generation tools, so they can quickly figure out which schema to use in certain situations. Whether or not you have the `discriminator`, the validation will still work, documentation can show available options, mocking tools can generate a sample response by picking one of the subschemas, everything will work just fine. Without it, you can still rely on the natural shape of the data. This particular JSON instance has `type: dog` in there, and that acts as the switch in the oneOf by matching against the `const: dog` in the subschema. ``` components: schemas: Animal: oneOf: - $ref: '#/components/schemas/Dog' - $ref: '#/components/schemas/Cat' Dog: type: object properties: type: type: string const: dog barkVolume: type: integer Cat: type: object properties: type: type: string const: cat whiskerLength: type: integer ``` If you have a property to act as a switch then you're fine. If you don't have one then discriminator could never have worked, because it's fairly rigid like that. The oneOf approach does not need a single specific field to provide a `mapping`, it can use any combination of values, so even without the `type` field you could have different required properties on `Dog` objects (like `barkVolume`), and `Cat` objects (like `whiskerLength`), and that would be enough to select a scheme. ### Fixing Examples of Discriminator Here's an example from another documentation provider who have an example of `discriminator` to help pick between different `powerSources` for a vehicle. ``` schema: discriminator: propertyName: powerSource mapping: electricity: "#/components/schemas/ElectricVehicle" gasoline: "#/components/schemas/FueledVehicle" human-energy: "#/components/schemas/PedaledVehicle" anyOf: - $ref: "#/components/schemas/ElectricVehicle" - $ref: "#/components/schemas/FueledVehicle" - $ref: "#/components/schemas/PedaledVehicle" ``` ``` components: ElectricVehicle: type: object properties: powerSource: description: How is the vehicle powered. type: string example: electricity ... ``` We could just delete that whole `discriminator` object, and change `example: electricity` to `const: electricity`. ``` schema: anyOf: - $ref: "#/components/schemas/ElectricVehicle" - $ref: "#/components/schemas/FueledVehicle" - $ref: "#/components/schemas/PedaledVehicle" ``` This will actually work better, and give more meaningful validation on `powerSource`. ### Picking Between Schemas I think people got a bit hooked on the idea of discriminators because it's "the way to do polymorphism in OpenAPI". That may have been true in OpenAPI v2.0, but OpenAPI v3.0 got the `anyOf` and `oneOf` keywords which handle that. Some tools perpetuate the reliance on `discriminator` by pushing users to use them in order to show selectors allowing users to switch between different `oneOf` and `anyOf` subschemas. Modern OpenAPI documentation does not need to do this. For example, [Bump.sh](http://bump.sh/?ref=apisyouwonthate.com) will take the end of each of the `$ref`, turning `"#/components/schemas/ElectricVehicle"` into `ElectricVehicle` and `"#/components/schemas/PedaledVehicle"` into `PedaledVehicle` in the interface. ![image.png](https://storage.googleapis.com/bump-blog-resources/2024-11-04-the-discriminator-in-openapi-is-generally-redundant-and-confusing/image%201.png) If you’d like more human-readable names you can add the `title` keyword to each of those referenced schemas. ``` components: ElectricVehicle: title: Electric Vehicle type: object properties: powerSource: description: How is the vehicle powered. type: string const: electricity ... ``` That will then update the selectors to use the provided name instead of the generated one. ![image.png](https://storage.googleapis.com/bump-blog-resources/2024-11-04-the-discriminator-in-openapi-is-generally-redundant-and-confusing/image%202.png) Between `oneOf`, `anyOf`, and `title`, most documentation tooling can create brilliant interactive selectors without forcing people to write out each possible mapping. ### Wrapping Up The `discriminator` in OpenAPI v3.x is less useful than it might seem at first. It doesn’t affect whether a payload is valid, doesn't help with documentation, mocking, or anything much else. The main reason for using it is to help tools like code generators and their resulting output code, which need to know which schema to pick quickly. In that case, sure, add these optimization shortcuts to your OpenAPI with `discriminator` speed things up, but in most cases you’re better off keeping things simple and letting the data itself decide which schema to use. ### Powerful HTTP/API Clients: Alternatives to Postman URL: https://apisyouwonthate.com/blog/http-clients-alternatives-to-postman/ Last updated: 2025-02-27T18:18:31.000Z Postman is an incredibly popular tool for testing and developing APIs. In [*Build APIs You Won't Hate, 2012*](https://apisyouwonthate.com/books/build-apis-you-wont-hate/) I called it the Adobe Photoshop of API development, the tool all of us would use all day every day to get our APIs off the ground, and make sure they were working beyond that. It started off life as a simple, free, Chrome extension for sending HTTP/API requests, managing collections of complex requests so you could send them over and over, and this was a rudimentary "test my API is working" workflow for those of us didn't want or need to get stuck into a full test suite, and let us poke-and-prod APIs for debugging. Since 2012 Postman have added so many more features, rewriten the app and UI several times, added so many features, covering API documentation, mocking, testing, and monitoring, that it became hard to use it as a simple HTTP client. Those of us who wanted a tool for simple poke-and-prod use-cases and sharing collections have many high quality HTTP/API clients to chose from. ## Insomnia [Insomnia](https://insomnia.rest/?ref=apisyouwonthate.com) is a powerful, open-source REST and GraphQL client that emphasizes simplicity and productivity. It offers a clean UI, collections, environment variables, cloud workspaces with team collaboration, testing, mock servers, and testing. ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2024/11/insomnia.png) Insomnia started off life open-source and was acquired by Kong, who have been putting in serious effort to get Insomnia onto feature parity with Postman, meaning it's got some of the same bloat problems as Postman. That does however make it an easy choice to anyone looking to move away from Postman but maintain most of the same functionality and workflow, especially with a [Run in Insomnia](https://insomnia.rest/create-run-button?ref=apisyouwonthate.com) button working similarly to the [Run in Postman](https://learning.postman.com/docs/publishing-your-api/run-in-postman/creating-run-button/?ref=apisyouwonthate.com) button. Insomnia is OpenAPI-compatible via an import feature, but the environment variable support was a little broken and lead to some difficulty creating a fluid workflow: [Read more here](https://docs.bump.sh/guides/bump-sh-tutorials/try-requests-in-insomnia/?ref=apisyouwonthate.com). - **Availability:** ✅ Web, ✅ macOS, ✅ Windows, ✅ Linux. - **Website:** [insomnia.rest](https://insomnia.rest/?ref=apisyouwonthate.com) ## Bruno [Bruno](https://www.usebruno.com/?ref=apisyouwonthate.com) is a newer open-source HTTP client, which aims to use Git (other other source control) to provide the collaboration on collections instead of a SaaS service. ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2024/11/bruno.png) Bruno supports OpenAPI Import, which pulls in an OpenAPI document and creates a simple collection of requests and responses out of it. It's a little fiddly, and I'm not sure how you'd handle reimporting an OpenAPI that has evolved over time, but it saves you having to build your own. $19 for a two-year license enables a few extras like the OpenAPI Designer which is - **Availability:** ❌ Web, ✅ macOS, ✅ Windows, ✅ Linux. - **Website:** [usebruno.com](https://www.usebruno.com/?ref=apisyouwonthate.com) ## Hoppscotch Formerly known as Postwoman, [Hoppscotch](https://hoppscotch.io/?ref=apisyouwonthate.com) is a fast, open-source API development tool available as a web app. It supports REST, GraphQL, and WebSocket requests, and provides real-time collaboration through workspaces. ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2024/11/hoppscotch.png) Great for those who prefer lightweight, web-based tools with real-time collaboration, who aren't looking for a desktop experience. The interface allows for import from OpenAPI, Postman, Insomnia, Gist, or HAR. - **Availability:** ✅ Web, ❌ macOS, ❌ Windows, ❌ Linux. - **Website:** [hoppscotch.io](https://hoppscotch.io/?ref=apisyouwonthate.com) ## RapidAPI (formerly Paw) RapidAPI provides a comprehensive platform for API development, testing, and sharing. Originally known as Paw, it is now integrated into RapidAPI, offering a wide range of API-related tools, but still completely free. This is a native macOS application, unlike many of the others which are desktop-wrapped web applications, so it should be a little lighter on your CPU. ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2024/11/rapidapi.png) Best for developers working within the RapidAPI ecosystem, which provides API catalogs, an API marketplace to enable discovery and monetization, and even an API gateway putting it on par with the Kong + Insomnia experience. It does have OpenAPI support, but the website calls it Swagger which suggests it will be older versions. OpenAPI v3.0 is supported by one of the many [extensions](https://paw.cloud/extensions?ref=apisyouwonthate.com) available. - **Availability:** ❌ Web, ✅ macOS, ❌ Windows, ❌ Linux. - **Website:** [paw.cloud](https://paw.cloud/?ref=apisyouwonthate.com) ## GetAPI GetAPI is another native macOS application which is very clear about what it is: a "dead simple HTTP client". It is designed to be easy to use, making it accessible for developers of all skill levels. Unlike RapidAPI this is not free, but freemium, with a pro license costing $39 to provide a years worth of updates. ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2024/11/getapi.jpeg) Created by [TablePlus](https://tableplus.com/?ref=apisyouwonthate.com), the developers behind the excellent SQL client of the same name, GetAPI has the feeling of many macOS applications; It's here to be easy to work with, help individual developers out, but isn't trying to get stuck into edge-case enterprise stuff, confusing everyone else in the process. - **Availability:** ❌ Web, ✅ macOS, ❌ Windows, ❌ Linux. - **Website:** [getapi.com](https://getapi.com/?ref=apisyouwonthate.com) ## New Contender: Scalar [Scalar](https://scalar.com/?ref=apisyouwonthate.com) cannot be called popular yet because it's brand new! This simple HTTP client aims to be the first entirely OpenAPI-centric HTTP client, with [online](https://client.scalar.com/workspace/default/request/default?ref=apisyouwonthate.com) and [desktop](https://scalar.com/download?ref=apisyouwonthate.com) flavours available. ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2024/11/scalar.png) All of the other tools support OpenAPI to some extent, but generally this takes the form of an "Import" step, only used to get an API collection built up. After that you've got to spend the rest of your life repeatedly updating changes in the API collection and the API description it's meant to be based off of. Whether you're using the [API design-first or API code-first workflow](https://apisyouwonthate.com/blog/api-design-first-vs-code-first/), you're going to be adding new endpoints and evolving requests and responses throughout the lifecycle of the API, meaning that there is never a stage when the OpenAPI is "done". Instead of trying to navigate the thankless task of keeping OpenAPI descriptions in sync with yet another proprietary collection format, Scalar takes a new approach, of making the entire process work off of your OpenAPI document. Open up a local `openapi.yaml` and make changes, as you add endpoints you'll see new operations pop up. As you add new examples they'll be saved to the document. The Scalar team have gone all in with OpenAPI, not just in this client, but are building an [ecosystem of handy open-source OpenAPI tools](https://github.com/scalar/scalar?tab=readme-ov-file&ref=apisyouwonthate.com#projects) that solve different bits of the lifecycle. This client in-fact started life as an interactive API client embedded into their API reference documentation offering. - **Availability:** ✅ Web, ✅ macOS, ✅ Windows, ✅ Linux. - **Website:** [scalar.com](https://scalar.com/?ref=apisyouwonthate.com) ## Honorable Mentions Here are some amazing HTTP clients that didn't make the article at first, but absolutely should be considered. - [Yaak](https://yaak.app/?ref=apisyouwonthate.com) \- Same developer behind Insomnia who has had another go at it. - [Hurl](https://hurl.dev/?ref=apisyouwonthate.com) \- A command line tool that runs HTTP requests defined in a simple plain text format. - [Posting](https://posting.sh/?ref=apisyouwonthate.com) \- A beautiful open-source terminal app for developing and testing APIs. ### APIs Around the World URL: https://apisyouwonthate.com/newsletter/apis-around-the-world/ Last updated: 2024-11-01T14:23:19.000Z **APIs You Won't Hate is proudly global**, both in community and in teamwork. Behind the scenes, our newsletter, podcast, and thriving community are run by just three people—spread across three continents! [Phil Sturgeon](https://philsturgeon.com/?ref=apisyouwonthate.com) in the UK, [Alexander Karan](https://www.alexanderkaran.com/?ref=apisyouwonthate.com) in Australia, and [Mike Bifulco](https://mikebifulco.com/?ref=apisyouwonthate.com) on the US east coast. We recently caught up across time zones for a new podcast episode you won’t want to miss. \-Mike --- ## The API Round-Up ### [RateLimit header fields for HTTP](https://datatracker.ietf.org/doc/draft-ietf-httpapi-ratelimit-headers/08/?ref=apisyouwonthate.com) The latest draft from the IETF’s HTTPAPI working group proposes standardized headers for rate limiting, which could mean fewer custom implementations and clearer communication with API consumers. ### [How OpenAPI Ensures Reliable API Communication](https://bump.sh/blog/how-openapi-ensures-reliable-api-communication?ref=apisyouwonthate.com) OpenAPI's standardized documentation transforms API reliability, from setting clear expectations to easing integration. This quick read explains why OpenAPI is a go-to for making seamless developer experiences. ### [OAuth 2.0 Security in OpenAPI](https://www.speakeasy.com/openapi/security/security-schemes/security-oauth2?ref=apisyouwonthate.com#authorization-code-flow-object?utm%5Fsource=apisyouwonthate&utm%5Fcampaign=241101) Ever wondered about describing OAuth2.0 in OpenAPI? Here's everything you need to know, all in one place. ### [The "Don't Have Time to Create API Documentation" Paradox](https://bump.sh/blog/the-dont-have-time-to-create-api-documentation-paradox?ref=apisyouwonthate.com) In a twist of irony, skipping documentation to save time often leads to even bigger time drains down the road. This post tackles why investing in clear API docs upfront actually saves you—and your users—time in the long run. ### [Speakeasy's $15M Series A](https://www.speakeasy.com/post/fundraising-series-a?ref=apisyouwonthate.com) Speakeasy, a longtime sponsor of our community, just hit a huge milestone with their Series A fundraise! Dive into their journey to learn the strategies and challenges they tackled to secure funding. We're thrilled for them and excited to see what's next! ### [The story of web framework Hono, from the creator of Hono](https://blog.cloudflare.com/the-story-of-web-framework-hono-from-the-creator-of-hono/?ref=apisyouwonthate.com) This behind-the-scenes look from Hono’s creator, detailing how the framework came to life and what makes it a powerful, minimal tool for developers. ### [Introducing Express v5: A New Era for the Node.js Framework](https://expressjs.com/2024/10/15/v5-release.html?ref=apisyouwonthate.com) Express.js, a staple in the JavaScript and TypeScript communities, has launched its long-awaited v5 update. This release introduces support for modern Node.js versions, enhanced security measures, and streamlined APIs, all while maintaining its minimalist approach. ### [Fuzz Testing Go HTTP Services](https://packagemain.tech/p/fuzzing-http-services-golang?ref=apisyouwonthate.com) This guide walks through fuzz testing—a method of bombarding your service with random data to uncover edge cases—for Go-based HTTP services. Learn how fuzzing can boost reliability and make your APIs more resilient. ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2024/08/speakez-1.jpg) ## Speakeasy: Write OpenAPI your users love A great spec is the beginning of a great API devex. The OpenAPI Hub has all the info you need to craft a spec ready for docs, SDKs, and delighting users [Learn More](https://speakeasy.com/openapi?utm%5Fsource=apisyouwonthate&utm%5Fcampaign=241101) ## APIs You Won't Hate The latest from the team at API's You Won't Hate. ### [The APIs You Won't Hate Starter Pack for Bluesky](https://go.bsky.app/BNkzSpM?ref=apisyouwonthate.com) There has been a large migration of tech folks from Twitter to Bluesky over the past few weeks. It's feeling a whole lot like the good old days on Bluesky - and we'd love to see you there. If you're on Bluesky, please [drop Mike a line](https://bsky.app/profile/mikebifulco.com?ref=apisyouwonthate.com) to get added to the Starter Pack! ### [🎙️ A pod worth casting for](https://apisyouwonthate.com/podcast/catching-up-with-phil-mike-and-alexander) Though we've been working together for a long while, it turns out that Alexander, Phil, and Mike had never *all* recorded a podcast together before. We caught up to catch up on what everyone's been up to these past few months, to talk APIs, and to gab about the future of this very community. --- ## From our Community Articles written and shared in our free [Slack community](https://slack.apisyouwonthate.com/?ref=apisyouwonthate.com). ### [Laravel API Course](https://apiacademy.treblle.com/laravel-api-course/intro-and-setup?ref=apisyouwonthate.com) From Steve McDougall, a major contributor to our community and past podcast guest, released a full course on setting up Laravel for API development. Steve is actively looking for his next role, so if you’re hiring a top-notch API DevRel pro, reach out! ### [Attacking APIs using JSON Injection](https://danaepp.com/attacking-apis-using-json-injection?ref=apisyouwonthate.com) Dana Epp delves into how attackers can manipulate JSON payloads to exploit API vulnerabilities, leading to unintended behaviors or security breaches. Understanding these tactics is crucial for developers aiming to fortify their APIs against such threats. ### [Deprecating REST APIs: A Developer's Guide](https://zuplo.com/blog/2024/10/24/deprecating-rest-apis?ref=apisyouwonthate.com) Zuplo's latest article offers a comprehensive walkthrough on gracefully phasing out REST APIs. It covers best practices for deprecation notices, managing client transitions, and ensuring minimal disruption during the process. A must-read for developers aiming to maintain robust and user-friendly APIs. ## Support APIs You Won't Hate When you become an member, you'll get access to members-only content while directly supporting our work. Your support helps us to keep making resources for the API community. [Become a member today](https://apisyouwonthate.com/#/portal) Thanks so much to our [members](https://apisyouwonthate.com/membership/): Al N, Kin L, Juxt, Alex R, Nolan S, Frank, James D, Bill D, and Ali S. Your support means the world to us! Until next time, Alexander, Phil & Mike ### Catching up with Phil, Mike, and Alexander URL: https://apisyouwonthate.com/podcast/catching-up-with-phil-mike-and-alexander/ Last updated: 2024-11-01T14:22:33.000Z Though we've been working together for a long while, it turns out that Alexander, Phil, and Mike had never all recorded a podcast together before. We caught up to catch up on what everyone's been up to these past few months, to talk APIs, and to gab about the future of this very community. #### Transcript \[00:00:00\] ****Mike Bifulco:** Hello friends, and welcome back to APIs you won't hate. My name is Mike Biko, one of your co-hosts of the podcast. For once I am hanging out with two of my favorite people in the world. \[00:00:09\] I've got Phil Sturgeon and Alexander Karen on the line with me. Alex, how are you, \[00:00:13\] ****Alexander Karan:** yeah. Good. \[00:00:15\] ****Mike Bifulco:** I'm, fully convinced. \[00:00:18\] ****Phil:** Yeah, same, same trouble here. I was out flyering for a charity, Crowdfunder all yesterday and in the morning \[00:00:23\] we started in the coffee shops and in the evening we gotta all the pubs. So I was very hyperactive and then very hungover. \[00:00:29\] ****Mike Bifulco:** I think of all the things I do, this is the most challenging to schedule. Just, just in case. Someone listening is new to this experience. I am on the east coast in the us. Phil is usually somewhere in Western Europe. And Alexandra is in Australia, which means we are basically spanning the globe which is why we only catch up maybe once a year. \[00:00:47\] It is really, really nice to see you both. I actually cannot think of the last time I've been on a call with the two of you at the same time. \[00:00:54\] ****Phil:** Yeah, I think the first time, \[00:00:55\] Yeah, \[00:00:56\] ****Alexander Karan:** it's definitely the first time. I mean, \[00:00:58\] I've spoken to Phil separately and Mike \[00:01:00\] separately, but now together. \[00:01:01\] ****Mike Bifulco:** Yeah. Now that, now that you've seen us both in the same room, you can tell we're not the same person too, which is kind of wild. \[00:01:07\] ****Phil:** I mean, it's similar \[00:01:08\] ****Mike Bifulco:** I'm, I'm interested in hearing what you've both been up to. \[00:01:10\] ****Alexander Karan:** Okay. So, Let's start what's keeping me busy outside of work because, so I, I've been looking for a new hobby recently. And 'cause you know, that's what you do when you're in your thirties. \[00:01:21\] ****Phil:** class right? \[00:01:24\] ****Alexander Karan:** yeah, that's it, right? For, for, for middle age, get ready for middle age. So I, I decided to get really into coffee. And like now, I, you know, work at Big Tech and get paid a bit more. Because I'm not working for a startup anymore, I decided to go like the full whole hog and like, spend like a used small, a small used car amount of money on like a proper barista coffee machine at home, a grinder, like all this special equipment. And then like, you know, and like I'm just making like really great espressos \[00:02:00\] and cappuccinos in the morning now and getting \[00:02:02\] like lo freshly roasted coffee. And then there's like that little nagging voice in the back of my head that's like, climate change is affecting the amount of area we could grow coffee. \[00:02:12\] And I'm like, wait a minute. There's a time limit on this hobby. \[00:02:15\] ****Phil:** Yeah. Right. Are you, are you going full fancy and getting the, getting the beans that have been passed through a spider monkey or something? Or, or what's your favorite choice of bean? \[00:02:22\] ****Alexander Karan:** So I joined, so in per, so I'm in the west coast of Australia. Just had to double check that for a second. So it's actually quite small, but we actually have like a ridiculous amount of like specialty roasters. Like for some reason we have lots of specialty roasters, lots of gin distilleries, lots of \[00:02:39\] breweries, like all these fans chocolatier \[00:02:43\] ****Phil:** not much else to do out there. Is there \[00:02:45\] ****Alexander Karan:** just eat a drink. That's it. So, so yeah, I joined this coffee club and they like send you freshly roasted coffee every four weeks from like different places. \[00:02:53\] Um, yeah, so I'm just experimenting at the most. That's my, my personal life is I've been one of those like pretentious \[00:03:00\] people that's like, takes 10 minutes to make a coffee and like, you know, cleaning the machine, it. \[00:03:10\] ****Phil:** I think that we've gone on a similar journey, but mine is such a, like a dumber, smaller version, which is I have finally decided that I'm bored of instant coffee and I decided to treat myself to an arrow. Press go, like the travel one. But I've had it for like six months and I've never got it out of the box. I've just still got like a massive jar of instant coffee because my theory was like in the van, I can take it, bike back in, I can take, you know, I can keep it in the van. I've got a little ho and a little stove and stuff, so I can, I can boil water in the van and then I can pull that over. But the problem is trying to keep I can. I don't get through enough van coffee that I can justify having like buying a whole bag of ground stuff. So myself, grinder minutes in this random \[00:03:51\] Tecos park. \[00:03:53\] ****Alexander Karan:** you do you, you do. You do. Because like you, like as I get into this hobby, you grind \[00:04:00\] fresh. That's how you get the best taste, right? Like you, \[00:04:02\] ****Phil:** Yeah. Yeah, yeah. \[00:04:04\] ****Alexander Karan:** but \[00:04:05\] a fresh is \[00:04:05\] ****Phil:** I got the. I had to, like, I had a Kelly kettle, if you've heard of those, like the little you put like little twigs in this. It's kinda like a rocket stove. You make a little fire and the, and the water is stored in the chamber around the outside, so the fire boils the water. And so I used to be like, \[00:04:19\] sat in a, a car park or wherever the hell I was sleeping last night, A random lay by on the side of the remote, just like wherever. And I'm there just making a fire with everyone going by. Like, what is this psycho doing? Um, so then the whole process of and grinding just felt like a little bit too much on top. \[00:04:34\] By that point I was like, it's taken me 20 minutes. I've got some boiling water. I'm putting some bloody instant coffee in there. But yeah, if you actually are a normal human being with like a house and a space to put things, you can get pretty nice for your coffee. \[00:04:46\] ****Alexander Karan:** Yeah, but I, I didn't used to be into coffee until I moved into Australia. Like when I moved here, that's when I really, 'cause they're really into coffee here. They're like really into literally just like really good food. It is. 'cause all you do is \[00:05:00\] chill out in the sunshine and like eat and drink and have a good life. \[00:05:04\] Right? Like that's kind of what Australia's all about. Like, it's not really about doing anything else. So yeah, that Australia got me into coffee. And now working at Big Tech has given me the, the funds necessary to \[00:05:16\] really get into coffee. \[00:05:18\] ****Phil:** Alright, I like that. And so I like that you picked that as your hobby. I, I've not really considered coffee as a hobby before, but \[00:05:24\] ****Alexander Karan:** feel like I have to say that now with the level of effort that I've \[00:05:27\] put in. Right. Like that's \[00:05:29\] why. \[00:05:29\] ****Phil:** You si similar money could have got you like a kayak and all the gear or like a pair of skis or something. But if it's, if it's up in that ballpark then \[00:05:37\] ****Mike Bifulco:** yeah. Alexander, I'm so excited to hear that now that you're part of the club, I can show you the secret handshake and all that. my \[00:05:43\] ****Alexander Karan:** You are. You are one of those too. Yeah. I found a \[00:05:45\] few with Atlassian. \[00:05:47\] ****Mike Bifulco:** So I'm very similar to you. I didn't actually drink coffee until. A few years ago like just never, never bothered. \[00:05:54\] But somewhere around starting my first startup due to something, my co-founder Sean said to me, I \[00:06:00\] started drinking coffee. 'cause he told me that successful startup founders tend to be early risers and that just like. Burrowed into my brain, and I was, you know, up earlier and earlier as a result of that. \[00:06:09\] However, it's gone very far from there. And I've, I've come to understand that I really only enjoy espresso for the most part. So I make myself an espresso every morning. And when Covid hit the US and I sensed that the world was shutting down, I took my coffee budget for the year and I bought an espresso setup for home. \[00:06:24\] ****Phil:** Nice. You went prepper. But for coffee, big time. I've got 20 years of coffee in the basement. No one's fucking getting anywhere near it. \[00:06:31\] ****Mike Bifulco:** That's right. I've got the spreadsheets to prove it, and I've been tracking my expenses with it ever since. And I'm, I'm happy to say that as of about the beginning of this year, it's, it's now technically paid for itself. You know, in the amount of money I've saved by not buying coffee and coffee shops. \[00:06:44\] I haven't stopped doing that. But I also, you know, I'm, I'm drinking additional coffee. So yeah, we will have to spin up the espresso you won't hate podcast next Alexander. And we can, we can go ape on that. \[00:06:55\] ****Phil:** For me, it's more of a capitalist thing, right? So I'm traveling about all over the place and I'm going up \[00:07:00\] and down in my van. And I love, I love, just like, I've made a few coffees. I got a little bioethanol stove that I made out of, like I cut a in half and like converted that into, you can make into a little, um. And so it's bioethanol, so it comes from plants. Super efficient. And yeah, I literally just made coffee on a, a bench, a service station near Scotland, somewhere on the Board of Scotland. And I was like right outside the Costa with people coming in and out buying their plastic cups full of shi. And I was just like, \[00:07:26\] big middle fingers up the costa. \[00:07:28\] I'm just made a, it was, it was still instant and it was a bit, but it was, I'd rather have a coffee than give what is it, Pepsi that owned that. So. \[00:07:36\] ****Alexander Karan:** Costa's not that good. Well, it's, it's four pounds now for a coffee at Costa. \[00:07:40\] ****Phil:** It's something ludicrous. I'm like, at this point I might as well be getting a pint. I know it's eight 30 in the morning, but I'm just gonna buy a pint. It's the same price. \[00:07:47\] ****Mike Bifulco:** I had Wait. That'll wake you up. \[00:07:50\] ****Alexander Karan:** yeah. That's more expensive than it is here. Like that's with, with the exchange, you know? That's crazy. \[00:07:56\] Wow. \[00:07:57\] ****Phil:** Anyway you have a bike in the background and I'm \[00:08:00\] excited \[00:08:00\] 'cause you have been on a Me and Voyage. You wanna tell me about that? \[00:08:03\] ****Mike Bifulco:** man. Yeah, it's been a whole journey. I if, if you've listened to past episodes of this thing, you'll know that Phil and I pro probably met originally because of our sort of overlap of tech stuff and cycling, but, that's the thing. Yeah. And it's genuinely been about two years since I've been able to ride. \[00:08:18\] I've been dealing with a hip injury since 2018\. And two surgeries and a whole lot of MRIs later. and a bunch of physical therapy and all those other things, I've finally just started riding again. And I, I am hesitant to say I'm back to fighting strength 'cause I'm definitely not. But I'm like able to get out and tool \[00:08:33\] around a little bit and \[00:08:35\] ****Phil:** Yeah, \[00:08:36\] That's good, man. That's gotta feel real good. \[00:08:37\] ****Mike Bifulco:** It feels really, really good. Yeah. It's, it's definitely been cool to see. I yeah, it's, it sits behind me as a reminder while I'm on work calls that I need to go do that more. But yeah, it went from like actually not being able to finish one pedal stroke in April to being able to get out and ride 10 or 15 miles at a time now, which is, which is. \[00:08:54\] Just life changing for me, like \[00:08:56\] the mental health part of life balances out a lot from that. You \[00:09:00\] know, I was re relegated more or less just to walking around the neighborhood, which is nice, you know, it's great. It's just not the same, you know what I mean? \[00:09:06\] ****Phil:** Yeah, it's \[00:09:08\] another one of those situations where like you can just casually make number go up. Like you don't need to worry about like busting out cheeky centuries straight away. Like you'd have to do a hundred miles right now, like you can just go and do, yeah, 10 and then 15, and then 20 or maybe smaller increments, but you, you get to see number go up and that just is progress again. \[00:09:24\] So \[00:09:25\] that's really nice. The difference between like zero miles and, and, and 10 is fucking infinite, isn't it? So now you've just gotta edge it up a little bit. \[00:09:32\] ****Alexander Karan:** Yeah, it's great riding a bike. There's like, it's just, you know, because you can go so far, you can do so much, you know, and I, I think also the best thing is when you've got a good bike, you can also like, just like hardly ever use a car, like just bike everywhere. It's great. \[00:09:47\] ****Phil:** Mm. \[00:09:47\] ****Mike Bifulco:** It is one of very few things I do that occupies all of my limbs in a way where I'm definitely not like pulling out my phone to distract myself, \[00:09:54\] ****Phil:** Handlebar Mount will fix that for you. You can get it \[00:09:56\] right up there on the top. you. \[00:09:58\] can be tweeting and.\[00:10:00\] \[00:10:00\] ****Mike Bifulco:** This is not something I'm looking to fix. Yeah, it's really nice. Like I, I throw my phone in my pocket and don't think about it until I'm back home, which is also a massive feature too, you know? \[00:10:08\] Yeah, man, I've missed it. I'm, I'm happy to see Phil, what, what's your life looking like as far as that all goes these days? \[00:10:14\] ****Phil:** Absolutely mental. I currently am \[00:10:17\] not riding as much bikes as I would like. I did a few nuts ones a month or two or three ago I did UK two Zurich, which was a good laugh. And yeah, that one was, Four 50 miles in four and a half days to get to a conference. So that was, that \[00:10:33\] was a good time. \[00:10:34\] ****Alexander Karan:** That's, that's insane. \[00:10:36\] ****Phil:** Yeah, it was a bit rough. The, the, it, every single day I was dealing with headwinds, mostly uphill. 'cause I'm going to the Alps from the sea, right? It was pretty much consistently up headwind and wet and some days were worse. Before it was, it was pretty wild. \[00:10:53\] But oh, I lost my, I lost my rain jacket. On with one hole with, with a day and a half to go. And as I was like \[00:11:00\] cycling through Alps and it was just pouring down with rain, it was when Switzerland had that, these really bad floods where like entire towns were getting washed away. And I'm just cycling around in like my hoodie that was just soaked through and just \[00:11:10\] shivering. \[00:11:11\] And I was like, well, if I get cold, I'll just pedal outta Problem solved. And. \[00:11:23\] ****Alexander Karan:** You did get ill from that. Did you? \[00:11:25\] ****Phil:** I'm generally ill of something, but that's just 'cause my life is madness. I mean the, the main thing I've got going on right now is my charity is buying 70 acres of land in bath, which is actually where I'm from. One of the, the two accents I flip between is the Bristolian West Country one and the rather posh bath accent that falls out now and then but I'm \[00:11:41\] from, I'm from Bath and it's really exciting to be working on buying some land where I'm actually from. People in Cornwall don't really like it when someone from not Cornwall buys some land. And people in Wales really don't like the English buying at Wales. But now if anyone says, what the heck do you think you're doing buying this land? I'll be like, I'm from here. Shut up. \[00:12:00\] And we're doing a which is 50,000 pounds. \[00:12:04\] I've been banging the drum for six weeks, but we're now like, so it gone over while I'm. \[00:12:13\] ****Mike Bifulco:** Wow. That's \[00:12:14\] incredible. Phil. Can, Can, you give a little more details on the charity? Like how, how long have you been doing it? What, what have you all done and how much, how many trees have you planted and how much land have you now confiscated for, for the earth? \[00:12:26\] ****Phil:** I confiscate for the Earth. I am gonna Google my own charity one moment. But so basically protect Earth has been going since the end of 2020\. So it's four years and we planted like 4,000 trees in our first year. And then like. We've got up to a hundred thousand trees after four years. So there's a, a big, a big number go up. and we are hoping to plant another a hundred thousand this year. So the growth is exponential, but we've, we've basically outstripped all of our existing funding partners. Which is wild. 'cause we used to be in a position where \[00:12:57\] we had more. Trying to for \[00:13:00\] trees than we had places to put them, and now we're the other way. So it's a bit wild. But we, we started off by just planting. There's, you know, a couple of farms nearby that, that had some corners they weren't really using for food production, and they thought it would be nice if it, if we welded that up. And now we've gone to much bigger stuff where we're literally buying our own land and still working with landowners and doing the sides of motorways and hotel grounds and working with councils to plant mini forests and parks and all these amazing things. But really exciting when we get our own. Degraded kind of sheep blasted hillside. And even the, even the, the, the sheep farmer there said this has been overgrazed. Like when sheep farmers are saying this has been overgrazed, you know, it's, it's having a rough one. And we're reforesting that and \[00:13:55\] another 27 acres that we're leasing, which is a bit of an experiment in like trying to \[00:14:00\] find a huge amount of money just to buy the land. And then start to do some work is a bit silly. Well, not a bit silly. We can protect it forever, but it's quite an expensive way to go about it. But yeah, this experiment, leasing a small bit, doing all the work, sticking a covenant on the land that says, Hey, whoever ends up with this in a few years time, you've gotta protect this woodland. \[00:14:18\] And then we can shove off and work on something else somewhere else. So our land is, is going. \[00:14:24\] ****Mike Bifulco:** That is truly staggering. I, I feel like I, I bump into people often who know you, Phil, but I also bump into a lot of people who don't know you, and that is more challenging because I have to then describe you to them. Like, oh yeah, this, this guy who I, I ki I work with, I kind of work with who I know I've never met. \[00:14:40\] I. And he lives in a van. He's smart, I promise. You know, and it's like, I, I feel like if I had a headline that was just continually updated where it's like the man has planted a hundred thousand trees and, you know, re forested this many acres of land, like if you could just give me that tweet length thing updated every week, it would help my \[00:14:57\] endeavors a great deal. \[00:14:59\] ****Alexander Karan:** actually doing \[00:15:00\] something about climate change. \[00:15:01\] ****Phil:** Yeah. Another thing or two about climate change. Yeah. I've been like currently getting in a fight with like the, the Dale Vince who's got like a net worth of 107 million or billion, I don't even know. Just this like really rich dude that runs a major energy company and he is like, oh yeah, heat pumps are bad. \[00:15:16\] We don't need heat pumps. We can just like. Turn 70 of all of England into like hay and then collect that to make, and we can. Then we've got all this about heat pumps, so. Yeah, trying to educate \[00:15:36\] people about the climate crisis and do a bunch of tree planting because the trees that I plant are somewhat pointless. If emissions keep going up, I don't want to, you know, capture some carbon from the increase of terrible things that are happening. I want my trees to be sucking up, you know, atmospheric carbon as line go down on how much we. \[00:15:55\] ****Alexander Karan:** Yeah. Yeah, it's, it's, I think that's difficult \[00:16:00\] anywhere, right? Like, you know, in Australia, w we have like, our emissions per capita is incredibly high, like \[00:16:06\] per person for such a small population. But also like we export so much fossil fuels, you know, like, you know, \[00:16:14\] like, and WW Woodside, the company that. I shouldn't name, but I'm going to anyway, like nwa, they just like, they're like, no, gas is the future. \[00:16:24\] We need more gas. And I'm like, but \[00:16:25\] you sell 95% of your gas offshore. We don't, we don't need the gas. Like, \[00:16:31\] ****Phil:** Right. \[00:16:33\] yeah, \[00:16:33\] We're doing the same thing in the UK with the, the British government the previous government. It was, it was now, it's now being deemed that the previous government handed out North Sea licenses illegally, like they were un unlawfully handed out because they \[00:16:46\] When they did their like emissions estimates, they forgot to take into account the burning of the gas they were producing. They're just like, nah, we stuck some solar panels on top of the drill. So it's basically carbon neutral. They're like, yeah, but what about when you burn it? They're like, ah, don't worry about it. \[00:16:59\] ****Alexander Karan:** We do that \[00:17:00\] too. \[00:17:00\] ****Phil:** the \[00:17:00\] instead of the new government, like immediately ripping up those licenses and saying, nah they've it's been left to a charity conglomerate to take the government to court to try and get those licenses revoked. And at least the labor government said, okay, we choose not to fight this in court ourselves, but they're leaving it to the gas fight. Said. Not only are we not gonna fight this, we're also gonna just like unilaterally completely remove all of those licenses. But they don't wanna do that. So \[00:17:27\] that's something. But we've \[00:17:28\] gotta, we all need to sort our shit out a whole lot more. And the answer \[00:17:32\] isn't vegan gas. \[00:17:34\] ****Mike Bifulco:** We face an oddly similar thing in the US We're a, as we record this, like 14 days away from the presidential election in the US and which means I get to hear a lot of politicians, you know, chattering about everything under the sun. And the \[00:17:45\] weirdest thing is hearing the left-leaning politicians in West Virginia saying like, we love the coal miners, we need the coal mining industry at Powers West Virginia. \[00:17:53\] But we \[00:17:54\] probably need to get you guys like. A Cisco certification or something just in case coal mining goes away. You \[00:18:00\] know it's, it's weird to hear them with a foot in both both worlds for sure. But that's what they have to do to sell boats, you know? \[00:18:05\] ****Alexander Karan:** so strange. So we actually did something here with native logging, which I don't know why they don't extend to like the coal and oil and, and and all that more aggressively, but like we were like, okay, we're gonna end native logging in these areas. 'cause we've got like incredible forests in Australia. \[00:18:25\] Like I cannot. Describe to you what the forests are like. Like they put England's forest to shame. They're just Wow. Right. And like we, we, you know what? Tens of millions of dollars, you know, into that community to help transition and bring other places there to transition. We gave them a time period to transition from the native logging, you know, we set up forests that was specifically for logging. Like, you know, I was like, why, why do we not do that in like the, the coal gas industry? Instead, we're like, nah, it's, it's part of the future. Let's, let's do \[00:19:00\] more. You know, it's, \[00:19:00\] it's such a weird, \[00:19:02\] ****Phil:** Well, the thing is the, a lot of the, a lot of the messaging takes a really long time to break through and then gets like held onto for far too long, right? So gas was a bridge solution. We're past it. Like you know, if, if, if in the nineties and in the two thousands people were saying like, Hey, let's get rid of some gas and we can let's get rid of some coal and we can replace it with some gas. 20, 30 years ago, that was absolutely the right thing to do. Sure. We've gone \[00:19:23\] way too hard for way too long and we forced it onto all these like, developing countries that can't afford to pay for it. So they're screwed and they can't afford to do an energy transition, and we also refuse to lend them any money at a discount to help 'em do that. But it kind of was a solution for a while in the same way that like biomass for a while was considered using \[00:19:42\] waste chip from, from logging, and. Shut down a coal plant and switch it over to to biomass. And that was a great idea in like the eighties and the nineties and two thousands, but they're still building more of that stuff now, and so they can't get enough of it. So they're starting to do dumber and \[00:20:00\] dumber things to get ahold of it, and they should just be getting rid of it. \[00:20:02\] And, and people were going, \[00:20:03\] well, all these experts back then said it was a good idea. It was back then. It's not then anymore. \[00:20:09\] ****Mike Bifulco:** Yeah, it \[00:20:09\] takes a long time to \[00:20:10\] steer that ship. \[00:20:11\] ****Alexander Karan:** It's so funny. Yeah. And we've got states in Australia, like south Australia, it's got like the world's biggest battery connected to the electrical grid. You know, \[00:20:20\] they produce so much renewable energy that their bills are so much cheaper, you know, meanwhile, like the \[00:20:26\] east coast. Where, where, where energy is also privatized, right? \[00:20:30\] It's not state run, but also just very heavy gas and coal base. Like it's just the prices are through the roof as well, you know, they're like, we need more. \[00:20:40\] ****Phil:** I've, I've seen some really good things about Australia. Have like entire entire states, provinces, what you calling them? \[00:20:47\] ****Alexander Karan:** states. So yeah, we're \[00:20:48\] well states and territories. States and territories. \[00:20:51\] So \[00:20:51\] there's like, \[00:20:52\] ****Phil:** Right. Well, whichever, there's big chunks of Australia that are kind of, you know, 90% renewable and aiming to be a hundred percent and some of them up to \[00:21:00\] 60%. \[00:21:00\] Like there's, there's areas doing really well and a lot of that without hydro, which is good 'cause they were just flooding entire salt marshes, which isn't very clever. \[00:21:06\] ****Alexander Karan:** Yeah, we \[00:21:07\] we do some good stuff. \[00:21:09\] ****Phil:** Doing really well, but then there's people saying like, oh, currently we've only got 50% of our energy being powered by wind and that's why it's bad. \[00:21:15\] It's like, well, you haven't built very much and it's doing quite a lot. If you some more it be doing more. Sounds like you wanna build \[00:21:20\] some more mate. \[00:21:21\] ****Alexander Karan:** that, that is it. So I'll do one thing in Australia's I people that like work at the energy companies. There is so much land in \[00:21:30\] Australia, like just, just so much land and like, it's really hard to negotiate like, with all those land owners and like get all the, you know, and then, and then also like all the lawyers that sit in the middle just try to make money off of like the negotiations as well. \[00:21:46\] And it's, it's real tough. It's real \[00:21:49\] ****Phil:** Yeah. Half the \[00:21:50\] cost of any of those projects is just fees. Yeah. \[00:21:52\] ****Alexander Karan:** Yeah. That's it. Yeah. It's just so big. \[00:21:56\] ****Mike Bifulco:** So this, this is maybe a good place to pivot some of the next things \[00:22:00\] I was interested in. So, Alexandra, last time you and I actually talked on the podcast, you were at Climate Clever doing \[00:22:05\] climate things. \[00:22:06\] ****Alexander Karan:** Trying to \[00:22:06\] save the world. So Climate clever got sold. And I mean, look, I'm not gonna complain because, you know, I ended up dating my co-founder. We, you know, moved in together. We had a baby, you know, life's great. So, \[00:22:20\] ****Phil:** Dating has changed. It's complicated. you gotta set up a company \[00:22:24\] together now. \[00:22:26\] ****Alexander Karan:** yeah, yeah, yeah. Who needs dating apps when you could just date your boss? \[00:22:30\] Right? Like \[00:22:31\] I I always, I always, I like to say she took advantage of me, but nobody cares when I say that they just high five her for some reason I. \[00:22:39\] ****Mike Bifulco:** I love that \[00:22:40\] man. \[00:22:40\] Australia's different. \[00:22:41\] ****Alexander Karan:** yeah, it's the other way around. So, you know, it was nice doing like community stuff and like having like a really good impact, like watching schools save money on their utility bills and reducing their emissions and using that money for like their library budget and stuff. And then we started working in businesses, so I was a little sad, like when. \[00:23:00\] 'cause we had some laws delayed in Australia that really affected climate, clever. Like they kept pushing back the mandatory reporting and it just had such a shockwave on us. So the business ended up getting sold and I just kind of felt really lost 'cause it was so weird, like actually coding and having purpose. \[00:23:15\] So I decided to do the middle life sensible thing and get a job in big tech. So I got a job Atlassian, and the reason I took the job Atlassian. Because like I figured they do the least evil, right? Like, you know, out of \[00:23:29\] all the, out all the big tech companies like Jira and Confluence aren't really trying to kill anyone. \[00:23:34\] So I thought, you know, that's a good start. \[00:23:36\] Yeah. Do I have to, why would this Jira ticket close? I keep pressing down and I refresh and it's still in progress. Right. But you know, like they've got a good work life balance remote work, which is great. \[00:23:46\] So I gotta spend more time with my son. And they had this like, interesting position open in finops, which, you know renewable energy actually falls under, well not \[00:23:55\] renewable energy, but you know. \[00:23:57\] ****Phil:** Yeah, the green \[00:23:57\] the green team doing carbon accounting and yeah.\[00:24:00\] \[00:24:00\] ****Alexander Karan:** yeah, so I, I, I joined finops and so like I work on like improving code and architecture and infrastructure setups to you know, make our cloud spend more efficient. \[00:24:10\] I also build tools to help teams across the company understand the cloud spend and their cloud usage. I'm \[00:24:17\] also working on some projects with our ESG team to help teams understand the footprint of their services, which is pretty fun \[00:24:23\] too. So I still get to have that little bit of like, sustainability. So like, yeah. \[00:24:28\] And then like on the side, like I'm still teaching, so like I, I, I teach at foundation called Sheet Codes, which is all about getting more women into tech. So, you know, I'm still tea still teaching there. That's my \[00:24:40\] like. Good bit for the community that I still do. You know, like that makes me feel all happy and warm inside. But yeah, so like mostly most every day is just like build more tools to help people understand costs, identify cost spikes, you know, A API wise, like most, like our front end is mostly \[00:25:00\] GraphQL, so like I deal with GraphQL when I'm dealing with user facing stuff. But like internally for our tools, there's a lot of rest APIs. Um, and then like, you know, which is, which is good. Like, 'cause it's all like the only downside I, I like, I really hope none of my colleagues hear this 'cause I don't want them to hear me bagging on it. But like, it's so much like Spring Boot and, and Java and Kotlin. Now Kotlin, Kotlin and Java aren't, aren't that bad actually. \[00:25:26\] Really? And I come from like, go and Node, so it's like a different, it was a \[00:25:29\] different world, but like there's so much of Spring Boot that I'm just like. Ah, it, I want to love Spring Boot. I really do. 'cause sometimes it does things That's great. And other times I'm like, why are you making building an API so difficult. \[00:25:42\] Why, why is documenting it so difficult? Like, you know, I, I wanna do, you know, design driven development, but you are forcing me to do code driven development. You know, like it's, there's just, there's aspects of it that make \[00:25:56\] ****Phil:** I've had really \[00:25:57\] weird moments with that, like trying to get docs out of a \[00:26:00\] spring, API, 'cause we work out a bunch of that and it was like you had to use an annotations based system. There was nothing else. And you had to like run the entire application in order to get that open API out, there's no way \[00:26:15\] to just be like. Generate, which all of the node, all of the go, all of the PHP tools, they all have that even, you know, even the annotation ones, \[00:26:22\] you just run a little command and it loads up just enough and does a little bit of static analysis and it grabs that and chucks it in a file for you. But this is like, \[00:26:28\] you need to set up a database so you can see what endpoints of API has and it absolutely fuck off. \[00:26:34\] ****Alexander Karan:** Yeah, it's, it's, it's, it's so, it's so weird. I'm like, why hasn't this been added? Right? Like, 'cause I'm, I'm used to like doing what you're saying, like, design an API, I've got an open API spec that generates a lot of boilerplate for me. And then like, I. In some of the node and go frameworks, it then takes those like you know, JSO schemas for your request bodies and speeds up passing as well. \[00:26:55\] Right? Like, and I just, yeah. Like, I, I miss that so much. But \[00:27:00\] yeah, I mean it's, it's pretty, it's a pretty cool job. We get to do some fun stuff, like the scale issues that you just. Don't have, unless you're a company Atlassian size. Right? Like, you know, one, one tiny compression change could like, you know, ramp up CP usage in the millions. \[00:27:15\] Right? Like it's, \[00:27:17\] it's, it's fun and, you know, I'm like using Tant Stack Query on the front for dealing with the rest API, which is like a, a really nice, beautiful wave just. Working with and like if you do the rest API really well, like you can just easily filter on stuff. Great search. \[00:27:35\] It's just, yeah, that's, so, that part's nice. \[00:27:38\] But yeah, I just, I miss being fully design driven because I feel like Spring Boot like really resists that like it's, \[00:27:45\] it's more code driven, which is the only thing that makes me sad API \[00:27:49\] ****Phil:** Which I think is a bit of a mindset of, of the average Java developer as well. It's like, why would I wanna write some when I could write a shit load of Java to do the same thing and.\[00:28:00\] \[00:28:01\] ****Alexander Karan:** Well, it's, it's, it's funny 'cause like Kotlin has come along and Right. Kotlin is like, it's a beautiful language, right? Like it's you know, I've been writing Kotlin now for a year and a half. Honestly, one of my top three languages fallen in \[00:28:14\] love with it, right? And it's mindset is so different. So. You'd think that there would be a different mindset around that sort of stuff, but because like people still use a lot of spring, I guess, which is still really more Java based, right? \[00:28:27\] It's just, yeah, it's different. \[00:28:30\] ****Phil:** For sure. I mean, it's one of the things I've been writing a lot for Bump, bump sh, the API, documentation providers and writing a lot for them, which is nice to, you know, they're hiring me to work on some really interesting topics. But we we just did a, a, a piece kind of really taking a whole bunch of steps back and, and not getting stuck in the weeds on really nerdy, obscure, open a. Why is an API contract a good thing? And that was a fun one to write of, just like, this isn't extra work. Oh, I did another blog post on a similar vein, which \[00:29:00\] was the, we don't have time to write API docs paradox where you basically end up. Completely forgetting how your a p works. And then a few months later end up, you just have to rebuild it because that team has moved on and someone quit and all of the documentation was in a random Slack channel full of now. And yeah, literally end up building a whole API to replace it instead of instead of just looking at the \[00:29:23\] docs to find out how it works at meaning. And yeah, like doing, doing stuff like that. I'm, I'm really hoping it feels like trying to explain to people that tests are a good idea. You know, it, it's like they're not more work. \[00:29:41\] It's, this is, this is something that you do once and it saves you time immediately and forever in more ways than you can think about. 'cause you've got an API contract, you can use it for contract testing, you can use it for SDK generation, you can use it for thousand things. but \[00:29:57\] yeah, they don't wanna hear it.\[00:30:00\] \[00:30:00\] ****Alexander Karan:** I miss contract testing. So much like, it's like it's, it's the best type of testing. It's, it's, it's, it's so good. It's so good. \[00:30:08\] ****Phil:** Right. Well, it's the most important thing for an API 'cause an API like the, the, the IPA interface, right? You wanna make sure that your interface is actually doing what you said it would be. Not just doing random, whatever it feels like, and not just changing every now and then. Like if you don't have contract testing, you don't have an API, you have some random shit on an endpoint somewhere. That's not helpful. \[00:30:29\] ****Alexander Karan:** I, I'm gonna go to work tomorrow and say that guys, Phil said this is just some random shit on an endpoint somewhere. \[00:30:36\] ****Phil:** Yeah, Yeah, Yeah, yeah, I've got this really cool thing. It's gonna be for our business. Call this url. It does some random shit that's. \[00:30:45\] ****Mike Bifulco:** So Phil, what have you been up to? \[00:30:47\] ****Phil:** Oh heck. Where have I been up to? So, apart from all the charity stuff I got, I set up a splinter faction at APIs API Days London. That was really fun. I was pestering Baptist for like a, a free \[00:31:00\] ticket 'cause I'm broken. I run a charity. 'cause I was actually gonna be in the area and had plans to meet some people and he, he's busy running a massive conference and didn't get back to me. \[00:31:08\] And then the other organizer was like, oh shit, Phil's around. We shoulda let him come. But I was like, whatever, I don't care. I'm just gonna set up a little pub meet around the corner. And we got loads of people. They were like 20, 30 people came by. All the see. There was, yeah, the, the, the bump team and Tom from Wok and Baba, some of the Smart Bear people, Frank Gil Cummins, that was it. \[00:31:30\] And Eric Wild loads of really cool people. All just Lorna Mitchell. Loads of people just chatting away and having a nice one. So that was really good. I literally came in from the woods. And had to UI chained together every single free charger that I knew in all these like rural countryside pubs. And I was like sleeping over in a car park on the slow triple charger so I could get all the way to London and then just met them for a pint and shuffled off. That was really funny. Uh, and I liked that, but. API stuff, like, I'm doing a lot of writing \[00:32:00\] at the moment, which works out really well for me because I, I just get, like, I've got a backlog of, of content that needs in for bump and for do little ad hoc pieces. Sometimes I like write an article and then see who would like it. \[00:32:11\] So the Tim Perry for h been doing a lot of bits for him here and there and basically, yeah, it's just letting me write more content. Then I would have time to do for just purely APIs you won't hate because we have got a few paid subscribers and I love you guys, but we do with a few more. But yeah, I, I've just really been enjoying writing more recently and I'm kind of using some of these articles to kind of ramp myself back up into building APIs you won't hate too. Oops. I did it again which is a working title, but I might go with a. \[00:32:42\] ****Mike Bifulco:** It has been nice seeing you get, get back into writing. So proliferally Phil I feel like one of the things that that happens is. There's like a network effect of you launching things into the internet where people start getting in touch about, like, ideas and tangential problems that they have. \[00:32:55\] ****Phil:** Sure. I mean as well, like Alexander, it's a lot of it's thanks to you as well. 'cause your, your, your \[00:33:00\] newsletters are amazing and they teach me things and it's really nice to kind of, not, I'm not doing as much stuff but I still am interested and I'm still trying to like, teach people about things and you seem to have your finger on the pulse a little bit more. \[00:33:12\] So I quite often like read. Read the newsletter and I'll be like, oh shit, I didn't know about that. That's really cool. That's a, where did find that? Sure. We put it on our, do we have more articles yet? \[00:33:29\] ****Alexander Karan:** The, the, the social pressure. \[00:33:31\] The \[00:33:31\] social pressure, yeah. \[00:33:34\] ****Mike Bifulco:** That's a good way to be. It's a forcing function we needed. \[00:33:38\] ****Phil:** I got more to do about green tech in APIs as well, which I'll have to talk to you about Alexander because yeah, there's, \[00:33:43\] I did that like introductory to green software in general for Eric, but I, I made big promises that I haven't kept about talking about how API design can impact carbon and, and a lot. \[00:33:56\] It's just. Article from Fastly about design \[00:34:00\] your APIs to be more, I'm pretty much redoing that, but you know, with a polar bear on it. But it, it is really about kind of, you can, you can make your API run incredibly. Quickly and serve a lot of requests that shouldn't need to be made. And that's still, you know, dumber than having a slightly less optimized API that you don't call very often. \[00:34:24\] So literally just kind of set your caching up properly. is is a really big thing. And then of course all the usual database optimizations that everyone needs to make. But there, there are things that we just don't think or talk about. And especially with the, the rush towards like Jason API include all the things and GraphQL like, let's fetch the whole bloody thing and one request, like, let's fire these massive uncatchable resources around all over the place. \[00:34:45\] And you want one more bit of data, you just load it all again, screw it. All of that \[00:34:50\] mindset of the last couple of years has been a problem. \[00:34:52\] ****Alexander Karan:** But it's architecture skills, right? Like this, this, \[00:34:55\] this, and this is also the thing that worries me with AI driven development or whatever you call it as well, \[00:35:00\] right? Like writing an endpoint is easy. You know, writing a a database is easy. Building a webpage is easy, but like architecture, an API layer, modeling your data in a good way. That sort of like you, you only learn that from like experience and failures and read and like proper reading and understanding of why other people have built stuff a certain way. Right? Like, \[00:35:23\] and, and and that's hard. Like it's easy to build an endpoint and whack it all in there. It's, it's hard to think about your resources and like actually split your API up correctly, put things on the correct roots, you know. Metadata linking, like it's, it's tricky. Like you need to actually sit and think and plan and that's \[00:35:46\] ****Phil:** If anyone, if anyone would like to do that, I really enjoy those sessions. So I've been doing a few contracting contracting consulting ad hoc stuff here and there, you know, two hour API, deep dives we call them. And yeah, I've had, I've had this conversation a hundred times where someone is trying \[00:36:00\] to plan their API, we had a university and we were saying like, right, what, what does your API need to do? \[00:36:05\] And immediately they jumped straight into. Hang on, let's not worry about what shape your data will take or how you describe that data. Let's back way up and like fundamentally, what was your API doing and I managed to kind \[00:36:20\] of say like, let's go through, are you gonna have one API or multiple? Like what do these do? What is the style of API, not even graph, like fundamentally is this a datadriven API, where you're essentially a generic database a. And then it was, how many APIs do you need? And they, they were trying to make an API that was basically one API that could, that had like teachers, students, organizational stuff and some other bits. And those were gonna be like, I guess four different endpoints. And then like depending on whether you were a user or teacher, it would show you some different embedded \[00:37:00\] data. \[00:37:00\] And it was all this like really weird super variable stuff. Like based on this context, I might see that permiss make this four completely separate APIs that have different different scopes. So you've got like a, a student, API that has my courses and my exams and all of the stuff related to me. And I can see some teacher information, but I can't see their home address and their phone number. You know, like you just make that be one thing for, for them. And there were lots of like individual resources and sub resources, and you could sign up for a course and all these little bits. And just kind of separating it out like that and having a a a a teacher's specific API for all the stuff that teachers would need in their, in their \[00:37:39\] scope. \[00:37:39\] And they had some similar but quite different resources, but they could all be per perfectly cached because it's not like all weird and variable based on the scope of the access token. So \[00:37:48\] trying to get, trying to \[00:37:49\] go through stuff like that is really hard. Yeah. \[00:37:52\] ****Alexander Karan:** Well, people just, it's, it's, so for me, it's like people are like, oh yeah, crud, that's, that's API, I'm like, no, no, that's like crud. Iss \[00:38:00\] not API, or, or it's like, okay, well we need this specific thing in our ui, so we're gonna build this endpoint for it. And I'm, I'm always like, but, but how do we reuse that? \[00:38:09\] Like, what's the \[00:38:10\] actual resource? What, what, you know, it's, it's, it's all, it's, it's, right. It's like people never take a step back and really think about it. And when you \[00:38:20\] think about it, that's when the speed comes in. That's when the caching comes in. That's when the optimizations actually, you know what, the optimizations are easy. \[00:38:28\] Because they're obvious. Because you've layered it well, right? You've designed it well, know. \[00:38:35\] ****Mike Bifulco:** I think it's a process a lot of people don't go through until they've felt the pain too, right? Like we, \[00:38:39\] we, I think in, in a weird way, a lot of developers spend a ton of time thinking about their edge based architecture and serverless stuff that will truly never be a problem for them with their, you know, even thousands of users. \[00:38:50\] You can get away with a, a small VPS but it is really easy to get yourself into. I, I think the technical term is a data mess with your API and, something that that \[00:39:00\] deserves a whole lot more thought than people realize, and the more upfront the better. 'cause it's a hell of a lot of trouble to unwind once you've already built things on top of something that becomes problematic, you know? \[00:39:09\] ****Phil:** Yeah, I think the. The reason that it's hard for people is that an API is seen as a developer's job. Just like you might have to go and write some functions or design a form. It's kind of seen as like throw an engineer at it. And what does an engineer do? They wanna write code. So that's, that's why API design first is so hard 'cause they wanna write code, not yaml. But you know, they're not used to stakeholder engagement. Need to. Get 10 potential users or beta users, or the front end team or whoever in a room and talk about their needs. And don't even try and translate those needs into, into Jason structures immediately or endpoint straight away. But you need to talk about like functionally in human words, what actions are you trying to take? And then once I've got a list of what actions that we're trying to take, we can then start to kind of. Do a, do \[00:40:00\] a mind map diagram of, of how these concepts might relate, and then we can \[00:40:04\] start to draw some boundaries around it, about which APIs are which, and then we can start to talk about, okay, maybe we're gonna talk about the resources and actions and endpoints that they might have. \[00:40:13\] But that's like, you know, day three of the planning process and not the first thing you do, which I'm, I'm. A hate says step one, write down list of things. You wanna turn, turn, \[00:40:31\] ****Alexander Karan:** Yeah. \[00:40:32\] yeah. \[00:40:33\] but it's, it's, it's the human problem. It's the human problem, right? Like it's, it's like, it's like the hard part about being an engineer isn't writing code. It's like talking to people and planning and navigating multiple stakeholders and different people and bringing them together, right. It's always the hard part. It's, it's, \[00:40:51\] it's never the \[00:40:52\] code. \[00:40:53\] ****Phil:** People are hard code's. Easy. I got AI to write for now. Don't worry. Just code. \[00:40:58\] ****Mike Bifulco:** Alright, so one other thing I wanted to get \[00:41:00\] to before we wrap the, the record today is I feel like the internet has fractured itself in the past couple years where it's harder to find people online and harder to find the places to listen to people online. I, I personally have given up on Twitter quite a while ago and no longer use Twitter. \[00:41:15\] I've been, I've been using threads for about a year and I've found it pretty satisfying. \[00:41:18\] It is of course another social network owned by another billionaire who is doing his best to humanize himself and whatever, \[00:41:25\] ****Phil:** Yeah, but he did promise to punch Elon Musk in the face, which I support. and he is got a cool chain now as well. Like what \[00:41:31\] ****Alexander Karan:** chain and a fro, right. \[00:41:33\] ****Phil:** He's got \[00:41:33\] bling. haircut. Yeah. I'm, I'm also, this week I feel like I've heard a lot about the tech community kind of jumping over to Blue Sky. \[00:41:42\] ****Alexander Karan:** Yeah, Kelsey Hightower jumped over. Yeah. So Kelsey Hightower jumped over and like the second he did, like, I felt like everyone else, I mean, like, he has a lot of sway, \[00:41:53\] but like he's really big in the infrastructure space and he's, he like, he's always really good at talking about things and being really \[00:42:00\] open and like, you know, really \[00:42:02\] focusing on fundamentals. \[00:42:03\] So the second he did it, I. Half of half of everyone. And I've been on there for a while and all of a sudden it was busy and, and \[00:42:11\] like I was, people were interacting with my own posts. And I was like, \[00:42:15\] this is, I mean, 'cause I, I'm still on Twitter because like, I stay on there because some tech people are still there and I need it for the newsletter, like, \[00:42:23\] like. That's fair. \[00:42:25\] ****Phil:** Yeah. \[00:42:26\] ****Mike Bifulco:** I still squat my account on Twitter, to be fair. Mine is marked as private so that no one can like create a new account under my name and do whatever garbage they're gonna do. I just don't log in any longer. So it sounds like Alexander, are, are you actor active on Blue Sky? Is that primarily your, your spot on the internet now or you \[00:42:41\] ****Alexander Karan:** Yeah, it's, it's me and that, that and my website and \[00:42:45\] I guess, I guess LinkedIn. I've seen a bit of the tech community on LinkedIn, which is, is is super weird 'cause like in between posts of people like being real estate agents with Lamborghinis and houses. You know how how to \[00:43:00\] structure a database and like all this other weird interaction, but yeah, most, mostly blue sky these days. \[00:43:05\] ****Mike Bifulco:** Yeah. \[00:43:06\] ****Phil:** Nice. \[00:43:07\] ****Mike Bifulco:** you? \[00:43:07\] ****Phil:** I'm kind of on all the things at the minute. On Mastodon, I'm still on X just because I actually, oh, I went back to pay for it. I had, I had, I was off for a month stopped paying for it for a month, but then like, just black hold and the fact that I'm trying to do a crowdfunder. You know what I mean? \[00:43:21\] Like the more people see the, my tweets, the more, the more money my charity gets. So I'm, I'm kind of locked in for a while, but I refuse to put the company, the, the charity credit card anywhere near that jerk. I'm, I'm still on Twitter for now, ri writing out the I'm trying to be king of the shit pile, but, I'm on Mastodon, on threads and on blue Sky. They all have slightly different feelings, but I I like how much of the climate community and now say community are getting over there. \[00:43:44\] ****Alexander Karan:** I was gonna say, I think we need to put APIs won't hate on Blue Sky. I feel \[00:43:48\] like there's enough of tech \[00:43:49\] Twitter \[00:43:49\] over do you want to do it? I can't. I have so many accounts. God dammit. I was so excited to like close down a few accounts on, on, on Twitter for like the charity and for other places because we're \[00:44:00\] like, screw this guy. I'm not doing seven accounts on here anymore, but I can't handle anymore. \[00:44:05\] I'm, I'm happy to set it up. Yeah, if you want. \[00:44:08\] ****Phil:** I'll just put you on the spot on live on air. But no, we should, we should do that. That'd be good. And Mastodon, I quite like it's still got this like really small community feel where like people I don't really know, just like are super friendly and supportive and get really excited about the charity stuff and like volunteer and come help out. \[00:44:23\] And it's just like I'm meeting new people in the village pub over there all the time. But I threads is, is such a dumb one. I felt like it got noticeably worth worse a month ago. But I posted a video of the SpaceX Rocket re landing itself, getting caught, and I put credit where credit's due. Elon Musk did a really good job of taking credit for this, and no one read it. No one read what I said. They all went didn't have anything to do with Elon Musk. Oh no, he didn't do it. The engineers did. And like 50 people, like 3000 likes, but 50 people just \[00:45:00\] replied being like, oh, I didn't read what \[00:45:01\] you said, and I'm angry, and I'm just like, this is not the replacement for X that we need. \[00:45:05\] Is it \[00:45:08\] ****Mike Bifulco:** Yeah, yeah. It's been, it is been frustrating. I'm, I am definitely still gonna be posting on threads. I feel like I've found a small group of people there that I really enjoy learning from. I. And I'm trying to find people on blue sky. The, the process of starting an account with zero anythings is always like, a little bit like, man, do I actually have the energy for this or not? \[00:45:27\] You know? \[00:45:27\] ****Alexander Karan:** So \[00:45:27\] they've actually got this cool list feature like in Blue Sky. They've got like this cool list feature where it's like database experts and you can click the list and just press \[00:45:36\] ****Mike Bifulco:** Yeah. \[00:45:36\] ****Alexander Karan:** and it just \[00:45:37\] like, \[00:45:37\] ****Mike Bifulco:** Yeah. It's a thing called starter packs that I started seeing where I found \[00:45:41\] the Ruby on Rails starter Pack the other day and I started following it. You click one button and you get all of the people there. I need to find more of those and also find my way onto some of those so that I I make some friends over there. \[00:45:50\] ****Alexander Karan:** Yeah. \[00:45:51\] An API starter pack. \[00:45:52\] ****Mike Bifulco:** Yeah, \[00:45:53\] ****Phil:** that's a really good shout actually. Yeah. \[00:45:55\] ****Alexander Karan:** It is just, just Phil. \[00:45:59\] ****Phil:** do you need? What \[00:46:00\] else do you need? That's funny. No, we should, that would be really good. If we set up a blue sky, then I'll, I'll help make a, a list of all the people what are cool. And we can go and talk to our friends at women and APAs and get a bunch of them on the list if they're interested, because that will help help people get started. Um, yeah, that. \[00:46:16\] ****Mike Bifulco:** There. \[00:46:18\] ****Phil:** Well, I was gonna say there's the, the best thing to do as well is to get the like follower bridges. So blue sky just released a follower bridge where you can kind of sign in with your X account and then also your blue, blue sky account and it'll kind of transfer the followers. So that really doesn't get you followers, but when people see someone they recognize come in, they go, yay, and click back. \[00:46:38\] So that's, I like. \[00:46:45\] ****Mike Bifulco:** That's a solid idea. \[00:46:47\] ****Phil:** Right. Well, the one thing I wanted to say is that we are taking a more concerted effort to do regular newscast, hopefully a bit more regular podcast. We're doing a lot more stuff and there's a bit of a chicken and egg problem, which is I would like to dedicate more time \[00:47:00\] to doing more stuff. \[00:47:00\] I would like to make more videos on whatever you ask us to do. I would like to do a lot more content for the premium subscribers, but we don't really have very many. Take time away from the charity and which doesn't pay me and the client work in order to do a, you know, four hours of recording a video for nine bucks. \[00:47:16\] So if a few more of you can sign up as a premium subscriber, I promise we'll get you a lot more cool stuff and you'll support the content that we're already giving out for free for everybody else. So you should go to \[00:47:25\] APOs. You won't hate, there is a smash that subscribe button and, and you just click on. \[00:47:33\] ****Alexander Karan:** and, and like, like Phil said, he'll do whatever you want, so. \[00:47:39\] ****Phil:** API OnlyFans coming your way \[00:47:40\] for no one that wants that. \[00:47:42\] ****Mike Bifulco:** That's hilarious. \[00:47:44\] ****Phil:** disgusting. \[00:47:44\] ****Mike Bifulco:** I. \[00:47:45\] ****Phil:** End podcast. \[00:47:48\] ****Mike Bifulco:** Well listen, both of you is fantastic, catching up on that extremely strange note. I will have a link in the show notes to our only fills where you can find everything you want about APIs you won't \[00:48:00\] hate. I'll, I'll drop as many links as I can to everything we've chatted about here and, and get this published soon. \[00:48:04\] And the three of us ought to do this more often. I'll put \[00:48:07\] my hand in the air as saying, I'm gonna try and organize the next one of these around the beginning of the, of next year. so \[00:48:11\] hopefully we can catch up in early 2025 for a shout. And if you're listening to this, please share the podcast with. \[00:48:17\] A friend. If you know someone who's interested in APIs in our world, we'd love to hear from them. We'd love to meet you. Chase us down on the socials. You'll find links to the mess of places that we just talked about in the show notes as well. until next time, thanks so much for listening to APIs. \[00:48:31\] You won't hate Alexander. Phil, thanks so much it. See you. \[00:48:38\] ****Phil:** That was a. \[00:48:40\] ​ ### Robots Are Coming For Us! URL: https://apisyouwonthate.com/newsletter/robots-are-coming-for-us/ Last updated: 2024-10-16T12:36:44.000Z It's been an interesting few weeks since the last newsletter, and robots are now a flag on my radar. I am not talking about Elon's clearly human-operated droids, but the [robot vacuums that recently got hacked](https://www.vice.com/en/article/ecovacs-robot-vacuums-hacked-slurs/?ref=apisyouwonthate.com). Hackers got into a relatively popular brand of these vacuums, which led to them flying around the house, spying on people and yelling slurs. A general reminder that security is an afterthought for many companies. I sprinkled in some API security content this week to ensure you are not responsible for the next robot uprising. \-- Alexander --- ## The API Round-Up ### [Microsoft OpenAPI Blog](https://devblogs.microsoft.com/openapi/welcome-post/?ref=apisyouwonthate.com) Microsoft has launched an all-new OpenAPI blog and plans to host many fantastic resources for working with OpenAPI. Their intro blog includes a complete list of everything they intend to provide, from libraries to guides and resources on specific API approaches. ### [Describing API Security ](https://learn.openapis.org/specification/security.html?ref=apisyouwonthate.com) OpenAPI supports a variety of security mechanisms to protect API access, and the official documentation is excellent for explaining them. Don't forget that OpenAPI allows you to set up multiple security schemas. Just make sure your API is using the security you describe 😉. ### [Scalar - Document, Test & Discover APIs](https://scalar.com/?ref=apisyouwonthate.com) I stumbled across another excellent API documentation tool a few weeks ago and kept forgetting to share it. It has a super cool playground feature for testing API endpoints and an offline first API client. Defiantly worth checking out. ### [Scramble – Laravel OpenAPI Doc Generator](https://scramble.dedoc.co/?ref=apisyouwonthate.com) I found another OpenAPI documentation tool, this time for Laravel specifically. It allows you to generate API docs without writing any PHPDoc annotations. It uses Stoplight UI to show the final docs and has a delightful developer experience. ### [Strengthen API Security with OpenAPI](https://blog.stoplight.io/api-security-can-be-stronger-with-openapi?ref=apisyouwonthate.com) This is an excellent short read from Stoplight about how OpenAPI and design-first approaches to API development can help strengthen security. Security comes down to planning and design, and building good OpenAPI docs requires moving through these phases. ### [Generate an OpenAPI spec with FastAPI](https://www.speakeasyapi.dev/guides/api-frameworks/fast-api/fastapi?utm%5Fsource=apisyouwonthate&utm%5Fcampaign=240701) Python developers love FastAPI for its developer experience, flexibility, and built-in performance. This guide shows how to use fast API to generate an OpenAPI spec, including some sweet customizations. ### [Deno 2](https://x.com/deno%5Fland/status/1844418020539846754?ref=apisyouwonthate.com) I know, *Alexander, stop talking about JavaScript*, but it's hard to stop talking about the best programming language 😜. In all seriousness, Deno recently dropped version two, which comes with some massive updates, making it easier to run JavaScript everywhere. Node powers some pretty cool APIs, and pretty soon, the same will be said about Deno. ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2024/10/apiywh_openapi-1.jpg) ## Speakeasy: Write OpenAPI your users love A great spec is the beginning of a great API devex. The OpenAPI Hub has all the info you need to craft a spec ready for docs, SDKs, and delighting users. [Learn more](https://speakeasy.com/openapi?utm%5Fsource=apisyouwonthate&utm%5Fcampaign=241015) ## From our Community Articles written and shared in our free [Slack community](https://slack.apisyouwonthate.com/?ref=apisyouwonthate.com). ### [The Complete Guide to API Monetization](https://zuplo.com/blog/2024/09/26/what-is-api-monetization?ref=apisyouwonthate.com) Adrian Machado recently shared a killer article about API monetization with our Slack community. The best part is that the article goes beyond pricing models and a few examples; it goes into tradeoffs, bill providers' choices, and more. Thank you so much for sharing, Adrian. ## Support APIs You Won't Hate When you become an member, you'll get access to members-only content while directly supporting our work. Your support helps us to keep making resources for the API community. [Become a member today](https://apisyouwonthate.com/#/portal) Thanks so much to our [members](https://apisyouwonthate.com/membership/): Al N, Kin L, Juxt, Alex R, Nolan S, Frank, James D, Bill D, and Ali S. Your support means the world to us! Until next time, Alexander, Phil & Mike ### OpenAPI for all URL: https://apisyouwonthate.com/newsletter/openapi-for-all/ Last updated: 2024-10-01T11:26:33.000Z OpenAPI is not perfect because specifications are complicated, and this is because *people* are complicated. The more people involved, the harder it is to match every use case and get everyone to agree. I personally never had too many issues with OpenAPI beyond conditional schemas because OpenAPI does such a great job. We should be proud of how far OpenAPI has come. You only have to look at something like Web Components to see how adopting something as an industry can cause a lot of drama. \-- Alexander --- ## The API Round-Up ### [The Complete Guide to OpenAPI v3.1](https://docs.bump.sh/guides/openapi/specification/v3.1/introduction/what-is-openapi/?ref=apisyouwonthate.com) Somehow, in between his spare moments of saving the planet, Phil has found time to write a complete guide to OpenAPI for Bump.sh. This guide covers anything you'll likely need to know for v3.1, from an introduction to responses to extensions and advanced parts of the specification. It's a handy tool and a great read. ### [OpenAPI 3.1 - The Cheat Sheet](https://bump.sh/blog/openapi-31-cheatsheet/?utm%5Fsource=linkedin&utm%5Fmedium=social&utm%5Fcampaign=openapi%5Fcheatsheet&utm%5Fcontent=seb) It does exactly what it says on the tin. Print it off, stick it on your desk, and see exactly what combination of keywords you need to solve common tasks. A huge thank you to everyone who helped put this together. It's a fantastic reference for everyone using OpenAPI. ### [Spectral Governance for Arazzo and API Workflows](https://blog.stoplight.io/spectral-governance-for-arazzo-and-api-workflows?ref=apisyouwonthate.com) Want to learn how Arazzo works but don't have a clue? Let Spectral show you the way! For those using Spectral (an open-source API linter), enable Arazzo linting, and you are good to go. ### [Content Engine for your API Docs and Product Guides](https://hashnode.com/products/docs?source=main-header) Hashnode recently launched a new product on top of its fantastic blogging tools: docs for APIs and products. Their blogging tools are a breeze to use, so hopefully, this same user experience extends to their docs. You can set up your API docs super quickly by importing your OpenAPI specs. ### [Generate an OpenAPI spec from tRPC](https://www.speakeasy.com/openapi/frameworks/trpc?ref=apisyouwonthate.com) tRPC gives TypeScript devs a type-safe pattern for creating and consuming apis - and is compatible with many frameworks from Remix to Next.js, Vue and SolidJS. This guide shows you how to generate a tidy OpenAPI spec from an existing tRPC config. ### [Fastify v5 is Now Officially Released!](https://openjsf.org/blog/fastifys-growth-and-success?ref=apisyouwonthate.com) Fastify just pushed a new major version. I often consider Fastify a modern alternative to Express. Its OpenAPI integration is beautiful and helps speed up your application by integrating your JSON Schemas. ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2024/08/speakez-1.jpg) ## Speakeasy: Build APIs Your Users Love Your API deserves a great developer experience. Get one by using Speakeasy to generate idiomatic, type-safe SDKs from OpenAPI [Start Generating](https://www.speakeasy.com/?utm%5Fsource=apisyouwonthate&utm%5Fcampaign=240701) --- ## From Our Community Articles written and shared in our free [Slack community](https://slack.apisyouwonthate.com/?ref=apisyouwonthate.com) ### [TypeSpec Reminds Us Why OpenAPI Exists in the First Place](https://passo.uno/typespec-openapi-api-design/?ref=apisyouwonthate.com) Following on from the theme of this newsletter's intro, Fabrizio Ferri Benedetti talks us through his thoughts on TypeSpec. His words, "Design is hard because it's a human problem", really hit home and highlight the reasons OpenAPI exists in the first place. ### [Analyzing the OpenAPI Tooling Ecosystem](https://modern-json-schema.com/analyzing-the-openapi-tooling-ecosystem?ref=apisyouwonthate.com) The incredible Henry Andrews has started a new blog series that reviews the tools from the API ecosystem. It's a fascinating read, and I can't wait for the next one. ### ## Support APIs You Won't Hate When you become an member, you'll get access to members-only content while directly supporting our work. Your support helps us to keep making resources for the API community. [Become a member today](https://apisyouwonthate.com/#/portal) Thanks so much to our [members](https://apisyouwonthate.com/membership/): Al N, Kin L, Juxt, Alex R, Nolan S, Frank, James D, Bill D, and Ali S. Your support means the world to us! Until next time, Alexander, Phil & Mike ### Introducing Arazzo: Describe API Workflows with this extension to OpenAPI URL: https://apisyouwonthate.com/blog/introducing-arazzo-describe-api-workflows/ Last updated: 2024-09-20T15:31:13.000Z Arazzo is a new specification from the OpenAPI Initiative for describing and documenting complex workflows throughout your API which touch multiple operations (a.k.a endpoints). The word “arazzo” means “tapestry” in Italian, which gives you a bit of an idea what it’s about, but let’s let the spec do the talking: > The Arazzo Specification provides a mechanism that can define sequences of calls and their dependencies to be woven together and expressed in the context of delivering a particular outcome or set of outcomes when dealing with API descriptions (such as OpenAPI descriptions). Much like the [Overlays specification](https://docs.bump.sh/guides/openapi/augmenting-generated-openapi/?ref=apisyouwonthate.com) we’ve been talking about lately, [Arazzo](https://spec.openapis.org/arazzo/latest.html?ref=apisyouwonthate.com) is the creation of a Special Interest Group within the OAI, made up of tooling vendors and experienced API folks who all have the same interests: creating standards which can solve a wide variety of use cases to push the API ecosystem forward, whether that’s testing, documentation, or even AI. [](https://www.youtube.com/watch?v=C%5FrkvN9zwKg&ref=apisyouwonthate.com) ## Describing Workflows APIs are rarely just one request. Maybe you need to log in with OAuth and use a property in the response to grab some data from another endpoint. Perhaps you need to create a few things before you can then fetch something else. It’s rarely clear from the API alone what needs to be done. API Reference Documentation is here to help with a lot of it, but it’s usually not enough by itself, with guides and tutorials picking up the slack. These can be produced manually, with lots of examples in code or curl showing the various steps, but these can suffer human error. Arazzo lets you create these in a declarative format, glueing operations together with inputs and outputs, referencing relevant parts of the OpenAPI to show how things all fit together. ## Example Arazzo Document If you’d like a quick look at how Arazzo works, here’s a workflow that builds on top of the [Train Travel API](https://github.com/bump-sh-examples/train-travel-api?ref=apisyouwonthate.com) we published earlier in the year. ``` arazzo: 1.0.0 info: title: Train Travel API - Book & Pay version: 1.0.0 description: >- This API allows you to book and pay for train travel. It is a simple API that allows you to search for trains, book a ticket, and pay for it, and this workflow documentation shows how each step interacts with the others. sourceDescriptions: - name: train-travel url: ./openapi.yaml type: openapi workflows: - workflowId: book-a-trip summary: Find train trips to book between origin and destination stations. description: >- This is how you can book a train ticket and pay for it, once you've found the stations to travel between and trip schedules. inputs: $ref: "#/components/inputs/book_a_trip_input" steps: - stepId: find-origin-station description: Find the origin station for the trip. operationId: get-stations parameters: - name: coordinates in: query value: $inputs.my_origin_coordinates successCriteria: - condition: $statusCode == 200 outputs: station_id: $outputs.data[0].id # there is some implied selection here - get-station responds with a # list of stations, but we're only selecting the first one here. - stepId: find-destination-station operationId: get-stations description: Find the destination station for the trip. parameters: - name: search_term in: query value: $inputs.my_destination_search_term successCriteria: - condition: $statusCode == 200 outputs: station_id: $outputs.data[0].id # there is some implied selection here - get-station responds with a # list of stations, but we're only selecting the first one here. - stepId: find-trip description: Find the trip between the origin and destination stations. operationId: get-trips parameters: - name: date in: query value: $inputs.my_trip_date - name: origin in: query value: $steps.find-origin-station.outputs.station_id - name: destination in: query value: $steps.find-destination-station.outputs.station_id successCriteria: - condition: $statusCode == 200 outputs: trip_id: $response.body.data[0].id - stepId: book-trip description: Create a booking to reserve a ticket for that trip, pending payment. operationId: create-booking requestBody: contentType: application/json payload: trip_id: $steps.find-trip.outputs.trip_id passenger_name: "John Doe" has_bicycle: false has_dog: false successCriteria: - condition: $statusCode == 201 outputs: booking_id: $response.body.id components: inputs: book_a_trip_input: type: object properties: my_origin_coordinates: type: string description: The coordinates to use when searching for a station. my_destination_search_term: type: string description: The search term to use when searching for a station. my_trip_date: $ref: "#/components/inputs/trip_date" trip_date: type: string format: date-time ``` This example shows how Arazzo works from a high level, defining a single workflow that shows how to find two stations, a train traveling between them, and shows how to use that data to book a ticket. It might feel fairly familiar to some of you. It feels to me a lot like Continuous Integration setup for tools like Travis CI, Circle CI, GitHub Actions, etc. It also feels a lot like the tool [Strest](https://github.com/eykrehbein/strest?ref=apisyouwonthate.com), which I used to love using for testing multiple interactions, but which has since been discontinued. ## Arazzo Syntax Just like OpenAPI you define a version: ``` arazzo: 1.0.0 ``` ### Info Object Then you define an `info` to contain relevant metadata about the purpose of this workflow. ``` info: title: Train Travel API - Book a Trip version: 1.0.0 description: >- This API allows you to book and pay for train travel. It is a simple API that allows you to search for trains, and book a ticket. This workflow documentation shows how each step interacts with the others. ``` ### Source Descriptions Object Then we have `sourceDescriptions`. OpenAPI is an API Description Format, which is stored in the form of an API Description Document, so this section is chance to mention which type of API description format is being used, and point to a specific API description document. ``` sourceDescriptions: - name: train-travel url: ./openapi.yaml type: openapi ``` Currently the types supported are `openapi` and `arazzo`, with the latter being a chance to extend other workflows, but for now let’s just stick to the main case of working with OpenAPI documents. The URL can be a relative file, or a full `https://...` to a document hosted elsewhere, for example: ``` sourceDescriptions: - name: train-travel url: https://bump.sh/bump-examples/doc/train-travel-api.yaml type: openapi ``` ### Workflows Then we move onto `workflows`. ``` workflows: - workflowId: book-trip summary: Find train trips to book between origin and destination stations. description: >- Find the right train traveling between your origin and destination, then book a ticket. inputs: $ref: "#/components/inputs/book_trip_input" steps: ... ``` Lots of this is familiar to OpenAPI fans, only instead of paths and operations we have workflows, with a `workflowId` to give this a unique reference instead of an `operationId`, the same short `summary` and long `description`, and even some `$ref` which you’ll remember from [splitting up your OpenAPI documents](https://docs.bump.sh/guides/openapi/advanced-ref-usage/?ref=apisyouwonthate.com). The inputs being referenced here are a standard JSON Schema, outlining what inputs should be given to this workflow, either by another workflow or by a user interface. These are defined inline or referenced to `components.inputs`. ``` components: inputs: book_trip_input: type: object properties: my_origin_coordinates: type: string description: The coordinates to use when searching for a station. my_destination_search_term: type: string description: The search term to use when searching for a station. my_trip_date: $ref: "#/components/inputs/trip_date" trip_date: type: string format: date-time ``` It would not be hard to imagine a documentation “try it now” interface, or testing tooling providing a UI for these schemas. This could be done with [JSON Forms](https://jsonforms.io/?ref=apisyouwonthate.com) or similar, allowing users to enter values with the type providing a relevant HTML input, and the description being displayed as a label to explain what values should go in there, along with other JSON Schema keywords being leveraged to allow for enum values, or examples. ### Steps Object Now we get into the main chunk of Arazzo: `steps`. ``` steps: - stepId: find-origin-station description: Find the origin station for the trip. operationId: get-stations parameters: - name: coordinates in: query value: $inputs.my_origin_coordinates successCriteria: - condition: $statusCode == 200 outputs: station_id: $outputs.data[0].id ``` Here the `find-origin-station` is a uniquely named step within the workflow, which defines a name that can be referred to elsewhere in the document. The `operationId` is referring to an operation inside the OpenAPI document, and the `parameters` match up with parameters in that operation. The parameters are similar to OpenAPI parameters, where `in` can be `path`, `query`, `header`, `cookie`. The new thing here is `value`, which can takes either a hard coded value, or refer to a workflow input defined earlier. Steps can define a `successCriteria`, where all [criteria](https://spec.openapis.org/arazzo/latest.html?ref=apisyouwonthate.com#criterion-object) must be passed in order to be considered a success. At the most basic level this should be checking for a successful HTTP status code, but can do any comparison using the [runtime expression](https://spec.openapis.org/arazzo/latest.html?ref=apisyouwonthate.com#runtime-expressions) syntax to grab a value and compare for any basic literals, operators, and loose comparisons on available variables like `$url`, `$method`, `$response.body`, etc. You can even user operators to do OR. ``` - condition: $statusCode == 200 || $statusCode == 201 ``` By default the `simple` conditions are used, but you can get more advanced with a `context` attribute to set the variable being used, then using `type: regex` for the condition. ``` - context: $statusCode condition: '^200$' type: regex ``` If the responses are JSON or XML you can get even more advanced with [JSONPath](https://docs.bump.sh/guides/openapi/jsonpath/?ref=apisyouwonthate.com) or XPath. ``` - context: $response.body condition: $[?count(@.data) > 0] type: jsonpath ``` The last part of this step example shows output, which takes values from various bits of the step and makes them available to other steps. ``` outputs: station_id: $outputs.data[0].id ``` Now other steps can refer to this output property for their inputs. ``` - stepId: find-trip description: Find the trip between the origin and destination stations. operationId: get-trips parameters: - name: date in: query value: $inputs.my_trip_date - name: origin in: query value: $steps.find-origin-station.outputs.station_id - name: destination in: query value: $steps.find-destination-station.outputs.station_id successCriteria: - condition: $statusCode == 200 outputs: trip_id: $response.body.data[0].id ``` This next step shows a mixture of parameters being sent to the next operation using a mixture of workflow inputs, and values defined as output from the steps before it. Chaining together workflow inputs and values from other steps you can create some amazing workflows, and have multiple workflow documents for different use-cases to describe all the important workflows that need to be documented and tested for your API. ## Tips ### Extending Other Workflows if you find there are certain operations, or groups of operations, getting repeated over and over again, you can make a step which runs a workflow instead. Instead of referencing an `operationId` you can define another workflow, and reference that `workflowId` in a step. ``` - stepId: find-origin-station description: Find the origin station for the trip. operationId: get-stations parameters: - name: coordinates in: query value: $inputs.my_origin_coordinates successCriteria: - condition: $statusCode == 200 outputs: station_id: $outputs.data[0].id ``` ``` - stepId: find-origin-station description: Find the origin station for the trip. workflowId: find-station # instead of operationId parameters: - name: coordinates value: $inputs.origin_coordinates # no `in` needed successCriteria: - condition: $statusCode == 200 outputs: station_id: $outputs.data[0].id ``` The main difference here is that you no longer need to specify where parameters are going, because they are then used as `inputs` in that workflow. The rest is the same. ### Add Operation IDs to OpenAPI Using an `operationId` is generally considered good practice because they’re used to make clean URLs for documentation, and help generate cleaner SDKs, but Arazzo creates a new reason for using them. If an OpenAPI operation does not have an `operationId` you are left using an `operationPath` which is a much uglier syntax, which will also break if paths change. ``` steps: - operationPath: '{$sourceDescriptions.petstoreDescription.url}#/paths/~1bookings~1{bookingId}/get' ``` Remember how to escape slashes in this syntax is horrendous, so before you start using Arazzo properly it would be a good idea to get all your OpenAPI documents ready by getting sensible consistent `operationId` into them. ### $ref vs reference There’s a new way to reference objects in Arazzo, and that’s the `reference` keyword, different from the `$ref` keyword you might be used to from OpenAPI. ``` steps: - stepId: find-pet operationId: findPetsByStatus parameters: - name: status in: query value: "available" - reference: $components.parameters.page value: 1 ``` This “expression based referencing mechanism” uses the same runtime expressions that we were using for inputs, outputs, and criteria and is available in the following parts of Arazzo: - successActions This is different to the the JSON Schema `$ref` keyword which uses JSON Pointer syntax, which might beg the question… why are there two different approaches to referencing things? Well, there has been confusion in OpenAPI as it attempted to completely align its schema objects with JSON Schema, which is a [very long story](https://apisyouwonthate.com/blog/openapi-v3-1-and-json-schema/) we can skip over here. Basically there are two different semantics for `$ref` depending on where it is, and they’re really subtle things like whether or not it can have other properties next to it... To maintain compatibility with JSON Schema whilst also creating functionality necessary for this new workflow specification, the authors of Arazzo decided to make a new `reference` keyword that would work as it needed to and make it available in limited locations. ## Tooling Support As with any new specification, the question is: what tools actually support this? Multiple tooling vendors are working on supporting this new specification. It’s also in the [Bump.sh](http://bump.sh/?ref=apisyouwonthate.com) Roadmap. In the meantime there is an early prototype of a test runner similar to the Strest tool I mentioned, called [arazzo-runner](https://github.com/AdrianMachado/arazzo-runner?ref=apisyouwonthate.com). This can help test the concept and help you build out some of the workflows before better tooling supports comes along to make it easier. ### Laravel's big raise URL: https://apisyouwonthate.com/newsletter/laravel-raises-57m-series-a/ Last updated: 2024-09-16T11:23:59.000Z In case you missed it, Laravel raised $57 million to help build Laravel Cloud. So throw away your React, NextJS and Vercel, and return to the good old days. All jokes aside, seeing alternatives for Vercel and others arise is excellent. PHP and Laravel have been a solid stack for a long time and deserve some love. However, I am slightly cautious of how VC investment can shape or change open source and their communities. \-- Alexander --- ## The API Round-Up ### [Accel invests $57M into Laravel Products & Framework](https://www.youtube.com/watch?v=BCV%5F357WGaM&ref=apisyouwonthate.com) If you need the full details of the Laravel investment, Taylor Otwell sits down with Aaron Francis to discuss all the juicy details. ### [How to: Generate an OpenAPI Spec for Laravel](https://www.speakeasy.com/guides/api-frameworks/laravel?ref=apisyouwonthate.com) Speaking of Laravel, this tutorial showcases the Scribe package for Laravel. Scribe is a documentation library for PHP apis, and has loads of great configuration options for optimizing your Open API output. ### [API Versioning at Monite](https://dev.to/monite/api-versioning-at-monite-3ba3?ref=apisyouwonthate.com) Monite outlines its approach to API versioning, highlighting the challenges of breaking changes in APIs and the need for versioning to prevent client disruption. It also introduces Cadwyn, an open-source versioning framework inspired by Stripe. While Cadwyn simplifies versioning, Monite stresses the importance of careful planning. Remember: API design *first* 😉. ### [HTTP 1 vs HTTP 2 vs HTTP 3!](https://www.youtube.com/watch?v=UMwQjFzTQXw&ref=apisyouwonthate.com) Alex Xu (the system design book guy) provides a beautiful breakdown of HTTP protocols, their evolutions, and how they work. The explanations are clear and straightforward, and incredible explainer animations accompany them. ### [Introducing Arazzo](https://bump.sh/blog/arazzo-describe-api-workflows-extension-openapi?utm%5Fsource=linkedin&utm%5Fmedium=guide&utm%5Fcampaign=arazzo) The one and only Phil explains the OpenAPI extension Arazzo and how it can be used to describe API workflows that touch multiple operations. Apparently Arazzo means tapestry in Italian. ### [Designing API Errors](https://httptoolkit.com/blog/designing-api-errors/?ref=apisyouwonthate.com) Phil is writing blogs for everyone this month and has spun out this concise article on API Errors. Phil provides a completed error spec and details why each part is important. ### [Express 5.0](https://github.com/expressjs/express/releases/tag/v5.0.0?ref=apisyouwonthate.com) A release for Express version five just dropped on Github, but the current version is still listed as `4.21.0`. It's great to see Express getting some much-needed love; however, let's wait till V5 is the stable release before upgrading. ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2024/08/speakez-1.jpg) ## Speakeasy: Build APIs Your Users Love Your API deserves a great developer experience. Get one by using Speakeasy to generate idiomatic, type-safe SDKs from OpenAPI [Start Generating](https://www.speakeasy.com/?utm%5Fsource=apisyouwonthate&utm%5Fcampaign=240701) ## APIs You Won't Hate The latest from the team at API's You Won't Hate. ### [API Designs Basics: Resources & Collections](https://apisyouwonthate.com/blog/understanding-resources-and-collections-in-restful-apis/) Phil returns to some basics of API design, covering collections and resources. Phil covers the meaning and then moves on to how these tie into HTTP methods and designing Rest APIs. --- ## From Our Community Articles written and shared in our free [Slack community](https://slack.apisyouwonthate.com/?ref=apisyouwonthate.com). ### [Detecting new API endpoints with oasdiff](https://danaepp.com/detecting-new-api-endpoints-with-oasdiff?ref=apisyouwonthate.com) Dana dives into using **oasdiff** to spot new API endpoints by comparing OpenAPI specs. This tool is perfect for bug hunters and security folks looking for fresh, untested paths to exploit. It automates the process with easy steps, spitting out differences in JSON or HTML formats, and it makes tracking API changes a breeze. ## Support APIs You Won't Hate When you become an member, you'll get access to members-only content while directly supporting our work. Your support helps us to keep making resources for the API community. [Become a member today](https://apisyouwonthate.com/#/portal) Thanks so much to our [members](https://apisyouwonthate.com/membership/): Al N, Kin L, Juxt, Alex R, Nolan S, Frank, James D, Bill D, and Ali S. Your support means the world to us! Until next time, Alexander, Phil & Mike ### API Design Basics: Resources & Collections URL: https://apisyouwonthate.com/blog/understanding-resources-and-collections-in-restful-apis/ Last updated: 2024-09-04T09:04:43.000Z HTTP APIs representing data generally split that down into two concepts: **resources** and **collections**. Let's take a look at these two concepts, as see how they’re used in real-world APIs. ## What is a Resource? In the context of REST/HTTP APIs, a **resource** represents a specific piece of data or object that can be accessed via a unique URI (Uniform Resource Identifier). This could be anything: a user, a blog post, a product, or an order. Imagine you have an API which handles invoices and payments. Each invoice would be a resource, and each resource would have its own URI: ``` GET /invoices/645E79D9E14 { "id": "645E79D9E14", "invoiceNumber": "INV-2024-001", "customer": "Acme Corporation", "amountDue": 500.00, "dateDue": "2024-08-15", "dateIssued": "2024-08-01", "items": [ { "description": "Consulting Services", "quantity": 10, "unitPrice": 50.00, "total": 500.00 } ], "links": { "self": "/invoices/645E79D9E14", "customer": "/customers/acme-corporation", "payments": "/invoices/645E79D9E14/payments" } } ``` Here, `/invoices/645E79D9E14` is the endpoint that uniquely identifies a single resource, in this case, the invoice with the unique ID `645E79D9E14`. The resource contains loads of data, including the customer name, an array of items on the invoice, various dates, and how much of the invoice is left to be paid. It also has "links", which can be related resources, collections, which could be pure data or could be actions, like a "pay" link which allows you to make a payment, a "send" link which helps you send an invoice, or the one we've gone with here "payments", which still allows you to create a payment, but also supports viewing a list of partial and failed payments. ## What is a Collection? A **collection** is a group of resources. It’s essentially a list or set of all the items of a particular type. Collections also have their own unique URLs. Using the invoices example again, if you wanted the API to let users retrieve all invoices, you would have an `/invoices` collection: ``` GET /invoices [ { "id": "645E79D9E14", "invoiceNumber": "INV-2024-001", "customer": "Acme Corporation", "amountDue": 500.00, "dateDue": "2024-08-15" }, { "id": "646D15F7838", "invoiceNumber": "INV-2024-002", "customer": "Monsters Inc.", "amountDue": 750.00, "dateDue": "2024-08-20" } ] ``` In JSON this collection is represented with an array, where each item in the list is a representation of a resource. Usually the API returns some basic information about each resource in the collection, and this example has links so the client can easily load up more data for each resource it's interested in. The vast majority of web APIs are built like this, but how can anyone know where the resources are? They could guess, go off searching around the Internet for some API documentation, or you could just... you know... tell them. ``` GET /invoices [ { "id": "645E79D9E14", "invoiceNumber": "INV-2024-001", "customer": "Acme Corporation", "amountDue": 500.00, "dateDue": "2024-08-15", "links": { "item": "/invoices/645E79D9E14" } }, { "id": "646D15F7838", "invoiceNumber": "INV-2024-002", "customer": "Monsters Inc.", "amountDue": 750.00, "dateDue": "2024-08-20", "links": { "item": "/invoices/646D15F7838" } } ] ``` If you have a collection of things, and want clients to load up more data from going to the URL, it makes sense to give them that URL so that they can do that. A website and a web API are not all that different, they both help access data, make lists of things, show more details for things, and make more actions available to users, and that's exactly how you can think about popping links in. ## HTTP Methods and How They Work with Resources and Collections REST APIs typically use standard HTTP methods to interact with resources and collections: **GET**: Retrieve data. - `/posts` \- Get a collection of all blog posts. - `/posts/123` \- Get a single blog post by its ID. **POST**: Create a new resource. - `/posts` \- Add a new blog post to the collection. **PUT**: Replace an entire existing resource. - `/posts/123` \- Update the blog post with ID `123`. **PATCH**: Update part an existing resource. - `/posts/123` \- Update the blog post with ID `123`. **DELETE**: Remove a resource. - `/posts/123` \- Delete the blog post with ID `123`. APIs are about a whole lot more than just CRUD, but when thinking about collections and resources this is a simple way to start thinking about it. ## Best Practices ### URI Structure The structure of URIs in REST APIs is crucial for consistency and readability. Here are some common conventions: - **Nouns over Verbs**: URIs typically use nouns (like `/posts`) rather than verbs (like `/getPosts`), because HTTP methods (GET, POST, etc.) already imply the action. - **Pluralization**: Collections are usually plural (e.g.: `/posts`), while resources are identified with a unique identifier (e.g.: `/posts/123`). ### Minimal Data in Collections When retrieving a collection, APIs often return minimal information about each resource to save bandwidth and speed up responses. This allows you to quickly scan the collection and then retrieve more detailed information if needed. ``` GET /posts [ { "id": 123, "title": "Understanding REST APIs", "author": "Bob Doe", "link": "/posts/123" }, { "id": 124, "title": "Introduction to HTTP Methods", "author": "Sally Smith", "link": "/posts/124" } ] ``` There's plenty of debate about how much detail you should put in your collections. If you put everything in there and bloat the collections horrendously, wasting time, money, and carbon emissions stressing your servers sending massive JSON payloads around. If you trim them down to the bare minimum then you force consumers to make more requests to get even the most basic data. Some even go as far as putting no information at all in their collections because it can all be fetched directly from the resources, which mean if cached data does change, there's not a strange outcome of having a collection and a resource showing different data. ``` GET /posts [ { "link": "/posts/123" }, { "link": "/posts/124" } ] ``` There is no one simple answer here, but if you are using a bit of common sense and talking to your consumers, you should be able to find something that works for you. I generally strike a reasonable middle-ground, where "summary" data is in the collection: name, ID, status, and a few key bits of data that you know from talking to consumers are the most important bits they want access to when they're building an index of data. Then if people want more data, they can go fetch it, but it's up to them. There's a lot we can do to make this more performant with [sensible HTTP caching](https://apisyouwonthate.com/blog/http-client-response-caching/) and better API design, but those are all topics for another day. ### Resources Linking to Related Resources & Collections Collections linking to resources is helpful, letting clients follow various links throughout your API like a user browsing a website, but resources can link to other related resources and collections, which might be data but could also be considered "actions", all handled through the same conventions. ``` GET /posts/123 { "id": 123, "title": "Understanding REST APIs", "author": "Jane Doe", "content": "This is a detailed tutorial on REST APIs...", "datePublished": "2023-10-01", "links": { "self": "/posts/123", "author": "/authors/jane-doe", "comments": "/posts/123/comments" } } ``` In this response: - The `self` link points to the resource itself, like a canonical URL, which is a handy convention for knowing where something came from even if you're just seeing a JSON blob of it or its available on multiple URLs. - The `author` link points to the resource representing the author of the post because it's quite likely you'll want to load that, but its also going to have its own caching rules and makes no sense to munge that data into the post resource. - The `comments` link points to a collection of comments related to this post if you want to load that, and any application loading that up is going to want to do it after it's got the post showing to users, so it doesn't matter if it loads later. Splitting up API data into multiple endpoints that can be grabbed if needed is really handy, upgrading a REST API from basically a set of functions which grab some data, into an Object-Relational Mapping (ORM) where relationships can be navigated easily, but we can go a step further. ***Later articles in the series will show you how to upgrade that ORM to a State Machine, so make sure you subscribe.*** [Become a member](https://apisyouwonthate.com/#/portal) ### Don't Confuse Resource Design & Database Design A key aspect of API design is not tying your resources and collections directly to the resources being designed. Your database needs to be able to change and evolve rapidly as data structures change, but your API needs to evolve slowly (or not at all), meaning the more tied your API customers are to your internal database structure the more they're going to have to rewrite their applications. So, the customer might be showing up in the invoice resource even though its in a separate table, and could be INNER JOIN'ed in the background (for those using SQL). Then if that query starts to get really slow you could reduce a level of normalization and bung that customer name directly into the invoices, which is going to help if the customer changes their name, because then you have a history of invoices with names correct at the time. There's lots to think about, but the quick point here is to avoid letting your database design influence your resource design too heavily. Your clients should always come first. ## Real-World Examples ### GitHub API When retrieving a list of repositories, each repository item includes a `url` field that links to the full details of that repository. ``` GET /users/octocat/repos [ { "id": 1296269, "name": "Hello-World", "url": "https://api.github.com/repos/apisyouwonthate/Hello-World" } ] ``` ### Twitter API When retrieving a user's timeline, each tweet includes a `url` that links to the specific tweet’s details. ``` GET /statuses/user_timeline.json?screen_name=apisyouwonthate [ { "created_at": "Wed Oct 10 20:19:24 +0000 2018", "id": 1050118621198921728, "text": "Just setting up my Twitter. #myfirstTweet", "url": "https://api.twitter.com/1.1/statuses/show/1050118621198921728.json" } ] ``` ### Stripe API Stripe has a collection which is a bit different, instead of returning a JSON array directly in the response, it wraps it in an object with a `data` property: ``` { "object": "list", "url": "/v1/charges", "has_more": false, "data": [ { "id": "ch_3MmlLrLkdIwHu7ix0snN0B15", "object": "charge", "amount": 1099, "amount_captured": 1099, "amount_refunded": 0, "application": null, "application_fee": null, "application_fee_amount": null, "balance_transaction": "txn_3MmlLrLkdIwHu7ix0uke3Ezy", "billing_details": { "address": { "city": null, "country": null, "line1": null, "line2": null, "postal_code": null, "state": null }, "email": null, "name": null, "phone": null }, "calculated_statement_descriptor": "Stripe", "captured": true, "created": 1679090539, "currency": "usd", "customer": null, ... snip because its HUGE... } {...} {...} ], } ``` They do this so they can add in various other bits of metadata, but much of this metadata comes down to pagination which can be handled other ways (like popping [pagination](https://apisyouwonthate.com/blog/api-design-basics-pagination/) into Links headers), so this practice is somewhat dying out. ## Summary - **Use Consistent Naming:** Stick to conventions like using plural nouns for collections. It shouldn't matter, but it drives people mad. - **Keep it Simple:** Start with basic endpoints and add complexity only when necessary. It's easier to add things to an API if they're needed later, than take them away once they're in production. - **API model is not a database model:** Do not try and recreate your database model over HTTP because it will be a big heaving waste of time and be almost immediately wrong making clients upset. By understanding and applying these concepts, you'll be able to design and work with RESTful APIs effectively, ensuring that your API interactions are intuitive, efficient, and scalable. ### 2024 State of the API Survey URL: https://apisyouwonthate.com/newsletter/2024-state-of-the-api-survey/ Last updated: 2024-09-03T12:11:31.000Z We care about the planet at API's You Won't Hate, and no one cares more than our very own Phil Sturgeon. His charity Project Earth aims to plant 100,000 trees this winter (northern hemisphere) but still needs sponsorship. Using the Shopify API and a Larvel serverless API, you can help sponsor trees in the UK. Check out the sponsorship options here: [https://shop.protect.earth](https://shop.protect.earth/?ref=apisyouwonthate.com) \-- Alexander --- ## The API Round-Up ### [Postman: 2024 State of the API Survey](https://www.surveymonkey.com/r/2024-state-of-api-survey?ref=apisyouwonthate.com) It's that time of year again. Postman has posted their annual survey looking for feedback on the API ecosystem. Share your thoughts on the future of APIs, daily issues, favourite tools, and more. The results of State of JavaScript are always interesting due to the amount of responses, so let's get this done. Not to mention, there are prizes up for grabs. ### [Improve OpenAPI specifications with Rate My OpenAPI](https://www.youtube.com/watch?v=i4tgrKe-RVs&ref=apisyouwonthate.com) The better the Open API document, the better the developer experience will be for anyone trying to use it. In this video, Martyn shows how you can get up and running quickly with RateMyOpenAPI using the CLI for instant feedback on the command line as you develop your Open API documents. ### [Increasing Resilience with Hypermedia Clients](https://mamund.substack.com/p/increasing-resilience-with-hypermedia) Mike Amundsen shares an extract from his book RESTful Web API Patterns & Practices Cookbook. If you have not read this goldmine of API knowledge, get on it. Mike talks about how balancing usability (the ease of use for an API) and reusability (the ease of using the same API for another task) is tricky and how abstraction improves reuse. ### [Deno 2 with Ryan Dahl](https://syntax.fm/show/815/deno-2-with-ryan-dahl?ref=apisyouwonthate.com) Deno has done a lot to shake up the JavaScript/TypeScript/Node ecosystem. The creator, Ryan Dahl, heads onto syntax to chat about Deno 2.0 and its new features. You are missing out if you have not tried building and deploying serverless APIs with Deno Deploy. ### [Using TypeSpec to Make OpenAPI Schemas and SDKs](https://www.speakeasy.com/guides/api-frameworks/typespec?utm%5Fsource=apisyouwonthate&utm%5Fcampaign=2409/01) TypeSpec is a powerful API-first framework which can streamline the design and documentation of your APIs. This tutorial walks through a developer-friendly approach to spinning up OpenAPI Schemas and SDKs. ### [Using the Swift OpenAPI Generator for the Jamf Pro API](https://dev.to/brysontyrrell/using-the-swift-openapi-generator-for-the-jamf-pro-api-p2k?ref=apisyouwonthate.com) This article is a fantastic walkthrough on using the Swift Open API generator to create a Swift client using the Jamf Pro OpenAPI spec. The tutorial walks you through the whole process, from Xcode setup to copying the OpenAPI doc to middleware and client code. ### [OpenAPI Initiative Newsletter – August 2024](https://www.openapis.org/blog/2024/08/27/oai-newsletter-openapi-initiative-newsletter-august-2024?ref=apisyouwonthate.com) Did you know the OpenAPI Initiative has its own newsletter? Catch up on the August edition, which covers specification news and provides a round-up of API-focused events. ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2024/08/speakez-1.jpg) ## Speakeasy: Build APIs Your Users Love Your API deserves a great developer experience. Get one by using Speakeasy to generate idiomatic, type-safe SDKs from OpenAPI [Start Generating](https://www.speakeasy.com/?utm%5Fsource=apisyouwonthate&utm%5Fcampaign=240701) ## APIs You Won't Hate The latest from the team at API's You Won't Hate. ### [Mocking APIs with Microcks](https://apisyouwonthate.com/blog/mocking-apis-with-microcks/) Phil walks us through using Microcks, a powerful API mocking tool, and integrating it with your existing API documentation flow. ### [🎙️ Generating API Specs vs. Hand-writing them with Joel Clermont from Mastering Laravel](https://apisyouwonthate.com/podcast/generating-api-specs-can-be-painful-with-joel-clermont-from-mastering-laravel/) Joel joins Phil to talk about his perspective on API design, and why even a single-person dev team should consider hand-writing their Open API spec rather than generating it. --- ## From Our Community Articles written and shared in our free [Slack community](https://slack.apisyouwonthate.com/?ref=apisyouwonthate.com). ### [Why API Security Testing Matters](https://danaepp.com/why-api-security-testing-matters-learning-from-tracfone?ref=apisyouwonthate.com) Dana covers Verizon's recent $16 million settlement with the FCC and why API security matters. I will not spoil the post, but it's a good read. Thank you, as always, Dana. ## Support APIs You Won't Hate When you become an member, you'll get access to members-only content while directly supporting our work. Your support helps us to keep making resources for the API community. [Become a member today](https://apisyouwonthate.com/#/portal) Thanks so much to our [members](https://apisyouwonthate.com/membership/): Al N, Kin L, Juxt, Alex R, Nolan S, Frank, James D, Bill D, and Ali S. Your support means the world to us! Until next time, Alexander, Phil & Mike ### Generating API Specs can be Painful, with Joel Clermont from Mastering Laravel URL: https://apisyouwonthate.com/podcast/generating-api-specs-can-be-painful-with-joel-clermont-from-mastering-laravel/ Last updated: 2024-09-03T12:09:48.000Z ## Summary In this episode of 'APIs You Won't Hate', the Phil discusses API development with Joel Clermont, a prominent member of the Laravel PHP community. They explore different approaches to building APIs, focusing particularly on the debate between code-first and design-first methodologies. Joel shares his experiences and challenges with generating API specifications and expresses his preference for manually writing specs. They also touch on challenges related to auto-generated specs, including the risk of drift between code and documentation. Various tools and strategies for improving API spec accuracy and integration are discussed, highlighting the importance of planning and thorough documentation in API development. ## Show Notes - **Joel Clermont** - website **\-** [joelclermont.com](https://joelclermont.com/?ref=apisyouwonthate.com) - github - [@joelclermont](https://github.com/joelclermont?ref=apisyouwonthate.com) - [**Why I don't like generating API specs**](https://masteringlaravel.io/daily/2024-07-08-why-i-dont-like-generating-api-specs?ref=apisyouwonthate.com)on *Mastering Laravel* - [**OpenAPI Overlay spec**](https://github.com/OAI/Overlay-Specification?ref=apisyouwonthate.com)on *GitHub* - [**Use OpenAPI Overlays Today**](https://apisyouwonthate.com/blog/use-openapi-overlays-today/)from *APIs You Won't Hate* - [**Scramble**](https://github.com/dedoc/scramble?ref=apisyouwonthate.com) for Laravel --- #### Transcript \[00:00:00\] Introduction and Guest Welcome \[00:00:00\] ****Phil:** Hello everyone, and welcome to another episode of APIs. You won't hate the podcast, it's the audio version Instead of blog posts, we're here talking about APIs. Joel Claremont's Background and API Experience \[00:00:11\] ****Phil:** And today I've got Joel Claremont with me, who is well known around the Laravel PHP community but might be new to some of you folks. So Joel, Hello. what's going on? \[00:00:23\] ****Joel Clermont:** Not too much. Yeah, I've listened to some of your episodes here and I'm like, I am definitely not in the API dev tooling space. That is not my specialty. , I'm coming at this more as the, the, in the trenches practitioner building, shipping, supporting APIs and not, not like the higher level tooling person that's \[00:00:40\] building things for other people to \[00:00:42\] ****Phil:** Well that's, that's fine. So we're not, we, we have, because I have been working in the kind of open API dev tooling space, a lot of the content ends up being quite specific to that. And that's not really what this \[00:00:54\] podcast is meant to be like. If you cast your \[00:00:57\] mind back to way back when of the origins of \[00:01:00\] this, of APOs, you weren't hate as a an anything before it was a community in a podcast. \[00:01:04\] I was writing that book. Build APIs you won't hate, which was pretty much how to make rest Dish APIs in Laravel and all of it was Laravel. So \[00:01:11\] it's lovely to be chatting to Laravel people and I still build APIs in Laravel and I think it's important that we don't, like, I don't want to kind of separate out that there's the, the API community and then like all of the, the programmers who are actually building APIs in various frameworks. \[00:01:27\] They're all, it's all overlap overlapping \[00:01:29\] circles. And, whenever I'm talking to people \[00:01:31\] about like. The Importance of API Specifications \[00:01:32\] ****Phil:** One of the main things in, in API derail, I always try and explain to folks is that like if you go to API conferences and you only ever talk about APIs and you sell your products to other people who care about APIs, it's a very small world. \[00:01:45\] But if you go to Laravel, PHB meetups and and things like that, then there's actual \[00:01:50\] people who might end up using your stuff who have never heard about any of the wonderful, lofty things you've been talking about and really should have. So that's kind of why I'm talking to you today of, yeah, you're \[00:01:59\] posting about \[00:02:00\] like how, how you like to make API specs with Laravel or how you don't, so you wrote an article called I have It right here. \[00:02:08\] Why I Don't Like Generating API Specs, which is written nail on the head. do you wanna tell me a little bit more about this? And you said there's a series of blog posts and like what led to this? Tell me more. \[00:02:18\] ****Joel Clermont:** Yeah, it, it actually started like, I think the, so, so, first of all, for a little context, I, I have a site called mastering laravel.io and I publish, for some reason, I committed to a daily newsletter Monday through Friday, and I, I cross post to the site. But so any, any random thought that pops into my head, I'm like, oh, that could make an email, right? \[00:02:38\] So I'll, I'll, I'll write it down. And I was working on a project recently building an API for a Laravel company. And we were writing the spec and, and the client, like they, they're an existing product and they're like, we've never written a spec. Like we only have an internal team. I. And so that kind of started the discussion, which is like, I think it's \[00:03:00\] useful. \[00:03:00\] In fact, I know it's useful to have a spec even if you don't have a giant team, even if you don't have like a separate frontend and backend team. I think like even if it's a full stack small team or a single dev, it, it is nice to have a spec. It, it sort of keeps you honest. And so that, that was the, the start of the whole discussion in, in the newsletter, but then it ultimately led to, okay, hopefully I've convinced you. Now how do I get started? Challenges with Code-First Approach \[00:03:24\] ****Joel Clermont:** And where most developers like to go is, well, let's use code to do it right? Like, that's what we do. And I personally have never had success with that approach outside of like demos and a proof of concept or maybe like breaking ground on something to get started. But the code first approach. Has been problematic for me. And so that, that was, that was where my experience was coming from and I was trying to share that, like, it's not that complicated to like just write the file yourself and, and I think there's actually some benefits to it. \[00:03:56\] ****Phil:** I mean, so many developers are running about, like, everyone has to learn \[00:04:00\] Kubernetes and everyone has to learn these 25 different things. Comically. Most of those do end up being slightly overloaded. yaml DSLs. But the idea of learning this one other yaml DSL is completely \[00:04:11\] unacceptable to anyone and we'll never do it. \[00:04:13\] And, That's pretty funny. So just in case, there are people who haven't heard me \[00:04:16\] banging on about open API in the past. Joel says spec, he means API specification, which is one other term for like API description or API design document. There's a million different names of this stuff, but yeah, basically it's \[00:04:30\] like. I'm gonna, I'm gonna describe the endpoints and properties and parameters and request bodies and all the bits of the, the interface of the API with some sort of machine language, which. tools can use generally for docs. Pretty handy for mocks and loads of other stuff. And I've built a spectral API scanner. \[00:04:51\] So literally you put your open API in and it goes, oh, that's gonna get you hacked buddy. Watch out for that. There's a million things you can do with, with an API spec when you've got it. But \[00:05:00\] yeah, when, when left are the developers. I'm picking up on what you said a minute ago. Like the, if you leave it to the developers they're just gonna write a whole bunch of code. \[00:05:08\] And if you, if they ask you, if you ask them for documentation, they'll go and write down like 10 lines somewhere and it'll be bad. So at some point over the last I. I don't know, eight five. Open API and Tooling \[00:05:18\] ****Phil:** Over the last five years specifically, I feel like open API has picked up in popularity massively. And that's partly 'cause people like me bang on about it all the time. \[00:05:26\] But it's bloody useful for everyone to know what's going on. And you've got that machine learnable format, so you, and, and you've got this, this spec where you can generate some docs and you can use it for contract testing in a million things to, and that helps keep those docs up to date. \[00:05:40\] But some folks don't like \[00:05:42\] to write. That YAML out by hand. 'cause that is pretty boring. There's a bunch of other ways of doing it. There was gooeys and things, but people don't like to write that by hand and instead they try a bunch of creative ways to avoid having to write that. You were talking about in Laravel Specific Tools and Practices \[00:05:57\] ****Phil:** Laravel, there's a few kind of Laravel specific tools. \[00:05:59\] How, \[00:06:00\] how do, how do people do it in Laravel? How do they get, how do they get their code spitting out open API so they don't have to write it. \[00:06:09\] ****Joel Clermont:** Yeah, it's what you might expect if, if, even if you're not in Laravel, you know, where, where there's some, some element of static analysis and then that can be enhanced with some sort of annotations or like inline documentation to, to enhance it further or to specify like examples of things. \[00:06:27\] ****Phil:** Mm. \[00:06:28\] ****Joel Clermont:** and I probably should say like I've. At least not super recently. I've never actually tried to use one of those packages in earnest. Like, I will try, somebody will recommend one, and I'll be like, oh, I'll give it a shot. And I usually get a couple days into it. I'm like no, this is, it is just, it ends up, you're just conforming to a different layer of abstraction in the sa in the same sort of specification. \[00:06:52\] And so, it's, it's really a personal thing. Like I'm sure there are teams out there that do this and it works great and I'm sure there are use cases where it, it's \[00:07:00\] beneficial, but I'm generally not working on massive, massive projects. You know, like, like maybe there's 50 to a hundred API endpoints and a lot of \[00:07:11\] times there might not even be a public consumer. \[00:07:13\] Is it \[00:07:15\] okay. I mean, it's not stripe. right now, but yeah. Fair enough. \[00:07:20\] Okay, well I'm counting like get versus post too, \[00:07:23\] right? So, you know, in terms of \[00:07:25\] resources it's probably a lower number, but, but yeah, it's, it's \[00:07:29\] not, I don't work on teams of like 50 developers and you know, that like, so there's probably some efficiencies that become more. Important and beneficial where the trade offs might skew the other direction toward code first tooling. IJI just am not there and \[00:07:48\] I here, maybe this is a question for you, Phil. Are there any tools that you've seen that do spec generation, that actually leverage the spec? Like things like refs and, you know, things like that Or is does it end up \[00:08:00\] spitting out some really flat, verbose document that no human would ever write in that way? \[00:08:05\] ****Phil:** Yeah, that's the problem. Most of those tools will just kind of just vomit something upwards, which is not \[00:08:11\] particularly. Helpful. I mean, it generally, it, that sort of stuff doesn't really matter if you are just putting it straight into a documentation tool. 'cause the documentation tools will flatten it. \[00:08:22\] so yeah, the, the way, the way that it seems to me is. Somebody has said, Hey, we've, we've started using open API to have one kind of documentation hub, a developer portal, whatever. We've got this, we've got this thing, and we need all the teams to make open API and we don't care how you do it. I've been that guy. \[00:08:43\] I've, I've done that. And I'm like, you know, some of them are like, oh, we've got it all in Postman, so we'll convert it into open api. I, I'm like, sure, whatever. And some of them are like, oh, we've got open API two or Raml or these different old kind of defunct formats. And I'm like, whatever, I can \[00:08:56\] convert that for you and, and or, you know, and then eventually you end \[00:09:00\] up with open API and everyone's like, great. \[00:09:01\] Some of it's weird and some of it's ugly and some of it's not entirely right. But that's a step one is we now have one documentation hub for the entire thing. Great. But then over \[00:09:09\] time it's a case of like, well, how do we get this to be more useful? And. And how do we use this for more verticals as well? \[00:09:15\] Like if you are just shoving it into docs, then it kind of makes that nice stripe, like three coal API stuff that everyone wants. But \[00:09:23\] if you try and use it for, you know, code generation and a bunch of other use cases, it, it kind of falls apart and you can't it gets a bit hard to like add stuff to it as well. \[00:09:32\] So. the one thing I've seen, which helps, if, so, if you've got a API spec that's being kind of just a magic appeared out of your API one thing, that can be pretty handy. Have you heard of open API Overlays? \[00:09:48\] ****Joel Clermont:** Mm. No. \[00:09:50\] ****Phil:** It's this like extension thing. So open APIs started doing like working groups or special interest groups. And these special interest groups create different things. So they're doing like \[00:10:00\] workflows so you can document entire kind of workflows of, of API calls and things like that. And one of them is overlays where you can basically. \[00:10:08\] It's kind of like a spectral rule set. If you're familiar, you kind of have like a J path and you're like, go to this bit of the document and then like add this or remove this. And so you can theoretically have some horrendously vomited out, open API and then be like, and I'm gonna add some descriptions in here, which are like 10 paragraphs long and have a bunch of markdown so you can have your technical \[00:10:29\] writers kind of wedging that stuff in. \[00:10:32\] And so. My, my initial like, no one should ever do code first ever, because then you have this useless thing is, is somewhat kind of abated a bit where it's like, okay, you, you've, you've given us this useless thing and now some technical writers can jump through some quite horrendous hoops to make it a bit less useless, but that still \[00:10:52\] isn't like the most shining endorsement for it. \[00:10:57\] ****Joel Clermont:** you know, like you mentioned, if if, if the only people \[00:11:00\] reading the spec are doing so via like a generated HTML portal, then it, it doesn't probably matter internally if it's using like refs for shared schemas and things like that. But \[00:11:11\] I, you know, just like I'm thinking and recommending you write it by hand at least once, at least know how to do it. I also. I think like, I like to look at the spec by hand too. Like \[00:11:20\] the, the tool, the tooling is great to auto generate fancy, nice branded documentation, but I, I do find some value looking at the spec directly and then especially seeing, oh, this, you know, this specific type of request or this, you know, form payload is \[00:11:36\] used more than one place. \[00:11:38\] Like that tells me something about the overall API design that you don't see if it's just all flattened out. \[00:11:44\] ****Phil:** Mm. Yeah, I mean, that's a useful thing. So kind of having intent. Written somewhere else outside of the code, not only helps you spot \[00:11:54\] obvious mistakes from a wider pool of people that don't necessarily know how Laravel works \[00:12:00\] but \[00:12:00\] it, it just becomes a, it becomes a lot more useful. Like you can have a bunch of people doing design reviews that literally don't know Laravel and don't know its random conventions. \[00:12:07\] And, and it also means that if \[00:12:09\] you accidentally have a slip of the keyboard and, and change some code. You didn't necessarily change all of the annotations that came with it then. I, the graphic I used on a blog post talking about this was like a, a drinks cabinet with a labeled that as milk and it's clearly orange juice. \[00:12:24\] Like people kind of confuse the like proximity with \[00:12:28\] accuracy if like these annotations are somewhat near the code I'm describing. So I'm sure someone will definitely remember to update them both. And so. Kind of having it in that second place means you can then kind of you, you have those extra checks where you, you use that open API as an \[00:12:42\] assertion in your contract test that says, does my code match what I think it does and what the design review team think it does, and what everyone \[00:12:50\] involved so far has thought it would. \[00:12:52\] And so that, that for me is kind of why I'm a big fan of the approach. But it's, it's just really hard to get a PA \[00:13:00\] developers to do another thing. They're like, I'm a coder. I write \[00:13:04\] code. I would like to write some code for \[00:13:06\] this. And yeah, it's, it's a weird one. \[00:13:09\] ****Joel Clermont:** One of the benefits I think there is in writing the spec separate from the code is like. Especially if you start with the spec is it forces you to think through the requirements in a level of detail that you don't often do. You know, if you're getting a requirement from some stakeholder and they're like, Hey, I want an endpoint to do this, this, and that. Okay, well, like who can do that? You know? \[00:13:34\] How long can that field be? Like there's all these constraints that. Somebody has to make a decision on, and like that spec becomes the forcing function to decide like, well, what is the maximum length? What, what is the format on this date time? Or, you know, \[00:13:47\] just things like that that otherwise you might get to later in the middle of implementation. \[00:13:52\] And a dev just like, well, I guess we'll just do it this way. And it wasn't actually discussed or decided on. \[00:13:57\] ****Phil:** That's a really good point. That's I think where I \[00:14:00\] was heading was that like the API spec being thought of as a documentation mechanism is kind of the problem here because it's it's basically. It's not always a problem. There are, there are use cases where you have an API and you need to like play catch up, right? \[00:14:18\] There's the there's, there's API design first. There's API code first, and the, and I've done polls where it's like, are using API code first. API design first. Switching from one to the other or some awkward combination. And every time I do this, every couple of years, the answer's always like 25% of each. \[00:14:35\] So people, there are some teams who are doing code \[00:14:37\] first, some are doing design first, and, and it's just all over the place. But yes, like those, those tools that can help you kind of grab a whole bunch of metadata out of your code through static analysis and maybe you add a few other things in there. \[00:14:52\] Can really help you get a head start. And then you have the open \[00:14:55\] API that you can go back and put assertions into your code base to make sure that \[00:15:00\] it's correct. And then you can expand that. \[00:15:01\] And if you want to add a new endpoint, then you can design first on that, on that open API file. You can add a new endpoint that's completely imaginary, start working on it. \[00:15:10\] Have everyone agree whether that's a good idea or not. Whether it solves, solves the needs or not. Run it through a mock generator, a mock server so that people can interact with it before it even exists. And then later on. Code it and make sure that it matches the contract test for that. But do you think, do you think that's how everyone should work? \[00:15:29\] ****Joel Clermont:** I like your use case because I, I do join a lot of, I'll call 'em legacy projects where you don't have documentation. In fact, \[00:15:37\] I hate to say this, sometimes you don't have tests, right? Like, \[00:15:40\] and so that I could see as a useful time to generate something just, just to get a starting point. And then edit it from there, and then like, you know, write contract tests around the little bits as you go. And, and so I, I gotta remember that because a lot of times when I join a project, like decisions have already been \[00:16:00\] made and now you're just trying to catch up. And I, I think code generation could be an excellent use case for that. Just to, to get you to where you wanna go faster. And to not have to write all of that by hand and, and reverse engineer it. \[00:16:11\] You know, like try to \[00:16:11\] figure out, well what are all the overlapping validation rules or schema constraints in the database layer or things like that. A static analysis can get you 80% there. \[00:16:21\] ****Phil:** for sure. I think. An interesting third way I've come across, and we've written about it a bit on the blog, is kind of using HTP traffic inspection, right? So I've, I've definitely inherited APIs where no one knows how the heck this works. You've gotta like decompile the iPhone app \[00:16:38\] and there's some third party client that you can't even look at the \[00:16:40\] code of and, and like no one can get into the compiled Java app on some bloody API that people haven't got the password for anymore. \[00:16:47\] It sounds silly, but I've been in these silly situations that help out a lot of \[00:16:50\] struggling \[00:16:51\] ****Joel Clermont:** boy. \[00:16:51\] ****Phil:** And everyone's like, right, we've gotta rebuild this API. And you're like, what API if I can't find any of the endpoints, like, everyone, check your browser history. What are we doing here? And \[00:17:00\] so being able to \[00:17:00\] kind of wrap it in a, in a traffic proxy that essentially says like all the requests. \[00:17:05\] Going in, or you can literally, you can wedge them into the browser. And, and then kind of run around clicking things. There's, there's a bunch of these different tools for sniffing actual traffic going through the system. And then \[00:17:15\] you can kind of take all of that and output an open API file and say, there you go. \[00:17:21\] And I feel like if we're, if we're talking kind of the catching up with design first, then I'm almost, I'm mu I'm, I generally feel more interested in going with the traffic sniffing approach because it's real. I. \[00:17:34\] Like you can still look at some, look at some stuff and, and get it wrong. Like if, if when you've got these complicated dynamic payloads where there's this or this or this, that could come through and you're like, it's definitely only this 'cause you've only seen it come, you know, you haven't noticed any null values in there. \[00:17:46\] So you said it, it is a required property. Mistakes like that can be made depending on how much \[00:17:50\] the sample size goes through. But I, I feel like the promise of having these kind of framework integrated static \[00:18:00\] analysis. Tools is potentially misleading unless they can hook into enough where you literally don't need to add other stuff yourself. \[00:18:10\] 'cause as soon as you ask the humans to get involved, that's when they're just introducing human error. Have you come across anything like that? \[00:18:17\] ****Joel Clermont:** Yeah. Um, Not directly. 'cause again, I've not worked on a project \[00:18:21\] recently that has done the automated tooling, but, but I've, I've experienced drift for sure. \[00:18:26\] And, and honestly that's one of my fears with Code Generation, I, you alluded to it earlier, is like if you make an unintentional change, well, and it just regenerates the spec to match it. \[00:18:35\] Now your tests are still passing, but I. Unless a human catches that in code review or design review of some sort, like you could unintentionally change your API even in a breaking way, which to me is like the worst possible outcome of of, of, of an API change. So I don't know if there's solutions to that, but yeah, the, either the intentional or unintentional drift is just, just an ongoing problem I have experienced, \[00:18:59\] \[00:19:00\] With that traffic inspection like that's super useful for like sort of those black box scenarios you were \[00:19:04\] describing. Like, I'm gonna have nightmares about that. But I, I'm almost wondering too if like that in conjunction, like when you're, when you're, when I'm joining a project, like I'm coming into some project, if I, if I just use static analysis, that doesn't tell me. How much dead code is in there, \[00:19:22\] right? And like now, maybe I'm wasting a week of work trying to document and test something and then I, ship it to the client. They're like, oh, we haven't used that in five years. Like that. Well, that was just a wasted week of effort. \[00:19:31\] So I could see pairing the static analysis with the traffic analysis just to like, even to prioritize. These are the paths that get the most traffic. Like \[00:19:41\] let's focus on those. These endpoints, they never got traffic. Oh, I'll ask the client even before I start working on it. Oh yeah. We can just delete that. Right? Like, so I could see a hybrid approach there being useful too. \[00:19:53\] ****Phil:** Yeah. Interesting. The I've seen a few frameworks that are. \[00:20:00\] me love the idea of code first a little bit more. I wrote \[00:20:04\] an article recently and I can't remember the names of any of these things. I think it's like Hu Hugo Oh, Hummer in, in Node and Fast. API I. I'll get some problem links in the show notes when I remember. \[00:20:15\] But there, there are, there's this kind of new idea of like open API Aware frameworks, right? Like built. Into the core of the framework is in the past, in the past you've got like swagger, PHP, which I hated. Like I, I, when I, when I first got involved with swagger, when I first tried to start trying to figure out how to document an API, I looked at swagger, PHP, and went, ah, for all the reasons we've discussed is literally you put a bunch of doc block \[00:20:39\] annotations in your code and hope that it's correct. \[00:20:41\] And, and I ran away and then I ended up doing other things. But that's still like the main premise. But with modern tools, like like Scribe is a really cool tool and like props to the people that have made this. But they, I. They've essentially got rid of quite a few of the annotations because the framework is kind of saying that this is a poster or, or \[00:21:00\] a get, and like they've, they know what the URL is from looking at the code. \[00:21:02\] They're doing a lot of that static, static analysis, \[00:21:05\] but it's still kind of hacked into the framework a bit. And there's still a certain amount of duplication. And so what I like about the frameworks that are. Coming at this from like the framework itself will tell you what your open API is, because just by using the framework, we are building up all of the bits that you need. \[00:21:25\] That seems really cool. Like when, when you write the validation rules for your models, they all line up with an appropriate validation in open API. \[00:21:33\] So when you're doing that, it sounds that that, that to me seems like it would be pretty much just fine because. The, the main, the main thing of like a PA design first is it's more about planning. \[00:21:47\] You should like plan this thing and spend a lot more time in the planning stage past the whiteboard, but before you start writing loads of code. But if the framework can \[00:21:56\] make itself simple enough that it, the framework is, \[00:22:00\] is. Just like a config DSL where you are, instead of writing open API in yaml, you're writing open API in this kind of a bit of PHB code and it's easy enough to read and it makes sense, and then you just shove a bit of business logic in there later that, that to me would be fine. \[00:22:15\] That kind of absolves the differences, removes the differences between open API. Well between design first and code first. It kind of makes them become the same thing. You just kind of like add your business logic some other time. But none of the tools in Laravel that I've seen are that, and it's quite rare to actually find them. \[00:22:35\] And at this point I worry that those projects are kind of just like hobby projects and not a whole thing. So \[00:22:43\] there's a bit of hope in there, but I don't think anyone's taken the idea too seriously at this point. \[00:22:47\] ****Joel Clermont:** Yeah. The, the reason this, this came on my radar recently is there is a newer package, like it's not even at 1.0 in Laravel called Scramble. I \[00:22:55\] don't, I don't know if you've seen this one. It will look at, like in, in LAR you have a form request \[00:23:00\] with validation rules. \[00:23:00\] It will statically analyze those rules and. Set like max links or field types or things like that, \[00:23:07\] purely based on that with like zero annotation, which is, I, I do like the promise of it. I really do. Like I, I get the appeal. Um, one, one other thing too I was thinking about is I. You know, when you mention whiteboards and upfront design, like I cringed a little bit because like I, I've come over the years to recognize like you never can plan super far in advance. \[00:23:30\] And, But I, you know, so one thing I wanted to clarify is like, even with the design first approach, I generally do it like literally end point by end point, right? \[00:23:38\] So I, I design it, I write the spec, we agree on it, I implement it. Then we repeat for the next one, not like, well, let's design the entire API and, and see what happens, \[00:23:47\] because that, that just never materializes correctly. \[00:23:51\] ****Phil:** man. Yeah, it's, it's something that I really, I need to do a whole episode on this, on like, how do you actually plan an API, because the stuff I said in my book was, was pretty immature at the time. I \[00:24:00\] think I was like, just think of all the endpoints you need and then like post and get next to 'em or whatever is, is a bit dumb. \[00:24:05\] But, but like the phases, the phases, of of planning and that's like literally stakeholder interviews, which is like a, a, a concept from user research and, and, and ux. and then kind of taking everyone's needs and requirements, and not necessarily too literally, but distilling down the essence of what. \[00:24:25\] They want and what they need and, and how multiple different stakeholders can ask for different things. But finding the middle ground that will be just fine for everyone, like doing all this really complicated work. And then another group of people or, or you, with a different hat on, can then start to say, well then these are the end points that I need. \[00:24:42\] But even then, like, is the, is jumping straight into endpoint endpoints premature. Like, do, do you need to think a little bit more about, about kind of the workflows and the, and the type of API that could be used and like, should this even be a, a endpoint based API or should, should it be event driven? \[00:24:56\] There's, there's so many stages to the planning and it's hard to figure out. \[00:25:00\] Where you should start and open API and design the spec first. Stuff like it doesn't help you with any of any of that. Like you, literally, your planning starts with Slack calls and whiteboards and like Miro boards and sequence \[00:25:13\] sequence flow diagrams and like chatting to a load of people and having a spreadsheet of feedback and all this crazy stuff that no one, no one that I know is really working on, apart from the tools that I've mentioned, but they're like generic. \[00:25:25\] And then once someone has says. Someone has said, you definitely need to make a HT PAPI and it should probably do these things. That's when it's time to get the open API out. 'cause you're literally turning the words they've just said right there of like, make an API that does this, this and this. \[00:25:39\] Then you kind of are going, well okay, we'll have that end point, that end point and that then end point and. Where design first gets really handy there is you can say, okay, I'm gonna turn those words into, like, I'm gonna reshape them into a little bit of yaml and I'm gonna like, the A PO will look like this. \[00:25:54\] Then it's much easier to change that little bit of YAML than it is to go and change a whole pile of \[00:26:00\] Laravel code. You're like, oh God, now I've got rename all of my controllers and that means I've gotta rename all my models and all the relationships are gonna be broken. And I, you know, \[00:26:08\] that's, that's really daft. \[00:26:10\] And so \[00:26:11\] pretending that. Code first is quicker 'cause you just add some annotations to some code that you wrote is predicated on the fact that you wrote exactly the right code first time. And when does that happen? \[00:26:24\] ****Joel Clermont:** It doesn't. Right, exactly. At least not for anything. Non-trivial. \[00:26:27\] ****Phil:** Yeah, well, alright. Pretty decent agreement on that. Screw. Screw that. \[00:26:33\] ****Joel Clermont:** Awesome. \[00:26:35\] ****Phil:** That was \[00:26:35\] ****Joel Clermont:** See, I I we're far too reasonable, Phil, in our, in our older age and maturity we're, we're mellowing and we're, we're just seeing things much more reasonably. \[00:26:43\] ****Phil:** Absolutely. Absolutely. I haven't had a fight about framework interoperability in a long time. Well, fantastic. Conclusion and Where to Find More \[00:26:48\] ****Phil:** Let's, let's call that where can people find more of you? You are, you're all over the internet doing all sorts of things. What, what what are you up to? \[00:26:54\] ****Joel Clermont:** Yeah, my, my main hub of activity now is mastering laravel.io. You know, on there we have a \[00:27:00\] podcast for Laravel and PP devs, the, the mailing list, \[00:27:03\] some books and courses if you wanna, like, keep me fed, like, you know, all, all that good stuff. But that's, that's really the central point for all of it. \[00:27:10\] ****Phil:** Nice. I will have to I need to get on that newsletter because I am using Laravel more and more and they've gone, they've gone on like a rampage of major versions and I'm like, I, I remember thinking four and five are quite nice and now version eleven's out. I'm like, I don't know what's happening. \[00:27:25\] Brilliant. \[00:27:26\] ****Joel Clermont:** Yeah, it does tend to move quickly. \[00:27:28\] ****Phil:** Cool. Alright, well thank you very much and yeah, pleasure having you on the show. \[00:27:31\] ****Joel Clermont:** Yep. Talk to you later. \[00:27:33\] ​ ### Mocking APIs with Microcks URL: https://apisyouwonthate.com/blog/mocking-apis-with-microcks/ Last updated: 2024-09-03T11:25:32.000Z *This was originally written for* [*Bump.sh*](https://bump.sh/?ref=apisyouwonthate.com)*, but I wanted to share it with a wider audience because Microcks is a very impressive OpenAPI mocking tool.* You’ve created beautiful documentation for your API powered by OpenAPI, what’s next? If you’ve been following the [API design-first workflow](https://docs.bump.sh/guides/bump-sh-tutorials/api-basics/dev-guide-api-design-first.md?ref=apisyouwonthate.com) the next phase is gathering feedback on the proposed design before investing loads of time building it. Sharing the documentation is a good start, but you can get even better feedback by giving stakeholders a mock server to interact with. A mock server simulates an API, allowing stakeholders to see if all the data they need is available, and give feedback on how easily their workflows can be solved based on the endpoints in the API. This can be thought of like a study group, something user researchers are used to doing for frontends, but is just as valuable for APIs. This used to be done with “prototypes” or manually made mock servers, but seeing as most Bump.sh users have already got the API described entirely by OpenAPI already, setting up mock servers is a whole lot easier using OpenAPI-powered tools like [Microcks](https://microcks.io/?ref=apisyouwonthate.com) to do the hard work for you. Using OpenAPI instead of code means changes can be made to the mock server far quicker than getting the API developments to change loads of code. Let’s take a look at using Microcks specifically, and see how we can tie it into your existing Bump.sh documentation. ## Step 1: Set up Microcks locally Microcks is a Java (Spring Boot) application, which comes packaged as a Docker container, so it can be deployed anywhere either of those environments are happy. Setting it up locally is probably the first step to see how it works. Whether you use [Docker](http://docker.io/?ref=apisyouwonthate.com) or prefer [Podman](https://podman.io/?ref=apisyouwonthate.com), grab one of those applications to install Microcks within a container. Assuming you’ve got Docker installed and running, you can use following command to get Microcks running on your computer. ``` docker run -p 8585:8080 -it --rm quay.io/microcks/microcks-uber:latest-native ``` When that’s done, open a browser tab and point to the [http://localhost:8585](http://localhost:8585/?ref=apisyouwonthate.com) endpoint, changing the port if you picked a different one. ![Microcks dashboard in a browser window running on http://localhost:8585](https://docs.bump.sh/images/guides/mocking-with-microcks/microcks-dashboard.png) ## Step 2: Add Your First API There are several ways to get OpenAPI into Microcks, but for the sake of simplicity we’re going to use the web interface to upload your OpenAPI. If you are just starting out and don’t have any OpenAPI yet, why not use the [Train Travel API](https://github.com/bump-sh-examples/train-travel-api?ref=apisyouwonthate.com) for now. Once we’ve got Microcks loaded in the browser, click on “Importers”, and click the “Upload” button. ![The "Direct upload artifact" modal window](https://docs.bump.sh/images/guides/mocking-with-microcks/upload-modal.png) The modal that pops up is asking for an Artifact, which is referring to various documents that could describe an API, like a [Postman Collection](https://docs.bump.sh/guides/bump-sh-tutorials/try-requests-in-postman/?ref=apisyouwonthate.com) and other “API description documents”. We can pop our OpenAPI in there, which is the same `openapi.yaml` document that you deploy to Bump.sh. The question about primary or secondary artifacts can be ignored for now. ![The API View contains the title and version of the uploaded API, a list of endpoints, and shows whether it found "samples" (examples)](https://docs.bump.sh/images/guides/mocking-with-microcks/microcks-api-view.png) Once you’ve uploaded, go to “APIs / Services” and click on your new API, which should have the same name as whatever was in the `info.title` of your OpenAPI. Below it you’ll see a bunch of operations, which are all the endpoints that Microcks found in your OpenAPI. ## Step 3: Try out the Mock Endpoints Open up one of the operations and see what Microcks thinks about it. ![The API view further down has a list of operations, which have been expanded to reveal more information about the mock URL and sample response](https://docs.bump.sh/images/guides/mocking-with-microcks/get-bookings.png) You should see a URL which you can copy, and the sample JSON it has pulled from the OpenAPI `examples` object. ``` http://localhost:8585/rest/Train+Travel+API/1.0.0/bookings ``` It’s basically: ``` /rest/// ``` If that operation has a GET method defined you can copy a curl command that’ll look something like this: ``` curl -X GET 'http://localhost:8585/rest/Train+Travel+API/1.0.0/bookings' -H 'Accept: application/json' ``` Running this command will then return the sample Microcks was showing the web interface. ``` { "data": [ { "id": "bfc5af2c-f477-43c4-8bdf-a00bdb939d65", "trip_id": "efdbb9d1-02c2-4bc3-afb7-6788d8782b1e", "passenger_name": "John Doe", "has_bicycle": true, "has_dog": true }, { "id": "b2e783e1-c824-4d63-b37a-d8d698862f1d", "trip_id": "b2e783e1-c824-4d63-b37a-d8d698862f1d", "passenger_name": "Jane Smith", "has_bicycle": false, "has_dog": false } ], "links": { "self": "https://api.example.com/bookings", "next": "https://api.example.com/bookings?page=2" } } ``` Getting a collection is a handy first example, but next lets try working with resources. Microcks uses examples with a particular naming contention, pairing up request examples and response examples with the same name. The naming convention is is `_`, so sticking with the Train Travel API, we could do this: ``` /bookings/{bookingId}: get: summary: Get a booking operationId: get-booking parameters: - name: bookingId in: path required: true description: The ID of the booking to retrieve. schema: type: string format: uuid examples: booking_1725ff48-ab45-4bb5-9d02-88745177dedb: value: 1725ff48-ab45-4bb5-9d02-88745177dedb booking_bfc5af2c-f477-43c4-8bdf-a00bdb939d65: value: bfc5af2c-f477-43c4-8bdf-a00bdb939d65 responses: '200': description: The booking details content: application/json: schema: $ref: '#/components/schemas/Booking' examples: booking_1725ff48-ab45-4bb5-9d02-88745177dedb: summary: John Doe value: id: 1725ff48-ab45-4bb5-9d02-88745177dedb trip_id: efdbb9d1-02c2-4bc3-afb7-6788d8782b1e passenger_name: John Doe has_bicycle: true has_dog: true booking_bfc5af2c-f477-43c4-8bdf-a00bdb939d65: summary: Billy Bikeless value: id: bfc5af2c-f477-43c4-8bdf-a00bdb939d65 trip_id: efdbb9d1-02c2-4bc3-afb7-6788d8782b1e passenger_name: Billy Bikeless has_bicycle: false has_dog: true ``` Upload that to Microcks again and it will now be aware of these two examples, and show you the mock URLs for both. ![](https://docs.bump.sh/images/guides/mocking-with-microcks/microcks-resources.png) Now you can call either of them, and get different HTTP responses with the appropriate example data being used for each. ``` curl -X GET 'http://localhost:8585/rest/Train+Travel+API/1.0.0/bookings/1725ff48-ab45-4bb5-9d02-88745177dedb' -H 'Accept: application/json' curl -X GET 'http://localhost:8585/rest/Train+Travel+API/1.0.0/bookings/bfc5af2c-f477-43c4-8bdf-a00bdb939d65' -H 'Accept: application/json' ``` This is already really useful, you could use this to build a read-only API client on your computer without needing the actual API to actually exist. If you need to send `POST`, `PUT` or `PATCH` requests you can do that too. Let’s try sending a `POST` request. The “Copy to clipboard” functionality in Microcks won’t automatically give you a predefined request body, but pop back over to your Bump.sh API docs to get a better curl example and it should look a bit like this. ``` curl -X POST 'http://localhost:8585/rest/Train+Travel+API/1.0.0/bookings' \ -H 'Accept: application/json' \ -H 'Content-type: application/json' \ -d '{"passenger_name":"New Passenger","has_bicycle":false,"has_dog":false,"trip_id":"4f4e4e1-c824-4d63-b37a-d8d698862f1d"}' ``` ``` { "id": "efdbb9d1-02c2-4bc3-afb7-6788d8782b1e", "trip_id": "efdbb9d1-02c2-4bc3-afb7-6788d8782b1e", "passenger_name": "John Doe", "has_bicycle": true, "has_dog": true, "links": { "self": "https://api.example.com/bookings/efdbb9d1-02c2-4bc3-afb7-6788d8782b1e" } } ``` That’s somewhat handy, but it’s showing me a generic response instead of the data sent in the POST request. Named examples would not help here, as we want to use client data from the request body and not just show predetermined examples. How can we New Passenger instead of John Doe, and how can we see the actual `trip_id` instead of the default one? ## Step 4: Customizing Responses Mocks can be made more powerful with [dynamic mocking](https://microcks.io/documentation/explanations/dynamic-content/?ref=apisyouwonthate.com), which is made possible through Microcks’ [templating system](https://microcks.io/documentation/references/templates/?ref=apisyouwonthate.com). Using a combination of predefined variables, built-in functions, and some JSON Pointers, we can provide a dynamic example in OpenAPI which put together a more useful response. ``` responses: '201': description: Booking successful content: application/json: schema: # snip examples: new_booking: summary: New Booking value: |- { "id": "{{ uuid() }}", "trip_id": "{{ request.body/trip_id }}", "passenger_name": "{{ request.body/passenger_name }}", "has_bicycle": {{ request.body/has_bicycle }}, "has_dog": {{ request.body/has_dog }}, "links": { "self": "https://api.example.com/bookings/1725ff48-ab45-4bb5-9d02-88745177dedb" } } ``` Go back to Importers, Upload the `openapi.yaml` document again, and it will merge changes into API so we can see how they look. ``` **curl** -X POST 'http://localhost:8585/rest/Train+Travel+API/1.0.0/bookings' \ -H 'Accept: application/json' \ -H 'Content-type: application/json' \ -d '{"passenger_name":"New Passenger","has_bicycle":false,"has_dog":false,"trip_id":"4f4e4e1-c824-4d63-b37a-d8d698862f1d"}' ``` Now we will see the values we sent showing back up, and the computed random UUID. ``` { "id": "a248a638-000a-44b4-b19b-0ca30507a940", "trip_id": "4f4e4e1-c824-4d63-b37a-d8d698862f1d", "passenger_name": "New Passenger", "has_bicycle": false, "has_dog": false, "links": { "self": "https://api.example.com/bookings/1725ff48-ab45-4bb5-9d02-88745177dedb" } } ``` Finally, that HATEOAS link at the bottom there is not right. The id is `a248a638-000a-44b4-b19b-0ca30507a940`, so how can we get that showing up in the link? If we used `{{ uuid() }}` again it would be generate a second different UUID, Microcks templating has a brilliant feature called capture. ``` examples: new_booking: summary: New Booking value: |- { "id": "{{ uuid() > put(bookingId) }}", "trip_id": "{{ request.body/trip_id }}", "passenger_name": "{{ request.body/passenger_name }}", "has_bicycle": {{ request.body/has_bicycle }}, "has_dog": {{ request.body/has_dog }}, "links": { "self": "https://api.example.com/bookings/{{ bookingId }}" } } ``` Upload that again. Run the command again. ``` { "id": "b4441c3d-b659-4f57-95d3-ce6ee592da7c", "trip_id": "4f4e4e1-c824-4d63-b37a-d8d698862f1d", "passenger_name": "New Passenger", "has_bicycle": false, "has_dog": false, "links": { "self": "https://api.example.com/bookings/b4441c3d-b659-4f57-95d3-ce6ee592da7c" } } ``` Perfect! Now you can do almost anything you need to do with the API. ## Step 5: Deploy Microcks Somewhere Working with Microcks locally was an OK way to get the hang of it, and could be useful for some solo developers, but the chances are you’ll want to get this off your laptop and share it with people. You can deploy Microcks anywhere that takes Docker/Kubernetes instances, and they have lots of documentation on your [deployment options for Microcks](https://microcks.io/documentation/explanations/deployment-options/?ref=apisyouwonthate.com). One approach is to deploy the Docker container into a managed Kubernetes cluster on [Google Kubernetes Engine](https://cloud.google.com/kubernetes-engine/docs/deploy-app-cluster?ref=apisyouwonthate.com), which has free trials and free tiers to ease you into the process. ![A screenshot showing Google Cloud's Kubernetes Engine section, on the Create a deployment page](https://docs.bump.sh/images/guides/mocking-with-microcks/gke-setup.png) This puts Microcks behind a publicly accessible load balancer, so you can stick some DNS on it and have everyone able to call the hosted version of the mock server. ``` curl -X GET 'https://mocks.example.com/rest/Train+Travel+API/1.0.0/bookings' ``` Once this is done we can replace the manual uploads with some automatic solutions too. ## Step 6: Automate Mock Updates Most Bump.sh users use some form of Continuous Integration (CircleCI, GitHub Actions, Jenkins, etc.) to push changes to their API documentation whenever the source code is changed, and you can work this way with Microcks, or there are some alternatives you can try out. ### Update Mocks with Continuous Integration Below is the standard GitHub Action used to deploy API changes to Bump.sh with one modification to also deploy changes to your Microcks server. ``` # .github/workflows/deploy-docs.yml name: Deploy API documentation on: push: branches: - main jobs: deploy-openapi: if: ${{ github.event_name == 'push' }} name: Deploy API documentation on Bump.sh runs-on: ubuntu-latest steps: - name: Checkout uses: actions/checkout@v3 - name: Deploy API documentation uses: bump-sh/github-action@v1 with: doc: 68ac0647-184a-4e9d-accc-682a5b1f7189 token: ${{secrets.BUMP_TOKEN}} file: api/openapi.yaml - uses: microcks/import-github-action@v1 with: specificationFiles: 'api/openapi.yaml:true' microcksURL: 'https://mocks.example.com/api/' keycloakClientId: ${{ secrets.MICROCKS_SERVICE_ACCOUNT }} keycloakClientSecret: ${{ secrets.MICROCKS_SERVICE_ACCOUNT_CREDENTIALS }} ``` You’ll need to set up some secrets on your repository for that Microcks service account, but then you’re done! A fully functioning mock server running on the cloud, which you can interact with internally or externally depending on how you set it up. > Learn more about [Microcks Automation](https://microcks.io/documentation/guides/automation/?ref=apisyouwonthate.com) to see how to push updates to Microcks using other CI systems, via the API, or using the CLI elsewhere. You can also use the [Microcks Scheduler](https://microcks.io/documentation/guides/usage/importing-content/?ref=apisyouwonthate.com#2-import-content-via-importer) instead, to pull content from a repo on a regular schedule instead of pushing. ## Step 7: Add Mock Server to API Documentation Once the mock server is up and running, you can help make it easier to find by adding it to your Bump.sh API documentation. To do this, we can add the server URL into the servers list like this: ``` servers: - url: https://api.example.com description: Production - url: https://mocks.example.com/rest description: Mock Server ``` Adding this second server URL will offer users a dropdown menu in the Bump.sh documentation. ![The select box apears on the Bump.sh API documentation allowing users to pick between servers based on server name](https://docs.bump.sh/images/guides/mocking-with-microcks/multiple-servers.png) The mock server is now an option, and all of the URLs and example HTTP requests will show up using the chosen server URL. ![A screenshot of the API documentation updated to contain the mocks.example.com after mock server has been selected](https://docs.bump.sh/images/guides/mocking-with-microcks/bump-mock-server-curl.png) If there’s no production API only the mock server is ready then only define that: ``` servers: - url: https://mocks.example.com/rest description: Mock Server ``` > Perhaps you cannot update the OpenAPI document to get that server added as it’s generated by somebody else or hosted online, in which case take a look [Overlays](https://apisyouwonthate.com/blog/use-openapi-overlays-today/). ## Summary Let us know what you think of Microcks as a mocking tool, and how it compares to [Prism](https://github.com/stoplightio/prism?ref=apisyouwonthate.com), [Wiretap](https://pb33f.io/wiretap/?ref=apisyouwonthate.com), and [others](https://openapi.tools/?ref=apisyouwonthate.com#mock). It also has testing functionality which we didn't get into here but will in upcoming articles if you're interested. Just let us know down below and we'll get more articles out on what you're interested in. ### Spotlight Studio in Swagger Hub? URL: https://apisyouwonthate.com/newsletter/spotlight-studio-in-swagger-hub/ Last updated: 2024-08-15T11:49:30.000Z If we have all learnt anything over the past few weeks, it's how good Australia is at breakdancing. As a fellow Aussie, I felt very proud of how many medals we snagged in the Olympics despite our small population. Sport aside, Smartbear recently gave SwaggerHub a facelift by shoving the Stoplight editor in there. It's great to see the experience improved, but I can't help but listen to that little voice in the back of my head asking what this means for Stoplight in the future. Maybe I am worrying for no reason, but we will see. \-- Alexander ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2024/06/treblle-assistant.png) ## Alfred AI: The Ultimate API Assistant Alfred AI is a cutting-edge, AI-powered assistant designed to revolutionize API documentation, discoverability, and integration. Alfred automates manual workflows, for faster API integrations and a superior developer experience. [Discover Alfred](https://treblle.com/product/alfred?utm%5Fsource=newsletter&utm%5Fmedium=apis-hate-sponsorship&utm%5Fcampaign=alfred%5Fai) ## The API Round-Up ### [Stoplight in SwaggerHub](https://www.linkedin.com/posts/frank-kilcommins%5Fapis-swaggerhub-openapi-activity-7226977321411366912-Lwra/?utm%5Fsource=share&utm%5Fmedium=member%5Fdesktop) SwaggerHub has new features, including the Stoplight editor with OAS, API Template, and API Domain form-based editing. Check out this video from Frank Kilcommins for all the details. ### [HTTP/0.9 From Scratch](https://kmcd.dev/posts/http0.9-from-scratch/?ref=apisyouwonthate.com) This is the first blog in a series focusing on building each major version of the HTTP protocol in Go. APIs are built on top of HTTP, so if you want to solidify your understanding, look no further. A post for HTTP versions 0.9 and 1.0 are already out. ### [How To Generate an OpenAPI Spec With Fastify](https://www.speakeasy.com/guides/api-frameworks/fastify?utm%5Fsource=apisyouwonthate&utm%5Fcampaign=2408/15) Fastify, a web framework inspired by Express and Hapi, is my go-to framework for building Node API's. Speakeasy talks us through using Fastify to generate your OpenAPI spec. The tutorial then takes it further by using Speakeasy to generate client SDKs for your API based on the spec you generated. ### [Using OpenAPI to Document the API Operations You Consume](https://apichangelog.substack.com/p/using-openapi-to-document-the-api-operations-you-consume) Bruno shares an interesting take on using OpenAPI for more than documenting your API. The idea is to use the OpenAPI spec to describe how you consume one or several API operations. It's a different viewpoint, but I wonder if the author knew about the Arazzo spec. ### [Microcks Spectral ruleset](https://github.com/microcks/microcks-spectral-ruleset?ref=apisyouwonthate.com) Microcks (a mocking tool) has built a spectral ruleset (an OpenAPI/API linting tool) to help you build OpenAPI docs that'll work nicely in their mocking tool. The style guide is more than just opinions; it can also push people towards compatibility with particular tooling. ### [Fetch in Framer](https://www.framer.com/updates/fetch?ref=apisyouwonthate.com) Framer has released a new feature allowing you to add APIs directly to your Framer site without writing any code. Adding dynamic data from any API to your site this easily is pretty impressive. Building sites just got a lot easier. ### [OpenAPI, Swagger and Spring Boot REST APIs](https://www.theserverside.com/video/OpenAPI-Swagger-and-Spring-Boot-REST-APIs?ref=apisyouwonthate.com) A handy guide on adding OpenAPI to a Spring Boot Rest API, giving you access to Swagger UI and better testing options. Swagger UI is not as friendly as modern tooling, but Spring Boot seems to favour it. ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2024/08/speakez-1.jpg) ## Speakeasy: Build APIs Your Users Love Your API deserves a great developer experience. Get one by using Speakeasy to generate idiomatic, type-safe SDKs from OpenAPI [Start Generating](https://www.speakeasy.com/?utm%5Fsource=apisyouwonthate&utm%5Fcampaign=240701) ## APIs You Won't Hate The latest from the team at API's You Won't Hate. ### [🎙️ A product to prevent billing nightmares](https://apisyouwonthate.com/podcast/supergood-ai-for-tracking-api-spend-in-real-time/) We've all woken up in a cold sweat in the middle of the night, worried about a rogue serverless function resulting in an outrageous AWS bill. In this episode of the Podcast, Alex Klarfeld, founder and CEO of Supergood.ai, shares the product's origin story. Supergood is out to help teams track API spend in real-time—something we can all get behind. --- ## From Our Community Articles written and shared in our free [Slack community](https://slack.apisyouwonthate.com/?ref=apisyouwonthate.com). ### [Effective API Error Handling](https://www.bitstreamapis.com/blog/implementing-effective-api-error-handling/?ref=apisyouwonthate.com) Hedley reached out to our community for input on this article. We love seeing you all working together and sharing learnings. Hedley discusses the importance of robust API error handling to enhance API reliability and user experience—a detailed overview covering error responses, status codes and error messages. ## Support APIs You Won't Hate When you become an member, you'll get access to members-only content while directly supporting our work. Your support helps us to keep making resources for the API community. [Become a member today](https://apisyouwonthate.com/#/portal) Thanks so much to our [members](https://apisyouwonthate.com/membership/): Al N, Kin L, Juxt, Vedran C, Alex R, Nolan S, Frank, James D, Bill D, and Ali S. Your support means the world to us! Until next time, Alexander, Phil & Mike ### No more API usage nightmares, with Alex Klarfeld from Supergood.ai URL: https://apisyouwonthate.com/podcast/supergood-ai-for-tracking-api-spend-in-real-time/ Last updated: 2024-08-15T01:09:32.000Z ## Show notes - [Supergood.ai](http://supergood.ai/?ref=apisyouwonthate.com) - OSS on [github](https://github.com/supergoodsystems?ref=apisyouwonthate.com) - [Alex Klarfeld](https://www.alexklarfeld.com/?ref=apisyouwonthate.com) - [GitHub](https://github.com/aklarfeld?ref=apisyouwonthate.com) - [LinkedIn](https://www.linkedin.com/in/alex-klarfeld-a2706848?ref=apisyouwonthate.com) - Threads / [Twitter](https://x.com/AlexKlarfeld?ref=apisyouwonthate.com) #### Transcript \[00:00:00\] \*\*Mike Bifulco:\*\* Hello friends, and welcome back to APIs you won't hate. My name is Mike Biko, API developer, consumer sufferer designer, all of those things. And coincidentally also hosted the show for quite a while now. I am excited to be sitting down today and chatting with a new friend to talk about a product that is very relevant to lots of API developers. \[00:00:17\] If you have built. A tool or product or service that relies on lots of APIs, you've probably also incurred spend and cost and you know, sent dollars across the wire having to do with using other people's service services. Like me, you've probably also heard nightmarish tales of people accidentally taking advantage of too much of a service and ending up with a, you know, $20,000 spend accidentally month over month because of a runaway. \[00:00:42\] You know, for a loop or something like that. And so I believe get into answering some of the questions and some of this sort of neck of the woods. Chatting with my new friend Alex Clarkfield who's founder of Super Good ai. Alex, thanks so much for joining me today. I really appreciate you being here. \[00:00:55\] How are you doing? \[00:00:56\] \*\*Alex Klarfeld:\*\* I am doing well. Thanks for having me. I'm excited to be here. \[00:00:59\] \*\*Mike Bifulco:\*\* Yeah, \[00:01:00\] of course. I'm really, really glad you're here as well. I, I think I hinted pretty strongly at super good and what you're doing. Why don't we start with the elevator pitch of super good and I'll work back through your story from there. But yeah, tell, tell me the 32nd pitch for what super good looks like. \[00:01:12\] I. \[00:01:13\] \*\*Alex Klarfeld:\*\* Yeah, for sure. It's pretty simple. So it's a couple lines of code you drop into your code base and we automatically monitor the cost and performance of your third party APIs. That's it in a nutshell. The performance and cost piece, you can go into like a rabbit hole of like exactly what that means. \[00:01:27\] I'm sure listeners of this podcast, like horror stories are popping into their heads right away when they talk about like APIs failing or causing problems and taking down parts of the business, causing executives to freak out over. Insane invoices. But those are all the problems we're looking to solve. \[00:01:43\] \*\*Mike Bifulco:\*\* Yeah. Got it. Well, you've certainly come to the right place. I am definitely one of those people who uses a lot of APIs that are sort of spend per usage or at least on some monthly cost based on. You know, capped use. And I think lots of the folks who listen to this show will be really interested in that as well. I also should add that \[00:02:00\] many of the folks who listen to the show are also builders of API based products and would probably be interested in what you have to say about offering spend management tools to people and creating user experience, end user experience for the people being billed. That is, I dunno, the most ideal perhaps. \[00:02:15\] Before we get into that, I would really like to hear about your story before Super good. So can you tell me a little bit about you before this and kind of how you got here? \[00:02:22\] \*\*Alex Klarfeld:\*\* Yeah, for sure. So, so before Super good, I helped found a PropTech FinTech company called Divvy Homes divvy really cool mission. We were basically trying to help renters transition into homeowners. So we take folks who like, maybe they couldn't get a mortgage today. We thought they could get one in like a couple years time. \[00:02:37\] And then they would pick a house out. We would buy it for them, and then they would, live in it. And each month, part of their monthly rental payments would go towards a down payment. And the idea is at the end of this program, they would we'd cash 'em out and we'd like help them get FHA mortgage. \[00:02:50\] So really cool mission. The sort of like nitty gritty inside was we built like an end-to-end home buying platform. So every part of the home buying process was all in-house software. \[00:03:00\] Which, you know, was a lot of my responsibility for building. And as you can imagine, we had actually sat down and counted, we had like 47 different. \[00:03:07\] Third party APIs that we had to integrate with. So it was the usual FinTech suspects like Plaid, Experian Equifax, TransUnion checker, you know, you name it. All the payroll APIs. And then there's sort of like a long tail of a bunch of like real estate specific ones. And, you know, two things would happen, I'm sure a ton of people listening can relate to this was, you know, my day job is like building code, trying to build the best product for our customers. \[00:03:32\] And what felt was like my night job was dealing with vendors so. One, like we'd get a bill at the end of the month. Not to name names, but I will. But like DocuSign would like suddenly say like, Hey, you guys are over and you owe us some crazy number of, of overages. And we're like, they would send us a graph, I remember very vividly. \[00:03:49\] And we're like, how do, how do we not have this graph? Like, it's like buried in logs somewhere and it's crazy. And the other part would happen is like a developer, these APIs would break in really weird ways and it wasn't like \[00:04:00\] outages are easy to detect, like it's down. What do you do about that? But like schema changes were a little bit more nefarious. \[00:04:07\] Like the vendor suddenly upgrades their version and everything breaks. Another one that like schema changes were like value changes, like enums changing. That was pretty bad. The other one that was like worse I'd imagine is like data quality issues, like all the, the hype that, you know, someone trying to sell you an API can give you like you really don't know until you run it in production and like against real customers. \[00:04:28\] And then once it's in there, you have to like continually analyze to make sure like you're paying like a non-zero amount of money to that. So better get the most bang for your buck. So these were the two problems I kind of ran into as super good was like. The sort of data quality, keeping up with like the version of the APIs and then they kinda like all boiled down to like the bottom line of our company. \[00:04:46\] Like we were running customers through this application flow, hitting a gauntlet of APIs and it was just like would make or break our business depending on which, like how the API performed and how much it cost. \[00:04:56\] \*\*Mike Bifulco:\*\* Yeah, I think what a lot of. People maybe haven't \[00:05:00\] experienced UN until they're sitting at the helm of a company is that a lot of your success or failure once you have a product that people are buying, is getting down to unit economics and driving each user's experience to profit for you and value for them in whatever way that means, you know, depending on the company. \[00:05:15\] And it's a really hairy problem to solve. And every dollar that goes out the door in the, in the name of making your customers happy is you know, working against you. And effectively in that sort of unit economics game, it's a tricky thing to wrap your head around especially if you know, you've, you've spent your days studying, say, computer science and worrying more about merge sorts than like. The realities of a consumer facing product or even a B2B product for that matter. \[00:05:37\] Um, I, I'm curious, I don't know if you mentioned before you built divvy homes and before you built super good. What was your background? Or you come from a prop tech background. Are you more on the business side? Are you a, a sort of straight up and down developer? \[00:05:48\] Like how, how has your, how has your career path shaped where you are now? \[00:05:53\] \*\*Alex Klarfeld:\*\* Yeah, for sure. So I'm a full stack developer through and through. I studied electrical computer engineering in college. Like made my way from like \[00:06:00\] embedded software into like web application software from the Midwest, moved out to Silicon Valley based on. Some, some friends that came out here and just got, kind of got fascinated by the ecosystem, but I've always been a full stack developer. \[00:06:12\] Sort of a master of none I like to say, but I just love the breadth that it, that sort of, sort of gives me and like really being able to like, dig into very specific problems. I also love product so just combining like the two, two parts of like, just like building things. I care a lot about the, the what that I'm building more so than a lot of like. \[00:06:30\] Much better developers than me care about, like the how. So that's, that's, that's kind of me, \[00:06:36\] \*\*Mike Bifulco:\*\* Sounds shockingly familiar. I feel, I feel like in a lot of ways I'm looking in a mirror there. Can you tell me then about ha have you worked on developer facing products before Divvy homes? Sounds like it was more consumer facing, but this is really built building four developers or dev teams. Is this your first time around the block with that? I. \[00:06:52\] \*\*Alex Klarfeld:\*\* S. P pretty much. I had a I worked at a company that was, it was very interesting. It was tools for like data engineering \[00:07:00\] teams. It was founded by one of the he was founded by the creator of Postgres Mike Stone Breaker. So we were building for fortune 500 companies basically, and like the development teams in there. \[00:07:10\] So that was an experience. And then at Divvy we were like, we had a very cool consumer facing application. But the, like, the sort of like iceberg below, it was building for ops teams actually. 'cause we had like an operations, like it was a, it was like software powered ops company. So no developers specifically, but I'm familiar with like working with like the in-house folks. \[00:07:29\] Those were usually the people I had to work with when I was like dealing with API issues because it was typically like an ops person or like someone on underwriting's. Like I don't, I don't know why a bunch of customers are starting getting stuck in their application flow. It's like, all right, well Experian is down. \[00:07:41\] And that kind of explains it. \[00:07:43\] \*\*Mike Bifulco:\*\* Okay, cool. Yeah, and I, I guess the other thing is as a you know, self-assigned full stack developer, you've probably been subject to many other developer products along the way too. So I \[00:07:53\] think that buys you some familiarity with it, at, at the very least. \[00:07:56\] \*\*Alex Klarfeld:\*\* Yeah, I've been a, I've been a huge consumer of these, these \[00:08:00\] APIs and there's, there's great ones and then there's not so great ones. And I just, my goal with super good, my long-term vision is I wanna make all the APIs great. So like the ones that are great are like. Killing it in the market like plaid, I'd say. \[00:08:14\] Like, they're so ubiquitous, but they're, they're ubiquitous for a reason. Like the, the documentation is incredible. The onboarding experience for developers are, are incredible. And like, I think also like Stripe is, is one of the top dogs in terms of like developer experience. And I like, there's just a long tail folks who like either just getting started out or don't have the resources to invest. \[00:08:33\] And I just wanna like, bring everyone up to the level that like the stripes and plaids of the world's have set. \[00:08:38\] \*\*Mike Bifulco:\*\* Sure, yeah. It's an extremely high bar to hit. Is one of \[00:08:41\] those things that I think a lot of dev teams aspire to, especially if you're building things for other developers to use. We as developers tend to be really good at sniffing out things that we don't like, for whatever reason. And Metaphor is a really powerful tool there, like saying, Hey, you know, like plaid's version of this is so much better wise than this. \[00:08:57\] More like plaid or, you know, what were they thinking when they got here is a very \[00:09:00\] common thing. And usually by way of comparison, like, oh man, I really wish that was like Stripes web hooking interface or whatever it may be. We, we also very nearly made it to 10 minutes into the podcast without me having to give the disclaimer that I also previously worked for Stripe, \[00:09:13\] \*\*Alex Klarfeld:\*\* Oh, cool. \[00:09:13\] \*\*Mike Bifulco:\*\* something I feel like I should call out on, on each and every mention of it. \[00:09:17\] For, for clarity and, and for transparency and all that. \[00:09:20\] \*\*Alex Klarfeld:\*\* Cool. okay, so we touched a little bit on the, the elevator pitch for super good. So the tagline I think on your site says something along the lines of like, curb your API spend before you get a bill which is super cool and. I, I get the mission from that. \[00:09:34\] \*\*Mike Bifulco:\*\* You also mentioned that it's something like a, a two or three line implementation to do that. So I'm curious about that. What does that look like? How do I integrate with super good and start getting benefits from it? \[00:09:45\] \*\*Alex Klarfeld:\*\* For sure. So super good is a, the way that it works today is for a free version, it's a passive interceptor. It's not a proxy. So the way it works, all the clients or SDKs are open source. You can check us out on GitHub. You drop in the few lines of code and we basically \[00:10:00\] apply a light monkey patch to like. \[00:10:02\] Your favorite HTP library. And then what it does behind the scenes is like, as calls get you know, made in your application, we kind of stow away the calls locally in memory. And then what we'll do is basically redact everything. Like there's, one of the big things that I take very seriously is we do not want to have any data that we should not have. \[00:10:20\] So everything is redacted by default. You can set that flag and then once on an interval, those logs will get shipped to our. Our, our service, which will start to analyze from a cost perspective, data quality perspective. We're sort of like just running live analysis on your HTP logs specifically in order to solve this problem. \[00:10:40\] \*\*Mike Bifulco:\*\* Got it. And so it sounds like it's a client side integration. I mean, I would imagine there's a service side element to this too, wherever API calls are happening. Is, I \[00:10:47\] feel like one of the, the, the things that my API dev friends would ask is what does that look like in terms of performance hit? \[00:10:54\] Is, is, are you getting in between my call and the service? Or is this truly something that is \[00:11:00\] just like queue safe for later? \[00:11:02\] \*\*Alex Klarfeld:\*\* Yeah, it's, it, I intent, we intentionally built it to get out of the critical path. So that is why it's not a proxy. So it doesn't mess with any like actual network traffic. There's no like network call between the call that you're making. The only limitation is like memory like the memory limitations of the machine that you're running on. \[00:11:19\] 'cause all it is, is dropping the call into like a memory store locally and then running. Redaction on the, the, the js ON. So it's very minimal. Latency hit just like completely memory bound to the machine that you're running. \[00:11:32\] \*\*Mike Bifulco:\*\* Yeah. Fascinating. That's a very interesting angle. I can imagine the challenge of trying to track billing for every, every API call is the every a API call part of that. So what, what have you done \[00:11:43\] to try and. Determine like which of these calls I'm making out of my service are ones that are metered in some way. \[00:11:50\] How do you determine who's billing me? \[00:11:53\] \*\*Alex Klarfeld:\*\* Yeah, exactly. So when the calls come in, we basically run it through categorization right away. So try to figure out like \[00:12:00\] easy ones, like domain is specific to the vendor. The harder one is each. API has like a different signature, for example, like S3 uses, like the sub-domain to like identify calls, but like maybe certain plaid calls use the path. \[00:12:12\] And then sometimes they'll stick a like a ID somewhere in there. So the first step that we do is sort of like categorize the calls. So as they flow in. We'll try to automatically categorize them best we can. We actually use LLMs for that. It's pretty good. And then sort of like we can manually re-categorize things if they're not like being grouped properly. \[00:12:29\] And then after that, the, we kind of had have like a toolbox in like the standard way that people bill. So like. We know how open AI and together, like all Bill and plaid, also bill specifically based on like accounts rather than usage. And then we sort of built this flexible framework on our end where we have the tools basically to just implement custom billing. \[00:12:50\] Like if you're like overage based where it's like you get certain amount of calls for a certain value and anything over that, it's more value. We can, we can basically fine tune it to your \[00:13:00\] specifications. But like for free users right away it's just like, drop us in and get usage. And then the sort of like more advanced metering for like the long tail APIs. \[00:13:08\] We, we, we charge for \[00:13:10\] \*\*Mike Bifulco:\*\* Yeah. Okay, I, I get that. So sorry for the pause there. I think one of the things that I am fascinated by is that it sounds to me like the hello world is drop a snippet in and the product sort of starts sprinting away and going to work for you. \[00:13:26\] \*\*Alex Klarfeld:\*\* that. That's right. \[00:13:28\] \*\*Mike Bifulco:\*\* yeah. Yeah. Okay. That's, that's really cool. How did you, let's see. \[00:13:31\] Yeah, many questions to come from there. What did the first version of this product look like? Like did you target one API in particular, or were you starting from categorization from day one? \[00:13:40\] \*\*Alex Klarfeld:\*\* Yeah, the first version of the API was, it was just a node library that like monkey patched fetch and then just logged all the calls. I think like a lot of people, like outta the gator, like, you know, I have Datadog or, or, or sort of century to do that. And like the first version that we built at Divvy was like trying to use Datadog to do this. \[00:13:58\] And \[00:13:59\] it's \[00:14:00\] really hard, like it's just a very, like both like expensive from like a cost perspective where you have to. Like triangulate everything, like every, every single thing that you need to, to meter or you need to like tag and, and instrument appropriately. Plus the sort of like dashboarding and setting up of it is like hard. \[00:14:15\] And sometimes like there's a lot of like, you know, native knowledge to the, to the team. That's like, it's one person that understands the Datadog logs and then you also have to like maintain it. So like not only is Datadog gonna charge you per like field that you're indexing, they're gonna, you're gonna like have to continue to maintain it. \[00:14:30\] So the first version of this honestly, was like, tried to build in Datadog super hard. Very difficult to maintain. Second version of it was like a custom logger which is more similar to what super good is today. And then really it was just a graph on a screen. There's just like, here's a bunch of vendors you use and then here's like their, their associated usage. \[00:14:47\] Get ready for the invoice and, and, and kind of like building on from there. \[00:14:51\] \*\*Mike Bifulco:\*\* Okay. And so. Let's say I, if I've integrated with super good, it's starting to measure metering for the various APIs I'm consuming. What does my \[00:15:00\] end user experience look like with Super good? Is it a dashboard I get? Do I get a series of emails? Is it ACL I? Like what, what's my how do I get insight into what this all looks like? \[00:15:09\] \*\*Alex Klarfeld:\*\* For sure. So we have a dashboard. Though as an engineer myself, I do not want to give engineers a. Another single pane of glass to look at. So it is, is meant to audit if you, if you want to. The two ways that we mainly interface with, with folks is through alerting. So right now we have standard slack alerting set up. \[00:15:26\] So if it's like we start to notice overages spikes in the bill, you'll get, you'll get a Slack alert and then also like sort of a weekly report. So at the end of the week you'll say, okay, this is sort of like the usage that we, we've been seeing. If you have uploaded like your contract information, we'll sort of be able to like. \[00:15:41\] Give you a heads up that like some sort of renewal is, is happening. And then the sort of the, the other suite of tools we have is around like data quality monitoring and like more of like the nitty gritty error monitoring, where it's like, okay, we see an error that we haven't seen before. We just wanna flag this to you to make sure it's not taking down another system. \[00:15:58\] 'cause a lot of these errors \[00:16:00\] that issues have can go quietly. Mostly because if you, even if you have instrumentation set up and like the best tools, which we, we use personally, like Century and Datadog, they get noisy because APIs might error for. A non error reason. For example, like if plaid, someone disconnects their account, you're gonna get errors all the time, every time you pull it. \[00:16:18\] But like, it's not really an actual error, so you tend to ignore it. But if you have like 80% of your calls to plaid totally disconnected, that's probably something that you, you shouldn't ignore to save some money, at least. \[00:16:29\] \*\*Mike Bifulco:\*\* Sure. Yeah, that all sounds very familiar. Actually. I spent a non-trivial amount of time, just this week debugging an error with Twilio. We were getting an error response from, from some API calls. And the API call is basically like Mike and Alex are texting each other. If I have a text conversation with you over SMS and I, if I send an API call to say, please add Alex to this conversation. \[00:16:48\] It fails sends a a 4, 4 0 9 back. And the docs for Twilio say like, if you get this error, probably just ignore it. But it, it's, you know, when you start getting that error a lot, it sends up a lot of \[00:17:00\] signals. And especially if that was something that was billed, it's something I'd wanna know about. But it may also just be a signal from my tooling that we're trying to do something we shouldn't be right. \[00:17:07\] It, it may be a a non breaking error or maybe a breaking error, but it's a little hard to ascertain from the beginning. And I would certainly wanna know if all of my, or 80% of my Twilio calls started failing in a hurry too. That's that's a great angle to take. \[00:17:21\] \*\*Alex Klarfeld:\*\* Yeah, and the, I mean like it just feels like pushing a boulder up the hill. Like even if you instrumented something to catch that specific error and you put all the instrument, like, okay, this is a thing that we need to like alert. If it's only exceeding a certain threshold, it might be another one that's like probably gonna come up in the meantime. \[00:17:36\] And it's just sort of never ending battle. That's why like as an engineer, it's like. Just drop us in. We handle like the instrumentation remotely. Also, like APIs are like a finite there's like a finite schema for each of them. Like there's documentation, there's a lot of data that sort of like confines the problem space. \[00:17:50\] So like why not utilize that rather than trying to set everything up on your own. \[00:17:55\] \*\*Mike Bifulco:\*\* Sure. Yeah. To, to that end, I'm curious for devs who may be listening who build API products \[00:18:00\] is there something they can do or is there a way for them to integrate from their side to ensure that Super goods understanding of their API is correct? Do you consume anything along those lines to say that, like, don't let the LLMs decide, let, let you know, the product owners themselves configure it. \[00:18:14\] \*\*Alex Klarfeld:\*\* Yeah, actually, so part of, part of our, our pitch is we wanna also help out the API vendors. We're starting to roll that out right now, where basically we wanna help give the same experience that the plaids and the stripes of the world give to their customers to those API vendors. So. That integration's a little bit different. \[00:18:31\] We basically don't wanna get in in the way of any sort of like live traffic as well. So we'll actually hook into the API vendors logs themselves and sort of start, start using our tool. Like the first version of this product is basically like building out internal dashboards if they don't already exist. \[00:18:46\] And then surfacing that information up to the sales team or supports teams so that like. They can provide a better experience for their customers. Like some of the best API experiences I've had are from like, honestly like Century and the Datadog, where they're like, Hey, \[00:19:00\] your usage is about to exceed what you pay for. \[00:19:02\] Just like wanna give you a heads up. And that's just like not the norm. So we want to kind of help, help the API vendors themselves, make a better experience for the customers, like engage customers when we they notice they're having issues engage when like there's spikes in usage. So that's also like a pretty big focus of ours is like building a product for the vendors. \[00:19:19\] \*\*Mike Bifulco:\*\* Yeah, as a vendor, there's many things to get right there. Pricing is a tricky challenge because it needs to ensure the vendor can make money, but also that there's an on-ramp for users who start at a reasonable level and that they, they don't have to, you know, invest a ton to prove that the thing is useful and valuable at the same time. \[00:19:36\] I would imagine some of it is also, trying to be competitive with the competition and making sure that comparable services, you know don't under underprice or overprice you or whatever the optimization challenge is there. \[00:19:47\] To that end, maybe one of the things I'm curious about and I had kind of noted to ask you about is tell me a little bit about Super goods pricing. \[00:19:53\] \*\*Alex Klarfeld:\*\* Right. So we, we just released a free tier. You can download it for free. We basically limit to like, we, we won't actually \[00:20:00\] block you because I don't believe in that sort of sales tactic just as like an API consumer who, goes overage. We're not gonna, we're not gonna overcharge you for that. \[00:20:07\] We'll hopefully have some conversations at the start, but the free tier is basically a hundred thousand calls a month. You can, you can set up pricing for three vendors automatically and then sort of like track usage from the rest of the vendors for free. Just to, the idea is like, we don't wanna charge you for the vendors that like work and aren't really expensive. \[00:20:25\] Like you shouldn't have to pay for like tracking, like. You know, maybe Google APIs or like Stripes APIs. 'cause those just work. So the best customers are the ones who are like, yeah, we, we like, have a specific set of APIs that we wanna keep an eye on from price and like, quality and usage. And those are the ones that like, we also want to provide value for. \[00:20:45\] So that's, that's the way that like philosophically, like I'm trying, we're trying to price is, is get that for free. And then everything else is sort of like the enterprise part where you have more than a million calls. You have all sorts of vendors that you want to sort of keep an eye on. That's, that's sort of like dependent on the \[00:21:00\] the, the, the enterprise itself. \[00:21:03\] \*\*Mike Bifulco:\*\* Okay. That makes sense. I, I feel like there's a pretty big. Delta between three APIs and a hundred thousand calls and enterprise grade. The question I would have there as a potential consumer is when I get to 100,001 calls and four vendors like what is, what is the on-ramp? Like there, am I jumping from zero to, you know, thousands of dollars a month? \[00:21:23\] Is it like, what's the scale of, of initial costs, I guess for someone who's incrementally adding. \[00:21:28\] \*\*Alex Klarfeld:\*\* Yeah. And, and, and like our goal is like onboard slowly, especially like with growing companies, like you just don't know how many APIs you're going to use. And I, for one, don't wanna be someone who's like, oh, surprise, you added like another LLM, this is like adding to your bill. So we basically try to charge, we basically try to give th 30 days free to monitor. \[00:21:47\] Every single API at any volume, just to make sure that like, okay, is this an API that is like worth monitoring and worth tracking? And the idea is at the end of the 30 days, you tell us and say like, yeah, I wanna keep tabs on that, or I don't, and then we \[00:22:00\] basically discard it. Or like, there's, there's a way that you can set, set to ignore certain vendors inside of our UI to just like, turn 'em off right away so it won't stop working. \[00:22:08\] We'll just stop tracking if you choose to, to stop tracking that API. \[00:22:13\] \*\*Mike Bifulco:\*\* Sure. Got it. I'm gonna press a little further only because I feel \[00:22:16\] like it's my solemn duty to do so, but like what was truly, what's the scale there? Am I looking at hundreds, thousands? \[00:22:21\] \*\*Alex Klarfeld:\*\* Scale in terms of like the volume. \[00:22:23\] \*\*Mike Bifulco:\*\* No, sorry, in terms of cost, right? So, so like, when, when your customers convert from free to paid maybe for the smallest ones who are growing, what does that tend to look like for their, their incremental ad? \[00:22:33\] \*\*Alex Klarfeld:\*\* Yeah, for sure. So it's, if it's pushing a million calls upwards we basically start at like 99 a month and then 10 million calls upwards, like, sort of like custom pricing based on that. But again, if it's a million calls to, like, Google doesn't, like, probably something you don't need to track. \[00:22:50\] So it's like more of like million calls, like, like open ai a month kind of thing is like a hundred bucks a month. \[00:22:56\] \*\*Mike Bifulco:\*\* Right. Okay. Yeah, that makes sense. That, that to me is a \[00:23:00\] narrative that I think is easy to sell too. Like the the scary thing is if the product you're using to manage your spending becomes a giant spend to \[00:23:07\] manage the spending, then it's like, you know, an or a boroughs of challenges to, to upsell through the product stack there too. \[00:23:14\] \*\*Alex Klarfeld:\*\* Yeah. I very, I'm very cognizant. Sorry, go ahead. \[00:23:16\] \*\*Mike Bifulco:\*\* no, please, please. \[00:23:17\] \*\*Alex Klarfeld:\*\* No, I'm very cognizant of that, like all the ways that vendors have hurt me with pricing is like not something that I wish to incur on like my own users. So the idea is to be transparent and like not screw up their businesses as much as possible. \[00:23:31\] \*\*Mike Bifulco:\*\* Cool. Yeah, I love that. I think that's a, a good angle to take and especially like I alluded to before, I think developers are really good at sniffing out when they're being toyed around with, and, you know, they're the first ones to head for the hills if something feels fishy. \[00:23:43\] \*\*Alex Klarfeld:\*\* I don't really blame him. I've done the same. \[00:23:45\] \*\*Mike Bifulco:\*\* Right. Yeah. You are one of 'em. So it, it \[00:23:47\] makes sense. Yeah. Okay. So tell, tell me what's next? Like, what other problems are you starting to think about or maybe do you have releases coming that you're excited about? What, what are you working on now? \[00:23:56\] \*\*Alex Klarfeld:\*\* yeah. So you can download the Super Good Tracker for free \[00:24:00\] at Super Good AI current account. One of the things I hate is. Trying to te test out a dev tool with like talking to a salesperson. So you can do it for free without talking to any of us. We'll add you to a Slack channel just to help you get onboarded async. \[00:24:13\] But drop us in check us out, kind of go off to the races from there. And then the sort of like longer term is we want to help the API vendors themselves. So starting to talk to the vendors that we're already monitoring today to try to make a better experience for the customers. The current integration today is if, if we have a special partnership with these API vendors, you get essentially premium support through us. \[00:24:35\] So you're the first to know if these, these vendors have issues. It's super good. We'll know before the status page updates just because we already have. A bunch of the data going. So like if you're having an issue, one other customer's having an issue, we joke, it's like the, the age old problem of like debugging an API is like, is it us, is it them? \[00:24:53\] We try to answer that, that pretty quickly. And yeah, just trying to make the vendors have the best experience for their \[00:25:00\] customers. And then also trying to get into larger enterprises. So we actually have a, eBPF agent also in the works, which is targeted towards like larger enterprises. \[00:25:07\] I'm happy to go into detail of that. It's an interesting technology, but it's like a little bit more robust than making a code change. It's more of like an infrastructure change. \[00:25:15\] \*\*Mike Bifulco:\*\* Yeah. Interesting. So that's more, more gateway level, presumably, or like more broad level where maybe you have dozens and dozens of different software stacks through an organization and it's hard to just add three lines in one place and be happy. \[00:25:26\] \*\*Alex Klarfeld:\*\* Yeah, exactly. So trying to, trying to gear up for that. So it's a agent that sits next to your box and does the same thing as the, the logger. But does it on like the, the kernel level and then performs the same way. It's just a little bit more effort to debug. It's usually teams that have. DevOps teams already set up. \[00:25:41\] But we, we had, we started with the, the logger because that was the easiest way I thought that we could integrate too. \[00:25:48\] \*\*Mike Bifulco:\*\* Yeah, it makes a lot of sense. I think one of the interesting things about what you're working on it super good is that you get to. Interpret and traverse and feel parts of the software stack, the protocol that a lot of \[00:26:00\] developers don't think about. Like, you're, you're making some really interesting network requests. \[00:26:03\] You're sniffing out for certain shapes of things. Patching, fetch is like an idea that I've never even remotely considered, right? But it's a super cool thing to, to be able to do and to kind of monopolize isn't the right word, but utilize as, as an opportunity creator for you and certainly a value add for developers. Maybe this is a good, good segue into one of the other things I was interested in is like, tell me about your team, the size of the team building this thing and the tech stack. What does that look like right now? \[00:26:28\] \*\*Alex Klarfeld:\*\* Yeah, for sure. So we're a really small team. There's three of us. We're all engineers. The tech stack is pretty straightforward. I'm a big fan of TypeScript, so all the web application stuff is done in TypeScript. And a lot of like data processing is also done in TypeScript. We actually utilize neon. \[00:26:44\] So I'm a Postgres stalwart. Like I just think it's the best. Hopefully that doesn't lose me too many favors, but there's a really cool technology. Called Neon that helps, like, does like serverless Postgres. I'm a huge fan of it. It's greatly increased our developer productivity. Being able to like clone a \[00:27:00\] database immediately in the cloud behind your, your VPN just makes developments so much faster. \[00:27:05\] So we use Neon for, for Postgres and you can also like set up multi-tenants very easily with them. And we deploy on a combination like render for the front end stuff and then GCP for a lot of the backend stuff. And then. Just using like the core GGCP stuff like cloud run and things like that. \[00:27:21\] Pub sub and GGCS. \[00:27:24\] \*\*Mike Bifulco:\*\* What I really like about your product and a thing that I really like talking to founders is when you hear about a product that has such an interesting and simple implementation story and that a lot of it is just good fundamentals and a solid idea for how to help people and not not having to go spend 10 years like researching a brand new, you know, I don't know, algorithm. \[00:27:42\] Not having to do what open AI did and figure out how to make an LLMA thing to begin with. There are loads of opportunities for, you know, enterprising people to go and solve problems without having to I don't know, level up computer science as a theory. Right. And it's, it's a really cool sign that you're onto something that is an \[00:28:00\] interesting value add and also just fundamentally probably a helpful thing too. \[00:28:03\] That that's super cool. \[00:28:05\] Uh, and the team size of three is pretty impressive for what you've built too. \[00:28:08\] \*\*Alex Klarfeld:\*\* Thanks. Yeah, they're, they're top engineers. I'm a, I'm a big fan of them. \[00:28:12\] \*\*Mike Bifulco:\*\* Yeah. Cool. Well Alex be, before I let you move on, I've got a, a couple of questions that I wanna make sure I don't skip. \[00:28:18\] Um, where's the best place for developers to find super good online and to get started with it? \[00:28:23\] \*\*Alex Klarfeld:\*\* Yeah, for sure. So you can check us out@supergood.ai. \[00:28:27\] \*\*Mike Bifulco:\*\* Cool. We'll send them that way straight away. And maybe what's a sign that someone should be considering using? Super good. \[00:28:33\] \*\*Alex Klarfeld:\*\* If you've ever got a bill that someone has, I. Could been very confused by that's one if you have a horror story in mind where you're like, this API caused me to wake up in the middle of the night and, and heartache. That is, that is a good sign that we might be helpful. If you have more than, I'm gonna call it like five APIs, like if you're, the best customers that we have are like, kind of like they've, they've started to think about their unit economics. Maybe they haven't like implemented things yet, but if you're like a \[00:29:00\] smaller company, you probably are just like, Hey, I'm just trying to, trying to grow. So I don't care that much about like, watch these APIs. \[00:29:06\] We're just gonna chuck stuff at it. Which, which I've been in that place too. So typically like larger companies that are just starting to think about this problem. And also if you're dealing with international APIs we actually have a pretty. Large contingent of, of international vendors that we've been working with to monitor. \[00:29:22\] They are working pretty hard to get up to snuff with some of the, the US based vendors. So that's a good, that's a pretty good one. So anything expensive too? Is, is, is great. Like the Twilio stripes plaids. Oh. If you're working with any credit bureau also I'd love to talk to you. They, they have a special place in my heart 'cause they've been pretty difficult to work with. \[00:29:40\] \*\*Mike Bifulco:\*\* No doubt. Yeah. Wow. That makes it super easy. If, if you have just listened to Alex and you, you fall into any one of those buckets get in touch stat. Alex, then where's the best place to find you online? \[00:29:51\] \*\*Alex Klarfeld:\*\* You can find me on LinkedIn, you can find me on Twitter. I respond to dms on both. So please, please hit me up. You can find me on GitHub too if you wanna watch me write code, but that's a \[00:30:00\] little less exciting. \[00:30:02\] \*\*Mike Bifulco:\*\* Well, I will make sure that there's links in the show notes to everything we've discussed ways to get super good and to you and Twitter and LinkedIn and all the other places. Alex, I really appreciate you coming and hanging out and telling us a little bit about Super Good. Thanks so much for joining. \[00:30:14\] I really appreciate it. \[00:30:15\] \*\*Alex Klarfeld:\*\* Yeah. Thank you so much, Mike. I appreciate it too. \[00:30:18\] \*\*Mike Bifulco:\*\* Of course, Alex Feld. We'll catch you next time. Thank you. \[00:30:20\] \*\*Alex Klarfeld:\*\* Alright, see ya. \[00:30:22\] \*\*Mike Bifulco:\*\* Bye. \[00:30:24\] ​ ### Back To Basics URL: https://apisyouwonthate.com/newsletter/back-to-basics/ Last updated: 2024-08-01T11:43:26.000Z There is nothing better than a clear and descriptive OpenAPI spec and a well designed API. Sometimes it easy to forget that good APIs allow others to build brilliant applications. Some of the best UI interactions for end users is only possible when we support them with a great API. Phil takes us back to basics in this issue, with a post on supporting pagination. Phil gives us a breakdown of different approaches and their tradeoffs. Always remember: a good API leads to great UI. \-- Alexander ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2024/06/treblle-assistant.png) ## Alfred AI: The Ultimate API Assistant Alfred AI is a cutting-edge, AI-powered assistant designed to revolutionize API documentation, discoverability, and integration. Alfred automates manual workflows, for faster API integrations and a superior developer experience. [Discover Alfred](https://treblle.com/product/alfred?utm%5Fsource=newsletter&utm%5Fmedium=apis-hate-sponsorship&utm%5Fcampaign=alfred%5Fai) ## The API Round-Up ### [The perks of a good OpenAPI spec](https://blog.val.town/blog/openapi/?ref=apisyouwonthate.com) Val Town talks us through their transition from ExpressJS to Fastify and how it allowed them to generate their OpenAPI spec from their code. The fun doesn't stop there. Thanks to their new spec, they improved their documentation, generated a new TypeScript SDK and helped LLMs use their APIs. ### [From SwaggerHub to Bump.sh, unlocking the power of Git](https://bump.sh/blog/from-swaggerhub-to-bump-git?utm%5Fcampaign=swaggerhub-to-bump-git&utm%5Fmedium=guide&utm%5Fsource=twitter) Phil walks you through moving your OpenAPI descriptions out of Swagger Hub, freeing yourself from the walled garden. Phil talks about the benefits of Git-based control flow and some of the fantastic tools from Bump.sh. ### [Introducing the Arazzo specification](https://www.linkedin.com/events/7220085969037705218/comments/?ref=apisyouwonthate.com) OpenAPI is constantly growing and something new has arrived the Arazzo Specification. This new specification helps API providers describe complex, multistep sequences of API calls. Find out more in the latest video of OpenAPI Hangouts. ### [Generate an OpenAPI Spec with Pydantic](https://www.speakeasy.com/guides/api-frameworks/pydantic?utm%5Fsource=apisyouwonthate&utm%5Fcampaign=240801) Python's most popular validation library has a superpower that often flies under the radar: it can be used to generate OpenAPI Specs. Check out this article for documentation on Pydantic's configurable, straightforward approach. ### [Generating HTTP clients in Spring Boot ](https://maciejwalkowiak.com/blog/spring-boot-openapi-generate-client/?ref=apisyouwonthate.com) When I had to pick up Spring Boot and Java, finding up-to-date, modern tutorials was hard. Maciej gives an overview of some OpenAPI spec tools for Spring Boot and a walkthrough of how he generated HTTP clients from an OpenAPI spec. ### [Open Source API Firewall ](https://github.com/wallarm/api-firewall?ref=apisyouwonthate.com) I stumbled across a nifty little tool this week: an API Firewall with request and response validation based on OpenAPI and GraphQL schemas. It is designed to protect API endpoints in cloud-native environments. It is also written in GO and is definitely worth checking out. ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2024/08/apiywh_2.jpg) ## Speakeasy: Build APIs Your Users Love Your API deserves a great developer experience. Get one by using Speakeasy to generate idiomatic, type-safe SDKs from OpenAPI [Start Generating](https://www.speakeasy.com/?utm%5Fsource=apisyouwonthate&utm%5Fcampaign=240701) ## APIs You Won't Hate The latest from the team at API's You Won't Hate. ### [API Design Basics: Pagination](https://apisyouwonthate.com/blog/api-design-basics-pagination/) Phil talks us through implementing pagination in a REST API, covering the three most common approaches page, offset and cursor. Phil talks about the trade-offs of each and links to some great examples of each. ### [Building an API for Global News](https://apisyouwonthate.com/podcast/building-an-api-for-global-news/) Join Mike Bifulco for a chat with Artem Bugara, CEO and co-founder of Newscatcher API. Artem shares his experience building an API-first product that provides global-scale insights from news sources worldwide. --- ## From Our Community Articles written and shared in our free [Slack community](https://slack.apisyouwonthate.com/?ref=apisyouwonthate.com). ### [Fuzzing JSON to find API security flaws](https://danaepp.com/fuzzing-json-to-find-api-security-flaws?ref=apisyouwonthate.com) Another article from one of the most active members of our Slack group, Dana. In his latest post, Dana talks us through fuzzing JSON payloads to find security flaws in an API. Fuzzing can be used to expose how certain properties can be tainted to allow further data manipulation. Another great read. ## Support APIs You Won't Hate When you become an member, you'll get access to members-only content while directly supporting our work. Your support helps us to keep making resources for the API community. [Become a member today](https://apisyouwonthate.com/#/portal) Thanks so much to our [members](https://apisyouwonthate.com/membership/): Kin L, Juxt, Vedran C, Alex R, Nolan S, Frank, James D, Bill D, and Ali S. Your support means the world to us! Until next time, Alexander, Phil & Mike ### API Design Basics: Pagination URL: https://apisyouwonthate.com/blog/api-design-basics-pagination/ Last updated: 2024-08-05T12:54:25.000Z Pagination is a massively important concept in REST APIs, often forgotten about at first as the API is juts fine with a few hundred records, but starts crumbling like shortbread as soon as there are thousands of records. Pagination is basically breaking down a large dataset into smaller chunks, which can be fetched incrementally by a client if they really want all that data. This helps improve performance of your web server, and makes for a far better user experience as they are able to work with some of the data sooner, instead of waiting for every record ever. Introducing pagination to an API after launch can be really difficult, and it's usually a breaking change, so get in there first and figure out your pagination strategy before you deploy. To help you pick a pagination strategy, let's look at some examples and talk through the pros and cons. 1. Page-Based Pagination 2. Offset-Based Pagination 3. Cursor-Based Pagination ## Page-Based Pagination Page-based pagination uses `page` and `size` parameters to navigate through pages of data. ``` GET /items?page=2&size=10 ``` This request fetches the second page, with each page containing 10 items. There are two main ways to show pagination data in the response. ```json { "data": [ ... ], "page": 2, "size": 10, "total_pages": 100 } ``` This is pretty common but forces the client to know a whole lot about your pagination implementation. As always when you want to move logic to the server-side instead of forcing clients to read lots of docs, you can add links, also known as HATEOAS, or "Hypermedia Controls" for short. ``` { "data": [ ... ], "meta": { "page": 2, "size": 10, "total_pages": 100 }, "links": { "self": "/items?page=2&size=10", "next": "/items?page=3&size=10", "prev": "/items?page=1&size=10", "first": "/items?page=1&size=10", "last": "/items?page=100&size=10" } } ``` [Hypermedia controls](https://apisyouwonthate.com/blog/rest-and-richardson-maturity-model/) are often seen as controversial with people chosing to skip them and pretty much ignore one of the main benefits of what makes a REST API a REST API in the process, but interestingly pagination seems to be the one use case where everyone is happy with it. If there's a next link, you can show a next button. If the next link returns data, you can show the data. You could even remove the meta object entirely and let people use links alone. **Ease of Use** - Pro: Simple to implement and understand. - Pro: Easy for users to navigate through pages. - Pro: UI can show page numbers and know exactly how many pages there are. - Pro: Can optionally show a next/previous link as you know if there are more pages available. **Performance** - Con: Involves counting all records in the dataset which can be slow and hard to cache depending on how many variables are involved in the query. - Con: Becomes exponentially slower the more records you have. Hundreds are fine. Thousands are rough. Millions are horrendous. **Consistency** - Con: If you load the latest 10 records, then a new record is added to the database, then a user loads the second page, they'll see one of those records twice. This is because there is no such concept as a "page" in the database, just saying "grab me 10, now the next 10" does not differentiate which records they actually were. ## Offset-Based Pagination Offset-based pagination is a more straightforward approach. It uses `offset` and `limit` parameters to control the number of items returned and the starting point of the data, which avoids the concept of counting everything and dividing by the limit, and just focuses on using offsets to grab another chunk of data. ``` GET /items?offset=10&limit=10 ``` This request fetches the second page of items, assuming each page contains a maximum of 10 items, and does not worry itself with how many pages there are. This can help with infinite scrolls or automatically "importing" lots of data one chunk at a time. There are two main ways to show pagination data in the response. ```json { "data": [ ... ], "meta": { "total": 1000, "limit": 10, "offset": 10 } } ``` Or with hypermedia controls in the JSON: ``` { "data": [ ... ], "meta": { "total": 1000, "limit": 10, "offset": 10 }, "links": { "self": "/items?offset=10&limit=10", "next": "/items?offset=20&limit=10", "prev": "/items?offset=0&limit=10", "first": "/items?offset=0&limit=10", "last": "/items?offset=990&limit=10" } } ``` **Ease of Use** - Pro: Simple to implement and understand. - Pro: Easily integrates with SQL `LIMIT` and `OFFSET` clauses. - Pro: Like page-based pagination this approach can also show next/previous buttons dynamically when it's clear there are more records available. - Con: Does not hel pthe UI build a list of pages if they want to show "Page 1, 2, ... 20." They can awkwardly do maths on the total / limt but its a bit weird. **Performance** - Con: Can become inefficient with large datasets due to the need to scan through all previous records. - Con: Performance degradation is significant as the offset increases. **Consistency** - Con: The same problems exist for offset pagination as page pagination, if more data has been added you could see the same record returned twice in two requests. **See this in action** - [YouTube Data API](https://developers.google.com/youtube/v3/guides/implementation/pagination?ref=apisyouwonthate.com) - [Reddit API](https://www.reddit.com/dev/api/?ref=apisyouwonthate.com) ## Cursor-Based Pagination Cursor-based pagination uses an opaque string (often a unique identifier) to mark the starting point for the next set of items. It's often more efficient and reliable for large datasets. ``` GET /items?cursor=abc123&limit=10 ``` Here, `abc123` represents the last item's unique identifier from the previous page, this could be a UUID, but it can be more dynamic than that. APIs like Slack will base64 encode information with a field name and a value, so you can send it a order by field, and an ID, all wrapped up in an opaque string of `dXNlcjpXMDdRQ1JQQTQ=` to represent `user:W07QCRPA4`. This avoids API consumers hard-coding values so your pagination logic can change, and consumers can pass the cursor around to do the job without any worry about what it actually involves. **Ease of Use** - Pro: API consumers don't have to think about anything and you can change the logic easily. - Con: Slightly more complex to implement than offset-based pagination. - Con: API does not know if there are more records available after the last one in the dataset so has to show a next/previous link which may return no data.\* **Performance** - Pro: Generally more efficient than offset-based pagination depending on your data source. - Pro: Avoids the need to count records to perform any sort of maths which means larger data sets can be paginated without suffering exponential slowdown. **Consistency** - Pro: Cursor based pagination data remains consistent, even if new data is added or removed, because the cursor acts as a stable merker identifying a specific record in the dataset instead of "the 10th one" which might change between requests. It can look a bit like this: ```json { "data": [...], "next_cursor": "xyz789", "limit": 10 } ``` Or again if you want to save the client doing the heavy lifting you can leverage hypermedia controls: ```json { "data": [ ... ], "links": { "self": "/items?cursor=abc123&limit=10", "next": "/items?cursor=xyz789&limit=10", "prev": "/items?cursor=prevCursor&limit=10", "first": "/items?cursor=firstCursor&limit=10", "last": "/items?cursor=lastCursor&limit=10" } } ``` **Update from the Slack community:** I mentioned cursors cannot conditionally show next/previous links only when there is more data available there, but Adam Altman suggests its possible to mitigate this con. If the backend requests *limit+1 (for example, 11)*, it can check if there are 11 results received, it can conditionally include a next link in the API response. If not, it can avoid the next link. This is a clever approach. Yes its a bit of over fetching, but it could be benchmarked against general usage to see what is producing more efficient page transactions. **See it in action** - [Twitter API](https://developer.twitter.com/en/docs/twitter-api?ref=apisyouwonthate.com) - [Instagram Graph API](https://developers.facebook.com/docs/instagram-api/?ref=apisyouwonthate.com) - [Slack API](https://slack.engineering/evolving-api-pagination-at-slack/?ref=apisyouwonthate.com) Choosing the right pagination strategy depends on your specific use case and dataset size. Offset-based pagination is simple but may suffer from performance issues with large datasets. Cursor-based pagination offers better performance and consistency for large datasets but come with added complexity. Page-based pagination is user-friendly but shares similar performance concerns with offset-based pagination. ## Where should pagination go? In all of these examples there's been the choice between sending some metadata back for the client to construct their own pagination controls, or sending them links in JSON to avoid the faff. Using links is probably the best approach, but they don't have to go in the payload. Using [RFC 8299: Web Linking](https://www.rfc-editor.org/rfc/rfc8288?ref=apisyouwonthate.com) might be the better choice. ``` Link: ; rel="first", ; rel="next", ; rel="last" ``` Popping them into HTTP headers seems like the cleaner choice instead of littering your resources with metadata. As well as the feels it's also [better compressed as of HTTP/2 using HPAK](https://blog.cloudflare.com/hpack-the-silent-killer-feature-of-http-2?ref=apisyouwonthate.com), and as it's a standard it can be supported by [generic HTTP clients like Ketting](https://apisyouwonthate.com/blog/ketting-v5-hypermedia-controls/). Either way, pick the right pagination strategy for your dataset, document it well with a dedicated guide in your API documentation, and make sure it scales up with the dataset you're expecting to have instead of testing with a handful of records, because if you want to change pagination later it could be a whole mess of backwards compatibility breaks. ### Building an API for Global News URL: https://apisyouwonthate.com/podcast/building-an-api-for-global-news/ Last updated: 2024-07-31T10:30:44.000Z ## Show Notes - Artem Bugara - [twitter](https://twitter.com/bugaralife?lang=en&ref=apisyouwonthate.com) - [LinkedIn](https://www.linkedin.com/in/artem-bugara/?ref=apisyouwonthate.com) - [Newscatcher API](https://newscatcherapi.com/?ref=apisyouwonthate.com) - [Y Combinator Launch](https://www.ycombinator.com/companies/newscatcher?ref=apisyouwonthate.com) - [Case Studies](https://www.newscatcherapi.com/case-studies?ref=apisyouwonthate.com) - On GitHub [kotartemiy/newscatcher](https://github.com/kotartemiy/newscatcher?ref=apisyouwonthate.com) [NewsCatcher: News API for risk and data teams | Y CombinatorNews API for risk and data teams. Founded in 2021 by Artem Bugara and Maksym Sugonyaka, NewsCatcher has 18 employees based in Kyiv, Ukraine, 02000\. NewsCatcher is hiring for 2 roles in sales and engineering.![](https://bookface-static.ycombinator.com/assets/ycdc/favicon-c8a914eeeba9fe6f7a863b35608b55aeedd7c1ff409c97b9ecb96b7a6c278d70.ico)Y Combinator![](https://bookface-images.s3.amazonaws.com/logos/7f6456b49401a2624b2a802796335ab66f26dcf3.png?1652602947)](https://www.ycombinator.com/companies/newscatcher?ref=apisyouwonthate.com) #### Transcript \[00:00:00\] ****Mike Bifulco:** Hello and welcome back to APIs you won't hate. I, of course, am Mike Biko, one of the founders of APIs you Won't Hate, and your podcast host. I am sitting down today for a conversation with a new friend of mine to talk through what is a really interesting API product with kind of a fascinating and unique story from inception to product two, where they're today. \[00:00:20\] And I'm really excited to sit down and talk to Artem Bora from News Catcher. Artem is a, like me, a yc founder and unlike me, has a really, really interesting story of the inception of his company and how he's really built something interesting through many, many interesting little bumps along the way. \[00:00:39\] And I'm really excited to get to chat with you. Artem. Thank you so much for joining. I, I appreciate it. How are you doing today? \[00:00:44\] ****Artem Bugara (NewsCatcher, CEO):** Hey, Mike. Thanks for having me. Yeah, I'm doing, I'm doing great. Happy to, happy to share something I've learned so far as a first time founder. Yeah. \[00:00:54\] ****Mike Bifulco:** So let's, let's start I think with, with Tell the World what let's tell the world what News Catcher is. So \[00:01:00\] what's your elevator pitch for news catcher? \[00:01:02\] ****Artem Bugara (NewsCatcher, CEO):** It is finding my elevator pitch for new sketcher changes almost every week, but I think, I think the latest one, I think the latest one based on like all the latest interactions and changes in the in the world is basically new sketcher. Think our assumption is, I'm gonna start with a problem, right? So my assumption is that companies don't really leverage information that can be found in news. \[00:01:28\] Surely if you ask and, and, and the most important thing is like, I think they do, but they don't. But they actually don't. If you ask like. The company, they're gonna obviously say like, yeah, CEO management reads like Bloomberg or FT, or something else. And obviously they like really looking into how like Trump Biden debate is going. \[00:01:49\] Like who's gonna win? How the interest rate gonna change, what the corporate tax, of course that's, and, and that's like important for big, it's important like as, as a big microeconomic shift, like as, as a big thing \[00:02:00\] going on, or like the war and conflict. But in reality, like the, that's not what usually makes company any like. \[00:02:07\] Money usually in a day-to-Day business. There is like much more like direct and applicable things that like are important and a lot of those actually can be automated or like semi-automated. And so our idea is actually for all the, at least the largest companies, like each department working for like each industry in each region. \[00:02:32\] Usually can benefit a lot by either losing less money or gaining more money by like being able, able to leverage stuff that can be publicly and openly available, not just like a big kind of things. So that's where we that, and that's where we, that's where we are today. But in more simple, in more simple terms, I would say that new sketcher basically. \[00:02:54\] Provides a news, API, we aggregate publicly available news articles from over the, over, all over the \[00:03:00\] globe. And not just like big media, but local news industry specific news. And then what we do is we basically take this, we have over 1 billion or one and a half billion articles indexed in our database. \[00:03:12\] All like index and, and, and easily and being easy to search. And what we do is we kind of try to meet an enterprise customer. Deeply understand what they need and then say like, look, we have the data that you kind of, that has insights and that has stuff that is important for you, but more of that we are gonna help you really like integrate it. \[00:03:36\] We are gonna do like a bespoke integration. So a little bit like paler like approach. And I think that's our, that's our, that's kind of our thing. I think one. Over the last four years doing new Sketch, I think the most important insight we realized is that the biggest problem is not like, you know, getting data and like getting articles through the same shape and form from like over 70,000 \[00:04:00\] new sources. \[00:04:01\] It's rather to be able to actually integrate it into like workflow into day-to-day of, business of some of, of some specific department in a specific, in a specific company. So yeah, that's, that's kind of where we are where we are right now. \[00:04:18\] ****Mike Bifulco:** When you and I first chatted, I was really interested diving in on your website, looking at some of the information on there. The, the heading of your website is something like news intelligence at a global scale or something like that. And from the sounds of it, so you've got a database with hundreds of thousands, millions, billions articles 70,000 sources and all focused on aggregating and providing intelligence around news. \[00:04:39\] ****Artem Bugara (NewsCatcher, CEO):** Yeah. \[00:04:39\] ****Mike Bifulco:** One of the first questions that my mind ran into is like, how does that, how do we get there? How does that start? That is a, a massive undertaking of a project, and I'm really curious about like the story that got you there. So can you tell me a little bit about maybe your career leading up to News Catcher and the reason that, that this came into existence? \[00:04:57\] ****Artem Bugara (NewsCatcher, CEO):** Sure. So basically News sketcher \[00:05:00\] has two founders me and Maxim. Maxim is CTO, I'm the CEO, but originally we are both non CAS degree data geeks. Kind \[00:05:10\] ****Mike Bifulco:** I like it. \[00:05:12\] ****Artem Bugara (NewsCatcher, CEO):** yeah. So, and, and probably like, I'm gonna start with like, so I know Maxim already for like 15 ish years, since like schooling back in Ukraine in the next. \[00:05:21\] Then we actually studied together in one university in Ukraine. Then we got accepted to the same university in Strasburg. And like, so yeah, we got a lot of actually years been studying together. And then just for the few years, I think we've been like a little bit apart from like doing different master's degrees and having different our first jobs basically, but originally. \[00:05:42\] So we got into, so like I said, we had like no CS degree engineers, so we like, we had to do all of it ourselves. And it actually started one day in Strasburg, France, where Maxim came to me and he said, I. I just like learn about this, \[00:06:00\] like Titanic dataset, Titanic, like machine learning competition on kale. \[00:06:05\] Kale is a website for machine learning people to like compete with each other. And it was like, I think 10 maybe, I think, yeah, I think it was 10, kind of 10 or nine or eight years ago. And Maxim and I both kind of, we studied eh, we studied economics and I think economics was the last part we actually enjoyed. \[00:06:25\] We enjoyed the most like mathematics and statistics and like game theory and probability and all this kind of stuff. And so like these Kaggle and like data science sounded like a nice trajectory for us to, to keep working on. And so we, that's how we got into like programming by. Learning RI think we, I think we actually, \[00:06:47\] ****Mike Bifulco:** Wow. \[00:06:48\] ****Artem Bugara (NewsCatcher, CEO):** yeah, I think we actually like studied on data camp. \[00:06:51\] I think there was the website and I actually remember Maxim one day was like waking, I think he, he, at some point he took all the courses, all the courses and, \[00:07:00\] and, and I dunno how much actually he learned from them, but he was like really dedicated to like. Studied and doing something with that. So yeah, we, we've been basically slowly just getting into this. \[00:07:10\] And for our master's degree, I went to Toulu School of Economics which is pretty solid economic school where the, I think the dean or the head of it has like a Nobel Prize in economics and Jeanti Hall, and I started the econometrics. And that's actually like, you know, like a, like a, like a more like applicable applied. \[00:07:32\] Economic economics economic, like economic statistics mix. And Maxim actually started like something actually that was called, I guess, data science in Paris. But yeah we've been there and one thing that happened, I went to, after the first year at. Doing my masters, I found an internship in a French company. \[00:07:52\] So basically the CEO is a TSE to School of Economics alum in Dubai, a vendor. And, and basically I was like an, \[00:08:00\] an intern there and then I said like, look, I don't want to really go back to, to university. I know how many more years of education I need to actually to actually work. I, I always, I always thought about it to be fairly honest with you. \[00:08:11\] And so I started with my very first job where I was like, just like, I think I had a lot of freedom and I could like, try to improve a lot. And I think that's where like I understood that like, I really like coding and one day I, I had a, I had to do some web scraping job and that's how I said like, okay, it's finally time to like open Python. \[00:08:30\] And I think this, in one day I started doing something in Python and then I came back to r. And yeah, and since then I, I was like really passionate about Python and I was trying to do a lot of like side projects as well as Maxim and we chatted about it. We really, we really liked it and I. I guess at some point I realized one thing, I don't know anything about Python. \[00:08:59\] \[00:09:00\] I know like Python, but I don't even know what server is, how internet works. And so I had some side project where I decided to go from zero to one, like following like Y Cator startup school thing. And so basically I, I, I've launched like a, like I was like scraping some news actually, like comparing some coverage for, for the, some something in Ukraine. \[00:09:22\] I think the, the, the presidential election. But basically it was like pretty cool 'cause I, I had to learn everything and I think I still remember this evening where I realized that, well, database is actually like, you can. Get it running with one Linux call. Like, it's like Postgres apparently doesn't have to be like, it's not a big building with like data. \[00:09:44\] It, it is pretty much, it's pretty much easier than that. And I think that that was like, like really I was working a lot and like I was doing my side project a lot and for like two months I, I kind of learned a lot of stuff about like how things work, how servers work, what \[00:10:00\] AWS is like, how, how everything works. \[00:10:03\] And I really kind of enjoyed it. And since then I felt like. It is pretty cool. I should probably do something I should probably do something something of, of my own with that. And I guess by the end of 2019 I was just looking for another like side project technically. Interesting. I just found this like other new EPI and I and I thought like, look, it sounds like a good idea. \[00:10:24\] Even though I don't really have any problem with it. I don't know anyone who needs it, but I just thought like. It's a really cool technical challenge. And so that was like a side new sketch. It was like a side project in the, in the beginning. And so I guess if you, if if there's anyone listening to me like it's having a cool technical challenge is a, is a great reason for the side project and it's probably like the worst reason for like starting actually a company. \[00:10:52\] If you dunno, you dunno what you're doing for who. Like, and that's what a lot of people do. They kind of like assume, I think. I think they assume that. Something \[00:11:00\] is needed. They build it, they ship the code, no one uses it. They select yes, we need more features. They add more features. Well that, that's not, that's not how it works. \[00:11:08\] But anyway, one thing, but the few things from startup school that we, so yeah, and I, and I asked Maxim to join me and, and we started like remotely, it was beginning of 2020 there was some rumors about this big kind of flu stuff going on in China. And so we thought like, yeah, that, that's probably like the really. \[00:11:25\] Which for people listening us from 2000 55 there was a thing called co covid. And so yeah, we, it thought like, okay, like the whole, the whole plan is gonna shut down, but that sounds like just the right time to quit my job and start, start new sketcher, any end. So we took the basics of Y Combin, which is like launch fast. \[00:11:47\] Talk to your users, focus on everything. What's important. So we just said like, okay, we gonna give ourself two months to build some kind of MVP. We're gonna release it. And that's how it all, that's how it all \[00:12:00\] kind of started. In two months, we actually released our MVP. We actually, I wrote a blog post. \[00:12:05\] There was like a blog post. How we've done it mostly on like. Credits from startups called and AWS and mostly using just AWS Lambdas and Dynamo. So it was like, and, and AWS Elasticsearch. So it was a lot of like cloud slash serverless solutions. So yeah, we just, we just kind of, we just kind of launched and surprisingly it was a really nice, it was a really nice thing to do because that got us some kind of like, attention from the crowd. \[00:12:33\] I think I've done a lot of like passing on Reddit and hack news. To figure out like if people like it. So I think at this stage we kind of realized that yeah, like people like it. And we thought like, this is an, even though no one was really like, yeah, take my money and, and give me the access. \[00:12:49\] But it felt like, yeah, that there's like some buzz around it. Like people, people think it's cool. So we should probably, we should probably keep doing that. \[00:12:57\] ****Mike Bifulco:** Sure. Yeah. \[00:12:58\] You've taken a really \[00:13:00\] interesting path. So I, I would guess you may be the only person I've ever met who started coding by learning R and found your way into building a full on product. Like I think there's probably a lot of data scientists, data engineers, whatever they're called in 2024. \[00:13:12\] Who, who learned R or Python or something like it first. It's definitely a fully inverted way to, to get yourself to learning about servers and the web and all that from there. And I'm, I am really intrigued by your ability to drum up interest too, right? Like, that's always an interesting part of the story for me as well. \[00:13:28\] So you mentioned writing some blog posts and getting attention for your, your MVP and your early prototypes. It's the chicken and the egg problem, especially for something like what you've ended up building is really fascinating to me. And it sounds like you found the right, right ways to get people pointed to it. \[00:13:43\] I, I saw earlier I was googling your name and saw your post about your trick to drum up. I think it's something like a thousand plus stars on your GitHub repos. Right. Things like \[00:13:52\] ****Artem Bugara (NewsCatcher, CEO):** yeah. \[00:13:53\] ****Mike Bifulco:** you, you've got a marketing muscle somewhere along the lines, and I'm, I'm curious how you. Landed on that? \[00:13:58\] Like is that just a natural \[00:14:00\] thing for you? Is that something that you've gleaned from people you followed? How did that happen? \[00:14:04\] ****Artem Bugara (NewsCatcher, CEO):** Yeah. So I guess the, I, I'm trying to think. Yeah, so we have, I think I have in total like maybe 5K stars. I've done two like things, some really. Like it's, honestly, it's like a hundred lines of code in Python, but like done something useful, very connected, basically like the RSS feed of New York Times with just name New York Times and you type New York Times and you get all the latest news from our, it's RSS. \[00:14:25\] It's something really easy, but no one thought about like doing, doing this, but I guess, yeah. So this, this kind of marketing or like, let's call it like go to market through developers is actually not, I, I don't think it's, it's right now it's not. It, it's, it's one of the, honestly, it's not what you should do in the beginning. \[00:14:47\] What you should do is you should like, call people and like talk to them and like email them. But I probably started it 'cause I knew that like, you know, early on I. That's the only thing I actually, I, I, I, I think I thought I \[00:15:00\] understand how to write. Yeah. I was, I was writing some technical blog posts before, like in the last six years and stuff. \[00:15:05\] So I thought, like, I feel like I understand really well how to write and how to talk to developers, and that's actually someone who I consider myself to be. So that was just the thing that I thought I will do the best, not. Maybe I also thought back then that this is like the right approach. So we were just more into like, you know, we'll just try, try something and let's say it's something that we probably assume that we know the best. \[00:15:30\] So that's kind of why, I think that's why we decided to talk to like no, try to try do like I think it's, you can call it like the bottom up approach or like trying to talk to which, which actually ended up I think not being. I dunno. I think, you know, it's really hard to go and like write emails and talk to people and try to get some like decision makers to talk to you people with real problems. \[00:15:57\] And I think, and it didn't help honestly, the fact that \[00:16:00\] we spoke just to developers. 'cause usually I. A lot of them, they themselves don't know what they do or what they need is like, you know, not really something critical. They just wanna play around with something. So I, I guess for some industry, maybe this works, but for our, that, that's, I don't think that's like kind of the, the right, the right approach. \[00:16:20\] And I think like we lost a lot of time trying to talk to wrong people, basically. \[00:16:25\] ****Mike Bifulco:** Yeah. \[00:16:25\] that is. Also an interesting angle too, and you, you've said this in a way that a lot of founders who have been through the process and have, have found their way into a product, say it, that in hindsight, there's definitely other things you could have done, but really the fundamentals that we hear a lot of, go talk to as many people as you can who you think might be your customers before you build something. \[00:16:44\] Validate the idea, then spend time building it, and then, you know, iterate, keep talking to them over and over and things like that. I'm really interested too, that you came to this through YC Startup School and then from, from our sort of prior chats, I, I know you had an interesting way of even getting into your Y Combinator \[00:17:00\] batch too. \[00:17:00\] Can you tell me a little bit about that? \[00:17:01\] ****Artem Bugara (NewsCatcher, CEO):** Yeah, so we applied to yc, I think first time We applied as soon as we started, and we applied in total of four times with new sketcher. And we got invited to interview twice with new sketcher, and we actually got accepted, got invited to YC twice. With new sketcher in summer 2021, we had to decline like going to Y Combinator because we already agreed to go to Tiny Seed, a small kind of program for Bootstrappers. \[00:17:31\] So we took some money from them and took this like year to basically just like. Do our small bootstrap thing as, as we thought, like it doesn't have a big enough kind of market. And then basically we reapplied in 2022 for summer 2022\. We got into interview again. It was with Michael sbo from 2021 to 2022. \[00:17:55\] We grew from I think three. So we started, when we started we had like zero MRR. \[00:18:00\] Then when we applied in six months, in 2020, I think we had like, like few hundred bucks, maybe a seven bucks. MRR then we first accepted to, so it was our third application in 2021\. We were at like three KMRR, but, and then, you know, they kind of like accepted us and we said no. \[00:18:18\] I said, like, we explained it was all fine. I think big kudos to Brett Flora, who, who were the partner there. And we reapplied one year later and we kind of was like, kind of really, like, I spent on a application, probably like a day, really, like at this point I didn't like spend that much time there. But I think we, we got really like, you know, nervous because like right now we already got some money, so we had to show actually some growth. \[00:18:41\] So we grew from three K to 20 K. And I guess, I don't know what, but like why C why C accepted us again and I was really happy we are like this Michael. 'cause I think. I was, I'm, I'm a big fan of I'm a big fan of his and I think few things for people who kind of wanna apply to yc. I know there's, I actually think I know why we got accepted, so I, I have some \[00:19:00\] like, kind of as assumption about why we got accepted. So a, a, a few things here. Like, first of all, we got really consistent, so we applied. First time we said we're gonna work on it. We applied Second time we said like, yeah, we actually worked on it. \[00:19:13\] We applied this short time. We said like, yeah, you know, like, you didn't get back to us first two times, but look, we get some progress and so on and so on. I guess it's really important to show kind of, you know, the consistency that you, that you have. I guess a lot of people think like. That's what YC kind of actually assume. \[00:19:30\] Like, okay, you have some idea, you say you want to commit to it, but like if we don't get, if we don't accept you, are you actually gonna work on it? 'cause like, usually I think you should, you should work on it. Whether you get to YC or not, YC shouldn't be. Or that's not the only way to start like a startup. \[00:19:47\] And I guess, you know, I, I'm, I'm, I'm, last time there was like applied to YC on high canoes. There was like a lot of crowds saying how YC is. Bad or how a lot of companies do like millions \[00:20:00\] of a RR without funding, which is again, true. I, I don't say anything about it, but YC is trying to invest into people who like, say, like we are gonna build a billion dollar companies, or like trillion dollar companies, and obviously. \[00:20:14\] That is a really different path from like, you know what you can do bootstrapping and you know, if you are solo founder and doing million dollars a year and spending on it, probably like three K bucks in a s bill. I mean, honestly that's, that's, that's absolutely fantastic. You just like cashing out all the money. \[00:20:31\] But it's not a billion dollar company still, even though you can get paid more than founders that actually started a billion dollar companies. And one thing that stood out to me on the hack news, there was like, I think someone saying like, how. People in, in yc they like, you know, they need to go to Harvard or MIT or this or that. \[00:20:50\] And, and I thought like, you know, it's, it, it's, it's probably true. It's true because the best kind of kids get accepted to M-I-T-M-I-T doesn't \[00:21:00\] really give them that much. Harvard, it's actually like they have such a high bar that obviously doesn't market for like kind of people, but YCI think has always been and still is a place for outsiders. \[00:21:10\] And I guess the biggest kind of. The biggest biggest companies are built from outsiders, and I guess that's something you can probably tell, you can probably like say about us. Maxim and I are like first time founders. We never studied computer science. We got into coding. We started new schedule by quitting our jobs. \[00:21:30\] We come from, we like kind of, we come from a country, from a region where the war started in 2000\. 14\. So we are like a typical exam and like we don't have any like cool kind of degree. The, the places we work at were like SMBs. We never worked at like Microsoft, apple. We had not like no good school, no good university, no good, like place of work. \[00:21:52\] Like none of this is really like a requirement. So it's rather being consistent. And I think the last thing I'm gonna say about YC \[00:22:00\] and people get who try to get accepted to yc. I recommend a lot of people to YC and then I, they get to the interview and they, they get rejected and I ask them why, and they tell me like, yeah, they told us about because of this or that, and I'm looking at them and say like, look, don't worry. \[00:22:13\] Like I don't have this figure out yet either. So that, that's like, that's one of the reason. But I guess the biggest one is like YC in people. They kind of, if you are like a great founder with like the, honestly, what you're gonna think is like the stupidest. Idea that never gonna work. They still gonna fund you. \[00:22:28\] 'cause they know that like you are smart enough to like validate it, understand that it doesn't work. Change your mind and build something great. So YC really invest in people, not ideas or anything like that. \[00:22:41\] ****Mike Bifulco:** I concur and it's a hard thing to convey. I. You will see a lot of things online about people applying to YC and getting no message back, but getting rejected, not hearing back, getting an interview, and getting rejected. And I think the, the through line in your story, and from what I've seen and observed with my company and with others that have been through the program, is that \[00:23:00\] not only is consistency important and people being invested in the problem they're looking to solve, but showing some, penchant to listen to feedback and make change based on your business. So when they say they're, they're investing in founders, it is really like the type of person who wants to build a company that the company will be successful and not the type of person who has an idea. And it has to be that idea. \[00:23:20\] Many YC companies have been through pretty wild, like 180 degree pivots that have moved them from, you know, one, one area of the world to something completely different. And, it's a subtle, weird thing. I like you. I'm also not an MIT or Stanford or Harvard or, you know, wherever Carnegie Mellon grad I went to a humble public state school here in the US and have stumbled through a lucky career at since then. \[00:23:40\] But I, I, I think a lot of the folks who I met, especially going through, the summer program at yc at the very beginning, last summer, we kicked off with a an outing, a week long weekend where all of the YC founders for the batch went. And one of the things that was consistently true with every founder that I met was imposter syndrome. \[00:23:55\] Everyone thought they didn't belong there for one reason or another. I'm too young, I'm too old. I haven't finished \[00:24:00\] college. I don't know if my idea is good. I'm not smart enough. All these things. That's, that's maybe hopefully a happy thing to hear for a lot of people. Like nobody really thinks they can do these things. \[00:24:09\] And it's an important thing to realize that we're all fallible in some way and we're all you know, just, just trying our best. It's not like there's some secret handshake that. \[00:24:17\] makes. YC founders better or different or whatever than everyone else? Yeah. \[00:24:21\] I don't know. Maybe I should get off my high horse there too. \[00:24:23\] There's definitely a lot of privilege wrapped up in what I'm saying, so yeah. Okay. So Artem you went through the YC program, that was your, your, you were summer of 2021 was your batch, so that was a few years ago now. Tell me about where news catcher is now. \[00:24:36\] ****Artem Bugara (NewsCatcher, CEO):** Yeah. So we in summer 2022, we went to yc. \[00:24:39\] ****Mike Bifulco:** 22\. Sorry. \[00:24:40\] ****Artem Bugara (NewsCatcher, CEO):** yeah. So two years, yeah, two years ago. So so yeah, when we went to yc, we didn't, we had a lot of. Different types of customers, basically, like people doing probably like the, like a lot of platforms who need an integrated news or something like that. \[00:24:55\] And I guess when I first meeting with Michael Sebel, he was like, okay guys, I like doing like, what is this? Like, is \[00:25:00\] it like for enterprises? And we said like, yeah, it's probably should be for enterprises. And he's like, okay, so what is your, like what are we trying to, like how many like calls and sales calls do you have? \[00:25:08\] And I'm like, well, we, we don't. And he's like. Yeah, so you should probably go and find four and, and get four pilots before YCR. So you have pilots with real companies, with enterprises. And I kind of, I thought I, I did actually that, but then I realized that I have practically like four people interested enough, but not actually like, signed anything or like, not even like agreeing on the price. \[00:25:33\] Well, long story short, it ended up really well we, we closed a few big contracts, I actually like, got into full y like I I, I flew to one guy to his like villa in Italy for like one day. And it was actually ended up being a lot of partying and then like next day, fortunately we, we got all him and I well, we both like woke up actually early, so we actually have like 20 minutes to discuss the business. \[00:25:58\] We did some kind of like \[00:26:00\] handshake agreement that they're gonna become our customers and yeah, by demo day we basically like. Had like, nothing much changed, but we just had more customers and like more kind of idea that, that it should be working. So by our demo day, we already had like 400 450 KARR. \[00:26:18\] Which, which isn't actually honestly that, that good. If anyone listen to me. It's really good if you have that after you, like just started yc. But if your company already exists for quite a while, that's also like, I don't know if, if that's a good result or not. But then basically we got we got great investors. \[00:26:34\] Like we, I spent fundraising like two days actually, that's one of the magic of yc. And yeah, we got, we got, we, we got a small round that we just needed really fast and get back to work. And I think we kind of, I'd say, didn't do that. I think what we should have done, we, we didn't, we didn't do that. And so the 2023 was somewhat \[00:27:00\] like us being like adding more customers, but struggling to find like, where is the market, where is the product? \[00:27:08\] Product, what is the problem? Where is the kind of feed? And I think only now. So we getting kind of hell of a year in 2024\. I think. Partially because like largely, which models got like much more. A lot of people right now understand that like how easy it is to work with tax and now they need great sources of data. \[00:27:29\] And one thing about us, we got really lucky 'cause we always doubled down on data quality while a lot of competitors like did on like NLP and stuff. And like with large language models, basically all UNLP is like now, like everyone is at the same level. You need to start from zero. And it got really easy. \[00:27:46\] And I think we recently, like we are hopefully like I. Soon gonna end up on like, having almost already a hundred percent growth since the beginning of \[00:28:00\] 2024\. And I think we, we, at the stage where I realized that what we should do is basically we should do like plenty of like deployments, more like onboardings. \[00:28:09\] We have great data. We know that in this data, there is a lot of value for a lot of enterprise customers. We know that no one does those like bespoke kind of integration, well, especially big companies that can't afford it for the same price as we do. Even though we charge like six digits, like, you know, Microsoft engineer with like a million dollar salary. \[00:28:27\] Like, you never gonna be able to like, you know, to, to afford the same stuff. And so basically what we do is like, we take our data we take a lot of like different LLM based like models. We take our customer's data, which is usually the, the trick is like, it's usually like different from, for each customer. \[00:28:44\] And we say like, look, based on the data you have and the data that we have, we're gonna try to build like a, like a connection. Then we will try to make sure that like our data, like it was really work like. In the most like excellent way possible. And so you actually can \[00:29:00\] have, can have something that works really great and works really great for you. \[00:29:04\] And the thing is like, and it's, and I don't see other way, like some of our competitors trying to build like kind of NLP and Reach news API and. What we hear from our customers, like, yeah, we try, we try this, but it doesn't work because like, yeah, it has a lot of different tools, but it's, it's never enough to like perfectly feed. \[00:29:23\] And so we said like, like, look, just try to believe us. We have the more accurate, like the coolest data that you need. And we actually gonna spend a little bit of time making sure that like we like. To do the integration initially and then it's gonna work. And I guess that's our, that's our plan right now. \[00:29:40\] And one of the beauty of this is like, it actually works really well because like, one, we can charge a lot for that. And like we can, we, you know, we can have, I. Like this onboarding, we can actually spend a lot of time and resources to make sure that it works, because again, we need to do it once and then it's like, you know, it, it, it, it just working, doing the same \[00:30:00\] thing with obviously doing some kind of like supporting and maintenance in between, but it works. \[00:30:05\] And two, that's like, I think like the only, like the big superpower that we have is that we can invest all those like. Custom thing and like a lot of like our time to actually make those connections. And that's one of the things that like the big companies can do. And that's one of the very few, that's what YC says. \[00:30:27\] Like you need to do. Like really, you need to make your customers really, really, really happy because you can and no one else can. And one of the best thing is like. It makes much more sense when you do like big checks because if you build in something that like, technically you'll have to scale to like, let's say like million of users you know, custom, you still need to like do a lot of like talk to your customers e each and every one. \[00:30:50\] But it's, it's really hard economically to do for really long when they pay you five bucks or like 50 bucks or 500 bucks and it's much easier \[00:31:00\] when they pay you half. You can, you can extend this like kind of, you can extend this period of time for much, much longer if you, if your customers pay you much more and like continuously learn and any end, we are at this stage where we're like trying to build a lot of those different connectors. \[00:31:15\] For a lot of different use cases, but at some point we, like, it's know every additional customer get like, it, it, it becomes easier and easier for us to do so. So hopefully at some point, which is actually like I, I do this, I just recently started to learn about like Palantir and that's what, like Palantir have been doing this for like 10 plus years and only recently they started doing actually products that kind of like somewhat scalable. \[00:31:38\] So I think Palantir is like a great, great example of like how. Given those bespoke integration, actually can, can still make you like, I don't know, like a, a multi-billion dollar company. \[00:31:53\] ****Mike Bifulco:** It is a the economic match there is really interesting. Like the Palantir is a, is a quiet giant I think \[00:32:00\] for a lot of people. Like I, I, I think possibly the audience of this podcast is, is more familiar with Palantir than let's say 95% of the world. But the way that businesses like that grow and become successful has a lot to do with their early plans and like the targets that they, they try and land and. \[00:32:15\] Getting, getting your business to a state where you know, you're growing by leaps and bounds is also something that requires that level of strategy. And as what I would imagine is, is a, a scrappy little company in comparison to Palantir. There's different ways to do that. Definitely a different ends to those means. \[00:32:28\] But it's certainly possible too. And, and I think it's a fantastic story and a really cool thing to see. Like, just, just how you've grown from like you know, a project in your flat and, and suffering through learning r and moving to Dubai and all these things. To something that's really a, a growing and inertia gaining project over the years and especially now, I would imagine being in a season where news is going to be more and more important. \[00:32:50\] And LLM certainly have gained an impossible amount of popularity. It's probably an interesting year for you is, is so. I should couch this in the fact that you and I are recording this the \[00:33:00\] morning after the very first US presidential debate which I could not watch last night. I just didn't have it in me to turn it on for, for a million different reasons. \[00:33:07\] But do you, do you imagine that the sort of political the, the election season in the United States will ratchet up user base for you? Do you think that'll have an impact on your end users? \[00:33:17\] ****Artem Bugara (NewsCatcher, CEO):** I think not really. And the reason for that is like, it's exactly what I was talking about. Like our customers, like, I think such thing as like. Like, you know, the presidential debate is, is usually something that like stakeholders, like chiefs should just read and understand. There's like nothing that much that you should do. \[00:33:39\] However, if. A company that owes your company money, has some small news that their branch getting closed, that they like, kind of had a fire or something, or that they actually like a five news local news this year that they are changing. The c or that's what you kind of, that's what is like a, like a, \[00:34:00\] like a stuff that people really don't think about, but then they, but one, but one day it like hits back on them and. \[00:34:06\] And then they come and see like, okay, look, every time someone doesn't pay us, we actually, so where one bank says like, they, they don't pay us. And we go and check like why this happened, how could we know this \[00:34:16\] ****Mike Bifulco:** Hmm. \[00:34:17\] ****Artem Bugara (NewsCatcher, CEO):** And they go, we go and check the news. And we realize that like, okay, probably if we were reading like the local newspapers, we would knew that months ahead that probably they not doing right. \[00:34:26\] And like, and sometimes, sometimes they're like, and we even give them more money when they ask, but like financial reports were like, fine, but. Financial reports are not something you get on like every day and like, you know, probably if, like, yeah, if, if if company changes management like really frequently, maybe there is like, something wrong with it or, or like, all of those facts. \[00:34:48\] So like all of those things like, it, it can like, you know. It's not probably like going to be like a billion dollar change for each company, but there is a lot of those small, \[00:35:00\] small little things that can be, that can be captured and like you honestly like missing those or. Directly, or like indirectly can, can cost like tens of millions of dollars for, for per department of big companies. \[00:35:15\] And that's kind of where we're trying to, where we're trying, but obviously like there's always gonna be like people trying to build like news, website coverage, comparisons, sentiment or, or Trump and Biden. And it never, and it never kind of like, it never monetizes into anything I feel like. But yeah, so we are like trying to do a little bit more like. \[00:35:33\] Not mainstream stuff. And that's why I say like, you know, some companies think that they follow news while invalidate, they follow like the biggest news that everyone follows, but they don't really follow news that can affect their business in a day to day. \[00:35:48\] ****Mike Bifulco:** Of course. And for you, having, having this strategy must have a leveling effect where seasonality is not something that you have to deal with quite as much. And from, from a founder's perspective, it's nice to have predictability and growth \[00:36:00\] and stability maybe in, in ups and \[00:36:02\] ****Artem Bugara (NewsCatcher, CEO):** Yeah. Yeah. And that's, that's, that's another really nice thing about like enterprises and like doing deep integration. 'cause like, honestly, if we succeed and if it works, I don't know what has to happen for, for them to like go to someone else or like to do something else. 'cause like, even like, because then even just to change it to someone, it is like already gonna be like a. \[00:36:24\] Like maybe a half a million investment for company, like in terms of like salaries and change and all. They, they will like customers, if you do something that fits like a glove and it works for them, customers, like they're gonna stick with you. Like your LTV is gonna be like huge because like, unless they, unless it doesn't bring them real value or unless they kind of run out of business, run out of money, you are kind of like you are married forever. \[00:36:49\] ****Mike Bifulco:** Yeah, no doubt. Yeah. Okay. So let's, let's jump into the nitty gritty of it. Tell me about the, your, your tech stack. Like the API developers are, are thirsty for nerdy \[00:37:00\] details at this point. So tell me what how you've built news catcher in the the bits and pieces you've tied together to make it all work. \[00:37:06\] ****Artem Bugara (NewsCatcher, CEO):** Okay. That's funny. So long story short, we have everything is like. Deployed in the, on-prem on Kubernetes. Mostly. One thing I wanna say is like, it's not something really you should worry about is, especially when you're like a small startup, you, you have to start as fast as you can. And that's why we started with AWS and Lambda function. \[00:37:32\] But then one thing we found out is like, if we get, like, if for, for whatever reason, you know, we are like a small customer to or GCP or anyone else, if we get. Kicked out of them. Not only are we gonna have a problem that like we just need to find other servers, but also gonna get bounded to like, like proprietary like services. \[00:37:54\] And it's sometimes really hard to like change those. So we said like, look, we're gonna do everything open source, everything's gonna be \[00:38:00\] infrastructure as a code. And you know, if something happens or like we need to scale, we can just like deploy the same kind of stuff on other clusters, on other, like on other hardware. \[00:38:09\] And that's what we do. So it's all Kubernetes and like a lot of dockers a lot of different microservices, I assume that like, are connected together. And our main stack is like Python MQ for like event processing. Elasticsearch for indexing the data. And to be fairly honest, there's probably like 50 other different technologies, but they like. \[00:38:29\] Small \[00:38:30\] ****Mike Bifulco:** That makes sense. \[00:38:31\] ****Artem Bugara (NewsCatcher, CEO):** Like fast API and, and we do, you know, like a lot of right now, like basic LLM, we've like fine tuned, we fine tune like mis trial seven B for example. Actually one, one thing I would, I would, okay, we know their customers. I think we're using their open source service, but I think they're really great company. \[00:38:48\] Open, open pipe by. It's a great company for help you fine tune \[00:38:56\] ****Mike Bifulco:** cool. Open \[00:38:57\] ****Artem Bugara (NewsCatcher, CEO):** yeah. Open pipe. Yeah. Yeah. I think \[00:39:00\] the guy, the founder, he's actually an ex y Combinator. He's an ex XY Combinator software engineer. Yeah. Oh, he has a, basically a great lending. He has a great lending page. It, it has some great ROI numbers, 25 times cheaper than Gvd four. \[00:39:14\] 5 million to start and $17 million saved by our customers this year. That's a great, great, great. \[00:39:20\] ****Mike Bifulco:** That's a solid pitch. \[00:39:22\] ****Artem Bugara (NewsCatcher, CEO):** So to be That's what I like. I, I like people like trying to understand the value. Yeah. So we use an open pipe, for example, for, for fine tuning some stuff and yeah, it's, it's all kind of open source. No, no kind of proprietary not much proprietary stuff. \[00:39:39\] ****Mike Bifulco:** Okay. And tell me about news catcher's story from a open source perspective. Have you, you made any of your software available for people to peek at? \[00:39:48\] ****Artem Bugara (NewsCatcher, CEO):** Yeah, we don't really have other software available just because it's not really kind of possible. But we have two things. I have a few open source project. One is actually called New Sketcher, the one that we done while we are just like doing our \[00:40:00\] MVP, which kind of gives you, and I don't maintain either of them by the way. \[00:40:04\] So News Sketch and I one py, Google News, like is basically like a, like RSS feed documentation of Google News, which I had to like hack around myself 'cause there was like no documentation. So basically Google News has r ss feed and it doesn't have documentation. There's like a lot of parameters that I had to like, kind of deduct and try to like, just hard, like try one after another one to see what is actually exists, what works, what doesn't work. \[00:40:28\] And another thing, we actually have free access for researchers. We have a bunch of research papers. I think the most interesting is one of the recent one by Berkeley where they see how well LLM can forecast the future using like news. And they use our, our API there. So that's like an interesting one. \[00:40:49\] So yeah, if you are like a researcher, if you're a student and you wanna mess around with like news and that's going to be your research, just go to us. We have free access for \[00:41:00\] researchers and we are really proud that we, that we do that and like, yeah, we, for example, also help to big nonprofit organizations, NGOs, like we should, I dunno, like, so we sign that we have like, we're gonna help Transparency International, catching all the news coverage about corruption and briberies and stuff like that. \[00:41:22\] So there's like a lot of cool stuff we do for free. \[00:41:25\] ****Mike Bifulco:** I love that. I think it's really an important thing to have some access to folks who can benefit from your product, who never could afford it, and that it speaks a lot to what you've built and certainly the value that you've provided for your customers that you can, you know, in some way subsidize people who are doing research and things like that. \[00:41:40\] Artem, I'll make sure to, to reach out to you and grab links to the research papers that you mentioned. 'cause I think that'd be something that'd be one fascinating to read through and I'm sure listeners would like to get to. But I think it may also give some indication for some of the folks who listen the sorts of projects that might be interesting for people to pursue if they're looking at news catcher or if they might be a good fit as a user of yours. \[00:41:59\] \[00:42:00\] I'm curious then. For the folks in the audience listening, how would they know that news catcher might be something that they should explore as a use case for their company? \[00:42:08\] ****Artem Bugara (NewsCatcher, CEO):** Gosh, that's a, that's a great, that's a great thing to, that we actually don't know answer yet really well. But, so globally, what I say is like is, is every industry has really different things like, some industries have to like do adverse media checks, for example. You know, like you open a bank account, they probably need to run like an adverse, like check that. \[00:42:29\] Like there's no news about you doing like, I don't know what, but, and there are some companies that just like can benefit and sometimes they just like don't know what they like, it, it just like, they use too much manual time for that. So what usually I try to say to everyone is like, look, if there is, like if you can. \[00:42:48\] If there are any news that are happening either locally or globally or in other countries, whatever it is on your industry that you, you know, like your business can directly benefit \[00:43:00\] from. For example, I don't know, like we have companies who like sales intelligence, they say like, like, look, we just need to know where, when our customers raise more money, open new business or something like that, and they say, we do it manually. \[00:43:13\] We lose a lot of time on this. As long as you can like articulate clearly what kind of news you have and like, as long as you can say like what kind of answers you need from each news article, we basically gonna make sure that you kind of receive those at the right kind of, at the right time, at the right place. \[00:43:31\] And a lot of people just like don't know how. A lot of stuff is possible. Like even one, one of my latest discoveries, like, like financial like, or like where people invest money. We have customer right now with like private equity in real estate and they wanna know if there's like any. Transport, public transportation infrastructure expansion going like, if, you know, like, if there's like a bus station opening somewhere, like the prices of the multifamily apartments gonna go up there. \[00:43:56\] So they want to know that. And there's like hundreds of those news, like every day, \[00:44:00\] for example, or like a financial institution. They not always invest in Apple and Microsoft or Palantir. Sometimes they buy bonds from like. Hospitals and universities and schools, and they kind of need to know local news about the schools, like if everything is happening there, if they're gonna fail on this bond. \[00:44:20\] And there's like a lot of those stuff where kind of, I guess people know that like there are news and they just like. Don't know how to scale it or they just never thought that they could do. And so we kind of try to help those that like figure out that there was like something in the news that is like important to their business and they come to us and we try to like make sure that like we integrate it like into their existing workforce, into the systems perfectly. \[00:44:44\] ****Mike Bifulco:** . \[00:44:44\] The summary I'm hearing perhaps is that if you, if your business or your strategy is swayed by. Signals in the news and, and you want to trigger some sort of reporting or workflows or decision making or whatever it may be. This is a good use case for plugging in the news catcher and trying to \[00:45:00\] listen to the, the broad swath of places where news can come \[00:45:03\] ****Artem Bugara (NewsCatcher, CEO):** Yeah, and I, and I would say, I would say like the only thing is like, it doesn't have to be some, like we work with some like Foresight Trends team. It can be that, but it's usually like much, much more. Specific, but which is still for some kind of, you know, companies is like, you know, like a hundred million dollars line of business that gets like affected by small news every day and they just like do, don't do anything about that. \[00:45:24\] And so just by changing this, just like a slightest beat can, can save you millions of dollars every year. \[00:45:30\] ****Mike Bifulco:** Well, Artem thank you so much for joining. Before I let you go, I have a couple of important questions for you. First of all, are you hiring, are you guys looking for anyone at the moment? \[00:45:38\] ****Artem Bugara (NewsCatcher, CEO):** Yeah, we kind of always hiring right now. So we are looking for, I think right now we're gonna look for two positions. We just hired a first technical like kind of account executives and we're gonna probably look for one more, so I. What I'm trying to do, like we do, like I said, a lot of like bespoke integrations. \[00:45:54\] So you need to be really, really technical. You need to, you're not gonna have a sales engineer \[00:46:00\] with you. You are actually gonna have to understand like how things work. You need to understand like customer problems and try to like come up with some smart solutions and things. So we are looking for technical account executive, AKA, the closer, the guy who, who actually does the sale. \[00:46:17\] And then we probably gonna look for very first, like LLM AI engineer slash data scientist, I dunno how you call them today. We do a lot of like, so basically we do a bunch of like fine tuning and stuff. We do really basic things. Surprisingly they work and probably right now it's the time for us to take it more seriously. \[00:46:35\] So, I don't know if you are passionate about AI and LLMs, we know, I cannot promise you like a million dollar Nvidia cluster. , actually, I can promise we're not gonna have that, but we can deliver something like zero to one with just. Little resources, but , measure the output, and just, you know, we have a lot of, you know, one thing I learned is like working with \[00:47:00\] enterprises, really few of them actually need this, like a hundred millisecond latency stuff. \[00:47:04\] Some of them can wait day or like 15 minutes. You can do a lot of right now with LMS, if you have like 15 minutes windows. So yeah, we are looking for like, for a great like AI engineer. We don't have those posts together. Probably by the time this podcast is live. You can, you can kind of apply. \[00:47:21\] Yeah. We are looking, we are looking for, and yeah, we mostly, I'm sorry for folks in the US who I think like the majority of the crowd, but we must likely need someone to work around the European time zone as we are all in, in Europe. Yeah. \[00:47:35\] ****Mike Bifulco:** . \[00:47:35\] Amazing. Artem, of course. , I'll throw those in the links to any job openings you have in the description here. The other thing I wanted to ask, and I don't actually think we've said it out loud on the podcast yet news Catcher, what's your website? \[00:47:45\] ****Artem Bugara (NewsCatcher, CEO):** Yeah, it's news catcher api.com. \[00:47:50\] ****Mike Bifulco:** Love it. And where's the best place for people to find you if they're looking to chat with you? Artem. \[00:47:55\] ****Artem Bugara (NewsCatcher, CEO):** Yeah. If you are ever in Warsaw, Poland. Please, \[00:48:00\] like really reach out to me. I don't have, like, it's not, not that many people here I know. So I'd be happy to ask you out for a coffee or something. \[00:48:09\] ****Mike Bifulco:** Cool. , that's amazing. I would hope that people take you up on that. I've heard Warsaw's an amazing place too. I'd, I'd love to visit sometime. Artem, thanks so much for joining, . I really appreciate it. It's been fantastic chatting with you and , what an interesting product journey you've had. \[00:48:22\] Anytime you're you're interested in coming back on the show, we'd love to have you. Thanks so much for joining. \[00:48:27\] ****Artem Bugara (NewsCatcher, CEO):** Thanks a lot. Thanks \[00:48:28\] ****Mike Bifulco:** All right. Take care. ### Postman's Turn For Layoffs URL: https://apisyouwonthate.com/newsletter/layoffs-at-postman/ Last updated: 2024-07-15T11:52:40.000Z Postman sent shockwaves over the last few weeks by letting go of the entire JSON Schema and Async API teams. While not surprising given the constant layoffs across tech, even in profitable companies, it is sad to see this happen. While I am usually suspicious of a company hiring this many people involved in a spec due to control issues, it was terrific to see funding put into work that, while needed, is always forgotten. Hopefully, we will continue to see further investment in these specs, and everyone who lost a job will soon find work. \-- Alexander ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2024/06/treblle-assistant.png) ## Alfred AI: The Ultimate API Assistant Alfred AI is a cutting-edge, AI-powered assistant designed to revolutionize API documentation, discoverability, and integration. Alfred automates manual workflows, for faster API integrations and a superior developer experience. [Discover Alfred](https://treblle.com/product/alfred?utm%5Fsource=newsletter&utm%5Fmedium=apis-hate-sponsorship&utm%5Fcampaign=alfred%5Fai) ## The API Round-Up ### [Generate your OpenAPI Spec With Zod](https://www.speakeasyapi.dev/guides/api-frameworks/zod?utm%5Fsource=apisyouwonthate&utm%5Fcampaign=240715) Zod is a super handy library for defining type-safe APIs for use with TypeScript, and has been adopted my many frameworks as a best-in-class developer experience. This article shows how to generate an OpenAPI Spec directly from Zod definitions using the `zod-openapi` library. ### [Async API Community Update](https://www.asyncapi.com/blog/2024-june-summary?ref=apisyouwonthate.com) While AsyncAPI has suffered from Postman's recent layoffs, the team is happy to announce IBM as the new gold sponsor for the year. A lot is happening in the community, with updates on the mentorship and bug bounty programs. The team also included details on how to help those affected by the job losses. ### [Ben Hutton JSON Schema Update](https://www.linkedin.com/posts/benhuttonuk%5Fsponsor-relequestual-on-github-sponsors-activity-7217101387216691200-AraZ?utm%5Fsource=share&utm%5Fmedium=member%5Fdesktop) Ben Hutton gives a brief update on JSON Schema and the future while also linking to others on the team affected by the job losses. If you or anyone can help out, please reach out directly to Ben and the others. ### [Source: OpenAPI to Request Handlers](https://mswjs.io/blog/introducing-source?ref=apisyouwonthate.com) MSW, a popular front-end library for mocking network requests, has launched a new tool called Source. It automatically generates request handlers from OpenAPI specs and more. They made some interesting choices with this library, which really make it shine. It's definitely worth a look. ### [A Guide to reading and writing Node.js streams](https://blog.platformatic.dev/a-guide-to-reading-and-writing-nodejs-streams?ref=apisyouwonthate.com) Every API/backend system has to tackle working with large datasets at some point. In Node, working with Streams is the best way to handle this; however, they are not always implemented correctly. This fantastic guide by Matteo Collina will have you streaming in no time 😉. ### [Moving from Express to Fastify](https://blog.val.town/blog/fastify/?ref=apisyouwonthate.com) For many years, Express has dominated the Node community for building APIs and other web systems. However, in recent years, its updates and support have been lacking. Fastify has gained a lot of traction and is a solid replacement. Val Town talks about the transition from Express to Fastify. My favourite part was how it allowed them to use their OpenAPI to generate types in Fastify. ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2024/07/apiywh--4--1.jpg) ## Build APIs Your Users Love Your API deserves a great developer experience. Get one by using Speakeasy to generate idiomatic, type-safe SDKs from OpenAPI [Start Generating](https://www.speakeasyapi.dev/?utm%5Fsource=apisyouwonthate&utm%5Fcampaign=240701) ## APIs You Won't Hate The latest from the team at API's You Won't Hate. ## 🎙️ [Convince me this is a bad idea: how Mercoa built a platform for accounts Payable](https://apisyouwonthate.com/podcast/how-mercoa-built-a-platform-for-accounts-payable) In this episode, Mike sits down with Sandeep Dinesh, cofounder of Mercoa, to talk about the journey from idea to fintech product, their experience going through YC, and advice for aspiring startup founders. --- ## From Our Community Articles written and shared in our free [Slack community](https://slack.apisyouwonthate.com/?ref=apisyouwonthate.com). ### [Finding Hidden API Parameters](https://danaepp.com/finding-hidden-api-parameters?ref=apisyouwonthate.com) Dana provides a great walkthrough on identifying and exploiting potential security vulnerabilities with hidden API parameters. As always, this is another excellent read from Dana. ### [API Fiddle](https://www.api-fiddle.com/?ref=apisyouwonthate.com) This promising new project is a browser-based tool for browsing and inspecting an Open API spec, with support for creating shared URLs for spec workspaces, as well as basic editing features. ### ## Support APIs You Won't Hate When you become an member, you'll get access to members-only content while directly supporting our work. Your support helps us to keep making resources for the API community. [Become a member today](https://apisyouwonthate.com/#/portal) Thanks so much to our [members](https://apisyouwonthate.com/membership/): Kin L, Juxt, Vedran C, Alex R, Nolan S, Frank, James D, and Bill D. Your support means the world to us! Until next time, Alexander, Phil & Mike ### Convince me this is a bad idea: how Mercoa built a platform for accounts Payable URL: https://apisyouwonthate.com/podcast/how-mercoa-built-a-platform-for-accounts-payable/ Last updated: 2024-07-15T06:00:27.000Z ## Show notes - Sandeep Dinesh - [LinkedIn](https://www.linkedin.com/in/dineshsandeep?ref=apisyouwonthate.com) - [GitHub](https://github.com/Thesandlord?ref=apisyouwonthate.com) - [Twitter](https://x.com/sandeepdinesh?ref=apisyouwonthate.com) - [Mercoa](https://mercoa.com/?ref=apisyouwonthate.com) - GitHub - [mercoa-finance](https://www.linkedin.com/company/mercoa/?ref=apisyouwonthate.com) - [LinkedIn](https://www.linkedin.com/company/mercoa/?ref=apisyouwonthate.com) #### Transcript \[00:00:00\] \[00:00:00\] \*\*Mike Bifulco:\*\* Hello friends. Welcome back to APIs you won't Hate. My name is Mike Biko, your co-host and APIs you won't hate. Co-founder leading you on this journey through API Tech in the world. I am super happy today to get to sit down with Sandeep Dineh from Meco to talk about the product he's building. And I think this actually may continue a bit of a weird streak I have where I have some lots in common with the folks who have been interviewing on the show. \[00:00:22\] Maybe that's a bit of self-selection, but in this case, I think Sandeep is working on something that is. At least tangential to a world I used to live in . And he also happens to be a YC co-founder. Sandeep, thank you so much for joining me. How are you doing today? \[00:00:33\] \*\*Sandeep Dinesh:\*\* I am great, Mike. Thanks for having me on. \[00:00:35\] \*\*Mike Bifulco:\*\* Of course. Yeah. It's, my privilege and pleasure to be able to chat with you. \[00:00:38\] Why don't we start here? So you, you are founder of mea. Tell me about Meco. What's the elevator pitch? \[00:00:43\] \*\*Sandeep Dinesh:\*\* Yeah, so Meco is an API for accounts payable. And so a lot of folks don't know what accounts payable is, but when you're a business, you got bills to pay and usually when you're at a certain scale, I. You have approvals and invoices and vendors to manage and this whole process that your \[00:01:00\] accounting team is probably really familiar with. \[00:01:02\] And so we help our customers launch accounts payable products to their customers. \[00:01:08\] \*\*Mike Bifulco:\*\* Got it. So a bit of a, whether that's a B two, B2B, or B two B2C or how, how does \[00:01:12\] \*\*Sandeep Dinesh:\*\* B two. B2B. Yeah. So I think the best way of looking at that is Stripe has Stripe Connect really popular product for marketplaces. So let's say you have like Uber drivers they need to get paid. So you pay, I. Uber and then Uber pays out to the drivers. That helps 'em get paid. We help make payments out. \[00:01:32\] So you have a bunch of customers that have their own vendors and they're managing their whole back office on your platform. We help you launch bill pay and accounts payable so your customers can start paying their bills on your platform. \[00:01:45\] \*\*Mike Bifulco:\*\* Sure. Okay. That makes a ton of sense. And I think especially for someone my, like myself with a background primarily in engineering and less so on the finance and business side of things, I'm always fascinated to hear a bit about the history of how you got there and like what, what caused this to \[00:02:00\] happen. \[00:02:00\] And I also think we just hit a world record or maybe a, a show record for quickest time to me needing to give a disclaimer that in a past life I worked for Stripe on the developer advocacy team. Had a great time there. It was a good job. I'm no longer at Stripe. But it's probably just worth calling that out to begin with since it'll likely come up here and there during the conversation. \[00:02:17\] So, Sandeep, with that being said, tell me about how you got here. What, what caused you to you know, take a path that had you build, has you building an accounts payable product? \[00:02:25\] \*\*Sandeep Dinesh:\*\* Yeah. So in order to answer that correctly, I think we gotta go back around 10 years. So 10 years ago I pretty much just graduated college. I was working at my first job and I was really bored. And so I actually met my current co-founder at that company. And we did a few hackathons together and we did our first startup way back in like 2014\. That crashed and failed real hard. But it was a great learning experience. And we've been working together for the past 10 years. But after that I actually joined Google on the developer advocacy team for Google Cloud. And so I was our \[00:03:00\] early hire on the cloud advocacy team there. And yeah, that was a great place to start my career as an engineer. \[00:03:07\] You know, I've been coding for a long time, building websites since I was a kid and all that kind of stuff. But I think just learning how some of the best developer advocates go ahead and talk about APIs and developer tools. And kind of really seeing Google Cloud grow from Yeah. They have a cloud platform, I think, to being in the same conversation as like Azure and AWS. \[00:03:29\] Was a really cool experience. And so I did it about for about five years. And right before the pandemic me and my co-founder launched a fully bootstrap startup. So beginning of 2020\. So we built that for about a year. And then I joined a small byc company as a lead engineer. And so I learned a lot about. Kind of the, the venture world and shipping fast and kind of. Doing that kind of stuff. And then I joined Stripe as well. So yeah, I think a \[00:04:00\] lot of career similarities with you, Mike. And yeah, so at Stripe I was a engineer on the revenue recognition team, and so we were building some cool products for accountants. \[00:04:10\] And so while I was there at Stripe, my co-founder was actually a product manager@bill.com. And so he was working for a company called Invoice to Go that did invoicing for small businesses. They got bought by Bill and one of his projects@bill.com was trying to build Bill dot com's account payable into invoice to go's platform. \[00:04:32\] So basically trying to turn bill.com into an API into an embedded, uh, API, and then embed it into like an internal project for that. And the thing that he really realized was other companies were coming to bill.com and asking for this kind of functionality. And bill.com really couldn't support them because their organization wasn't designed to sell APIs and developer tools. \[00:04:59\] And so we \[00:05:00\] started MEA because we saw this opportunity of like, if we are API first, if we're developer first, can we go ahead and help people launch their own accounts payable products? In a, in a, in a way that kind of competes with all these different AP systems out there. \[00:05:14\] \*\*Mike Bifulco:\*\* I'm really interested in all of this because it seems like there's a, there's a real confluence of things happening where you're both working at companies where, you know, opportunities are ripe in, in sort of the payments world, especially around this time period. Right? The past few years has been pretty crazy and, and all this stuff. \[00:05:27\] But also to be to have you at Stripe where developer journey is everything. And your co-founder at a company where they, they're desperately in need of a solid developer journey, feels like the, the birth of your product is a sensible outcome. From there I'm curious for yourself and your co-founder did you either of you have sort of like business education background or were you both sort of traditional engineers going into this? \[00:05:48\] \*\*Sandeep Dinesh:\*\* So my co-founder has an economics background. He's not an engineer. Lot of product experience, sales support, customer service. So he's, he's had the whole go to market kind of \[00:06:00\] experience from a career standpoint. For myself I technically have a business background. I graduated with a degree in marketing and computer science. \[00:06:08\] I don't know how useful that degree in marketing was. \[00:06:11\] \*\*Mike Bifulco:\*\* You a job in developer advocacy, I think it probably went a a really long way. Yeah. \[00:06:15\] \*\*Sandeep Dinesh:\*\* Yeah. So I think that really helped getting that job as a dev in Dere. But yeah, I think so to your point about kind of things just kind of converging together, I think learning about developer tools and dere at Google, learning about the accounting side of things at Stripe having my co-founder also super interested in FinTech. \[00:06:35\] Having the startup experience prior getting into yc, all these things kind of just combined in a way that made starting meco pretty, pretty obvious to do. Yeah. \[00:06:46\] \*\*Mike Bifulco:\*\* So you've, now come together with the idea you have people with, interesting backgrounds that sort of knit together to build something that makes a whole lot of sense. What was your story for chasing down your first actually, maybe let's talk about it this way. Your first use case, like what was the first thing you built and then \[00:07:00\] how did you convince your first users to jump on? \[00:07:02\] \*\*Sandeep Dinesh:\*\* Yeah, so we started talking to customers before we had anything built. I think. We've gone down the road of like building a bunch of stuff and then figuring out you built the wrong stuff plenty of times. So we started having phone calls with folks through, in our network, through LinkedIn. \[00:07:18\] We've been very fortunate that we've done a lot of people coming in talking to us through LinkedIn. So yeah, we had a lot of folks, we talked to them, try to really understand what problem they're trying to solve what's important what's difficult about this. And we kind of sprinted to create a demo. \[00:07:35\] And so we had like a nice happy path of like, you can embed this into your, into your product and now your customers can start paying bills. And that first demo was basically an I frame that you drop in and the word I frame kind of like burns the ears of many folks out there. They hear that and they're like, I don't like that. \[00:07:55\] But it was kind of the fastest way for us to validate that. Hey, drop this in, see if it \[00:08:00\] works. We had an API, it was hand documented. Hand rolled pretty bad. But, you know, we just wanna validate does this, does anyone even care? And so we started getting folks who were interested. We started signing some pilots. \[00:08:16\] And then we got started getting really serious about how do we deliver this product. So that was kind of the beginning. And I can go into more, but yeah. \[00:08:23\] \*\*Mike Bifulco:\*\* Yeah. Okay. I'm, I'm already hearing things that make me think that maybe having a marketing background was probably pretty helpful there. But, but maybe also having built and been through the ringer with. PR past companies, especially bootstrapped ones is really interesting there. I talk about this a lot and I almost feel a little I don't know. \[00:08:38\] I, I always feel like it's hard to really land this with people until you've actually experienced it in some way. But many first time founders will go and build a product with an idea that is, I. Whether good or not not told to enough people and then they don't go and chase down enough feedback before building the thing. \[00:08:53\] And \[00:08:53\] \*\*Sandeep Dinesh:\*\* Yeah, \[00:08:53\] \*\*Mike Bifulco:\*\* you know, the, the trope for me is someone who talks about like, well, it's not ready yet. I haven't shared it with anyone 'cause it's not ready. \[00:08:59\] \*\*Sandeep Dinesh:\*\* Never \[00:09:00\] ready. \[00:09:00\] \*\*Mike Bifulco:\*\* just now Yeah, it's never ready. Nothing is ever done, ever. That's the whole whole thing with software. What you've just described for me is it. \[00:09:06\] Entirely the inverse of that, right? Like we didn't build anything. We went and talked to a bunch of people first, and then built something that was no, nobody's favorite idea, but at least better than what existed before. That, that's a skill that takes some time and is definitely a, a nuanced approach, especially as compared to, I don't know what I would call maybe the traditional, like first time founder thing. \[00:09:24\] And so. When you do that, what are the signals that you were looking for to even pursue further? So I'd imagine you got some early signals that people were using it and liking it. Is it, Hey, I want more, or, Hey, we think we could make this better for you, or, or find a thousand more of you, or what was, what was what you were looking for at that point? \[00:09:40\] \*\*Sandeep Dinesh:\*\* Yeah, so it's actually really funny. I think we went about Meco in a way. Completely different from our other products. So I think one thing is we are building financial infrastructure. So that takes quite a lot of time to build and it takes quite a lot of time to build trust in that product, right? So like, we're moving millions of dollars. \[00:09:57\] You're not gonna wanna just \[00:10:00\] trust who, random guys who build a product in a weekend to do that, right? So I think for us, we were really trying to figure out. Is there market demand for something like this? First? Can we find those people in a, in a repeatable process? And then what are those people's biggest like need? \[00:10:19\] Right? And so at the end of the day, our product is pretty complicated. We call it like the eight components, and each one of those components can be used a la carte. By itself together. So it's a pretty like complicated platform. So the first thing that we really wanted to do was validate, Hey, are there people out there? \[00:10:38\] This is like a market that even exists. And I think we did that by just trying to talk to as many people as possible in the industry. Are you building this yourself? We got a lot of, yep. This is on my roadmap for next year. For next year. I don't have any resources against it, but I wanna do it. And we're like, okay, that sounds really interested. \[00:10:55\] It's like, do you wanna partner with someone? It's like, yeah, absolutely. If someone would build \[00:11:00\] this for me, I would definitely wanna talk to that person. We're like, okay, that's great. And then the last challenge was how do we find these people? And I think for that it was really a struggle. 'cause there's no like, list of vertical SaaS companies and that's who we primarily sell to is vertical SaaS. \[00:11:15\] So it's really hard to find those folks 'cause there's a bunch of founders starting them all the time. They have a lot of deep expertise in like, let's say the construction space or dental or home services, right? And so they maybe they've worked in that industry for a while. Maybe they've run Airbnbs and they wanna start a business to help other folks in that industry. \[00:11:35\] Like how do you find these folks? Was really difficult. And for us, we really try to solve that by being the most helpful folks. Out there. So a lot of people starting these companies don't have FinTech or finance backgrounds. And so we started writing content on how do you monetize this kind of like platform. \[00:11:51\] What are your customers looking for when it comes to like payments and just trying to be really helpful. And so we actually wrote a 28 page guide \[00:12:00\] on how to build Meco from scratch. And so it's like. If you wanna just build this yourself here is like the PRD for you. And yeah, we got a lot of hits on that. \[00:12:09\] I think people were like, this is really cool. It was just a Google doc. And so that was, that was our first, I think, real sign of, okay, there's something here. Let's invest more. \[00:12:17\] \*\*Mike Bifulco:\*\* I think that sends a strong signal too, that it's like, well, we don't have secrets. This is just a hard thing to do. Like you're, you're welcome if you want to go and, you know, go on this journey. But it's probably the sort of thing that you need a few people and many, many person hours of engineering time to get through. \[00:12:31\] And even then your industry background probably helped quite a bit there \[00:12:35\] \*\*Sandeep Dinesh:\*\* Yep. \[00:12:36\] \*\*Mike Bifulco:\*\* Right on., I'm super into that as a path to go and very, very much a show of strength there too. Like, cool. Here, have all our secrets. Good luck. So where, where does Y Combinator come into the picture? \[00:12:46\] \*\*Sandeep Dinesh:\*\* Yeah, so I was at Stripe. I. Basically just joined. I was about almost like a year in I felt like I was fully onboarded at this point. And my co-founder was, Hey, let's apply to yc. And we would apply to YC plenty of times for other \[00:13:00\] ideas and like, Hey, let's throw in an application. So we threw one in very last minute. \[00:13:04\] And then we got in, which is a very happy surprise. We took the weekend and we're like, yep, this is, we've been waiting for this opportunity for a long time. I don't wanna live my life thinking, what if I think this is a huge opportunity. Like, let's go. They're gonna give us some money, let's go try it out. \[00:13:21\] And we just hit the ground running really hard. So we quit our jobs and started building, started talking to customers. Basically like as my co-founder was lining up these meetings trying to validate that idea like I was talking about. I was like, what is the smallest demo we can throw in front of people to kind of give them an idea? \[00:13:38\] So we did Figma to start and then very quickly moved to like a clickable demo. And that was all before we actually started the, started yc. So we were winner 23 and so we were doing this in like November and December. And so YC started in January and, I think within the first month we signed our first contract. \[00:13:58\] And so that was like a big validation for \[00:14:00\] us. \[00:14:00\] \*\*Mike Bifulco:\*\* yeah, of course. Again, to, it is funny. I don't know this off the top of my head, but to clarify, winner 23 is. Starting January, 2023 ish, or was that into, okay. \[00:14:10\] \*\*Sandeep Dinesh:\*\* yeah, yeah. First three months of, yep, yep, yep. Mm-Hmm. \[00:14:14\] \*\*Mike Bifulco:\*\* And so you've, you've obviously come a long way since then. One of the things that I found valuable when Kraftwerk went through Y Combinator was we were really focused for a few months during YC on set, set an ambitious and, and hard goal that is meaningful and do everything you can to build a narrative towards that. \[00:14:30\] It sounds like you, you headed into YC with a working clickable demo and were maybe chasing down customers. What was sort of your North Star at that point? \[00:14:37\] \*\*Sandeep Dinesh:\*\* Yeah, so we wanted to close 10 customers, very aggressive. We did not get there. We got pretty close. I. I think again, our goal through that three months was can we kill this idea of like, what do we do to lose conviction that this is the business we wanna build? I think both of us are, you know, kind of mid-career. \[00:14:55\] We both are married and all that kind of stuff, and so it was like. \[00:15:00\] You know, it's not our first rodeo doing a startup. We know how difficult it can be and how easy it is to lose motivation. And so it's like, let's do that fast rather than slow. So I guess like fail fast. I don't know if you wanna call it that. But yeah, I think that was kind of the whole goal was like, I. Can we just lose conviction on this idea and kind of just shut it down? And everything that we did went the other direction of like, oh wow, okay. We've kind of gained conviction. And I think at the end of that, we were like, yeah, this is what we wanna spend a long time building because we think it's gonna be really big. \[00:15:35\] And I think we have the right team to do it. And so that was kind of our internal goal at yc, obviously. I think they talk about this a lot. You know, you have the two week check-ins with the group partners. And so we had pretty aggressive goals on trying to close new customers and leads. And so yeah, that was kind of like the external goal, but I think really internally it was, can we kill this idea? \[00:15:54\] And not even launch it. \[00:15:56\] \*\*Mike Bifulco:\*\* Sure. \## Debate me! \[00:15:57\] \*\*Mike Bifulco:\*\* You're absolutely the first founder I've talked to who is, \[00:16:00\] is going about this by way of like the debate me route. Like, you know, convince me this is a bad idea. I, I really like that, especially in the case where like you're, you're being forces you to be maybe intellectually honest with yourself which can be really hard to do. \[00:16:12\] A lot of, a lot of people get almost delusional about their idea and we'll go a really long way before they decide that like. Oh, you know, it turns out the thing that I'm building is just not a big enough molehill to be scalable at, at the scale that we need it to be. Especially once you start taking on investment, that becomes a much, much bigger molehill to tackle too. \[00:16:30\] Okay. So you finished YCS program at the end of last winter, so March-ish of last year. What, what's happened since then? \[00:16:37\] \*\*Sandeep Dinesh:\*\* Yeah, so I think we've learned a lot about what is kind of the, what is an MVP versus an MLPA minimum lovable product. So I was talking about the iframe. Yeah. No one likes that. Our API has gotten a lot more complicated. So I think you've had Danny from Fern on on this podcast before. So we are big users of Fern. \[00:16:57\] We use 'em for server side, \[00:17:00\] client side and docs. And so we basically moved our whole. API over to Fern during the batch. And so we were batch mates together, so very glad I met them early in the business. So I think one thing that we've learned is just how important developer experience and API documentation is for our sales process. \[00:17:20\] And so I think a lot of folks in FinTech the bar is very low for good APIs in FinTech. I think Stripe is like way up there. And everyone kind of assumes that everyone also has like stripe level documentation, SDKs, all this kind of stuff. And no, it is like pretty bad. And so I think like the bar is low, but we wanna go really above and beyond to kind of provide that really good experience. And I think the big reason why is, well twofold, one. We sell to product teams, but a technical co-founder or VP of engineering or lead engineer, whoever it might be, depending on company \[00:18:00\] sides, does have a very big say in the go, no go. Right? If they look at our SDKs and APIs and they're like, this looks like my team is gonna struggle to implement, I don't think they're gonna be serious about this. \[00:18:13\] We are not gonna win that deal. Right? So like the features and all the check boxes might look really cool, but at the end of the day, because we are a embedded product, we are API first, we don't make money until you start using us, right? And so like that implementation's all about the technical kind of coupling of our service to yours. \[00:18:33\] So the API SDKs are a huge part of our Go-to market movement. So that was a big learning. I think we. We're trying to build as fast as possible and the docs went outta sync and they didn't look good, and our guides weren't great and we invested a lot of time into like trying to make that onboarding experience really nice. \[00:18:53\] And so that was a big thing. I think Fern was a big help there. So for folks who don't know what Fern is, it \[00:19:00\] basically lets us define our APIs in a Fern yaml. I think they also support open API specs and it actually generates our server side routes and client SDKs for a bunch of different languages. \[00:19:12\] As well as our docs. So every time we make a change, all three of them are always in sync. And so we can get basically guarantee to our customers that like, everything is well documented, always up to date. And it's like what you, what you see is what you get and people really like that. So that was a big part of our investment. \[00:19:29\] The other side on the dev side was our front end. So we talked about the iframe, iframes. I, I love iframes. They let you move really fast. Uh. With things like JWTs where you can pass in custom props, you can actually customize iframes pretty easily. I think a lot of companies that deliver their solution with iframes do that, but I think a lot of her customers were like, this is even with, this is not good. \[00:19:52\] I don't like, it's a black box. My, like, observability tools don't show me what's going on. If a customer has a question, \[00:20:00\] I'm just kind of stuck on you guys to, to figure that out. And so we had built that iframe with react components underneath the covers. And so moving that from internal React components to an external library was a really big deal. \[00:20:15\] And, you know, I think like react components as a delivery mechanism. I think like Algolia does a really good job. It's a company called Quill. They do embedded analytics. They does a really good job, but it's kind of like a, a new. Way of delivering an embedded product. And so. \[00:20:31\] \*\*Mike Bifulco:\*\* think so. Yeah. \[00:20:33\] \*\*Sandeep Dinesh:\*\* There's really not that much like best practices out there. \[00:20:35\] So we've had to like learn a lot. So we're using storybook for a long time to do kind of a documentation, but that's really great for like UI libraries and that's not really what we're doing. And so we're actually migrating to a fully manual documentation for that where we put our components, we talk about how you can use them. \[00:20:52\] All of our components take can take in a child component. Then we just pass you a bunch of hooks, right? Like that's not really a \[00:21:00\] common use case for like UI libraries. So I think like that's been a big learning of like, how do we help our customers move fast? And that's the reason why they wanna partner with someone like us and not build it in-house is time to market, right? \[00:21:13\] And so I think the backend is really complicated. That's, you know, kinda expected. We have a lot of good documentation tooling for APIs. But then the front end, the react component side, I think that's kind of like a new world that we're exploring and trying to build a lot of tool that kind of tooling ourselves. And yeah, so that's been a big learning for us. \[00:21:32\] \*\*Mike Bifulco:\*\* Yeah, that's a really fair point. I don't think I've really heard it put that way, but it's definitely something I've felt maybe as a beneficiary of it, as an end user of these sorts of things. That many of the SaaS products that have come to love are ones where I essentially consume it through NPM install. \[00:21:47\] And become an end user that way. And I can think of a few off the top of my head. Clerk does authentication very much in that way where they ship a bunch of React components that you just toss it into your React app and suddenly have authentication. Post hog does a really good \[00:22:00\] job. They have some really nice React libraries. \[00:22:01\] They'll let you add. Product analytics and do a whole bunch of tracking and ab tests and all this stuff that you know, in prior days would've been a much deeper build out with loads and loads of code and docs that you could imagine, and certainly a few others that I've used. And there's something to be said for that too, that, that is scaling your developer experience by way of taking advantage of the massive amount of time and love and care that folks from these other companies are putting into their. \[00:22:24\] A deeply focused slice of the thing. You know, like authentication is something that I never wanna become an expert on. And I'm sure I've said that on the, the podcast a billion times by now. But being able to effectively hire out smarter people than me for a price that scales with my success is like, sign me up, let's do it. \[00:22:39\] That's great. \[00:22:40\] \*\*Sandeep Dinesh:\*\* The interesting thing with these like embedded, so clerk embedded authentication like the thing there is, it's the same for everyone, right? Like almost 90% of what you're trying to build with these embedded providers, it's the same as everyone else. \[00:22:53\] And I think that's just kinda the same thing for us. We're say like, you know, 80% of bill pay is like the same. It's an accounting process. Like there are gap \[00:23:00\] principles that everyone follows to like pay their bills. \[00:23:03\] \*\*Mike Bifulco:\*\* Right. \[00:23:03\] \*\*Sandeep Dinesh:\*\* Just add your secret sauce on top that's like in specific for your industry or your vertical or like your, your customer base. \[00:23:09\] But why are you trying to reinvent the wheel? And I think that's kind of like the, the bread and butter of embedded SaaS, right? We're fully white labeled. You don't know that we exist, but we let you move fast and like not worry about like the things that you don't wanna worry about. \[00:23:22\] \*\*Mike Bifulco:\*\* Yeah. And again, to that point, you, you meco is successful only when your customers are, and there's an alignment of incentives there. That's really interesting too. \[00:23:30\] \*\*Sandeep Dinesh:\*\* I'll try. \[00:23:30\] \*\*Mike Bifulco:\*\* Another case where maybe solving the boring problem, so to say is great for everyone involved. You can stay focused on your thing much the the way that, you know, say a clerk would be focused on authentication. \[00:23:41\] So I wanna talk a little bit more about how Meco is built. So Fern is, is super interesting because they give you so much of that sort of love and care in the world of SDK Cogen. So now Meco is available sort of. Pseudo natively in a bunch of languages, I'd imagine as, as installable packages, SDKs, whatever you wanna call 'em, and you get documentation out of it and \[00:24:00\] all that. \[00:24:00\] What, what is the underlying system that you're building then that that speaks to Fern? Like, what's your architecture look like, maybe what's the size of your team? Those sorts of things. \[00:24:07\] \*\*Sandeep Dinesh:\*\* Yeah, so bit of background on me At Google Cloud. I was on the Kubernetes team, so microservices were like my bread and butter. And so very specifically not doing that with Meco. So team is very small. It's basically on the engineering side. Right now. It's just me. So we've kept the team pretty small on purpose to move very fast, kind of find product market fit, and then start scaling it out. \[00:24:29\] And so it's pretty simple. We are running on Google Cloud Run, so fully serverless. And our database is on CockroachDB, which is also fully serverless. So that's for folks who don't know what CockroachDB is. It is a Postgres compatible database kind of based on Google Cloud. Google Spanner and Spanner is a. \[00:24:48\] Horizontally scalable asset compliant database that uses like crazy atomic clocks and all this kind of stuff to keep transactions always in sync as you scale out. And so we've kind of \[00:25:00\] chosen technologies that let us scale by default because I don't wanna deal with that. So really simple. It's a monolithic TypeScript, no js backend. We use kind of. The, the normal tools, GitHub and Sentry and all that stuff for monitoring. But yeah, we kept it very simple just so we can move fast. I've worked at teams where, you know, everything was split out into its own microservice and it just kind of, as a small team, very hard to deal with that. \[00:25:24\] So yeah, we just kinda, how can we move really fast and how it deliver a product that, you know, customers really want. And then we can think about breaking it down in the future. \[00:25:34\] \*\*Mike Bifulco:\*\* Right, and you mentioned you, you're the only engineer, is it just you and your co-founder currently as well? \[00:25:39\] \*\*Sandeep Dinesh:\*\* Yeah, full time. It's just me and my co-founder. Yep. \[00:25:42\] \*\*Mike Bifulco:\*\* Amazing. Definitely very small and lean and mean at that scale. And what I, if you're listening to the podcast, one of the things I would encourage you to do, which I think is maybe a good data point to like put side by side with that is me's site. Does not convey a two person team even slightly. It looks pretty, pretty intense. \[00:25:59\] There's a lot going \[00:26:00\] on there. So you know, a, a applause is due to you and your co-founder. There's loads of stuff there, and I'm sure you're standing on the shoulders of giants in many ways by using lots of interesting tech there. But it speaks to what you can get done with, with some focus and maybe a bit of a marketing polish on things too. \[00:26:15\] That's, that's super cool. \[00:26:16\] \*\*Sandeep Dinesh:\*\* Thank you for the compliment. I think, yeah, definitely building on the shoulders of giants. I think like we try to use as much tooling as possible. We try to partner with vendors that can move us really quickly. But I think at a small, small team size, you know, our secret sauce is just customer support. \[00:26:32\] And I think YC says this also where it's like, what can you do that like Stripe or Google can't? And that is just like. Really, really focusing on the customer and really delivering solutions to them. You know, we don't have a lot of red tape. If someone really needs something, we can ship it tomorrow. We wanna move very fast. \[00:26:51\] We wanna be very thoughtful and, back to the original point of no secrets. I think like even if you don't end up working with us, we still want to make you successful. 'cause maybe in the future \[00:27:00\] you will, maybe you won't, but at the end of the day, we wanna move the whole industry forward. It, it's really painful when you see folks still using like pen and paper or checks or spreadsheets to do their finances. \[00:27:10\] Like can we just move the world away from that is kind of like the goal, right? Like maybe it's us, maybe it's not, but let's move the world forward. Is kind of like the big picture. Yeah. \[00:27:20\] \*\*Mike Bifulco:\*\* Yeah, you, you've painted a very cool story as well around like having a focused idea and building something that addresses a direct problem that you understand really well. Often on this show we talk a little bit about, like, talk about a problem, a project, a product. Fern was one, Danny's been on the, the show where we've talked to him before about what they're doing and sort of the pitch for API developers at that point is something along the lines of like, how do you know if you are someone who should be interested in using Fern? \[00:27:45\] I would imagine there's probably a very direct story there for Meco too. So for, for a, for devs who are used to integrating with APIs and building products what's a sign that Meco might be solving problems that they should be interested in? \[00:27:56\] \*\*Sandeep Dinesh:\*\* Yeah, so I think the big one with MEA is if you're trying to \[00:28:00\] build the financial back office for your customer. I think a lot of folks start with invoicing to start, right? So helping your customers get paid. Then kind of the next natural thing is banking. So helping them store those funds, manage those funds and then you can go into payroll, you can go into spend and expense management, and you can go into accounts payable. \[00:28:18\] And so basically we wanna help you create that financial one-stop shop for your customer. And so that's money in money management and then money out. And so we plug into that money out piece. And I think the reason why a lot of folks don't really understand the opportunity in the money out side of things is I think when you're thinking about money in, it's like, Hey, Stripe charges me 2.7% plus whatever. \[00:28:44\] Okay, cool. I can charge like a fee for invoice processing, right? Like. That's pretty like cut and dry. Like I don't understand how that works. When I'm storing funds, you know, I can make some float interchange interest maybe I can like give you some like credit cards or debit cards to spend and I can make something called \[00:29:00\] interchange on those cards. \[00:29:01\] But then when you get a invoice for a thousand dollars, you gotta pay a thousand dollars, right? Like there's no middleman to take a fee there. And so I like think a lot of people are like, there's no opportunity here to like, monetize the payments. Maybe I can charge like. Some SaaS fees, like 20 bucks a month or something to my customer to like have them manage their bills through my system. \[00:29:22\] But kind of the secret here is that all these big account payable companies have found is your customer who's using your product, really wants to keep cash with them as long as possible, right? But the folks they're paying wanna get paid as fast as possible. So we basically have a mismatch in, in incentives here where. \[00:29:44\] The payer wants to hold onto money as long as possible, and the vendor wants to get that money as fast as possible. And so you can offer financing on both sides of that equation either accelerating the payment to the vendor or delaying payment from the payer to help both sides manage their cash flow. \[00:29:58\] And that's really \[00:30:00\] where the unlock comes in on monetization. \[00:30:02\] \*\*Mike Bifulco:\*\* Oh, super interesting. So you're, you're running financial mechanisms underneath this then that people are keying into on both sides. Really cool. Wow. And so, man, I, I that's a part of the world that I know enough about to be scared of. Does this mean you also then had to go chase down, I'd imagine compliance and certification and all sorts of things to be able to offer those. \[00:30:20\] What's that journey look like? \[00:30:21\] \*\*Sandeep Dinesh:\*\* Yeah, so I think we have a lot of flavors on how we do this. So right now we're using a lot of partners to kind of do those services. And so our big value prop is. one is gonna take your fancy financing product until they have a workflow that matches what their accounting and finance teams really want to use. \[00:30:38\] Right? So we spent a long time kind of building those workflows, those react components, those backend APIs so that you can deliver a product that'll actually be used, right? Because if you just have like a. Pick a financing button, like it's not really, it doesn't fit into that workflow. So our first step is kind of building something that your customers will actually use. \[00:30:57\] And then we work with a bunch of vendors to provide \[00:31:00\] that financing, to provide those payment rails. And eventually we'll bring some of that in-house. But I think our big value prop is we will go ahead and integrate with all of these platforms. And so then you just have one API to implement, drop it in, and then you get best in class payment rails out of the box. \[00:31:15\] \*\*Mike Bifulco:\*\* Yeah. Nice, clear story there too. That's fantastic. Okay. I do wanna ask, maybe the inverse of the question I asked before is you know, what, why would an API developer recognize that meco is something they should be interested in? I also think there's a good number of people that listened to this show that may be sitting in shoes similar to, to yours from a year or two ago, where they may be getting signals that they're. \[00:31:36\] Experiencing things or learning things at work or seeing problems that might make their way into a good company at some point. Do you have any advice for people like that? For how to test an idea how to, how to prove something out as a business \[00:31:48\] \*\*Sandeep Dinesh:\*\* That's a great, great question. I think experiencing that pain yourself at a big company definitely a good sign. You know, at Google and Stripe, we have so many internal \[00:32:00\] tools that like I wish existed outside. Good, great example of that is, you know, both companies have like a launch process that you had to go through to launch anything public. And it's pretty, it's, it's, it's one's process, but it's like really well thought out. Like on the Stripe side, like if you do anything that touches an API, someone from the developer relations team will come review, make sure it's consistent across the board. So like that kind of stuff, like is there a product that you could build externally? \[00:32:25\] There's also really simple things like, I think there's a company called Go Links. We use 'em at mea I think like Google had that originally. And it's a whole company now where you just like go slash and you put a link name in and then it redirects you to a page and it's so useful and you're like, I had this at Google now I don't like, let's go build a company about this. \[00:32:41\] And so I think like, how do you validate? That was kind of the question that you asked. I think having it yourself is great. And then going to, talking to your friends, going and talking to your people in your network, right? Like if you're an engineer and you're building a developer tool, I'm. I think you probably know some other engineers, like either work at your company, \[00:33:00\] work, went to school with you, you worked previously with Met at a meetup. \[00:33:04\] I don't know. Right? Like, but you probably know somebody, so just go talk to them. People like talking about their problems. I think that's something that especially first time founders are really afraid of. They're afraid of getting, being rejected, being like, I have this great idea. I don't wanna talk to anyone because what if they say, say it's bad? And for me it's like, that's the best thing then I don't know how to waste my time building this thing anymore. Right. Because you know, it's so easy to burn out building something that no one uses. Like you can spend years and years and years. Building something and then you launch it and then you, it just crickets and then it just feels super bad. \[00:33:37\] And then maybe your next idea is gonna be the billion dollar idea, but you're just so tired from working on something that no one used that you don't even try it out. Right. And so my advice to anyone kinda listening who wants to dip their toes in startup is just talk to people. Validate it. Get your idea in front of as many people as possible. \[00:33:55\] No one's gonna steal it. In fact, they might give you ideas on how to make it better. Right. And \[00:34:00\] honestly, being part of yc something like that I didn't realize before I joined was, you know, it's a filtering function for really great founders and we're, we are fully in person in San Francisco. \[00:34:09\] There's a lot of other YC companies here too. There's a lot of events where we meet up and I think talking to other folks, they give us ideas that we're like, oh my God, this is gonna. I didn't think of this. This is like a easy thing, low hanging fruit that just unlocked like some like big win for us, right? \[00:34:26\] And yeah, just sitting in your, in, in your room coating is not going to to help you. You do have to do that. A lot of coating, but you also had to get out there. Yeah. \[00:34:37\] \*\*Mike Bifulco:\*\* Yeah, that's, I, I will pile on and add to that, that at some point it's not about coming up with good ideas, it's about being able to execute on them and identify the problem space that matters. And one thing that I wish, like I get a lot of folks who ask me questions about starting a company or they wanna talk about an idea they have. \[00:34:54\] And I, I always wish I could talk to people like a year or two or three or four before they, they start with \[00:35:00\] pursuing a business to, to plant a seed with, with you, whoever you are, wherever you are, that one of the most important things you can do is learn how to find. People find the network, find the problem space, get embedded in the conversation. \[00:35:11\] And for a lot of developers who are maybe a little more introverted or you know, reticent to communicate openly online that creates a problem. If, if you woke up tomorrow and said, I wanna start a startup, and you have. No you know, no following on LinkedIn. You don't have a public GitHub, you don't have a website, you don't have anyone who you talk to about these things. \[00:35:28\] You're starting at a massive disadvantage. So I'm not saying you need to go and be an influencer, like, I don't even think that's even remotely the answer, but I do think you should be able to participate in open source, right? You should be comfortable with jumping into GitHub conversations, looking to Reddit for answers, helping people out diving into things and, and learning. \[00:35:45\] And you'll start to knock out the bad ideas just as well as meeting people who have really interesting, good ideas and. Don't have time to pursue 'em, you know? \[00:35:52\] \*\*Sandeep Dinesh:\*\* Yeah. I think just to add on to that one, if, if you're doing a startup because you were like, I wanna try a startup, may, maybe not the best reason. \[00:36:00\] I think like \[00:36:01\] \*\*Mike Bifulco:\*\* Yeah, yeah, \[00:36:02\] \*\*Sandeep Dinesh:\*\* the, to your point of like being in these communities, like you probably wanna care about this problem enough that you're already there or like. \[00:36:10\] You want to, you like, you naturally want to go and like help those folks. And then like the startup comes after, right? Where it's like, oh, I see there's a bunch of people to help. I can probably build a business to help these people. Then it kind of makes a lot more sense versus like, I'm gonna take this and shove it down their throats and make them buy for me. \[00:36:28\] Maybe not the best way to do it. \[00:36:31\] \*\*Mike Bifulco:\*\* For all but the most extreme outliers, your startup idea at best will take 10\. 10 years, right? You'll be working on a problem and it has to be something you can be interested in for that long. I. Odds are, it's probably already something you're interested in or at least related to a problem you've had that would, would make life better. \[00:36:46\] Otherwise you're you know, punching from the back foot or whatever the phrase is there. Sandeep be, before I let you go, I want to hear a little bit about what's coming next for Meco. So what are you working on now? \[00:36:56\] \*\*Sandeep Dinesh:\*\* Yeah. So we are relaunching our React documentation. \[00:37:00\] So stay tuned for that one. We're pretty excited. We're. Always trying to improve our developer experience. So that's a big one. We have a few cool payment partnerships coming out too, so hopefully we'll be announcing that. But yeah, aside from that, you know, just come check it out. \[00:37:13\] If you are at all interested in FinTech APIs developer experience send me a note on LinkedIn or Twitter. Happy to chat. However I can be helpful, I'm happy to help. \[00:37:25\] \*\*Mike Bifulco:\*\* I appreciate that. I will make sure I put both your LinkedIn and your Twitter profile in the show notes here for folks to find you. And where's the best place to find Meco online? \[00:37:34\] \*\*Sandeep Dinesh:\*\* Mea.com. This is a, it's, it's exactly how it sounds. It's spelled M-E-R-C-O-A. And then we do a lot of content on LinkedIn. So we're gonna start doubling down on that as well. So if you want any of our content, you can probably find it there. But yeah, our website is probably the best place. \[00:37:50\] \*\*Mike Bifulco:\*\* Amazing. That will obviously also be in the show notes Sandeep Dinesh from mea. Thank you so much for chatting with me. I really appreciate it. It's been fantastic to learn from you and really cool to hear about your story. Feel \[00:38:00\] free to join us anytime to give us updates on where things go between now and then. \[00:38:04\] \*\*Sandeep Dinesh:\*\* Thank you. Yeah, Mike, this was great. I had a great time, so I really appreciate you having me on. \[00:38:08\] \*\*Mike Bifulco:\*\* Of course. Take care. \[00:38:10\] \*\*Sandeep Dinesh:\*\* Thanks. ### Leveling up API Docs URL: https://apisyouwonthate.com/newsletter/leveling-up-api-docs/ Last updated: 2024-07-01T20:04:43.000Z What is the next step for API documentation? Even if something has documentation, it does not mean it is easily understood. I stumbled on [Mintlify](https://mintlify.com/?ref=apisyouwonthate.com) over the weekend and fell in love with the docs you can produce. With OpenAPI type generation tools providing linting and types, I leave the IDE to read endpoint docs less. When I reach for visual docs, it's for the recipes or cookbooks of specific flows within an API. Documentation is an art of technical storytelling that not everyone is great, but tools like Mintlify can help get us there. \-- Alexander ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2024/06/treblle-assistant.png) ## Alfred AI: The Ultimate API Assistant Alfred AI is a cutting-edge, AI-powered assistant designed to revolutionize API documentation, discoverability, and integration. Alfred automates manual workflows, for faster API integrations and a superior developer experience. [Discover Alfred](https://treblle.com/product/alfred?utm%5Fsource=newsletter&utm%5Fmedium=apis-hate-sponsorship&utm%5Fcampaign=alfred%5Fai) ## The API Round-Up ### [How To Generate an OpenAPI Spec With FastAPI](https://www.speakeasyapi.dev/guides/api-frameworks/fast-api/fastapi?utm%5Fsource=apisyouwonthate&utm%5Fcampaign=240701) Learn how to configure FastAPI's OpenAPI spec generation to be more usable and cross-compatible with industry tools. The interactive docs page gives a step-by-step guide, holding your hand the entire way. ### [Behind the Code: A Discussion with Backend Experts](https://sentry.io/resources/behind-the-code-a-discussion-with-backend-experts/?utm%5Fmedium=paid-community&utm%5Fsource=reactstatus&utm%5Fcampaign=fy25q2-backendworkshop&utm%5Fcontent=newsletter-backendworkshop-register) Sentry has put a killer event together for the 17th of July, featuring a fantastic lineup to talk all things backend. Topics will range from the impact of APIs on modern software development to building scalable and secure databases. Speakers from Sentry, Supabase, Prisma, Node and the founder of Larvel. ### [Visualizing algorithms for rate limiting](https://smudge.ai/blog/ratelimit-algorithms?ref=apisyouwonthate.com) Every API needs rate limiting; as I am writing this, I can't think of one where you wouldn't, maybe an internal API... Anyway, settle in for an engaging read with beautiful visuals on the different algorithms you can use and the implications of each. ### [An interactive study of queueing strategies](https://encore.dev/blog/queueing?ref=apisyouwonthate.com) Over the last few years, Sam Rose has produced some of the most visually engaging blog posts I have ever seen, and his latest on queueing is no exception. Sam covers the reason for needing ques, different approaches and types, and how they compare. It is a beautiful read with incredible interactive visuals. ### [](https://nodeweekly.com/link/156735/866f6b5735?ref=apisyouwonthate.com)[How to Setup *Serverless Framework* with AWS and Node.js](https://nodeweekly.com/link/156735/866f6b5735?ref=apisyouwonthate.com) Serverless is a great framework for building serverless applications, such as API powered by Lambda functions on AWS. In this walkthrough, you will learn how to deploy and set up Serverless on AWS using NodeJS. I love it when people provide great configuration walkthroughs so I can stop getting frustrated when AWS docs are lacking. ### [Node.js is Here to Stay](https://blog.platformatic.dev/nodejs-is-here-to-stay?ref=apisyouwonthate.com) Over the last few years, Node.js has had a lot of competition, first Deno and now Bun. Personally, I find this competition to be a good thing, driving much-needed discussions and pushing Node forward. In this post, Matt explains why Node is here to stay and provides some facts to back it up. ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2024/06/apiywh--2-.jpg) ## Speakeasy: Build APIs Your Users Love Your API deserves a great developer experience. Get one by using Speakeasy to generate idiomatic, type-safe SDKs from OpenAPI [Start Generating](https://www.speakeasyapi.dev/?utm%5Fsource=apisyouwonthate&utm%5Fcampaign=240701) ## APIs You Won't Hate The latest from the team at API's You Won't Hate. ### [OpenAPI.tools](https://openapi.tools/?ref=apisyouwonthate.com) Are you looking for modern tools to work with OpenAPI? Check out the most up-to-date list driven by the community. It offers the best high-quality list of tools around. --- ## From Our Community Articles written and shared in our free [Slack community](https://slack.apisyouwonthate.com/?ref=apisyouwonthate.com). ### [What makes a good REST API?](https://apitally.io/blog/what-makes-a-good-rest-api?ref=apisyouwonthate.com) This is a great share from one of our community members, Simon Curcke. Simon walks us through what makes a good REST API, covering areas such as design, OpenAPI spec, validation, and more. Thanks for the share, Simon. ### [Hacking APIs with HTTPie](https://danaepp.com/hacking-apis-with-httpie?ref=apisyouwonthate.com) Dana's API articles on hacking are always a great read, teaching me things I had never considered learning. In this article, Dana discusses his go-to tool for API exploration and testing of HTTPie and how he uses it to hack APIs. ## Support APIs You Won't Hate When you become an member, you'll get access to members-only content while directly supporting our work. Your support helps us to keep making resources for the API community. [Become a member today](https://apisyouwonthate.com/#/portal) Thanks so much to our [members](https://apisyouwonthate.com/membership/): Kin L, Juxt, Vedran C, Alex R, Nolan S, Frank, James D, and Bill D. Your support means the world to us! Until next time, Alexander, Phil & Mike ### GraphQL Is Dead URL: https://apisyouwonthate.com/newsletter/graphql-is-dead/ Last updated: 2024-06-17T12:46:17.000Z To be honest, GraphQL has never been my thing. I've never enjoyed writing it or consuming it on the client. For me, the tradeoff has never seemed worth it. So you can imagine how amused I was when a post titled "Why, after six years, I'm over GraphQL" landed on [Hacker News](https://news.ycombinator.com/item?id=40521518&ref=apisyouwonthate.com). Before my comments spark a flame war, let me say that GraphQL has its place, and there are parts I enjoy. I hope you find this slightly GraphQL-themed issue interesting, and feel free to overlook my cynicism. \-- Alexander ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2024/06/treblle-assistant.png) ## Alfred AI: The Ultimate API Assistant Alfred AI is a cutting-edge, AI-powered assistant designed to revolutionize API documentation, discoverability, and integration. Alfred automates manual workflows, for faster API integrations and a superior developer experience. [Discover Alfred](https://treblle.com/product/alfred?utm%5Fsource=newsletter&utm%5Fmedium=apis-hate-sponsorship&utm%5Fcampaign=alfred%5Fai) ## The API Round-Up Our monthly round-up of articles, news and updates from APIs You Won't Hate and around the web. ### [Why, After Six Years, I'm Over GraphQL](https://bessey.dev/blog/2024/05/24/why-im-over-graphql/?ref=apisyouwonthate.com) This article kicked off my week of giggles. Matt Bessey talks us through his frustrations with GraphQL, including attack service, authorisation, rate limiting, coupling, performance, and more. I can't entirely agree with all the negatives, but this article provides insight into some of the tradeoffs when using GraphQL. ### [Why, After Eight Years, I Still Like GraphQL Sometimes in the Right Context](https://www.magiroux.com/eight-years-of-graphql/?ref=apisyouwonthate.com) This is a solid reply to the first post, putting some of the complaints under scrutiny. Marc-Andre Giroux argues that Graph works best when you persist queries and are building an internal API with multiple clients. ### [Happy Compromise Between Customization and Cacheability](https://apisyouwonthate.com/blog/partials-compromise-between-customization-and-cacheability/) One of the main issues I have always had with GraphQL is that people seem to think moving API design to the client makes things easier. The truth is that very customizable APIs are almost impossible to optimize. Most of the time, APIs don't need to be as customizable as they are, and you don't need to force strong coupling between an API and front-end design. To help build on this, Phil talks us through making an API customizable but still cacheable. ### [Enable API Caching](https://www.fastly.com/blog/optimise-api-cache-improved-performance/?ref=apisyouwonthate.com) Another mistake people make is mixing catchable and uncacheable data. Fastly walks you through splitting up data, for example, not mixing invoice and payment details. Clients only fetch what they need by creating different endpoints with different capabilities and caching. With good API design, you might not need GraphQL as much. ### [Make your APIs Discoverable with APIs.json](https://bump.sh/blog/make-your-apis-discoverable-with-apis-json?ref=apisyouwonthate.com) Phil is back writing another article for Bump. Phil explains how to enhance API discoverability using the APIs.json specification. The specification involves creating a well-known URL (`/apis.json` or `/apis.yaml`) containing metadata about the API, which helps various tools and marketplaces locate and interact with your APIs. ### [Scalar OpenAPI Parser](https://scalar.com/changelog/2024-06-14-@scalar/openapi-parser?ref=apisyouwonthate.com) A new OpenAPI parser from the Scalar team. It aims to be a Swagger and JSON Schema reference resolver. Phil is excited about this one, as it may replace many defunct tools. --- ## APIs You Won't Hate The latest from the team at API's You Won't Hate. ### 🎙️ [Designing the API for building Voice Assistants, with Nikhil Gupta from Vapi](https://apisyouwonthate.com/podcast/designing-the-api-for-building-voice-assistants-with-nikhil-gupta-from-vapi) Mike sat down for an interview with Nikhil Gupta, founder of Vapi, to talk about designing and building a product around Voice AI for developers. They chat about the right type of onboarding for voice assistants, building tools that API developers can test easily, and the future of Voice Assistants. --- ### From Our Community Articles written and shared in our free [Slack community](https://slack.apisyouwonthate.com/?ref=apisyouwonthate.com). ### [7 Deadly Sins of API Security Testing](https://danaepp.com/7-deadly-sins-of-api-security-testing?ref=apisyouwonthate.com) Dana is back with another killer article, this time on API Security. Dana highlights pitfalls such as poor timing, lack of API inventory, inadequate surveillance, chaotic planning, over-ambition, blame culture, and over-reliance on tools. He stresses early, varied testing, detailed API knowledge, structured planning, iterative testing, collaboration, and thoughtful tool use. Dana also linked his [talk](https://www.youtube.com/watch?v=fSUP6HRMzh4&ref=apisyouwonthate.com) if videos are more your thing. ## Support APIs You Won't Hate When you become an member, you'll get access to members-only content while directly supporting our work. Your support helps us to keep making resources for the API community. [Become a member today](https://apisyouwonthate.com/#/portal) Thanks so much to our members Kin L, Juxt, Vedran C, Alex R, Nolan S, Frank, James D, and Bill D. Your support means the world to us! Until next time, Alexander, Phil & Mike ### Designing the API for building Voice Assistants, with Nikhil Gupta from Vapi URL: https://apisyouwonthate.com/podcast/designing-the-api-for-building-voice-assistants-with-nikhil-gupta-from-vapi/ Last updated: 2024-06-17T12:43:41.000Z ## Chapters - 00:00 Introduction and Guest Introduction - 00:52 The Future of Voice AI - 01:56 Wapi's Elevator Pitch - 03:27 Challenges and Opportunities in Voice AI - 05:11 Building Voice AI Solutions - 11:00 The Genesis of Wapi - 16:39 API Design and Developer Experience - 21:23 Docker vs Kubernetes: A Developer's Perspective - 22:28 Hello World with Wapi: A Hands-On Experience - 23:34 The Developer Experience: Making APIs Accessible - 26:09 The Future of Dev Tooling with AI - 28:43 Use Cases and Real-World Applications - 33:23 Technical Deep Dive: Building with Node and Kubernetes - 37:59 Team and Future Plans - 39:15 Conclusion and Final Thoughts ## Show notes - [vapi.ai](http://vapi.ai/?ref=apisyouwonthate.com) - [github](https://github.com/vapiai?ref=apisyouwonthate.com) - [openapi spec](https://docs.vapi.ai/api-reference/openapi?ref=apisyouwonthate.com) - [twitter](https://x.com/Vapi%5FAI?ref=apisyouwonthate.com) - [linkedin](https://www.linkedin.com/company/vapi-ai/?ref=apisyouwonthate.com) - [blog](https://blog.vapi.ai/?ref=apisyouwonthate.com) - [docs](https://docs.vapi.ai/introduction?ref=apisyouwonthate.com) [hyperbound.ai](http://hyperbound.ai/?ref=apisyouwonthate.com) \- call bots to practice selling #### Transcript \[00:00:00\] \## Introduction and Guest Introduction \[00:00:00\] \*\*Mike Bifulco:\*\* Welcome back to APIs you won't hate. My name is Mike Biko, one of the co-hosts of the show, and I am sitting down for an interview this afternoon with someone who I'm really, really excited to chat about. You, you may know if you've been listening to the show long enough for hearing me p along on the internet for long enough that I. \[00:00:15\] Worked for a while on the Google Assistant team in the world of voice tech and voice software and the land of assistance and iot and automation, all that stuff. That is one of the reasons that I'm super excited to, to chat with this founder. And the other reason, of course is that Nikhil Gupta is joining me from Wapi. \[00:00:31\] He's. Also a Y Combinator founder. So we're, we're we have many sort of related parallel lives seemingly going on here. I'm super excited to talk about what you're building. The pitch on the website, the, the H one on your site says Voice AI for developers, which I think paints a lot of story for a lot of people listening. \[00:00:46\] But Nikhil, thanks a ton for joining. I appreciate you being here today. How are you doing? \[00:00:50\] \*\*Nikhil Gupta:\*\* Thanks for having me, Mike. Excited \## The Future of Voice AI \[00:00:52\] \*\*Nikhil Gupta:\*\* to chat about Wapi and so curious to hear kind of the stories of voices like Google Assistant and what's happening \[00:01:00\] there Now, everyone but everyone is wondering now too, if you have any needs to share with us that would be great. And it's super fascinating the, the whole world around us, just like. It's gonna change, right? Like the way we talk to com, like the way we interact with computers, the, maybe we interact with like microwaves. I think that's all about a change in the next like five years. I like the fabric of like interaction. And I think that was pretty clear to people when they saw that G four oh demo. So it \[00:01:26\] decided to get into it. Yeah. \[00:01:28\] \*\*Mike Bifulco:\*\* of course. Yeah. So we're, we're a couple weeks out, out from GBT four Oh which is the O stands for O Omni. Channel Omnis Omni. Yeah, just omni in general. So this is the first like multimodal GPT from OpenAI which I think has really changed the way a lot of people are thinking about the software. \[00:01:45\] And, and so we're definitely in a changing landscape, especially since, so I, I left Google Assistant two years ago at this point. And at that time things were very different. But before we get into that, let's start with you here. \## Wapi's Elevator Pitch \[00:01:56\] \*\*Mike Bifulco:\*\* Give, give me first of all the elevator pitch for, for Wapi. What's \[00:02:00\] the, the, the story? \[00:02:00\] What do you sell it to people as? \[00:02:01\] \*\*Nikhil Gupta:\*\* Yeah, sure. The elevator pitch is, yeah, voice effort, developers. The you wanna add voice spots to your phones, website apps, we provide like a modular interface that's easy to use, easy to configure, and get you up and running very quickly. The, the, the, I guess like the, the, the favorite thing for developers is like the modularity of it. You don't like open ai, you wanna use Entropic, you can use Entropic. You don't like 11 Labs. You can use play hd. There's like different options for like transcribers hey, you want to like integrate, like make Zapier, like tools, like multi assistant, like all that kind of configurability that I think exists a lot out there for like chat you can bring to voice. And the future I guess, is pretty clear to people, right? When you call into a restaurant, most likely you'll be talking to an ai, and that's kind of what we're building. The, the, the bricks and pieces the \[00:03:00\] shovels to help you get that up and running in a day, in a couple hours instead of like months. \[00:03:06\] \*\*Mike Bifulco:\*\* sure. Something that would've sounded like an absolute miracle just a few years ago, maybe even a few months ago. If you think about it, \[00:03:12\] Definitely compelling for, for anyone working in a world where your end users are especially consumer focused. I'd imagine there's lots, lots to be said there, but in service industry, that's really interesting. \[00:03:22\] And customer service of any sort myriad use cases for that. That's, that's super cool. \## Challenges and Opportunities in Voice AI \[00:03:27\] \*\*Nikhil Gupta:\*\* I mean, I think one thing I want to highlight there is like, I think everyone dreads like calling into United, right? Like, like at t, like tax like IRS, but I, it's pretty conceivable that we would love calling them in the future. Because they, they'll be like, they have like such great ai, they're so patient, they have so knowledgeable and you just call them in and like they can immediately answer like, what's happening with their flight? \[00:03:51\] Like, can they like cancel, move it around. So there is an interesting kind of change there too. Like one, the experience improves, but does it also \[00:04:00\] mean like the, the world went from like phone calling to like chat \[00:04:03\] because like these, it, it's very expensive for these companies to like host voice experiences because it, it involves humans in the background where it's like chat could be more structured. And now it's like what is the inherent desire for people? They wanna do, use a voice or chat. But that I think is also gonna be interesting in the future where you can just call into at t and it'll be like a great experience. Which is weird to think, but it. \[00:04:29\] \*\*Mike Bifulco:\*\* Sure. Yeah. I love the one word you used there, which is really interesting to me, is patience. Like having an operator on the phone who's patient is kind of also a pretty big game changer. I, I don't think people think of it this way, but usually if you call in customer support somewhere, or even if it's a restaurant for example, you call in and speak to someone. \[00:04:46\] They're a complicated person with a busy day and probably lots going on, and you're almost certainly not the first chat they've had that day and. I would imagine many customer service people have had negative experiences just about every day with someone who's mad about something. And the nice thing about \[00:05:00\] building a voice agent programmatically is that they don't get tired. \[00:05:03\] They might not experience that, and if they're kind and polite and all that, it really changes the way that people interface with things. That's super interesting. \[00:05:09\] \*\*Nikhil Gupta:\*\* I, yeah. Yeah. \## Building Voice AI Solutions \[00:05:11\] \*\*Nikhil Gupta:\*\* And maybe I can talk a bit about, like, you know, right now, if you were to kind of try to create this future maybe you wanna build a company, maybe you wanna build, like a small agency or maybe you're a small business, like there's kind of different personas here of like who would, who would wanna use, like why ai? There's a couple options, right? Like you can take 11 labs deep Graham and open AI off the shelf and like try to switch them together. That process will take you at least a couple weeks, like if not months. Once you have stitch it together you would have to kind of figure out how to like scale that system because these are what's interesting about like multimodal and like, I guess voice specifically as soon as you enter like rounds outside our text is these are stateful long lived jobs and maybe this \[00:05:58\] technical center, but. \[00:06:00\] When you hear stateful long lived, I think that's like alarm bells go off for people. It's like, oh, that sucks. Like, it's like, but that's the reality, like where these GPUs are running, which contain the context of the conversation. So ultimately when a person is calling, they have to be pinned to something like a resource in the backend. \[00:06:20\] So scaling that out as a system is like a very complex undertaking, and that's kind of the value we bring in where if you harken back to like. Twilio or Stripe, right? Like early days payments could be done where you just had to fill like this ENT form with like Wells Fargo and like you could get set up like to accept payments on your website. Same for like Twilio, like if you wanted to send a text, like it's totally possible you could. Had a contract with like Verizon at and t. But Twilio just offered like, here's an API give me the number you wanna call, gimme the text you wanna send them done. \[00:06:55\] So that kind of ease of use is what we are trying to bring to like \[00:07:00\] voice AI world where here's the number I wanna call, or here's my number and here's the position prompt I wanna use. And that's it. Done. \[00:07:09\] \*\*Mike Bifulco:\*\* I, what I love about this is that it represents such a big change for, for this world. So clearly I'm gonna go pretty, pretty deep and nerdy on the, the things that have changed in voice tech in the past few years. But thinking back to the dawn of the first voice assistants the I, I'm reticent to say their names on a recording because I'll kick off actions on people's phones around the world. \[00:07:29\] But you know, Apple's, assistant and Amazon's and, and Google's are all the initial programming interfaces for those were almost quite literally like, imagine a tree structure for the types of conversations you want to have and build that tree for everything anyone could possibly say. And have escape hatches for everything. \[00:07:46\] And that's changed so dramatically by now that AI is way, way better at picking up with the context of the conversation and inferring things. And even like if you stutter into an ai, it won't freak out. It can understand that as well. It's, it's super interesting and you \[00:08:00\] really are at the dawn of something where people can build very compelling things without a lot of work without having to catalog the whole conversation. \[00:08:07\] \*\*Nikhil Gupta:\*\* Yep. Yep. \[00:08:08\] It's exactly like, that's, I think, the common thing we hear. So there's it, it's, it's interesting because like when we started Vai about a year ago, we were doing something else before, and when we started Vai, we were expecting that there's a new platform being built kind of the same way that iPhone kind of ushered in this like apps era. \[00:08:27\] And then no one predicted like Uber would be the thing that like. Was a killer app. We were expecting like voices capability, that thing unlocked as a platform. There would be like some really cool apps. And I still believe that, you know, like there'll be completely crazy apps, infrastructure like applications out there that are voice experience based. \[00:08:46\] And I think we are seeing that in learning a little bit. They're, they're hitting multimillionaire or I know they're hitting like. They're on track to hit like, you know, a billion dollar companies or multi-billion dollar, even ars. But that was like our kind of \[00:09:00\] pieces where we wanted to. I'll be this kind of infrastructure for this new platform where a lot of like new experiences are gonna be formed around voice. But the, to tie in what you were saying, oh, a lot of use cases and surprising amount of like volume, like an insanely large volume exists in like the old world of telephony. And. In like systems that people are built, like people have like teams of like 5,000, like they're designing like these like flows, these conversations. And clearly that's like shifting to like prompt, like here's the system prompt. And making better bots faster is like a, is a thing that resonates with like this crowd of like, telephony, which we are seeing a lot of adoption is I, I feel like, I don't know, I'm like hopeful. I really want, like, new experiences, new interfaces. \[00:09:51\] That's why we have like, for example, like a Python, SDK, but you can like, set up like a AI toy with, it's not used \[00:09:58\] as much as like \[00:10:00\] the polyphony stuff, but like I would love for that stuff to come. You know, as things happen. \[00:10:06\] \*\*Mike Bifulco:\*\* Some of this I feel like people need to, I. Come to the understanding that there is this like creative space where, where you can build things. Now that used to be much harder to do. And the fact that someone could pick up a Python, SDK and in a few hours build a pretty impressive demo is like not necessarily obvious. \[00:10:23\] Even if you've used chat GPTs, you know, new, new audio features in the interactivity built into that from their app, it, it still feels kind of like magic is happening on your device. It's, I feel like it's that new, you know. \[00:10:34\] \*\*Nikhil Gupta:\*\* it could be, it could be, it could be the, the, just the newness of it. I also wonder like how much of it is like there's a bit of like stasis, I think because open air is moving so fast \[00:10:42\] that people are like not sure what to build because AJ is around the corner, right? Like, what would be, what even is the point of like building anything. \[00:10:49\] It's totally fair. But like at the same time, everyone knows there's like so many huge opportunities. I'm just like not seeing them being capitalized as I guess as much which is, which is interesting. \[00:10:59\] \*\*Mike Bifulco:\*\* let's \[00:11:00\] take a step back. \## The Genesis of Wapi \[00:11:00\] \*\*Mike Bifulco:\*\* You mentioned a minute ago that, that you're only about a year old into building this product, or you've only been building it for about a year. Can you tell me about how Vapi got started, like what you were doing beforehand? Where and, and what was the genesis of all this? \[00:11:13\] \*\*Nikhil Gupta:\*\* Yeah. Yeah. So we were building ai, meaning note taker and it was called Silver Power. It was great. Hit profitability and we kind of realized the next milestone for us was like to get it to like 10 mil error, which is the usual, you know, milestone when you hit that kind of number. And we just, like, we didn't wanna spend our twenties like selling meeting software. It \[00:11:37\] was. Not the most challenging not the most like forward-looking impactful thing we could be doing. We just felt like there's like a bigger technical challenge we could take on and that like we were engineers and like, we just love technical problems. So started stumbling around on like finding what we could do. The, this my co-founder one day in his. \[00:12:00\] Depression of like pilot pivot, depression made this thing because we over like like over the course of like our startup, we have spent around like 30 to 50 KI think on like coaching, like founder coaching. We \[00:12:13\] started at the company when we were like right outta college immature. Didn't know how to work with emotions, communicate the, the usual thing of like founder complex and worked with a coach who was saved our company. Like he helped us articulate what we need from each other and \[00:12:32\] worked through ourselves and our own mind, and that that coaching was transformative. So transformative. \[00:12:39\] We were like, oh, that, like when, so my co-founder Jordan, he created that same experience basically with an ai where that AI could like help him. Reflect back his thoughts, his emotions, kinda like a Yeah. Therapist. \[00:12:52\] The experience got better and better. But ultimately we realized there's still like, like the company that's focusing, that's focusing on like, \[00:13:00\] making this experience better. It needs to be that needs to be the company itself. Like we cannot take out the problem of like making a great voice experience and making it like a great therapy. But, those have to be separate companies. And we are like, oh, that's actually kinda great because we are, we love technical, we are technical, we love technical problems, we love technical customers. Like we, we just wanna work with startups and like really technical people. That it was just like a perfect alignment where oh yeah, like this is, this feels right, like making APIs that feels right for people. \[00:13:32\] And that's how we got into it. \[00:13:34\] \*\*Mike Bifulco:\*\* sure. Yeah. I think, I think especially if you've had some coaching around like trying to reach scale and get to a point where if, if at that point you had taken on investment, like you really need to tackle some gargantuan problems. And to your point before about not wanting to spend your twenties selling meeting software like. \[00:13:52\] You would need to sell all of the meeting software to really hit the scale that a lot of investors are looking for. And that has happened, right? Zoom certainly had a, a bit of a rocket \[00:14:00\] launch over the past few years, but you know, how many times can that happen? And almost the meta problem here is, is also really interesting too. \[00:14:06\] Yeah, that, that's really fantastic. I. \[00:14:08\] \*\*Nikhil Gupta:\*\* Yeah, you're right. Like there's like, you know, something about like the space you're in which limits, like, what you can do as a feeling. I, I do find like, I, I, I go between my, in my mind between, like, I think the, the tam is never the market. The tam is the founder, which is like, \[00:14:23\] but you wanna just keep finding new ways to like, go, like expand Tesla, you know, like robotics now it's like, it says random as hell. Like SpaceX became starlink. Cool. So I, I do find like the tan isol as a founder, but yeah, like focusing, like for us it was just like, we wanna spend our time solving hard technical problems and offering great experiences for developers. \[00:14:45\] \*\*Mike Bifulco:\*\* There's a lot of wisdom in your words. It sounds like you, you've probably been through, like ingested the coaching in a way that was really healthy and have lived through a, a hard pivot with another founder. Having been down this. Startup wrote a few times myself. I've had the good, bad and middle experiences \[00:15:00\] of, of you know, many lifetimes I feel like wrapped up in building startups. \[00:15:03\] It's always interesting to hear someone who, who acknowledges that challenge as part of building the, the company too. That's, that is a big thing that we don't talk about a lot in general, I think. \[00:15:13\] \*\*Nikhil Gupta:\*\* Oh yeah. I mean the, we've been trying for like four and a half years now. Like at it, \[00:15:20\] We were Winter 21\. \[00:15:21\] \*\*Mike Bifulco:\*\* Good for you. \[00:15:23\] \*\*Nikhil Gupta:\*\* It is kind of, I, I, I mean, I think there's something to, like, something to, like, I think PJ and Michael, lets to say this where pg where it's like what sometimes like founders are founders because they don't have anything else to do. \[00:15:38\] Like I, I don't know what else I would do. \[00:15:41\] \*\*Mike Bifulco:\*\* Yeah. Agreed. \[00:15:42\] \*\*Nikhil Gupta:\*\* yeah, so, so we just gotta figure it out. That's, we \[00:15:47\] had another choice but to figure it out and that's where it's been a lot of fun. So I think for people, I guess kind of wondering, like develop, I'm guessing obviously very technical, wondering like what to do. \[00:15:58\] I, it is, \[00:16:00\] I guess like the, the, the seed, the seed of like, that feels so good. I know I'm spending my time serving people I like a lot. \[00:16:11\] And that is enjoyable in itself, like no matter what happens to Wapi. So that like the feed of advice that people give around, do keep doing the things you really like, that you feel like especially passionate about. \[00:16:20\] And like that the moat or competitive advantage will come from that because you \[00:16:25\] just have like so much experience, like so much love for that thing. So much love and serving that thing. \[00:16:30\] So that's been \[00:16:31\] \*\*Mike Bifulco:\*\* It has to be a, a problem you can care about for 10 years, you know, not just a, a 18 months or something like \[00:16:36\] \*\*Nikhil Gupta:\*\* yeah, a hundred percent. And like. \## API Design and Developer Experience \[00:16:39\] \*\*Nikhil Gupta:\*\* Like the, when I think about like, the common thing I hear, and maybe this could be a bubble, is that people love our API. It's like people \[00:16:47\] love, like the design of it. It's like it's easy to use. It makes sense. The validation is great. And that's just like, I'm like really in know about like the little details where like it \[00:16:57\] has to be like, for example, like. \[00:17:00\] Singular, not plural. Like it has to be like consistent in this format. There has to be like the same crud interface to all of these things. It has to be a noun. Like all these best practices that I'm sure people that I'm talking to here know about. But it's like, you know, it's surprising to me when I look at other APIs, it's like, just like, what was the design process here? \[00:17:18\] Like, what the fuck was going on in that head? Like making this interface like, like makes no sense to me. Where. Having this opinion of like a good developer experience, like really caring about that gives me joy. So that \[00:17:32\] I've been a lot of fun \[00:17:33\] building. \[00:17:34\] \*\*Mike Bifulco:\*\* you and I are just meeting for the first time, but I came into the world of APIs by way of studying user user experience and human computer interaction. And so being, being deeply embedded in the world of design, one of the things that I often say to people is that learning design is interesting because anyone can learn it. \[00:17:48\] It's not something you're born with, \[00:17:50\] but the curse of it is as you learn these rules and as you learn the, the. Things that break the rules. You see it everywhere, right? So like by focusing on building an API that follows all these \[00:18:00\] rules and uses plurals correctly and is thoughtful about the way interfaces are presented, you start to see all of the inadequacies of your own product, also other products. \[00:18:07\] And they become more and more like little red irritations across your life. And, \[00:18:12\] \*\*Nikhil Gupta:\*\* Yeah. \[00:18:12\] \*\*Mike Bifulco:\*\* it's a blessing and a curse I think. \[00:18:14\] \*\*Nikhil Gupta:\*\* What's your favorite API you think out there? \[00:18:16\] \*\*Mike Bifulco:\*\* So I'm, I'm definitely biased here because I also used to work at Stripe, so I was on the Derell team there. \[00:18:21\] \*\*Nikhil Gupta:\*\* Yes. Fair. \[00:18:22\] \*\*Mike Bifulco:\*\* obviously Stripe is the gold standard in many ways to work with. And I've, I've, you know, through the course of my career, especially working in developer advocacy for a while, played with many APIs, small and large to kind of build demos and show people how to do things and. \[00:18:36\] I think a a So Stripes, API is wonderful. The documentation is incredible. The team is top notch, like super cool to see, but there are also smaller companies making things that are really interesting to use as well. Lately I've been looking at weather APIs for various purposes for my current company for craft \[00:18:51\] \*\*Nikhil Gupta:\*\* Yeah, yeah, yeah. \[00:18:52\] \*\*Mike Bifulco:\*\* the most thoughtful APIs that I encounter are ones where I don't have to become a meteorologist to understand how to use it. And that's, that should be the bar for everything. Like \[00:19:00\] my, my lowly self should be able to get to your API and understand what's going on, you know? \[00:19:04\] \*\*Nikhil Gupta:\*\* Yes, yes. Yeah. Yeah. And like yeah, like that's. Also would love to hear from people. It's like, like I always wonder like when people show up, like like we have a swagger, right? Like they show up to this. Does everything just make sense? Like it just, if you \[00:19:17\] just to look at the top level concepts that exist in the API, does everything come together? Yeah. Strip is great because you don't strive is very painful, but it rewards you like you go through the pain \[00:19:28\] and because it's so complex. But everything kind of does make sense. I think that's like where. A lot of like teams like lose, I guess, like you, the more complexity you add can everything still seem coherent? Like as, as you learn? Does the world, does the world of your, does your world make more sense or does they make less sense because there's contradictions everywhere, like different ways to do things? Or is it just like there's one way to do thing and like, yes, it's very complex, but like if you learn everything, it'll make sense. \[00:19:59\] \*\*Mike Bifulco:\*\* \[00:20:00\] Totally. \[00:20:00\] \*\*Nikhil Gupta:\*\* does a great job. Great job of that for sure. I, I think there's something about like probably like, I don't know if like AWS counts, but I think a Ws \[00:20:11\] is like is also kind of the same thing where, pretty complex, like very hard to work with. But like once you learn it, it's pretty okay. \[00:20:19\] Everything makes sense. Actually, you know what my favorite API is Kubernetes. \[00:20:24\] That's, that's Kubernetes? Yes. Kubernetes. I, I might even say I like it better than Stripe. \[00:20:30\] \*\*Mike Bifulco:\*\* Oh, that's interesting. That's a new take for me, for sure. Yeah. \[00:20:34\] \*\*Nikhil Gupta:\*\* It is, I find like the documentation when you read through it, like they, they read your mind. I think maybe that the problem is that Stripe has too many customers. \[00:20:44\] \*\*Mike Bifulco:\*\* Hmm. \[00:20:44\] \*\*Nikhil Gupta:\*\* So when, when you, when you enter Stripe documentation, like they're trying to walk you through how to handle like subscriptions, they're trying to walk you through how to handle the invoicing. They're trying to walk you through how to handle usage based billing. And they try their best and it's \[00:21:00\] great. Kubernetes, you enter the doc and they'd already know what you're trying to do. They know that you're \[00:21:05\] trying to deploy something that can scale, so they talk you through different concepts. Like the first thing they'll talk about is here's pods, here's services. If that's, it's just the documentation gives me joy reading it, \[00:21:19\] like I, it's a conversation with like, the person who wrote it is \[00:21:22\] awesome. \## Docker vs Kubernetes: A Developer's Perspective \[00:21:23\] \*\*Mike Bifulco:\*\* I, I wonder if you'll think of this as a hot take. But I, I see Kubernetes documentation and the, the way that they describe things in stark contrast to the way that Docker's developer experience is and was for a very long time. My, my feeling on Docker and figuring out how to make Docker work for, for a really, really long time was Docker can't tell you what Docker is, and you have to figure it out by trial and \[00:21:45\] \*\*Nikhil Gupta:\*\* No. Yeah, I think Doctor has excellent like dev interface, like, you know. \[00:21:51\] \*\*Mike Bifulco:\*\* Yeah. Yeah. \[00:21:52\] \*\*Nikhil Gupta:\*\* Docker form makes it really easy to like, get things up and running. Like Kubernetes is harder to get up and running. But yeah, like, and \[00:22:00\] Docker logs for example, like everything sort of works, but yeah, like it's, they have no clue what they're doing. \[00:22:05\] It's kind of the vibe that you get for sure. \[00:22:07\] \*\*Mike Bifulco:\*\* Right. Yeah. You, you kind of have to know that Docker compose and docker's form exist to be able to figure out what they do. And it's not obvious what either does, you know, from the name alone. So let's talk about your, your use case. \[00:22:19\] So you've, you've spent all this time building a, a great API. What is it like to use wapi? Like, what, what does hella world look like? \[00:22:25\] \*\*Nikhil Gupta:\*\* Yeah. The hello world is and I'm curious to get your thoughts on it. \## Hello World with Wapi: A Hands-On Experience \[00:22:28\] \*\*Nikhil Gupta:\*\* The hello world is you go to a dashboard, you don't even touch the API first, right? You go into the dashboard and the first thing you'll see is like and prompt to create an assistant. And you, you select like there's some templates options there. \[00:22:42\] Like, Hey, I wanna make like a game NPC, or I wanna make like a customer service. I wanna make a, on my scheduling. You click one of those assistants, you get a template, and after that one click you should be able to, there's a button, right, that will show which like talk to assistant. And that is like \[00:23:00\] kinda the hella world there. \[00:23:00\] It's like you have quit an assistant that you like. And you can talk to it. And the, what you'll see front, right front and the center is like, what's the configuration? Configuration of the assistant. It's like, oh, like my name is not Domino's. My name is like Pizza Hut. You know? It's like, welcome to Pizza Hut. \[00:23:18\] Like that's changing \[00:23:19\] that, that, that should give you the first like satisfaction of like putting, cutting the system and being able to talk to it. Then you, you know, the dev experience, which is like, okay, where do you wanna put this assistant? Do you wanna put it on a phone number? Do you wanna put it on your website? \[00:23:32\] Do you wanna put it in your app? \## The Developer Experience: Making APIs Accessible \[00:23:34\] \*\*Nikhil Gupta:\*\* I guess like that's where APIs and the experience, like dev experience enter. I haven't figured out the hello world for it, but that then you would go to our documentation and then they're kind of like. It's like, you know, choose your own journey. \[00:23:46\] It's like I'm gonna create a phone number. Then you go to this doc, you wanna create like a put on a website, go to this doc. \[00:23:52\] \*\*Mike Bifulco:\*\* I think it's an interesting case because there's a lot to be said for being able to just poke around with a thing and try it, especially with the case of what you're \[00:24:00\] building. Kubernetes is a great example. If I know that when I get the API calls correct for Kubernetes, that I'm going to have something that does all this nice orchestration for me, and the, the outcomes of that are clear from the onset. \[00:24:11\] But for the product you're building, the outcome is like. A little more ambiguous and perhaps less clear in that, okay, if you execute these calls correctly, you now have a voice service. Is that any good? Right? Like the first thing you need to sell is, is the outcome of all this dev work going to be valuable? \[00:24:25\] And I think inverting the experience in that sense is honestly probably a really good choice. \[00:24:29\] \*\*Nikhil Gupta:\*\* That's like smart. Yeah. That's pretty smart observation. I've never thought about it that way. All I know is like as a developer, I think the thing that probably like now I know like I have to do Stripe, right? But like, I, I, wish Stripe just gave me like more joy, like before I even get started, like, do I, will \[00:24:46\] stripe even work for me? I think it takes me a long time to figure that out, that answer to that question for most dev tooling. Whereas I think we are trying to answer the question like, yeah. Bobby, will this work for you? This is, here's a assistant, and then it's your, the, the job is to \[00:25:00\] just figure out how to like, get that assistant in the right place that you need, which we have all the AP ideas for. \[00:25:05\] But that I, I think about the law too where like, it's like, what is like in my, like where dreams, like what would I love as a developer from this experience? You know, it's like, what is like the, the, the most beautiful, amazing experience coming into Wapi? I don't have an answer to that question, but like, yeah. \[00:25:21\] The hello world is don't code. Just go in, \[00:25:24\] try something and then if it looks good, get, get, get cracking. \[00:25:29\] \*\*Mike Bifulco:\*\* I think it's pretty compelling, and I, especially for this audience that we're talking to. So for API developers, you get two things out of the box before writing your first line of code. One is you can go and try the thing, right? You, you go to the dashboard, do that, do your own experience. But naturally, I think most of the people listening to the show will also go to your API. \[00:25:47\] Reference, right? Go to the docs themselves and see what's there. And like, there's a lot of good, juicy stuff in there that's, that in itself is exciting. And you can kind of see some of the thought and, and usability that comes outta that straight away. And to your credit too, there's a handy \[00:26:00\] little button on a lot of pages on your site, maybe all of them that says ask ai where you can maybe get some answers, some questions answered pretty directly too, right? \[00:26:07\] \*\*Nikhil Gupta:\*\* Yeah, that's awesome. \## The Future of Dev Tooling with AI \[00:26:09\] \*\*Nikhil Gupta:\*\* Like too, where I think now this like dev tooling, like, like there's, so, I think there's gonna be so much, like, so much rich experience around like dev tooling because a lot of it more, a lot more can exist thanks to like ai. Like, \[00:26:23\] like before, I think like if I was to think about like integrating a new API, it'd be like, oh man, I don't wanna figure out the documentation. \[00:26:30\] It just seems like insane. Like, that sounds like a lot of work. Now I can go to the documentation, talk to the AI and be like, okay, like I need to do this. Can you gimme the code for it? It gives me the code I just put in my code. Like that \[00:26:40\] is transformative. \[00:26:42\] I think to like composing, like composable APIs, I think we're gonna see a. So many APIs, I think in this like next, like two, three years it's pretty, pretty exciting for me. But yeah, I think for people like, who are considering, I guess like yeah, like what would \[00:27:00\] for API builders, I guess, like the question you, the reason, the way you would, the reason you. It's like, maybe you should ask yourself if, like, what are you trying to build? \[00:27:09\] Like, are you, A lot of our users are startups, right? Because they're trying to explore prototype, whether they wanna sell into like a small, like a particular niche, like Vertical sa or I guess like the, like everyone knows Activity four Oh will be everywhere. Like that experience that this showed, I think that's going to be in every phone. Every microwave, every car those \[00:27:33\] assistance. So now the question is do you wanna bring that experience? Do you wanna help get that experience deployed in different places and you can wait for four to come out, or you can use to get started on that right now. And then when we do have gbd four, we'll just plug it in and then you'll just have, won't have to do any work to switch it over. And, and then I guess it is for us to figure out like what other value we can add, \[00:28:00\] which will be like observability, tooling. Like you feel like a million calls that you're doing, are you gonna really use like the raw for opening API or are you gonna use like. Some production ready tooling around to make sure that your calls are going well. \[00:28:16\] There's a visibility, tooling, but that kind of thing is I think people, what people would use this for and that's what we are here to provide. So yeah, I think I'm drawn to people like if you think voice cool. If you think multi model is cool, give it a try. Like make something prototype. It's so easy to prototype nowadays. \[00:28:33\] \*\*Mike Bifulco:\*\* Sure, yeah. The imagination should run wild and, and go build the things that you definitely wouldn't have been able to, you know, in 2022 or, or however you wanna look at that. \## Use Cases and Real-World Applications \[00:28:43\] \*\*Mike Bifulco:\*\* Can you tell me about maybe an interesting use case or two that you've seen? With api, I. \[00:28:47\] \*\*Nikhil Gupta:\*\* sure. I can tell you the UCSA. Yeah. WC this is our first customer that we worked with. They do sales training. They're called hyper bound. And you can go to their website. It's really fun. It's \[00:29:00\] called hyper bound. I don't know the actual domain, but so hybrid, it will show, it should show up and you can call these boss to practice selling. So the founder, you can imagine, like, it's pretty learning how to sell is like a pretty key skill and pretty hard. You can practice on real humans, but then you're leasing losing real deals. So with practicing these bots, and these are bots are really hard to sell to. Like they would like hang up you on you all the time. So that, that's an interesting kind of role play is a very interesting use case that we see where like having AI to represent like hard conversations, having represent like sales training. So that's like one category. The other one that's pretty obvious is customer service, where people call into this, this restaurant, it's like, I want pizza here, or I wanna book a reservation. \[00:29:46\] Like, here's an availability for the reservation. That kind of like function calling, tool, calling use cases are another, what are other popular ones? \[00:29:55\] \*\*Mike Bifulco:\*\* I am. I'm curious you mentioned your founder created a coach bot early on. \[00:30:00\] Do, do you or your co-founder have any that you use for yourselves? Maybe internally, externally, personally, I. \[00:30:04\] \*\*Nikhil Gupta:\*\* I, I use for testing my, my Kill bot. \[00:30:07\] But I don't know if I do, I have like I mean, there's one thing which is I don't like the experience of like CGBT, like voice. \[00:30:17\] As much, 'cause I mean, right, right now they haven't deployed it before, I think in production for voice and being able to kind of brainstorm because like I'll get stuck on a problem and I just need to \[00:30:31\] like, talk to something. It's like. I'm, I need to think out loud with someone that's like intelligent and used, I used to do this with my co-founder all the time, and now I need to do \[00:30:40\] less with him, just with the ai. I dunno if that, what that speaks for human connection, but it's just like, yeah, just, I just loop it out and then I talk to it, but like I'm just holding it down because I'm still processing and I just like pick up a thing and it response. So brainstorming is a very common use case for me. \[00:30:56\] And yeah. \[00:30:58\] \*\*Mike Bifulco:\*\* have a I have a weekly \[00:31:00\] newsletter that I write on my, my personal site under my own name, Mike, by full code.com. I publish a newsletter for startup founders and JavaScript developers. And often I'll sit down in the morning and just brainstorm, like talk, paddle through some ideas and try and you know, expand the idea to something that kind of has a beginning, a middle and end. \[00:31:16\] And yeah, I, I agree with you. I don't think 4.0 is deployed yet to that audio experience, and it's a little irritating when it doesn't work well. The, the biggest change and the most valuable thing for that is that now it has memory context where I had previously been like, fine tuning things over and over or re-uploading transcripts. \[00:31:31\] And now, now at least that part is getting better and that's been a big helpful change. \[00:31:35\] \*\*Nikhil Gupta:\*\* That's awesome. Yeah. Yeah, exactly. \[00:31:36\] I, I think my co-founder and I have a bet around like whether as like years go by or months go by, people like. Will there be a wide widespread love for AI or widespread like skepticism around ai? My, like, it's open, open question because we will be like, oh, AI's getting better, taking my job, or a lot of negativity or responsible, like, I just think like when you look at activity four, \[00:32:00\] oh, like talking of the thing is like magical like that demo and it'll be hard not to like that experience and like want that thing. \[00:32:09\] Everywhere. So very curious to see that, that sentiment shift on ai what will happen. But thanks. Four. \[00:32:18\] \*\*Mike Bifulco:\*\* It is hard to tell, and I think that the mindset is very split right now in many directions, old and young. I think men and women have different feelings on this. I think different generationally, you know, gen Z and millennial and, and all that. There's, there's lots of different opinions on this stuff, and I think \[00:32:32\] we'll figure it out over time. \[00:32:33\] Yeah. \[00:32:33\] \*\*Nikhil Gupta:\*\* Yeah, but I guess like, I don't know if you run into this, but like, for example, my mom, she's like not very technical savvy. And for like, for example, like she needs to, wants to learn how to, like, she's figure this out now, which is great. Like how to post something on Instagram. And the process for that is like you sharing your screen and you're just kind of pointing her to a button, like press this. Talking about it, like, to me it just \[00:33:00\] seems like, like yeah. Would be so good at that. Like, I think all people would love it. Like, it's like it, it's patiently just kind of looking at her screen and like talking you through like how to like, do something \[00:33:11\] \*\*Mike Bifulco:\*\* Yeah. \[00:33:11\] \*\*Nikhil Gupta:\*\* that is pretty compelling. \[00:33:14\] \*\*Mike Bifulco:\*\* Yeah, a very a big potential for a tender and loving experience that helps you grow at your speed which is super interesting. Yeah. Yeah. So cool. \## Technical Deep Dive: Building with Node and Kubernetes \[00:33:23\] \*\*Mike Bifulco:\*\* Tell me a little bit more about how you're building Matthew. \[00:33:25\] Like what, what is underlying your your software? \[00:33:28\] \*\*Nikhil Gupta:\*\* So, yeah, we decided to use Node because the thing, so what's that? Like, why is voice harder than chat? Right. And like open A is now running into this, like getting, trying to deploy multimodal systems is because the chat it's synchronous like, it's like one thing after the other thing that after the other, the voice the person is talking, they might stopping stop again at any point and then you start talking, but they might interrupt again. \[00:33:53\] So there's like a very dynamic environment which makes it hard to write like code for it. \[00:34:00\] And so for us, like given that there's so much like, like there's, you're getting audio every 20 milliseconds. You given that there's so much activity, you clearly need the event loop, right? You need like event loop, like audience coming in, like process that is there something you wanna say? \[00:34:15\] Like send that out? If you need an event loop now for an event loop, you could try to build, use Python, you could use something else. You can make custom event loop in rust. But, but I was like, no, no is such an obvious answer for me because. I'm standing on the shoulders of Giant, like this event Loop has been optimized the heck out of it. \[00:34:38\] Like, because the entire world browsers running this thing. But the event loop is really good, basically like that. I cannot, I can't imagine all to do anything better. So we use Node to manage the asynchronous nature of our workload, which is like you're getting on audio base all the time. You, you're sending out audio all the time, and there's like so many checks between those two. \[00:35:00\] So the a harass nature node, these stack for the API interface, we use nest js. \[00:35:06\] Because Nest has like an excellent experience around like structuring, like making sure they get very opinionated interface around like how to get data moving and get, get, get operational. And then also exposing that, like, exposing that in docs, exposing that in like exposing, having validators on it. So Nest js is like kind of our choice for an opinion and framework in Node, like we can run in node. And then for the database side of things, like I love Postgres. I've always loved Postgres. It's just awesome. Never had any issues with it. And then so we use base for it, I think has an awesome managed offering that. Kind of, I think, gets to the heart of it where you have nest node. And then of course we have like, kind of a we have many different like GPU services that need to run. Like there's \[00:35:55\] things that you can often turn on, like, for example, you wanna turn on emotion detection. That's a good GPU \[00:36:00\] service. You wanna turn on Denoising, that's some other G service. So we, our cluster is like this main API gateway. Sharding to like different workers and different services for different stuff. And we use Kubernetes so as everything because That's great. Oh, one thing I think that's kind of underrated that we love is Lummi. \[00:36:23\] So if you use, you've probably used Terraform before to spin up everything. Lets you do that in your programming language of choice. And we, I love it where it's like, you can imagine like, so for example, we have many different reds that we use. We specify, we specify a list of them and then we can just literally write that code. Or cons, Redis of red. Like, it's like, before I do this, like it's like, and it'll actually do it. It will, it just works. It's crazy to me, like that, that is possible. So Lummi for orchestrating things \[00:37:00\] yeah. \[00:37:00\] \*\*Mike Bifulco:\*\* That's an unsung hero. And so I, I think you mentioned before you publish an open API spec with swagger. Is that right too? \[00:37:06\] \*\*Nikhil Gupta:\*\* Yes, yes. \[00:37:07\] Do you want me to link to it? \[00:37:09\] \*\*Mike Bifulco:\*\* Sure. Yeah. I'll make sure it's in the show notes. You, \[00:37:11\] \*\*Nikhil Gupta:\*\* Yeah. Yeah, \[00:37:11\] \*\*Mike Bifulco:\*\* welcome to drop it in here. \[00:37:13\] \*\*Nikhil Gupta:\*\* Yeah. \[00:37:14\] \*\*Mike Bifulco:\*\* That's something that our audience will be super interested in. It's we talk a lot about open API in that world here for \[00:37:20\] \*\*Nikhil Gupta:\*\* I love open a PII like, I think this is the concept of like, like I just envision, you know, everything to live in my code. I editor, like, I just create this thing and then this open API like is like kind blossoming into like this types everywhere. Kind of the, the dream world for me it doesn't exist right now as much. \[00:37:39\] I don't think there's like as yeah, it, it getting there, but open a open API is awesome. \[00:37:45\] \*\*Mike Bifulco:\*\* Yeah, it's, it is super helpful. We we have lots of hard opinions on, on when and how and why to use it around here. And this is one of the things I like about this community is I'm always learning from people smarter than me about, you know, tools and things available in that world too. Cool. So, Nikhil I'm, I'm curious what's next for you? \## Team and Future Plans \[00:37:59\] \[00:37:59\] \*\*Nikhil Gupta:\*\* \[00:38:00\] Yeah. For us, I think more observability, more monitoring kind of like going into Wapi and then being able to see how your calls are doing knowing that they're doing well. That is kind of like what's next for us, like giving people security that like, like, their voice, AI is doing the work for them. \[00:38:20\] I think that that would be a big thing going forward too, where like, you have all these agents, but like how can you be sure that they're actually doing what you want them to do? \[00:38:27\] \*\*Mike Bifulco:\*\* Yeah, making the black box a little less black is a very, very valuable feature especially when your business relies on it. If I'm standing up a customer service endpoint of some sort, I want to know that it's doing great customer service. \[00:38:38\] \*\*Nikhil Gupta:\*\* So for us it's \[00:38:39\] like all about like that, that satisfaction of knowing it's, it's working well. \[00:38:44\] \*\*Mike Bifulco:\*\* Super, super cool. so we've talked about how you build things. We've talked about yourself and your co-founder. How big is the team right now? \[00:38:50\] \*\*Nikhil Gupta:\*\* We are a team of like six right now. \[00:38:52\] And definitely always looking for the next great hire. If you love making, if you love \[00:39:00\] voice ai and are curious about like this whole world of modality and getting the deployed. Which we feel really passionate about and like making great experiences for developers to make it easy. We get it deployed everywhere. Yeah. I would love to chat. \[00:39:13\] \*\*Mike Bifulco:\*\* Yeah. Perfect. That's great. \## Conclusion and Final Thoughts \[00:39:15\] \*\*Mike Bifulco:\*\* And so just, just to wrap things up then, where can people go to find. \[00:39:19\] \*\*Nikhil Gupta:\*\* Ai, we, we are kinda everywhere where you can find us on Twitter, LinkedIn, ai. We have a blog. Again, if you're a developer and you love docs, you know, this is actually, this was a learning for me. People don't go to our website. Like they'll just go straight to go to our docs. Like, they don't go to the dashboard, they don't go to anything. \[00:39:35\] They, there's like, oh yeah, just like read through the docs. I'm like, oh, cool. That's that's, that's interesting. So docs, we have ai, just Wapi V be \[00:39:46\] the. \[00:39:46\] \*\*Mike Bifulco:\*\* have definitely found yourself in the right place to get to people who want to go straight to the docs and read. Of course, will make sure that there's links to everything you just mentioned in the show notes. \[00:39:55\] \*\*Nikhil Gupta:\*\* Thanks. \[00:39:55\] \*\*Mike Bifulco:\*\* Nikhil. Thanks so much for joining today. It's been a real pleasure chatting with you, and I'm, I'm really excited \[00:40:00\] to get in and poke around with some voice assistance myself. \[00:40:02\] Please, please feel free to join us anytime if you have more news to share or things that you're interested in chatting through and sharing with the audience. We'd love to have you back. Thanks a ton for joining today. I appreciate it. \[00:40:12\] \*\*Nikhil Gupta:\*\* Having, ​ ### Trees and Surving Other People's APIs URL: https://apisyouwonthate.com/newsletter/trees-and-surviving-other-peoples-apis/ Last updated: 2024-06-05T12:05:24.000Z Phil joins the [Leanpub Frontmatter Podcast](https://www.youtube.com/watch?v=KxDT3kXS82w&ref=apisyouwonthate.com) to discuss his and Mike's book, "Surviving Other People's APIs." Let's be honest—other people's APIs can sometimes be quite sketchy. So, prepare for an insightful episode that will enhance your skills as an API developer and consumer. Are you curious about the writing process? Phil dives into the details, sharing his experiences and explaining what it's like to use Leanpub. Phil also spends a lot of time working for his charity, Protect Earth and gets the chance to talk about it. He delves into their critical work and highlights some of the challenges in the green space. \-- Alexander ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2023/12/newsletter-2.jpg) ## Your Ultimate API Governance Tool Elevate your API game with API Insights. Visit [apiinsights.io](https://apiinsights.io/?utm%5Fsource=newsletter&utm%5Fmedium=apis-hate-sponsorship&utm%5Fcampaign=api%5Finsights) and experience instant analysis and detailed breakdowns, all designed to enhance your API's performance, security, and design. Join the API governance revolution [Get Started Now](https://apiinsights.io/?utm%5Fsource=newsletter&utm%5Fmedium=apis-hate-sponsorship&utm%5Fcampaign=api%5Finsights) ## The API Round-Up Our monthly round-up of articles, news and updates from APIs You Won't Hate and around the web. ### [API Testing Tool](https://chapar.rest/?ref=apisyouwonthate.com) A native API testing tool built with GoLang, currently in beta, offers robust features for working with REST APIs. Future updates will include support for additional protocols outlined in the roadmap. The tool also supports dark mode and has a meaningful name with an interesting backstory. ### [Your API Shouldn't Redirect HTTP to HTTPS](https://jviide.iki.fi/http-redirects?ref=apisyouwonthate.com) A short blog post on why you should not be redirecting HTTP to HTTPS and instead returning an error response. Some APIs already do this or disable HTTP entirely; however, the post lists a bunch of APIs that do not. Don't forget that if an API key gets sent over HTTP, it is not encrypted. ### [GraphQL Yoga](https://the-guild.dev/graphql/yoga-server?ref=apisyouwonthate.com) Who doesn't like practising yoga while writing GraphQL queries? Yoga focuses on a quick and easy-to-set-up GraphQL server. Yoga runs everywhere and delivers a smooth developer experience, allowing you to write, test and validate queries in an IDE. ### [Learn HonoJS](https://www.youtube.com/watch?v=gY-TK33G6kQ&ref=apisyouwonthate.com) Have you heard of HonoJS yet? I won't shut up about it, so you should have by now. If you have been hesitant to try out this new JS web framework, you can now do so with this learn-with-me video. HonoJS lets you quickly deploy an API to Cloudflare, so it's worth checking out. ### [Minimal GraphQL Client](https://javascriptweekly.com/link/154992/bbfce21b92?ref=apisyouwonthate.com) A minimalistic GraphQL client for Node.js and browsers, designed for scripts and simple applications. Offers a lightweight, promise-based API that works well with async and await and supports TypeScript. --- ## APIs You Won't Hate The latest from the team at API's You Won't Hate. ### [API Glossary](https://github.com/apisyouwonthate/api-glossary?ref=apisyouwonthate.com) Phil has compiled a glossary of terms (a shared vocabulary) to avoid confusion when creating educational material, tooling, and other content. In the OpenAPI community, people use many different terms to mean the same thing. To be honest, that is a problem in all of tech. Hopefully, Phil's attempts at some unification will help. ### [Surviving Other Peoples APIs](https://apisyouwonthate.com/books/surviving-other-peoples-apis/) As featured on the Leanpub podcast, APIs You Won't Hate Cofounders Phil & Mike have been producing a book to help API Developers learn to collaborate with and understand the perspective of their APIs' consumers. The book is available through Leanpub, and a portion of each sale supports Phil's climate charity Protect Earth. ### [Documenting PHP APIs with OpenAPI](https://apisyouwonthate.com/blog/documenting-php-apis-with-openapi/) What are the best practices in writing OpenAPI contracts while we're working with PHP? From high level recommendations down to code examples, these guides should help you out, but where to start...? ### [How to use JSONPath with OpenAPI](https://apisyouwonthate.com/blog/jsonpath-and-openapi/) JSONPath is a query language that can be used to extract data from JSON documents, and it's becoming increasingly useful in the OpenAPI ecosystem. This guide shows how to use them together. ## Support APIs You Won't Hate When you become an member, you'll get access to members-only content while directly supporting our work. Your support helps us to keep making resources for the API community. [Become a member today](https://apisyouwonthate.com/#/portal) Thanks so much to our members for supporting APIs You Won't Hate: Kin L., Juxt, Vedran C, Alex R, Nolan S, Frank G., James D., and Bill D. Your support means the world to us! Until next time, Alexander, Phil & Mike ### Documenting PHP APIs with OpenAPI URL: https://apisyouwonthate.com/blog/documenting-php-apis-with-openapi/ Last updated: 2024-06-05T11:45:11.000Z Over on Bump.sh I’ve been writing infinite guides helping people learn how to get OpenAPI out of (or into) their various programming languages and web frameworks, and I wanted to share a summary of that work for the PHP crowd over here with some links. What are the best practices in writing OpenAPI contracts while we're working with PHP? From high level recommendations down to code examples, these guides should help you out, but where to start? ### **Code-first and Design-first, with OpenAPI** There’s [a lot of strong opinions](https://apisyouwonthate.com/blog/api-design-first-vs-code-first) around the best approach here, but whichever approach you take, what matters is that you get the right tools and practices. The main thing is getting an OpenAPI document to work with. OpenAPI stands as the optimal method for publishing comprehensive API descriptions, accessible to both humans and machines alike. These guides will lead you through generating API descriptions, utilizing them for documentation, and delving deeper into how OpenAPI accelerates the development of robust APIs. ### **Generating OpenAPI docs from your code** If you’d like to create an OpenAPI documentation for an existing API, one approach is to pop some annotations into your code, helping flesh out what the requests and responses are, covering validation rules, examples values, and as much description as you can fit into your code. [This guide will cover](https://docs.bump.sh/guides/openapi/code-first-laravel/?ref=apisyouwonthate.com) using [Swagger-PHP](https://zircote.github.io/swagger-php/?ref=apisyouwonthate.com) you can do this with old-school “docblock” style code comments, or PHP 8+ users can use the newer Annotations approach, and either way you’ll be able to export OpenAPI from your actual API. The guide is written for a Laravel PHP audience, but Swagger-PHP works just fine regardless of what framework is being used, and for native PHP applications too. Combine this with Continuous Integration and Bump.sh CLI’s `deploy` command, and you’ll never have out-of-date API documentation again. ### **Building robust API descriptions and APIs** Writing API descriptions before writing code is a great practice because you can describe the API you would like to build, not just the one you have built. The two benefits of this approach are: 1. Why not leverage OpenAPI to handle request validation instead of writing a bunch of code to do that. 2. Why not let OpenAPI handle the contract testing for you, so you don’t have to write a bunch of code to do that as well. At first, it sounds wild, but when you realize how many places API developers repeat the same API contracts, you start realizing OpenAPI is already covering that, and you can start treating it as a single source of truth for most of the API lifecycle. [You'll walk through](https://docs.bump.sh/guides/openapi/design-first-laravel-php/?ref=apisyouwonthate.com) how to use the `membrane/laravel` and `hotmeteor/spectator` packages to handle validation and contract validation, and finally have confidence you have control over the docs vs code drift problem. Hopefully these guides help, but if that’s not, we’ve also knocked up sample OpenAPI documents available on public GitHub repos, AND deployed on Bump.sh. Find them at the end of the guides. The main takeaways emphasize the critical role of API contracts and the OpenAPI specification. Regardless of whether it's your initial step or a catch-up process, major programming languages offer a wide array of (Open Source) tools to maximize its benefits. Creating an OpenAPI description isn't about adding extra work; it’s preventing technical debt accumulation, and for the API Design First workflow it even helps speed up the development process. ### How to use JSONPath with OpenAPI URL: https://apisyouwonthate.com/blog/jsonpath-and-openapi/ Last updated: 2024-06-05T11:40:39.000Z A few years ago most API designers, developers, and technical writers would have had very little reason to bump into JSONPath, but its starting to get more and more relevant as more tools and standards start relying on it. So what is JSONPath, what is it used for, and how can you get up to speed with using it? JSONPath is a query language that can be used to extract data from JSON documents, which at first might not sound very exciting, but remember… OpenAPI is just a JSON (or YAML) document, so you can use JSONPath to poke around in OpenAPI and do various things. You can use [JSONPath for OpenAPI Overlays](https://github.com/OAI/Overlay-Specification?ref=apisyouwonthate.com), to patch OpenAPI documents with extra documentation content, code samples, or whatever else. You can use [JSONPath in Spectral](https://docs.stoplight.io/docs/spectral/d3482ff0ccae9-rules?ref=apisyouwonthate.com#given) to write incredibly advanced linting rules which can power your [automated API Style Guides](https://apisyouwonthate.com/blog/automated-style-guides-for-rest-graphql-grpc/). You can even use [JSONPath in AWS Step Functions](https://aws.amazon.com/blogs/compute/using-jsonpath-effectively-in-aws-step-functions/?ref=apisyouwonthate.com). JSONPath is popping up all over the the place these days, and if you work with OpenAPI it’s definitely a handy tool to have on your belt. ## How does JSONPath Work? JSONPath is one of several query languages which will let you filter, query, and traverse through a chunk of JSON, not just to pull bits out, but to navigate complex data structures, with syntax for getting into specific array indexes, filtering through an objects properties or array values before continuing on to its children. Here’s a sample JSONPath from the RFC. ``` $.store.book[?@.price < 10].title ``` > Anyone familiar with XPath in XML will be thinking “hmm, this looks pretty familiar!” and you’re spot on, JSONPath is [inspired by XPath](https://www.rfc-editor.org/rfc/rfc9535?ref=apisyouwonthate.com#inspired-by-xpath). If you’ve never heard of XPath no worries, we’ll start from scratch here. To see how this works we’ll need some JSON to run it against, so here is an example of some JSON from the RFC. ``` { "store": { "book": [ { "category": "reference", "author": "Nigel Rees", "title": "Sayings of the Century", "price": 8.95 }, { "category": "fiction", "author": "Evelyn Waugh", "title": "Sword of Honour", "price": 12.99 }, { "category": "fiction", "author": "Herman Melville", "title": "Moby Dick", "isbn": "0-553-21311-3", "price": 8.99 }, { "category": "fiction", "author": "J. R. R. Tolkien", "title": "The Lord of the Rings", "isbn": "0-395-19395-8", "price": 22.99 } ], "bicycle": { "color": "red", "price": 399 } } } ``` Now when you run that through any sort of JSONPath tool, you could expect to see these results. ``` [ "Sayings of the Century", "Moby Dick" ] ``` ### Syntax [#](https://docs.bump.sh/guides/openapi/jsonpath/?ref=apisyouwonthate.com#syntax) There is a whole lot of syntax to learn, but once you figure out the constituent pieces you can start to compile them into really advanced queries. | Syntax | Element Description | | ------------------ | --------------------------------------------------------------------------------------- | | $ | root node identifier (Section 2.2) | | @ | current node identifier (Section 2.3.5) (valid only within filter selectors) | | \[\] | child segment (Section 2.5.1): selects zero or more children of a node | | .name | shorthand for \[‘name’ \] | | .\* | shorthand for \[\*\] | | ..⁠\[\] | descendant segment (Section 2.5.2): selects zero or more descendants of a node | | ..name | shorthand for .. \[’ name’ \] | | ..\* | shorthand for ..\[\*\] | | 'name' | name selector (Section 2.3.1): selects a named child of an object | | \* | wildcard selector (Section 2.3.2): selects all children of a node | | 3 | index selector (Section 2.3.3): selects an indexed child of an array (from 0) | | 0:100:5 | array slice selector (Section 2.3.4): start:end:step for arrays | | ? | filter selector (Section 2.3.5): selects particular children using a logical expression | | length(@.foo) | function extension (Section 2.4): invokes a function in a filter expression | *Overview of JSONPath Syntax, from* [*RFC 9535*](https://www.rfc-editor.org/rfc/rfc9535?ref=apisyouwonthate.com#tbl-overview)*.* ### Examples If that isn’t making too much sense, here are some examples to help you visualize. | JSONPath | Intended Result | | ------------------------- | ----------------------------------------------------------------- | | $.store.book\[\*\].author | the authors of all books in the store | | $..author | all authors | | $.store.\* | all things in the store, which are some books and a red bicycle | | $.store..price | the prices of everything in the store | | $..book\[2\] | the third book | | $..book\[2\].author | the third book’s author | | $..book\[2\].publisher | empty result: the third book does not have a “publisher” member | | $..book\[-1\] | the last book in order | | $..book\[0,1\] | the first two books | | $..book\[:2\] | the first two books | | $..book\[?@.isbn\] | all books with an ISBN number | | $..book\[?@.price<10\] | all books cheaper than 10 | | $..\* | all member values and array elements contained in the input value | *Example JSONPath Expressions and Their Intended Results When Applied to the Example JSON Value, from* [*RFC 9535: 1.5\. JSONPath Examples*](https://www.rfc-editor.org/rfc/rfc9535?ref=apisyouwonthate.com#name-jsonpath-examples)*.* By combining these bits of example syntax together you can do amazing and powerful things with JSONPath, so let’s look at how to do those amazing things in OpenAPI. ### JSONPath & OpenAPI Take an OpenAPI document, like the [Train Travel API](https://github.com/bump-sh-examples/train-travel-api?ref=apisyouwonthate.com). ``` git clone github.com/bump-sh-examples/train-travel-api cd train-travel-api ``` Then install [jsonpath-cli](https://www.jsware.io/jsonpath-cli/?ref=apisyouwonthate.com) just so we can try some things out. ``` npm install -g @jsware/jsonpath-cli ``` Optional, if you’re working with YAML, you might want to convert from YAML to JSON in the CLI too. ``` brew install yq yq eval -o=json openapi.yaml > openapi.json ``` Don’t worry this is just for playing around, all of the tooling that uses JSONPath will support YAML without bodges like this. Let’s just get on the same page for this guide. ### Querying OpenAPI with JSONPath Once you have a JSON file to work with, we can use the `jpp` command, pass in a JSON/YAML document, and provide a JSONPath expression to query the document for specific parts. ``` $ jpp --pretty '$.info' openapi.json [ { "title": "Train Travel API", "description": "API for finding and booking train trips across Europe.", "version": "1.0.0", "contact": { "name": "Train Support", "url": "https://example.com/support", "email": "support@example.com" }, "license": { "name": "Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International", "identifier": "CC-BY-NC-SA-4.0" } } ] ``` In this example `$` refers to the root JSON document, then `.info` is using dot notation to access the `info` key in that object. We can get a bit more advanced, and pull up a list of paths. ``` $ jpp --pretty '$.paths.*~' openapi.json [ "/stations", "/trips", "/bookings", "/bookings/{bookingId}", "/bookings/{bookingId}/payment" ] ``` This uses the `.*` syntax which is basically grabbing all children of the paths object, then using `~` to grab the keys instead of the values. What sort of query language would JSONPath be if we could not do queries? Let’s pull up a list of paths which are a `get` or a `post`, but ignore all the `put`, `patch`, `delete`, etc. ``` $ jpp --pretty '$.paths[?(@.put || @.post)]~' openapi.json [ "/bookings", "/bookings/{bookingId}/payment" ] ``` ## OpenAPI Overlays powered by JSONPath One of the main uses for JSONPath will be for working OpenAPI documents, often by technical writers or other folks in the API governance space to check or improve OpenAPI documents. Overlays are a list of actions, which make up a “target” which is a JSONPath, and an operation of either “update” or “remove”. Let’s look at an update command. ```yaml # overlays.yaml overlay: 1.0.0 info: title: Overlay to customise API for Protect Earth version: 0.0.1 actions: - target: '$.info' description: Update description and contact for our audience. update: description: > A new and much more interesting long form description, which has all sorts of Markdown, or more specifically [CommonMark](https://commonmark.org/) which is _like_ Markdown but **better**, because it's an actual standard instead of a series of sometimes vaguely consistent conventions. Anyway, this is a good place to write all sorts of helpful stuff, link to other getting started content, link to where people can find access tokens, or even paste some code samples for getting your first API request off the ground. contact: name: Support Team url: https://example.com/contact email: support@example.org ``` This overlays file is pointing to the JSONPath target `$.info`, then updating the object with the new bits of OpenAPI for `description` and `contact`, as per the [OpenAPI specification](https://spec.openapis.org/oas/v3.1.0?ref=apisyouwonthate.com). This can be handy for improving the quality of all sorts of descriptions, not just info, and for popping in support team contact information if the API developers inevitably forgot to mention that sort of thing. Instead of using those `yq` or `jpp` tools we grabbed just to practice, we can use the [Bump.sh CLI](https://github.com/bump-sh/cli?ref=apisyouwonthate.com) which has support for Overlays built in, and thankfully it’ll work just fine with YAML or JSON. ``` npm install -g bump-cli bump overlay openapi.yaml overlays.yaml > openapi.new.yaml ``` If we were to run that overlay on the Train Travel API, the resulting `openapi.new.yaml` would like like this: ```yaml openapi: 3.1.0 info: title: Train Travel API description: > A new and much more interesting long form description, which has all sorts of Markdown, or more specifically [CommonMark](https://commonmark.org/) which is _like_ Markdown but **better**, because its an actual standard instead of a series of sometimes vaguely consistent conventions. Anyway, this is a good place to write all sorts of helpful stuff, link to other getting started content, link to where people can find access tokens, or even paste some code samples for getting your first API request off the ground. version: 1.0.0 contact: name: Support Team url: 'https://example.com/contact' email: support@example.org license: name: Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International identifier: CC-BY-NC-SA-4.0 # snipped ``` When combined with more advanced queries you can start to get really specific with bits of the OpenAPI document you’d like to update, enabling all sorts of random use cases like cleaning up the servers list for publishing an API Catalogue, removing Development and Staging servers not accessible or relevant to API consumers. ```yaml # openapi.yaml openapi: 3.1.0 servers: - url: http://localhost:3000 description: Development - url: https://api-staging.example.com description: Staging - url: https://api.example.com description: Production ``` An overlay can target the servers array with `$.servers` then query through them with `$.servers[?(@.description=="Development" || @.description=="Staging")]`, which is looking through objects in the array, and looking through the children for `description: Development` or `description: Staging` using basically JavaScript syntax. The Overlay for this would combine that JSONPath target with `remove: true` operation like this: ```yaml # overlays.yaml overlay: 1.0.0 info: title: Overlay to customise API version: 0.0.1 actions: - target: '$.servers[?(@.description=="Development" || @.description=="Staging")]' description: Remove Development and Staging servers but leave anything else. remove: true ``` That would leave this resulting OpenAPI. ```yaml # openapi.yaml openapi: 3.1.0 servers: - url: https://api.example.com description: Production ``` Then the Developer Experience folks decide to roll out a Mocking or Sandbox experience, where consumers can play around with requests without actually triggering real emails, real data, or spending real money, but how can we show everyone where that is? Do we have to go and pester all the API teams to add it? Nope, just add another action. ```yaml # overlays.yaml overlay: 1.0.0 info: title: Overlay to customise API version: 0.0.1 actions: - target: '$.servers[?(@.description=="Development" || @.description=="Staging")]' description: Remove Development and Staging servers but leave anything else. remove: true - target: '$.servers' description: Let everyone know about our amazing new hosted mocking/sandbox server. update: - description: Sandbox url: https://api-sandbox.example.com/ ``` ### Leaning more about JSONPath JSONPath made it to IETF “proposed standard” RFC status in 2024 ([RFC 9535](https://www.rfc-editor.org/rfc/rfc9535?ref=apisyouwonthate.com)), but before then it was in a similar position to [Markdown in the days before CommonMark](https://philsturgeon.com/state-of-markdown/?ref=apisyouwonthate.com), in that there are a few different variations of JSONPath as a concept. - [JSONPath “The Blog Post”](https://goessner.net/articles/JsonPath/?ref=apisyouwonthate.com) \- Written by Stefan Gössner in 2007. - [jsonpath.com](https://jsonpath.com/?ref=apisyouwonthate.com) \- An online evaluator which as far as I can tell matches the blog post. - [JSONPath-Plus](https://www.npmjs.com/package/jsonpath-plus?ref=apisyouwonthate.com) \- A popular (but now abandoned) fork which expands on the original specification to add some additional operators. - [Nimma](https://www.npmjs.com/package/nimma?ref=apisyouwonthate.com) \- A fork of JSONPath Plus created by the Stoplight team for Spectral to handle more advanced use cases. A list of caveats can be found here. Then to further compound this confusion, all of the implementations have different support for certain features, and have filled in the grey areas differently due to their own interpretations and community requests. The amazing [JSONPath Comparison project](https://cburgmer.github.io/json-path-comparison/?ref=apisyouwonthate.com) has collated all of the differences into a massive test suite and published the results, which was really helpful in shaping the new standard. Hopefully this will help tools converge, and we can forget all about this incompatibility. For now, try to follow the RFC 9535 syntax, and use tooling which lines up with that syntax. Unfortunately that means not using `jsonpath.com`, and even the `jpp` CLI tool we used earlier is JSONPath Plus, which has a few differences to the RFC… The Bump.sh CLI `overlays` functionality is JSONPath RFC 9535 compliant, and if you spot any valid RFC JSONPath syntax not working as expected please [create an issue on GitHub](https://github.com/bump-sh/cli/issues?ref=apisyouwonthate.com) so we can get that sorted out. ### AI APIs: is this the End of Coding? URL: https://apisyouwonthate.com/newsletter/ai-apis-the-end-of-coding/ Last updated: 2024-05-15T14:34:52.000Z Over the past year, we've witnessed a surge in AI tools, particularly in applications built around ChatGPT. Most of these wrappers of ChatGPT don't stand up to the wonderous things we are told we can do; however, there are plenty of AI-powered API tools I use today to increase my own output. Amidst the hype, concerns about AI replacing software engineers have surfaced. For a balanced perspective that cuts through the market speel, check out this article from [The Pragmatic Engineer](https://blog.pragmaticengineer.com/ai-developer-marketing-stunt/?ref=apisyouwonthate.com). \-- Alexander ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2023/12/newsletter-2.jpg) ## Your Ultimate API Governance Tool Elevate your API game with API Insights. Visit [apiinsights.io](https://apiinsights.io/?utm%5Fsource=newsletter&utm%5Fmedium=apis-hate-sponsorship&utm%5Fcampaign=api%5Finsights) and experience instant analysis and detailed breakdowns, all designed to enhance your API's performance, security, and design. Join the API governance revolution [Get Started Now](https://apiinsights.io/?utm%5Fsource=newsletter&utm%5Fmedium=apis-hate-sponsorship&utm%5Fcampaign=api%5Finsights) ## The API Round-Up Our monthly round-up of articles, news and updates from APIs You Won't Hate and around the web. ### [NotificationAPI - Add Notifications to any software in 5 minutes](https://www.notificationapi.com/?utm%5Fsource=apisyouwonthate&utm%5Fmedium=newsletter&utm%5Fcampaign=sponsoredlink&utm%5Fterm=05012024) **(Sponsored)** All-in-one user notification system for B2B SaaS products. Design, send, and monitor product notifications from one place with a powerful editor and API. ### [Describe API's](https://typespec.io/?ref=apisyouwonthate.com) Generate OpenAPI spec and JSON Schema with TypeSpec, inspired by TypeScript. This interesting library helps you maintain type consistency with one source of truth. Its validation annotations remind me of Spring Boot. It's worth checking out. ### [RESTful Web Services and HTTP: Standards and Sources](https://restful.ws/?ref=apisyouwonthate.com) Scrolling through Twitter the other day, I finally got a decent recommendation (rather than the usual nonsense I have no interest in), a compilation of Restful API resources. The site has it all: semantics, defining resources, headers, and more. It's a great webpage to keep in your back pocket. ### [Traefik 3.0 Has Landed](https://traefik.io/blog/traefik-3-0-ga-has-landed-heres-how-to-migrate/?utm%5Fcampaign=traefik-proxy-3&utm%5Fcontent=291477898&utm%5Fmedium=social&utm%5Fsource=twitter&hss%5Fchannel=tw-4890312130) [Traefik ](https://traefik.io/?ref=apisyouwonthate.com)is a tool for running APIs anywhere, and version three has just been released. The new version is a huge step forward for Traefix, adding support for WASM, OpenTelemetry, Kubernetes Gateway API, and SPIFFE. The intro blog to version three comes with a handy migration guide. ### [Express V5](https://github.com/expressjs/discussions/issues/233?ref=apisyouwonthate.com) I have mentioned this before, but Express is waking up from the dead. I wonder if the advancements by Deno, Bun, Hono and others have caused a stir in the ecosystem, pushing things forward. I hope this push does not fizzle out; so many projects are so deep into Express that migration away from it would be hard. Version five and its updates are desperately needed; we are now on the final push. ### [Node 22.1](https://nodejs.org/en/blog/release/v22.1.0?ref=apisyouwonthate.com) Hot on the tails of Node 22 comes 22.1\. If you are not up to speed with what's in Node 22, check out this fantastic[ video overview](https://www.youtube.com/watch?v=eZfLkVDJPTg&ref=apisyouwonthate.com) by Matteo Collina. ### [Rusty Go](https://borgo-lang.github.io/?ref=apisyouwonthate.com#error-handling) A new programming language that compiles to Go? It's not something I have on my wishlist, but it caught my eye, and I could not help playing around with it. Borgo is supposed to be halfway between Go and Rust, whatever that means, lol. [Hacker News](https://news.ycombinator.com/item?id=40211891&ref=apisyouwonthate.com) did seem very excited about it. --- ## APIs You Won't Hate The latest from the team at API's You Won't Hate. ### 🎙️ [Unkey: Globally Distributed API key generation, with cofounder James Perkins](https://apisyouwonthate.com/podcast/unkey-globally-distributed-api-key-generation-with-cofounder-james-perkins/) In this episide of the podcast, James Perkins from Unkey sits down for a chat with Mike Bifulco about Unkey's unique approach to delivering an API product that provides value to its users out of the gate, and the OSS story that got them here. --- ### From Our Community Articles written and shared in our free [Slack community](https://slack.apisyouwonthate.com/?ref=apisyouwonthate.com). ### [Reverse Engineering Electron Apps to Discover APIs](https://danaepp.com/reverse-engineering-electron-apps-to-discover-apis?ref=apisyouwonthate.com) Dana is off on another learning adventure, this time reversing engineering an Electron app to see what APIs it uses. As always, Dana's itch for learning leads to a great read. ## Support APIs You Won't Hate When you become an member, you'll get access to members-only content while directly supporting our work. Your support helps us to keep making resources for the API community. [Become a member today](https://apisyouwonthate.com/#/portal) Thanks so much to our members for supporting APIs You Won't Hate: Nolan S., Rüdiger S., Alex R, Vedran C, Danny S, Juxt, Kin L., and Olekshii T. Your support means the world to us! Until next time, Alexander, Phil & Mike ### Unkey: Globally Distributed API key generation, with cofounder James Perkins URL: https://apisyouwonthate.com/podcast/unkey-globally-distributed-api-key-generation-with-cofounder-james-perkins/ Last updated: 2024-05-15T14:13:56.000Z ## Show Notes - Unkey: - [https://unkey.com](https://unkey.com/?ref=apisyouwonthate.com) - OSS on GitHub: [https://github.com/unkeyed/unkey](https://github.com/unkeyed/unkey?ref=apisyouwonthate.com) - James Perkins - [GitHub](https://github.com/perkinsjr?ref=apisyouwonthate.com) - [Twitter (@james\_r\_perkins)](https://twitter.com/james%5Fr%5Fperkins?ref=apisyouwonthate.com) #### Transcript \[00:00:00\] ****Mike Bifulco:** . Hey friends, Mike from NPI. As you won't hate here chiming in from the future. I messed up during this interview there's about five minutes of this interview at the very beginning where I am using the microphone on my earbuds to record. \[00:00:11\] And you can really tell. So I apologize for that in advance. But I promise I did my best to make it better using all the tools available at my. Disposal using AI and audacity and things like that to make the audio a little nicer. It's as good as it can be roughly given the skills that I have, but I just figured I'd let you know. \[00:00:27\] The entire interview is not as miserable sounding on my end. As it seems at the beginning. So zip ahead if you need to, but otherwise enjoy the interview and thanks for watching the indoor listening. \[00:00:36\] ​ \[00:00:37\] ****riverside\_mike\_bifulco\_raw-synced-video-cfr\_podcast\_recording w\_0027:** Welcome back to another episode of APIs you won't Hate. My name is Mike Bico. I'm one of the co-founders of APIs You Won't hate. And I'm excited to sit down today for an interview. Stand up today for an interview, whatever, hop on the internet for an interview with my my old pal James Perkins to talk about what he's building. \[00:00:51\] And James, it's really nice to have you on the show today. I'm, I'm interested to hear about UN Key and your story there. Thanks a ton for joining us. How are you doing? \[00:00:57\] ****riverside\_james\_raw-synced-video-cfr\_podcast\_recording w\_0028:** Yeah, I'm good. Thanks for having me. It's been \[00:01:00\] a while since we've talked. Yeah, I'm, I'm happy to be here. Happy to chat about UN key and what we're building and how we're trying to make APIs suck a lot less than they, they do most of the time. \[00:01:10\] ****riverside\_mike\_bifulco\_raw-synced-video-cfr\_podcast\_recording w\_0027:** Oh, you found yourself in the right place then I think for that discussion. James, you and I have sort of a serendipitous internet relationship. I feel like every time I turn a corner online you're poking your head around saying, hi. I, I don't remember exactly where we met, but I know we crossed paths a bit while I was at Stripe. \[00:01:23\] I. Doing developer relations there and later while you were doing developer relations and now you're starting and building a product. So I, I'd love maybe if you start just giving us, like, the lowdown on your career. How did you find your way to what you're doing now? And then let's talk a little bit about. \[00:01:36\] ****riverside\_james\_raw-synced-video-cfr\_podcast\_recording w\_0028:** Yeah, for sure. So I've been around a really long time. I think if my co-founder listens to this, I'm really sorry, but I have to do this. So I've been around for 16 years. I started way back doing Java development for a few different companies. First one was like a really small stage startup that did online registration. \[00:01:54\] So in America, if you buy a car and you get from the dealership. And they give you like a temp tag in the dealership or the \[00:02:00\] actual tags depending on where you live. That software was built by me and a, and a few people. And then I moved on to things like gambling and the lottery. Did that for a while, did digital banking, then I moved on to FinTech startups. \[00:02:13\] Tina CMS was my real break into where people really know me. That's when I started doing developer relations and doing all the sorts of YouTubes and podcasts and all the other things that I've done throughout my career. And then the real thing that everyone really knows me for was Clark which is where I've met a lot of different people across, across different industries. \[00:02:36\] I think that's how we actually met. \[00:02:38\] ****riverside\_mike\_bifulco\_raw-synced-video-cfr\_podcast\_recording w\_0027:** I think it may be, yeah. \[00:02:39\] ****riverside\_james\_raw-synced-video-cfr\_podcast\_recording w\_0028:** I think you were working on a project with Clark and you had questions and then I think, you know, bouncing around and then, you know, you moved on to, to what you're doing now. And we chatted a bit about that. But yeah, that's most of my career. And then I left Clark in September which is when the funding for Key was basically almost finished at that point. \[00:02:59\] \[00:03:00\] We, we got a few checks in, but we were still finishing up. And then that was when I went full time. \[00:03:04\] ****riverside\_mike\_bifulco\_raw-synced-video-cfr\_podcast\_recording w\_0027:** Yeah. Got it. Wow. Well, quite the career. You've, you've touched many industries. You know, like, like I guess a lot of successful developers tend to do getting your fingers deep in all sorts of problems and getting your hands dirty with all, all sorts of industries. I guess that brings us to sort of present day and maybe September of 2023, when, when un key sort of raised you round, but also started making the presence online known. \[00:03:26\] Why don't you tell us about UN key? What's the story there? \[00:03:28\] ****riverside\_james\_raw-synced-video-cfr\_podcast\_recording w\_0028:** Yeah, so on Key is essentially a. API management platform that we're kind of redefining what API management is and how it's built and how we're using it today. So if you think about API management platforms today, you're talking things like Kong or Tyke or even, you know, Google's offering or AWS is offering through gateways and things like that. \[00:03:50\] And when we were building APIs, just in general, my co-founder just built a bunch of APIs. I've done it throughout my career. We found that it was \[00:04:00\] hard. To protect that API, however that might be. Whether it was through giving people API keys 'cause you want them to be able to interact with it or rate limit that API endpoint for some reason, or it was just a bunch of things that you don't think about until you've already built the API and then it's like, oh, I need to do this and this and this, and there was no easy way to do that. \[00:04:20\] It was all or nothing. It was, you need the entire gateway to do a one API key for a user, or you need to, you know, do some sort of ridiculous amount of work to make it globally distributed or whatever it might be. Or you build it in-house, which is where we find some people are still doing. So we kind of had this idea originally to be a blog post for Clark and UPS Dash because my co-founder used to work at UPS Dash. \[00:04:49\] We were like, Hey, we could just build a blog post around globally distributed API keys that would like used Clark, and then used up dash. And then basically you link them together and then you give them an API \[00:05:00\] key and then you can kind of work with it. And after about 10 minutes of talking, I was like, no, this is just a business idea that I think there's actually a business idea that we could build. \[00:05:09\] And then the rest is really history. We, for the first commit was in June. The beginning of June, we launched at the end of June, and then everything else has been a rocket ship since then. Lots of users, lots of feedback, lots of different iterations, lots of improvements, lots of new features, you know, all the stuff you kind of do when you're in a small stage startup. \[00:05:31\] And then we raised our fund, which was the weirdest thing I've ever done. Worked in all these startups, never seen the other side, and now I understand how that works. Yeah, it was really, really crazy and it was just like you know, let's try us out and see what happens, kind of thing. And here we are today. \[00:05:46\] ****Mike Bifulco:** So the rocket ship metaphor is definitely something that sounds apt given the path you've taken to get here. And I am also living on the the, the raising money side of the founder story for the first time. So I can definitely relate to that. If you're listening to this and you haven't. \[00:05:59\] Raised \[00:06:00\] money for, for a project of your own at some point. It definitely changes the way you think about things and requires a pro, a project, a problem, an audience that can hit a certain level of scale. And so hopefully the rocket ship that you've now strapped yourself to is headed in a direction where that scale is really important really important and also \[00:06:15\] ****James:** Right. Yep. \[00:06:16\] ****Mike Bifulco:** What I'm really curious about there then is like, this is a pretty big problem space. Like building API management is not a, a you know, a small rock in project manager parlance. What was the first cut at this? Like, what, what was the first thing you built to prove that you had something that people would \[00:06:30\] ****James:** Yeah, so the first thing, so we decided to go in reverse. So with all these companies, basically doing all or nothing, it was do a gateway and then you can have these features. We decided that the best idea was to give people what they actually wanted, which was just API keys. If you're a small startup, a small business, medium sized business, maybe even a large business, you probably don't need a gateway because you can probably handle most of the infra. \[00:06:55\] If you're using serverless, for example, like you can scale your infra pretty well without \[00:07:00\] having to worry too much about the other features that Gateway brings. So the first iteration was API Keys that had. Per key analytics. So anytime a key was used, you had the analytical data that you needed to figure out, you know, was it a successful request? \[00:07:17\] Was it rate limited? Was it use, succeeded? Like a few things, but you could do it per key. So you could really deep, deep dive into your analytics as you were building. And that was the first iteration. So we had rate limiting keys that expired by a certain day. And then we had key, the keys you could attach data to. \[00:07:37\] So owner id, metadata was all an option that you could just attach to a key and we would return that data if you needed it. And it was, you could create a key, you could delete a key and you could verify key. And that was it. That was the first iteration. It had a dashboard that had very rudimentary analytical data on it. \[00:07:56\] And that, that was basically V zero that we shipped in June. \[00:08:00\] Then we slowly iterated on things that we thought people needed at the time. \[00:08:05\] ****Mike Bifulco:** Yeah. I can imagine just from that first set of features quite a bit of value, right? Possible or at least promise of value possible for folks. From everything from just proving that like, hey, someone is using my API, this person is using my API over and over is a good signal for lots of teams building API products, but also preventing abuse or things like that and, and looking at where costs are being driven up. \[00:08:26\] That's super cool. How did you, well, I, I guess you are, someone who's been on the internet for a long time. I'm curious like how you found your first audience for this. How were you \[00:08:34\] ****James:** Yeah, so we, both of us are quite known on Twitter for different various things. Like my co-founder is very well known for like open source projects and building a lot of stuff, and I'm very much known for like the YouTube stuff and Clark and a few other things. So when we launched the product, we did a single tweet. A single video, which were spaced out like a day apart. And that really like \[00:09:00\] gave us the I that actually solidified when we did the launch in June, that solidified the fact that we were gonna raise money because we were getting signals from other founders that we knew in the industry saying like, Hey, this is actually something that people need. \[00:09:15\] Because the original project was just a side project. It was just something we wanted to build together. We weren't even charging people at the time. It was all open source. It was just like a fun project for us to build over like a few weeks. And when we did the tweet, it did like 150,000 impressions. \[00:09:33\] It brought us like 700 users. Some of those users were just, you know, people checking out the platform. Some of them were actual people that wanted to try the platform. And then, yeah, just a bunch of feedback came through that about like, Hey, this is a cool idea. This is really something we need. No one's really done globally distributed API keys before, like, how does that work? \[00:09:52\] And it was very much like, oh, here's the open source project. You can go and look and see how we're building it. And then that brought in some of our first real \[00:10:00\] users and real contributions at that point. \[00:10:02\] ****Mike Bifulco:** I can imagine those things adding up pretty quickly to be the signals you would wanna see. Had you been entertaining the idea of building this into something larger? Were you just hoping to get the open \[00:10:11\] ****James:** No. Yeah, we were basically like, cool, this is a cool side project that we can work on together. Andreas and I had never met outside of the internet, like Discord and stuff like that, and we were both very happy with the roles that we had. Like I was super happy at Clark, he was super happy at Up Stash. \[00:10:25\] So we used this as more of an opportunity of just like working together, collaborating, building something, having fun. And even when we were being approached by VCs, we have a discord message that we both have the image of, that we occasionally just randomly share in our slack now, which is just us saying like, no, I'm good. \[00:10:44\] Part-times fine. Like, I'd rather do this part-time. I'm happy with my job. And yeah, we like, we like to remind ourselves of that occasionally. \[00:10:50\] ****Mike Bifulco:** It seems like, \[00:10:51\] uh, the universe had different plans for you. Signals like that are hard to ignore. Certainly. And I guess, so at this point, that was last June. This would be, we're we're recording in \[00:11:00\] early May, I guess you'd call it of 2024\. So it's been almost a year. I'm, I'd imagine you've had several chapters of existence since then. \[00:11:07\] So tell me, tell me how things sort of fared since then and where you're at \[00:11:10\] ****James:** so we're definitely a lot different than where we were. The core product still lies a lot in API keys. Except from now we have a lot more features and we're expanding into more of the API management platform in general. So we can talk about keys first and then we can talk about some of the other stuff we've done. \[00:11:27\] So the keys, now we, with the feedback that we've got through customers and just generalized people, we've added other features. For example, now you can do keys with rate limiting that are cost based. So. If you've got like a specific endpoint that you need to rate, limit cost-wise, because maybe it's an expensive resource that maybe it's an AI project and you are giving keys out, you can, you can actually say, Hey, this is gonna cost three instead of the usual one for a rate limit so that you can kind of push \[00:12:00\] around and, and make sure that like your expensive resources aren't being abused as much as like a cheap resource where you're just looking something up. \[00:12:06\] We also have keys that now have. Usage base. So you can just say this key has 100 uses. After that a hundred uses, you can no longer use it anymore. Basically, the only way that it can be reused is if the developer then goes and does an update. So it works really well for token based usage or you know, a monthly subscription that you get a hundred uses of anything like that where you really just wanna limit the amount that someone can use A key, and then we've introduced audit logging and we've introduced better analytics. So like it's more rich in your analytical data. We've added API endpoints so that you can pull the analytical data yourself and then build your drone dashboards or build whatever you need there. And that's kind of how we're fit in, in, into the key space. \[00:12:51\] So that's still the core. Real core part of UN key, and it'll always be core regardless if we, you know, once we get into gateways and things like that, it'll still be \[00:13:00\] core part of our products. And then we introduced standalone rate limiting this year. So if you've ever had like A-T-R-P-C route or something like that, that you just want to rate limit, but you don't really need public API Keys or something like that, you can do that with UN key and it will just be globally distributed. \[00:13:17\] So it'll be really, really fast. And you can choose how you want to use it. So we have two options, which is synchronous requests. So that obviously you have to wait for us to actually respond and say, Hey, yeah, they're good, or they're bad, or you can use our asynchronous version, which you just give up a small amount of accuracy. \[00:13:34\] So our anchor accuracy is like 98%, even with Async. That will just be like, yep, I, we, we believe this is good. We assume that this is good based upon like the data that we have. So you can have a really, really fast rate limiting system and that's fully globally distributed. It has things like overrides, so if you've got a specific user or maybe an IP address that they need higher limits because maybe you have a deal in place or maybe \[00:14:00\] you know it's an internal user or whatever it might be, you can manually override. \[00:14:04\] In our dashboard and say like, Hey, if this user comes through, they have higher limits. Don't use the usual ones. And you don't have to make any code changes. We'll just say, okay, yep, this person is coming through. Cool. All right, let's give them these limits instead and let's check against those. So basically it gives you this really big amount of flexibility around how you wanna rate limit. \[00:14:25\] And those are the big main features that we've added, like since the beginning. We've had lots of iterations behind the scenes of different code base and like we've changed the way that we do keys and what providers we use. And we were originally on servers and now we're all serverless. Like we did a bunch of stuff to like really optimize how fast we could we could get it. \[00:14:44\] And yeah, we're really happy with where we're sat right now. \[00:14:49\] ****Mike Bifulco:** Certainly, it's a very interesting set of features because you've added a few things that very quickly give any product that takes advantage of it, like a much more polished feel. And \[00:15:00\] from, from my perspective, they're pretty hard to build. Those are things that you would probably need. If I was going to build that for scratch for an API product, it would take. \[00:15:07\] Dozens of hours of engineering time and probably revisits every so often when I un uncover a bug. And, and is the kind of thing that in a world where a AI based products are becoming more and more valuable those features are super important. Like being able to have a customer that gets advanced access or faster access is something that's really massive. \[00:15:25\] That's, that's a, a, a great set of features. And I, I like that it's like fairly minimal, but provides, I mean, immeasurable value to, to teams that really need it. That's \[00:15:33\] ****James:** Y Yeah. Our whole focus was really around like how as a, as a developer, if I get this product, do I need everything or do I need specific things? Right. And the, the, the problem usually lies that you get everything, even though you don't want it. Then you're like, oh man, I'm paying for stuff I'm not even using. \[00:15:52\] Or man, I have to implement this because of the way that it works. We are, we are really focusing on like, Hey, here's a package for rate limiting. Just \[00:16:00\] use it. Here's a package that gives you the entire API use that, and then issue keys, create keys, do whatever you want. We want people to have the option to pick the features they want versus being forced to use features they don't need. \[00:16:12\] ****Mike Bifulco:** It makes a ton of sense and it lets people bite off a, a size of the product that they want to use and that makes sense for their business. And hopefully one of the things I like about businesses like this is that as you are, your customers are successful, you become more successful, you know, as they get more use or more. \[00:16:26\] People driving their product and revenue up. Then you're, you're benefiting as well, invariably because they're using more of on key and everyone wins. That's, that's aligning incentives, which is always a good sign too. I'm, I'm curious to hear a little bit about your first customers or maybe your first year of customers. \[00:16:41\] Like what are the use cases that stand out to you? How did you find your, find your way into people's hearts as a, a product that they should \[00:16:47\] ****James:** Yeah, for sure. I'd love our first ever customer was a crypto trading platform, which is like odd and also kind of interesting. So they were a crypto trading \[00:17:00\] platform. The name is Premier. They were building an API and we launched pricing right as they were doing their API. So we're like, Hey, we're actually gonna charge for this. \[00:17:09\] Here's the pricing. If you know, we'll, we'll happily talk if you're doing big volumes, thinking like, no one's gonna do big volumes on this platform. No one's ever heard of them before kind of thing. And they've been our biggest customer since basically day one. They do millions of requests a month. \[00:17:24\] They're pretty popular. They drive a lot of like, resources and we see them, you know, spike when crypto spikes and they dip down when crypto dips down. So we're like, we can see when runs are happening 'cause like our usage will spike and then it will dip back down. \[00:17:39\] ****Mike Bifulco:** Sure. \[00:17:39\] ****James:** they were an interesting first customer and also just an interesting customer in general. \[00:17:43\] Like thinking about how web free is all decentralized and then using a centralized system for your API keys was something that we were still struggling to wrap our head around a bit. But yeah, they've always been, they've been super happy with the platform since basically July. And then a lot of AI \[00:18:00\] customers coming in where they're building, you know, those AI products that need either API keys or, or something to kind of help them drive forward. \[00:18:09\] And it's been interesting to see there that a large majority of our actual customer base that are paying customers are in the AI space. So it's usually ai. And then we have a few Web3 customers, one of them being the crypto trading platform and a couple others. And then we have others like cal.com uses UN key to power their rate limiting behind the scenes. \[00:18:33\] Which is, they're, they're a newer, newer customer. They were fairly recent. And then we've just had some side projects where they've come in and been like, I was building an A API and it would've taken me six months to do this work, but with you guys, it took me like a week. And most of that was just like me spinning up API endpoints and then introducing on Key into that. \[00:18:53\] And that, that's basically been our customer base. And then people check us out fairly regularly, like two, 300 people a month come in \[00:19:00\] and check us out and, and play around with the platform and, and see kind of what we're doing. But yeah, that's been most of our customer base. And then we have some other open source friends that also use us for their a their API endpoints too. \[00:19:11\] ****Mike Bifulco:** that's something maybe that we've glazed over a little bit here is that you, you have a pretty interesting open source story as well that this is not a, you know, fully private closed source product. So tell me a little bit about that, the, the nature of open source and un key and sort of your \[00:19:24\] ****James:** Yeah, so un key is a hundred percent open source. Every part of our platform is there, including our landing pages. The API, there's no trade secrets that are hidden somewhere. Everything is available for someone to go and look at and learn from. The open source nature of un Anki was from day one. We made the decision that if we were gonna build something, we wanted to build it with open source in mind. \[00:19:51\] And people are seeing more and more people kind of build on open source, like build successful products. I mean, if you [look@cow.com](mailto:look@cow.com "mailto:look@cow.com"), they're a really \[00:20:00\] successful platform. And the only thing that isn't open source, I think is just their marketing site. And we just believe that open source makes better products. \[00:20:09\] It makes it easier to do security checks, right? Like if you're a big business, you can just go and look through the source code and make sure we're not doing anything nefarious or we're doing something dangerous. It makes it easier for people to collaborate on an idea. So if they have an idea that like, maybe this is a cool feature, they can create an issue and say like, Hey, I'm thinking about this for Anki. \[00:20:30\] What do you guys think? And then we have our community. We also have our team. And we can get to a point where an idea may not be the best idea in the beginning, but by the time we've all talked through iterations of it, we've come out with this next feature or an extension of a feature that could really be useful for more than just one person. \[00:20:50\] And we, we just found that we just wanted to be as open as possible. So even internally as founders, we are transparent with our team. \[00:21:00\] So they know exactly what's happening. What are we doing? What kind of deals are in the pipeline, what deals have fallen through or are successful? And then again, like things like revenue and all that kind of stuff and how much money we have in the bank. \[00:21:13\] All that stuff we're completely open about. And it just makes for better overall feeling every day knowing that nobody is in the dark and there's no secrets that are being hidden. And we believe that works for software too. Building that way. Makes it really easy for someone to come and learn or, or contribute or whatever it might be. \[00:21:32\] ****Mike Bifulco:** Clearly it does. I think you've established as, as a pretty impressive product and one that people are providing value to. Very quickly while we've been chatting, I pulled up your GitHub repo, and I know this isn't exactly a direct litmus, but it looks like you've currently got over 1400 pull requests merged into the product between, you know, now and its inception. \[00:21:50\] Which is, you know, from any number of, of contributors you know, between internally and externally. But one of the things I think that comes along with that is you end up having to do almost like crowdsourced product management \[00:22:00\] too. And, and you touched on that a little bit, but how, how do you balance, like taking feedback from open source contributors versus paying customers versus what your team internally thinks \[00:22:09\] ****James:** Yeah, we spend a lot of time talking about this just in general, so like being such a young company, being less than a year old, it it. A lot of our direction comes internally around we believe this is a feature that people want or need because that's something we want or need. Right? Like it's one of those things where you're like, I'm playing around with AI and it, wouldn't it be cool if we had this feature? \[00:22:30\] And, and so over the ti over the time period, we've had people ask for feature requests and, and whether that's through open source or being a paying customer. And it's just a fine balance of deciding whether or not it's for the. Greater good of the product, or is it a very niche thing that somebody needs? \[00:22:50\] And if it's more niche, we try and pry a bit more into what they're trying to achieve because it may be something we already have on the roadmap, but it's like, you know, \[00:23:00\] phrase in a different way or, or what they're actually asking for is not what they really want. And we found over time that we, we've, we've really got a fair balance there of like, Hey, this is a cool feature, but. \[00:23:11\] Like, here's five different edge cases where this probably wouldn't work properly. And like, what would you think about this instead? And kind of talking to the community in general. And that's been really good for us. I think it's a tough challenge and I think anyone that has an open source project, it's just tough to be like, we are not gonna build this, or we are gonna build this, but it's gonna be six months from now. \[00:23:31\] But if you wanna open a feature request and like do it yourself, like here's how you can contribute, and it, it's just tough. I think it's just one of those things, you just gotta balance it the best you can and decide what's better for the greater good of the company and, and the business side of things too. \[00:23:45\] ****Mike Bifulco:** By nature you have a lot of constituents through opening the product up, and it's usually a good thing to listen to feedback from, you know, customers and people who maybe just want to use your product aspirationally, but are blocked by, you know, something not existing. But definitely also certainly have to \[00:24:00\] balance this out as like it needs to run as a business and be a functioning business too. \[00:24:03\] There's, there's. Well, almost the curse of open source is that when you're successful, you start to get a massive amount of signal to noise problems. But the benefit of it too is when you strike the balance, like your, your product also a little bit maintains itself or maybe grows itself. And, and your team can, be more open about that, you know, like, let's have a discussion about this feature before we go and implement it. And it sounds like from your ethos of keeping everything, you know, open with the team and, and being sort of approaching the problem with open arms you can ask a few whys, like, okay, you want this very niche feature, but I. \[00:24:33\] Why, you know, and why to that and why to that until you get to do something that maybe is a little more generic or you know, I'm sure in, in some cases you're ending up saying no, but you've had the discussion and hopefully publicly too, so everyone can kind of learn from it and chime in as well. \[00:24:46\] I'm, I'm I'm, I'm impressed how quickly this has come together. Like it feels like you've got a a timeline that makes no sense to me because it feels like you're traveling through time but also a product that feels really, mature, right. Given that like there's a lot of polish in what you've built and a lot \[00:25:00\] of really thoughtful touches across your site, across documentation, things like that. \[00:25:03\] So I'm, I'm interested a little bit in hearing about the developer experience of the product and from a couple of angles. So first why don't we start with, tell me a little bit about like, the implementation side of this. I, I have a, a AI based product and I want to use un key for you know, managing my API keys. \[00:25:18\] How do I get started? \[00:25:19\] ****James:** Yeah, so it, we've tried to make on key as easy as possible with the smallest amount of implementation. So if we just use TypeScript, 'cause it's probably the easiest example because we have like official SDKs for that. Essentially the idea is you sign up for an account through, through on key. And we give you an onboarding experience depending on what you want. \[00:25:38\] So if you are looking for API keys, we, that's one onboarding. Onboarding experience. If you just want straight rate limiting, we give you a different one. But when you go through that experience, essentially what you get is something called a root key, which is basically the keys to the kingdom. You can set it to be able to do a bunch of different things, but the idea is that root key gives you the ability to create a key.\[00:26:00\] \[00:26:00\] Or update a key or delete a key, whichever one of those kind of operations that you need. So the idea would be that you have an API endpoint somewhere that basically just says, you know, when a customer clicks this button, create a key for them. And it's just a simple, a simple SDK call to like un key client dot. \[00:26:19\] Keys create and you give us some sort of information based upon that. So it's just like our API ID and then maybe you need a user id 'cause you wanna reference it to someone. We give you the ability to do that and then we return you two things. The key itself and a key id. And the key is only ever returned one time and we'll never be able to retrieve it. \[00:26:40\] We'll never be able to show it to you again. Basically. It's just a one time thing. 'cause we, the way that we store the keys securely. And in that point you give it to the customer and the customer now has this key that whatever it might do to interact with your system, then we just have another piece that's called verify key. \[00:26:57\] And Verify key takes two pieces. It's the key \[00:27:00\] itself and then the API ID that you want to verify against. And then you just make a call to that and we'll tell you whether or not they should have access. So if you've got rate limiting, for example, and maybe someone's really hammering an API endpoint and they've used. \[00:27:14\] More than you've said that they can use for rate limiting. We'll actually return back that you sh basically it's a true or false statement and we'll just return false. And then at that point you can handle it however you want. You can reject them, you can tell 'em to calm down whatever you want to do through that. \[00:27:31\] And that's basically the way that it works. And then if the user is using something like token, you know, usage based, and they're like, oh, hey, this person's actually upgraded their account. I need to update the key. We have another call that you can update key and then just give us whatever you need to update, whether that's maybe their rate limits are higher now, plus they are, they, they get more usage or whatever it might be. \[00:27:54\] And the way that On Key's built, it's built on a really simple rest, API. So if we don't have an \[00:28:00\] SDK for it, the rest API is really, really easy to use. We have an an open API spec. So if you want to use generator to generate all the endpoints you can. And we've made it as easy as possible to get started. \[00:28:12\] In theory, if you wanted to do a next GS app, for example, that had this a like an API endpoint, and then you need to verify the key beforehand, you could probably do the entire UN key implementation in. Less than 20 lines of code, including verifying your keys. It's very, very simple. We've made it as simple as possible and it's really easy to get started. \[00:28:37\] ****Mike Bifulco:** I am sure many of our listeners are thinking about the subtleties implied by everything you've just said with a small amount of code. 20 lines of code, you know, for next are probably similar for other platforms. You've just described quite a bit of. Work that is not easy to achieve, right? It's like a very humble description of a super cool product, securely issuing and reissuing keys and mapping them to access \[00:29:00\] rights and rate limiting and all that is a crazy amount of scope and like a very cool thing to be able to just kind of you know, excuse the slightly sarcastic, joke, but like NPM install and have, you know, your API managed. Like, that's wild. That's, that's such a cool thing. You also touched on something before that I think is really interesting and maybe subtle if, if folks listening haven't touched on this or needed to implement this for themselves, but usage-based API usage when it comes to pricing is a really challenging problem. \[00:29:25\] And for the reasons you were saying before synchronous versus asynchronous is a really interesting challenge. So maybe I'll set the scene here and I'm curious maybe you can describe like the problem space and why it's important. But again, let's say I'm building an API product and you've, you've paid me for access to it. \[00:29:39\] So you've given me, you know, 10 bucks a worth of API usage, and I have a usage based API call. There are a few things that can happen as you're, you're. The, the value of your credits with me drain having to do with like race conditions and depletion and refunding and all of those sorts of things with my account. \[00:29:56\] So how do you convey all of that information to end \[00:30:00\] users when you're \[00:30:01\] ****James:** Yeah, so it is very hard to do if you're not using something like UN key. 'cause usually it's like, oh, I have to use my database to basically store a token number that then I decrement and increment, and then I take the risk of like my DB being slow and now they've burst through and now they're in negative numbers. \[00:30:17\] Like how does that all kind of work? So with UN Key, we basically give you the ability to, to put in a number, whatever that number may be. Let's say they pay for a hundred uses for $10\. At that point, you have a few options with un key to decide how you want to handle the future of that key. So for example, let's say they decide, oh, this product's terrible and I wanna refund, and you give them a refund. \[00:30:43\] They technically still have access to that key when you do the refund. And so what we can do is we have a few different ways to kind of basically disable a key. So you can do it right through the dashboard. So if you're still manually billing and you're manually doing that, you can just go into the dashboard, find the key \[00:31:00\] that's linked to that user and disable it, and then they won't be able to use that. \[00:31:05\] If you are doing automation where, let's say they wanna refund and then you basically scrub their account. You can use either the delete key, if you know for sure. They're never, ever gonna come back permanently, delete the key. And now the, the key has become sort of, you know, unusable or you can update their key and just say it's disabled. \[00:31:24\] Maybe they'll come back. And then what we do to protect you from things like race conditions and things like that when you're using our key system. We don't let you do an asynchronous request. We like, you have to wait for us to actually respond. You can't just be like, I'm just gonna assume everything's fine and continue on. \[00:31:41\] And that protects you because one, it basically stops your. Anything else being processed until UN key responds. And because of the way that we work behind the scenes, essentially what you get is the correct and valid number. Like we are heavily cashed. We know exactly what the number is at the time when the \[00:32:00\] user makes the request. \[00:32:00\] So there's less of this worry of like, I have to wait for my DB to respond. Oh, they've burst through. There's no way for them to actually physically burst through that request. And then what you can do is like once they deplete and get to zero. Maybe they wanna pay you $10 more and say like, oh, I want another a hundred users. \[00:32:16\] This is super cool product. Well, you can do the same thing. So you can either manually just do an update call and say, give, update remaining, and say, give me a hundred more. Or if it's like a monthly subscription, we have the ability to refill automatically once a month. So every month, if they've billed, you can just basically say, every month, just unie, handle this for me. \[00:32:38\] I don't care about it. Just handle it. And that makes it really, really easy. And then the final piece to the puzzle is in the very near future we, we will have web hooks for you to hook into. So we can tell you this user has done this much usage if you're doing usage based billing on your side, and we can send that to \[00:33:00\] you in specific timeframes or usage base or however you wanna really do it. \[00:33:05\] And so that you can bill via like 30 day invoices. So like, let's say it's similar to us, where once a month you get a bill from us and we tell you how much you owe us. You can now do that with UN Key and not have to worry about implementing the entire thing on your stack and use whatever billing partner you want, Stripe, paddle, lemon squeezy, whatever it might be. \[00:33:23\] ****Mike Bifulco:** I suppose that means somewhere under the covers you're probably using UN key to \[00:33:26\] ****James:** Oh yeah. Un key is built on un key. That is for sure. Yeah, we, we everything that we do is powered by un key. It's always the classic trope of like, you know, Clark is using Clark behind the scenes, right? Like, if you go to Clark's dashboard, you have to log in with Clark to get into Clark. It's the same thing for us. \[00:33:42\] So when you. Bill when you like create your workspace and you create your API and you create that first root key, that root key is then using UN key to actually build you a root key that you then use for a specific account. It's very similar to that. And then like when you do a verification, we verify that key against one of our \[00:34:00\] own keys. \[00:34:01\] ****Mike Bifulco:** That creates, a paradox of its API keys all the way down, but that's, that's a great way to build the product, making sure that you're close to it and using it. I I also wanted to talk a little bit about maybe the other side of, of your product in DevX. And particularly with your background in developer advocacy it feels like you've done a really good job of, you know, making a splash and socializing this because of the reputation you built for yourself. \[00:34:21\] But I'm curious what what other I. Benefits your work in developer advocacy has, has brought to engineering a product and building something that actually used. And maybe the additional caveat that I'll add there is that I, having worked in developer experience for, for a while myself, one of the things you hear, especially while interviewing is that developer advocates aren't engineers. \[00:34:39\] Like, don't build products, can't build products. And, and I think that. Obviously is a, a trope and something that has a lot of faults to it, but it's something that also just doesn't get talked about, right? Like you have jumped from the world of talking about building for developers to building for developers, and now you've got a product that's out there. \[00:34:54\] So yeah, tell me your secrets, man. I wanna hear it all. \[00:34:56\] ****James:** I, I think, yeah, that classic trope of like. \[00:35:00\] Dev advocacy never can build a product. Is is, is something that I think was probably true when dev advocacy was new, right? Like it was just like more of an educational kind of part than it was about anything else. And less about being involved in like moving a product forward or pushing a product forward, or helping engineering or whatever, helping customers, and, and because my development background was way before I was dev advocacy, they gave me some of that skillset immediately, right? Like I could build a product and don't have to worry. But I think having the background in dev advocacy and being unafraid to be open is something that's scary for a lot of founders, right? \[00:35:45\] Like you see these cool products like one time and then you never see or hear from them again. And then like two years later you find out, oh, they, they shut the doors 'cause they couldn't get another round of funding or they just couldn't get traction in the space that they're in. And it's because you never \[00:36:00\] see them actually doing anything on either social media or blog posts or being on podcasts because they're afraid that whatever they put out in the world. That's the be all and end all. Like it's gonna be cemented in history and I can never go back on what I've said or I can never change my mind or I can never improve the products because we've said this thing. We took the opposite approach was just like, just even if we're in the middle of building sand, just post it on social media and say like, Hey, this is coming soon. \[00:36:28\] Or Hey, we built this and it's like in alpha, like, love some feedback. Then doing podcasts and, and, and blog posts about things that we have opinions on is a huge thing. 'cause you know, the internet is full of people's opinions and having our own as we've built this product. Like we have a blog post that's about you UIDs and how like people will use them wrong and like, this is how you should really try and use them and here's why. \[00:36:52\] And that post is still one of our biggest drivers to the product. And most of it comes from a \[00:37:00\] place of like, either, wow, this blog post is amazing, it's a really great read, and now I wanna know more about products. Or, these guys have no idea what they're talking about. They have no experience. Like, they're clearly wrong and like, I have a really bad opinion about this. \[00:37:14\] Right? So like you have these two opposing opposite ideas and, and, and we just don't care. We're just like, cool. Thanks for coming and checking us out. Like. That's your opinion and you know, you rightfully can have that opinion. And building all those things together kind of really helps kind of cement the products as a whole. \[00:37:34\] And then having this engineering background and, and not being afraid to like be wrong about something. I am nowhere close. My ability is nowhere close to my CTO's ability in engineering. So there's times where I'm just like, ah, I just put this together and see what happens. And I push PR up and he is like, what are you doing? \[00:37:53\] Like, now this makes sense? And like, okay, well let me go and try again. Like, kind of thing. And not being afraid to, to, to \[00:38:00\] learn as we go. Because a lot of the stuff that we built, we had no idea what we were doing in the beginning. We were like, we think this is gonna work, but we won't know until we try. And, and just being unafraid of that. \[00:38:11\] And I think dev advocates are the perfect people for that too. A lot of the time, you spend a lot of time building content or education or talking to people or whatever it might be, and there's a big chance that whatever you put out in the world, either people aren't gonna like, they hate the, like they really love it, it's not very good, or it miss them up completely and being able to like get back up and be like, oh well, like we did our best with that one. \[00:38:38\] Let's move on to another post or another video, whatever it might be. That's the same I attitude that, that you need to have, if you're gonna build a product, it's just like, this might not work, but let's give it a shot and see what happens. \[00:38:51\] ****Mike Bifulco:** I love that, that's like a masterclass in both developer advocacy and startup founder school, you know, kind of slammed together. There is I feel like \[00:39:00\] the, the, one of the, the hidden superpowers of developer advocacy is learning to embrace opening and working in the open and in public and, and benefiting from the. \[00:39:09\] Internet's magical superpower of every time you share something online, someone smarter comes along and tells you why you're wrong. That's fantastic. Like as, as long as you look at that as, Hey, I can learn from this. That's, that's a superpower, that's a skill for sure. And I think a lot of founders, I, I mentor a handful of startup founders and kind of give advice to folks who are like early on in the journey. \[00:39:26\] And one of the things that I think people miss out on a lot. That a lot of building a product, especially a developer focused product, is that you've gotta love building it and rebuilding it and iterating and like every step along the way is part of the thing. It can't just be like, I did it once. It's done. \[00:39:39\] It's never done. And that's, that's the fun part, right? You've gotta learn to love that too. It seems like you've married the two together really well, and certainly your team has too. We actually haven't really talked about your team much apart from yourself and your, your co-founder. But how big is the team right now? \[00:39:51\] What, what are what's the scale you're at? \[00:39:53\] ****James:** five people total, including myself and Andreas. We have two engineers and then one design \[00:40:00\] engineer who does both design and also a lot of front end work for us. \[00:40:05\] ****Mike Bifulco:** Yeah. Got it. Yeah. A small but mighty team building something that's really, really pretty wild. Are you expanding at the moment? Are \[00:40:11\] ****James:** are not hiring right now. We just hired just recently, actually April tonight, design engineer came on board. So we'll probably be hiring again core, kind of close to the end of the year. We definitely don't wanna expand too far, too early and then be like, well, well I don't know what you can do right now. \[00:40:29\] 'cause we're like, we're all doing these tiny pieces, you know, or the, you know, the, the scary worry of being a founder is like, what if we run out of money? I don't wanna have to tell you. Sorry, like you just got a job with us now we have to let you go. But yeah, we'll definitely be hiring kind of close to the end of the year again which is when we started our first hire round. \[00:40:48\] So probably like October, November, we'll probably start hiring again. \[00:40:52\] ****Mike Bifulco:** Yeah, that's great. I will make sure by the way, to include links to UN Key and to James and to all of the social, everythings in the show notes, \[00:41:00\] as well as actually the blog post you mentioned before. Before I let you go, I'm curious to hear what's next, like, what are you thinking about, what's the next exciting feature \[00:41:06\] ****James:** Yeah, so we have a couple of different things. So we're currently working on a special project for AI users. Essentially the idea is semantic caching for ai. So the idea is that we can save you both money and also just speed and latency by using the same tech that we use behind the scenes but give you the ability to, to really have some really nice responses if it's something that's already been asked before and you don't have to go and ask open AI the same thing or some variation of that. \[00:41:37\] We're working on that right now. That should be released probably in June. If all goes to plan. And that will be our first real, like push into kind of the gateway esque product. And then the really big thing in the next probably 12 months will be gateways in some variation, but gateways that developers can use instead of gateways that you need a whole DevOps team for. \[00:41:59\] We wanna make \[00:42:00\] it really easy. A few clicks, gateway, or here's an API call and now I have a gateway. That's the idea. We're pushing towards the same devex experience that you get with the rest of the UN key products. And then we just have a few more things coming. Like we have a cash package coming that's not really related to On Key, but is where we can help you do a lot of like the heavy cash work in CloudFlare workers and things like that that we use. \[00:42:23\] Now we're actually building that out to be just a standalone package that you can use for whatever you want and don't have to use on key. And then, yeah, it's just some generalized improvements, more analytical data for you guys to ingest and play around with. Ingestible audit logs is coming so you can actually push stuff to on key and then do something with it afterwards. \[00:42:42\] Lots of things just to try and help API development in general \[00:42:46\] ****Mike Bifulco:** There's never any shortage of new features to build, especially when you've got an audience of \[00:42:50\] ****James:** Yep. There's never, there's never a time where we're like, Hmm, what should we build now? We definitely know that there's things to build and, and we're happy to build them. \[00:42:58\] ****Mike Bifulco:** james, that's a really good spot to be in. I \[00:43:00\] appreciate you coming to hang out. Last thing where's the best place to find you online and what's the address to find UN \[00:43:05\] ****James:** best place to find me is probably Twitter. It's where I spend most of my time. It's just James R. Perkins. You can find me there. And then for Un Key, you can now find [us@unkey.com](mailto:us@unkey.com "mailto:us@unkey.com"). We just launched the new site that you see today, which will be whatever anybody looks at right now. We launched that a couple of weeks ago with un key.com domain. \[00:43:23\] ****Mike Bifulco:** it looks great. Congratulations on the launch James Perkins. I am thrilled to be able to talk to you and super excited to hear about your story. Please come back anytime when you've got launches to chat through or things you wanna ask the community, or if you're just interested in getting yelled at about open API specs, I can do that too. \[00:43:38\] Thanks so much for joining me today. I really \[00:43:39\] ****James:** Thanks, Mike. Really appreciate you. Bye. \[00:43:43\] ****Mike Bifulco:** Soon. ### Killing Your Products URL: https://apisyouwonthate.com/newsletter/newsletter-1-5-24/ Last updated: 2024-05-01T12:28:08.000Z It's not API-focused, but I read an article titled "[The Man Who Killed Google](https://www.wheresyoured.at/the-men-who-killed-google/?ref=apisyouwonthate.com)" this week, and it left such an impression that I had to share. The article and the linked emails were fascinating, but they reminded me how competing priorities, internal politics or individuals can easily ruin products. It's worth the read. However, feel free to skip and jump straight into this issue's round-up of API goodness. \-- Alexander ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2023/12/newsletter-2.jpg) ## Your Ultimate API Governance Tool Elevate your API game with API Insights. Visit [apiinsights.io](https://apiinsights.io/?utm%5Fsource=newsletter&utm%5Fmedium=apis-hate-sponsorship&utm%5Fcampaign=api%5Finsights) and experience instant analysis and detailed breakdowns, all designed to enhance your API's performance, security, and design. Join the API governance revolution [Get Started Now](https://apiinsights.io/?utm%5Fsource=newsletter&utm%5Fmedium=apis-hate-sponsorship&utm%5Fcampaign=api%5Finsights) ## The API Round-Up Our monthly round-up of articles, news and updates from APIs You Won't Hate and around the web. ### [NotificationAPI - Add Notifications to any software in 5 minutes](https://www.notificationapi.com/?utm%5Fsource=apisyouwonthate&utm%5Fmedium=newsletter&utm%5Fcampaign=sponsoredlink&utm%5Fterm=05012024) **(Sponsored)** All-in-one user notification system for B2B SaaS products. Design, send, and monitor product notifications from one place with a powerful editor and API. ### [Optic Acquired by Atlassian](https://www.useoptic.com/?ref=apisyouwonthate.com) Optic, a tool for automating OpenAPI developer workflows, has been acquired by Atlassian. Optic will be integrated into Compass, Atlassian's developer experience platform. Optics had some excellent CLI tools, so hopefully, this will accelerate their offering. ### [Breaking APIs with Naughty Strings](https://danaepp.com/breaking-apis-with-naughty-strings?ref=apisyouwonthate.com) There is a list of naughty strings out there that can cause severe issues for your API. Dive deep into this very naughty list and discover how to use them to find validation failures, buffer overflows, or injection attacks. ### [Stripe Workbench](https://www.youtube.com/watch?v=cfejnY-sTtQ&ref=apisyouwonthate.com) Stripe has released a new tool, currently in beta, that gives you a fantastic UI to help you debug and work with their tools and API. Honestly, I can not begin to explain how cool this is in such a short summary. It gives you an overlay on their UI, shows API logs based on what you see, calls APIs without writing code and more. It's a fantastic tool, making working with their API effortless. ### [Learn OAuth 2.0](https://annotate.dev/p/hello-world/learn-oauth-2-0-by-building-your-own-oauth-client-U2HaZNtvQojn4F?ref=apisyouwonthate.com) At some point, when building an API, you're going to have to deal with OAuth. On the surface, the protocol seems easy enough, but it always trips you up at some point. This fantastic tutorial teaches you OAuth by helping you build your own OAuth Client. ### [Building BlueSky](https://newsletter.pragmaticengineer.com/p/bluesky?utm%5Fsource=substack&publication%5Fid=458709&post%5Fid=114113498&utm%5Fmedium=email&utm%5Fcontent=share&utm%5Fcampaign=email-share&triggerShare=true&isFreemail=true&r=1v83fz&triedRedirect=true) This article takes a deep dive into building BlueSky and what it takes to build scaleable federated software. It is an exciting read that covers many areas, including the database layer and moving from AWS to on-prem. --- ## APIs You Won't Hate The latest from the team at API's You Won't Hate. ### [How To use JSONPath with OpenAPI](https://apisyouwonthate.com/p/aa047309-1646-4517-8a7b-88d0e56f69c2/) Learn how to use JSONPath, a language that allows you to query JSON documents to poke around in an OpenAPI doc. You can use JSONPath to patch OpenAPI docs, write advanced linting rules, and more. Phil breaks down what it is and how to use it in his usual charming way. --- ### From Our Community Articles written and shared in our free [Slack community](https://slack.apisyouwonthate.com/?ref=apisyouwonthate.com). ### [Is Bruno a good Postman alternative for API hacking?](https://danaepp.com/is-bruno-good-for-api-hacking?ref=apisyouwonthate.com) I have already mentioned Bruno a few times in this newsletter. If you are not caught up, Bruno is an open-source alternative to Postman that allows you to save your requests in Git. Dana breaks down its features, from requests to testing and more, giving it a fair review. Thank you for the share, Dana. ## Support APIs You Won't Hate When you become an member, you'll get access to members-only content while directly supporting our work. Your support helps us to keep making resources for the API community. [Become a member today](https://apisyouwonthate.com/#/portal) Thanks so much to our members for supporting APIs You Won't Hate: Nolan S., Rüdiger S., Alex R, Vedran C, Danny S, Juxt, Kin L., and Olekshii T. Your support means the world to us! Until next time, Alexander, Phil & Mike ### OpenAPI 4: Project Moonwalk URL: https://apisyouwonthate.com/newsletter/openapi-4-project-moonwalk/ Last updated: 2024-04-15T11:42:27.000Z The OpenAPI committee set a release date of 2024 for version four. We're now a third of the way through the year, so the release date is getting closer. Thankfully, the one and only Phil has taken the time to explain what we can expect in his latest article: [OpenAPI v4.0 (AKA "Project Moonwalk")](https://apisyouwonthate.com/blog/openapi-v4-project-moonwalk/). The spec will support a broader range of URL design patterns, different response bodies based on query parameters, and more. Phil also covers some of the ongoing discussions in the spec. I highly recommend you read this post. \-- Alexander ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2024/03/vscode-ad-100--1-.jpg) ## Introducing a new VS Code Extension for API Insights! Streamline your API development with quality and security assessments directly in VS Code. Enhance your work effortlessly and stay ahead in the industry. Try it now and elevate your APIs to A-grade. [Download the VS Code Extension ](https://apiinsights.io/?utm%5Fsource=newsletter&utm%5Fmedium=apis-hate-sponsorship&utm%5Fcampaign=vs%5Fcode) ## The API Round-Up Our monthly round-up of articles, news and updates from APIs You Won't Hate and around the web. ### [HTMX Revives RESTful APIs](https://netapinotes.com/tired-of-javascript-fatige-htmx-revives-restful-apis/?ref=apisyouwonthate.com) By now, you've likely encountered HTMX, and if not, you've probably stumbled upon the killer meme or engaging shitposting about it on Twitter. Author Mathew Reinbold provides a comprehensive rundown of how HTMX revitalizes proper REST, shedding light on the issues introduced by JavaScript and Single Page Applications. ### [Treblle API Documentation](https://blog.treblle.com/treblle-api-documentation-phil-standards-review/?ref=apisyouwonthate.com) Crafting API documentation from scratch might feel intimidating, especially after completing the API development or for an existing API. Luckily, Treblle offers a remarkable self-documenting feature driven by traffic. But the real question is: does it meet the lofty standards set by our very own Phil Sturgeon? ### [Bruno Releases a New Version](https://github.com/usebruno/bruno/releases/tag/v1.13.0?ref=apisyouwonthate.com) Bruno is an open-source API client that works with Git, and they just pushed a new version. Environment secrets are now masked for added security alongside OAuth 2.0 fixes. Features like Request History and Export Collection as OpenAPI Specs have been added, with numerous bug fixes. Excitingly, 23 new contributors have joined Bruno's vibrant community! ### [Cloudflare Go](https://github.com/cloudflare/cloudflare-go/releases/tag/v2.0.0?ref=apisyouwonthate.com) Version two of the Cloudflare Go library is out. It's a complete rebuild of the first version, but the best part is that it is all generated from their OpenAPI spec. Is there anything OpenAPI can't do? --- ## APIs You Won't Hate The latest from the team at API's You Won't Hate. ### [Use OpenAPI Overlays Today](https://apisyouwonthate.com/blog/use-openapi-overlays-today/) When I first saw this title, I wondered what an overlay is—I had never heard of these before. Overlays help you change parts of an OpenAPI spec using JSON paths. Imagine you have a base spec but maybe need to ship slight variations for different clients. Overlays make this process easier and save you from writing multiple specs. ### [API Code-first: How to Generate OpenAPI from Code](https://apisyouwonthate.com/blog/code-first-how-to-generate-openapi-files-in-2024/) Phil has updated the API code first vs design first debate, covering a more comprehensive range of tools, giving you better code-first workflows and improved tooling for the design-first approach. ### [🎙️ APIs for websites that don't have APIs, with Suchintan Singh from Skyvern](https://apisyouwonthate.com/podcast/apis-for-websites-that-dont-have-apis-with-suchintan-singh-from-skyvern/) Suchintan Singh, founder of Skyvern (Y Combinator S23) stops by the podcast to chat about using LLMs and AI Agents to automate workflows in the cloud. Skyvern's approach to building a business follows a classic startup path: find painful, boring tasks, and automate them! --- ### From Our Community Articles written and shared in our free [Slack community](https://slack.apisyouwonthate.com/?ref=apisyouwonthate.com). ### [Writing API Security Tests in Postman](https://danaepp.com/the-beginners-guide-to-writing-api-security-tests-in-postman?ref=apisyouwonthate.com) Dan, a frequent contributor to our API Slack community, shares a concise guide on API security testing with Postman. Dan's article equips you with essential tools and knowledge for safeguarding your APIs, from understanding vulnerabilities to implementing authentication and data validation tests. Check it out to enhance your API security practices! ## Support APIs You Won't Hate When you become an member, you'll get access to members-only content while directly supporting our work. Your support helps us to keep making resources for the API community. [Become a member today 🥳](https://apisyouwonthate.com/#/portal) Thanks so much to our members for supporting APIs You Won't Hate: Nolan S., Rüdiger S., Alex R, Vedran C, Danny S, Juxt, Kin L., and Olekshii T. Your support means the world to us! Until next time, Alexander, Phil & Mike ### OpenAPI v4.0 (A.K.A "Project Moonwalk") URL: https://apisyouwonthate.com/blog/openapi-v4-project-moonwalk/ Last updated: 2024-04-26T17:59:46.000Z What is coming next for OpenAPI, as v4.0 of the OpenAPI Specification gets closer to being released? What major changes are coming, how easy will it be to upgrade, and how do tooling companies feel about it? ## Recent History of OpenAPI OpenAPI has been around for a long time, but only hit the mainstream when OpenAPI v3.0 was released in 2017\. It made it a whole lot easier to describe the majority of REST/RESTish APIs, and brought in major investment from tooling developers big and small who jumped at the chance to add value to the community. OpenAPI v3.1 came out in early 2021, and focused on fixing issues that appeared with this influx of new users in v3.0\. A large part of the focus was solving the subtle but problematic [differences between JSON Schema and the OpenAPI Schema Object](https://apisyouwonthate.com/blog/openapi-json-schema-divergence/). These two specifications looked similar enough that many users (and even tooling) treated them as interchangeable, but OpenAPI Schema Objects were a subset *and* a superset of JSON Schema, which caused confusion for years. OpenAPI v3.1 aligned Schema Objects to be a valid “dialect” of JSON Schema 2020-12, which not only solved this long standing issue, but brought lots of useful new keywords like `if/then/else` to replace awkward nested `allOf` \> `oneOf` usage. Other useful functionality like support for Webhooks expanded the type of APIs that could be described. So what is OpenAPI v4.0 about? ## Introducing OpenAPI Project Moonwalk With OpenAPI v3.1 released and chugging along happily, the OpenAPI TSC spent about a year keeping an eye on feedback, patching the specification with fixes and clarifications, then started collating ideas for the next version. Would it be 3.2? Would it be 4.0? Both? It wasn’t entirely clear for a while which would be the next step, but in late 2022 the OpenAPI Technical Steering Committee set up a Special Interest Group to start hashing out what the community needed out of a new major OpenAPI version, and that resulted in [Project Moonwalk](https://github.com/OAI/sig-moonwalk?ref=apisyouwonthate.com), a repository where ideas could initially be hashed out and potentially abandoned without confusing everyone with OpenAPI v4.0 being “cancelled” and forcing them to skip a version like PHP’s missing version 6… 🤣 The primary goal of Project Moonwalk is to make the next version of OpenAPI more approachable, and less complex for both humans and computers to write and understand. ### Reduce nested structures to improve readability and editability Let’s be honest, OpenAPI v3 can sometimes feel like a confusing pile of YAML. It’s grown to add excellent functionality which is a large part of how it achieved dominance against earlier API Description Formats, but it’s got pretty complex to write by hand. This complexity has lead to a plethora of open-source and paid tooling aiming to assist the editing experience, from DSL’s to write OpenAPI in easier language, to visual editors to avoid ever needing to look at the YAML. More recently LLMs like Copilot are helping remind users, myself included, how to write out all the right response, mime type, content, status code, schema, keywords… when core contributors of OpenAPI need to melt the ice caps with AI just to remember what keyword comes next, it might be time for a simplification. Here’s the new structure (as it stands). ![OpenAPI v4 diagram representation](https://storage.googleapis.com/bump-blog-resources/openapi-moonwalk/openapi-v4-diagram.png) Diagrams like this can fail to convey much meaning to folks like myself, so lets create an example before (OAS3) and after (OAS4) by taking a snippet from the [Train Travel API](https://bump.sh/blog/modern-openapi-petstore-replacement?ref=apisyouwonthate.com) ([source on GitHub](https://github.com/bump-sh-examples/train-travel-api?ref=apisyouwonthate.com)). ``` /bookings/{bookingId}: parameters: - name: bookingId in: path required: true description: The ID of the booking to retrieve. schema: type: string format: uuid example: 1725ff48-ab45-4bb5-9d02-88745177dedb get: summary: Get a booking description: Returns the details of a specific booking. operationId: get-booking responses: '200': description: The booking details headers: RateLimit: $ref: '#/components/headers/RateLimit' content: application/json: schema: $ref: '#/components/schemas/Booking' application/xml: schema: $ref: '#/components/schemas/Booking' ``` Now let’s try that in OpenAPI 4.0 Project Moonwalk: ``` /bookings/{bookingId}: parameterSchema: bookingId: description: The ID of the booking to retrieve. type: string format: uuid examples: [1725ff48-ab45-4bb5-9d02-88745177dedb] requests: getBooking: method: get summary: Get a booking description: Returns the details of a specific booking. responses: Ok: status: '200' contentType: application/json contentSchema: $ref: '#/components/schemas/Booking' ``` The nesting has been reduced by making `status` and `contentType` a new property in the response instead of another key and another level of nesting. I can remember keywords like `status` and `contentType` a lot easier than I can remember the order of that many keys and objects. Similarly the API requests definition have received the new `method` property avoiding yet another level of parent nesting. Another nice benefit is the OAS “Parameters Object” being replaced with `parameterSchema`. This is a simple JSON Schema object, which is a lot easier for tooling developers to work with than stitch together object maps of “name” and “in”, where parameters are unique based on their location in the URL… This can be confusing for users too, when you get messages about “Parameter foo is not unique” because you had a `foo` defined in the Path’s Parameters Object, but a `foo` also defined in the query string in the Operations Parameters Object. This is just one place, with one list of parameters that can all be reasoned about together. A handy bonus is seeing Operation ID moved from in object to being a key. Operation ID is super handy for lots of tooling, from documentation to SDK generation, but it is rarely used. This means everyone will be defining it! ### Support APIs that have different responses based on query parameters, headers and request bodies. For starters, there is an explicit goal of allowing more types of API than only REST/REST-inspired APIs. This will now support all sorts of RPC API, which often uses fewer URLs, and sends a “method”, “action”, or “command” as a parameter in the request body or query string. This would not have been possible in OpenAPI v3.x as a URL is expected to have one purpose only, and the shape of the response is expected to be the same regardless of what is sent. Here’s an example of an API that sends `service.{serviceName}` as a request header to decide what to do. This is a bit of a contrived example, but the goal is to show the flexibility of the new approach, where requests and responses can be defined to work in all sorts of ways, other than REST(ish) best practice. There’s nothing wrong with RPC, and the more APIs that can be described by OpenAPI the better. ``` openapi: 4.0.0 info: title: RPC API version: 1.0.0 paths: "/service": requests: createFoo: method: post parameterSchema: type: object properties: header: ## We can either use this specially named property or create a first class headerSchema property on the Request Object type: object properties: path: const: service.createFoo ## path Header field used to convey the RPC method contentType: application/json contentSchema: $ref: "#/components/schemas/foo" responses: ok: status: 201 contentType: application/json contentSchema: $ref: "#/components/schemas/foo" getFoo: method: get parameterSchema: type: object properties: header: type: object properties: path: const: service.getFoo ## path Header field used to convey the RPC method responses: ok: status: 200 contentType: application/json contentSchema: $ref: "#/components/schemas/foo" deleteFoo: method: post parameterSchema: type: object properties: header: type: object properties: path: const: service.deleteFoo responses: ok: status: 200 contentType: application/json contentSchema: $ref: "#/components/schemas/foo" ``` Here the `parameterSchema` is playing a role in deciding which method is being called, because the path property could be anything, but by defining `const` (constant value) it’s like writing a switch statement, where the case is the value of the const. Basically, if the path value matches this const value, a request will match this “operation signature” for this specific request. This can be used for data validation, documentation, SDK generation, and everything else, it’s just a bit of a different way to think about things. A more JSON Schema way. ### Support a broader range of URL design patterns As well as being able to work with more types of API than REST(ish), there are plenty of times REST API designers have had their hands tied by limitations on the sorts of URLs and parameter structures allowed in OpenAPI. For example, in OAS3.x all path parameters have to be required, there is no way to have optional path parameters, or multi-segment path parameters. Neither of these sorts of URLs would be allowed in OAS3.1: 1. `/files/{mypath}/{filename}` \- The `mypath` parameter would need to have slashes escaped and be considered one parameter. 2. `/reports/{reportName}/{nonDefaultFormat}` \- The `nonDefaultFormat` would need to be provided every time, or two different operations would need to be defined with and without it, which can lead to a lot of duplication. All of these things are common enough that the [RFC 6570: URI Templates](https://datatracker.ietf.org/doc/html/rfc6570?ref=apisyouwonthate.com) popped up back in 2012, and whilst OpenAPI paths look a bit like URI Templates they are only a subset of what it can do. Project Moonwalk once again says “hey there’s a standard for that, let’s use it!” and defers all path logic to the URI Template RFC. Using this RFC brings loads of powerful syntax, and adds support for loads of existing tooling which can make use of it. ``` paths: "/files/{/filepath*}": # filepath can contain as many URI segments as needed "/reports/{reportName}{/nonDefaultFormat}": # this last one is now optional ``` You can do all sorts of advanced stuff, like define custom server names for particular operations should you have some need to do that, or work with all sorts of complex arrays or objects in the query string without having to decypher the arcane (and rarely supported) [style/explode parameter combinations](https://spec.openapis.org/oas/v3.1.0?ref=apisyouwonthate.com#style-values) in OAS3.x. ``` {/list} /red,green,blue {/list*} /red/green/blue {/list*,path:4} /red/green/blue/%2Ffoo {/keys} /semi,%3B,dot,.,comma,%2C {/keys*} /semi=%3B/dot=./comma=%2C ``` Learn more about [URI Templates](https://datatracker.ietf.org/doc/html/rfc6570?ref=apisyouwonthate.com). ### When will OpenAPI v4.0 Be Released? It’s still being worked on, but the OpenAPI Initiative have set the target release date for “sometime in 2024”. There is always a delay between a specification being released and tooling vendors adding support for it, but unlike the v2.0 to v3.0 transition, the v3.0 to v3.1 was much quicker, and much easier. Whenever a specification like this ditches a unique snowflake of a concept and replaces it with an existing battle tested standard, the tooling migration gets a lot easier, as tooling vendors can wrap a few thousand lines of their code in an if statement, and the else is an existing library that has a whole bunch of tests. There is already a lot of tooling out there helping with migrations, and the word on the grapevine from some of the modern tooling companies is that they’re already experimenting with OAS4 to make sure they’re ready when it drops. ### Ongoing Discussions There are still lots of ongoing discussions occurring, and things are likely to change somewhat before the final release, but now is a good time to be having a look around. Take a look at some of the [examples](https://github.com/OAI/sig-moonwalk/tree/main/doc/initial-proposals/examples?ref=apisyouwonthate.com), then a look at the ongoing [discussions](https://github.com/OAI/sig-moonwalk/discussions?ref=apisyouwonthate.com). **Replacing $ref with “imports”** The `$ref` keyword is [incredibly powerful](https://docs.bump.sh/guides/openapi/advanced-ref-usage/?ref=apisyouwonthate.com), and can be used to “include” bits of OpenAPI and JSON Schema via a filepath, or a URL. This not only helps avoid repetition in a single OpenAPI document, but can be used to share components between multiple APIs, and can enable schema reuse across an entire organization (or even be published publicly and shared to others). Sadly `$ref` has a history of not quite doing what people expect, not quite lining up with JSON Schema, and trying to keep up with changing expectations as those communities all continue to evolve. The reliance on the filesystem structure makes it hard to share documents with $ref, and lots of tools either don’t understand them, only support some of the functionality, or implement things incorrectly… The latests versions of JSON Schema have standardized some really useful `$ref` resolutions that work outside of the filesystem, with an approach of “assuming you’ve got all these files from a repo, zip, floppy disk, whatever, they’ll all declare their `$id` and you can `$ref` on that. This has been great for the JSON Schema community, but the OpenAPI community has struggled to see any take-up on this at all. A new approach is being considered, conceptually referred to for now as “Moonwalk imports”. OpenAPI users can define a document like this, which for example purposes is purely using components. ``` openapi: 4.0.0 self: https://example.com/fooComponents components: pathItems: Foos: {...} schemas: Foos: {...} ``` Then in another document they can define all the paths, and import than components document, referencing them with the `namespace` value to differentiate them from components defined in the local document. ``` openapi: 4.0.0 self: https://example.com/fooPaths components: pathItems: Bars: {...} imports: - namespace: foo href: fooComponents paths: /foos: foo:Foos /bars: self:Bars ``` This is an interesting approach, which relies on [RFC 3987: Internationalized Resource Identifiers (IRIs)](https://www.rfc-editor.org/rfc/rfc3987.html?ref=apisyouwonthate.com), keeping with the theme of replacing OpenAPI-specific things with existing standards. **Better Tags** I am keeping a close eye on a discussion to improve Tags, because they’ve always felt a bit overloaded and confused with different tools having wildly different expectations for what they’re for and how they should be formatted. [Better tags · OAI sig-moonwalk · Discussion #67Tags in OpenAPI 3.x are fit for many purposes, sadly they’re not really fit for the purposes that they actually get used for. Having seen a bunch of discussions about how to work around the tags si…![](https://github.githubassets.com/assets/pinned-octocat-093da3e6fa40.svg)GitHubOAI![](https://opengraph.githubassets.com/fa12eee4944f5d24097dee231d5d01181508b16e7ae54793c8558e159e39362e/OAI/sig-moonwalk/discussions/67)](https://github.com/OAI/sig-moonwalk/discussions/67?ref=apisyouwonthate.com) **Removing Discriminator** Then there’s the discussion about removing discriminator entirely, because its been so poorly defined it’s been confusing people for years. It adds nothing on top of oneOf and does not exist in JSON Schema, so it’s hard to add support without custom writing an extension for specifically just that. Ditch it. [Replace or remove discriminator · OAI sig-moonwalk · Discussion #57There are a number of issues with the current definition of discriminator (as well as a number of open issues). It is not JSON Schema keyword-independent It claims to hint/short-circuit the evaluat…![](https://github.githubassets.com/assets/pinned-octocat-093da3e6fa40.svg)GitHubOAI![](https://opengraph.githubassets.com/c3c5a5d55b9c1f07e2e68d76d5c5423f16d73e3db2ba954e3e1bb34a96b7f230/OAI/sig-moonwalk/discussions/57)](https://github.com/OAI/sig-moonwalk/discussions/57?ref=apisyouwonthate.com) **Improving Links** Links, can we upgrade the OAS 3 concept of Links from a rarely used or supported “Next Step hints” idea into something a bit more useful? Being able to link backwards as well as forwards. Being able to link to other servers using a full URL not just jump around the current API. It’s a lot more HATEOAS this way. [Proposal for links in OpenAPI v4Proposal for links in OpenAPI v4\. GitHub Gist: instantly share code, notes, and snippets.![](https://github.githubassets.com/assets/pinned-octocat-093da3e6fa40.svg)Gist262588213843476![](https://github.githubassets.com/assets/gist-og-image-54fd7dc0713e.png)](https://gist.github.com/mikekistler/1983a8249c0df56a46f94b4df1b7181d?ref=apisyouwonthate.com) If you’re interested in Moonwalk, give it a shot. You can try out the concepts on your own OpenAPI documents to see how it looks. You’ll have to do that manually for now as tooling is waiting for the plan to settle, but the best tooling developers will be keeping an eye on things and adding experimental support as soon as the time feels right. Until then keep getting feedback to the OAI so they can make this as good as possible. ### API Code-first: How to Generate OpenAPI from Code URL: https://apisyouwonthate.com/blog/code-first-how-to-generate-openapi-files-in-2024/ Last updated: 2024-04-11T06:09:39.000Z We've posted about [API Code-first vs API Design-first](https://apisyouwonthate.com/blog/api-design-first-vs-code-first/) a fair few times over the years, but this article aims to be an updated look at some of the current day tooling which makes both workflows nicer to handle. 🫶 Thank you to [Bump.sh](https://bump.sh/?ref=apisyouwonthate.com) for sponsoring this post, and giving me time to write down some updated thoughts on this topic. API Code-first is the art of building an API and then popping some annotations or metadata in there to pop out API documentation in a API description format like OpenAPI. Is that how you want to keep doing things? Maybe, there's some tooling out that's made it a lot easier. There's also a few tools knocking around mean you might not need to. For those of you used to the API Code-first here are the three main workflows you should be thinking about going forwards. If you've been documenting your APIs entirely manually with some sort of content management system, wiki, or Word Document, then these ideas might save you from that nightmare. - [Annotations](#Annotations) - [OpenAPI-aware Frameworks](#OpenAPI-aware-Frameworks) - [Traffic Sniffing](#Traffic-Sniffing) - [Switch to API Design-first](#Switch-to-API-Design-first) ## Annotations The classic approach to the API Code-first workflow is to use code comments or some other form of annotations as extensions or plugins to write OpenAPI mixed in with the code it's describing. Here's how these annotations look in Go: ``` // @title Swagger Example API // @version 1.0 // @description This is a sample server celler server. // @termsOfService http://swagger.io/terms/ // @contact.name API Support // @contact.url http://www.swagger.io/support // @contact.email support@swagger.io // @host localhost:8080 // @BasePath /api/v1 // @securityDefinitions.basic BasicAuth func main() { r := gin.Default() c := controller.NewController() v1 := r.Group("/api/v1") { accounts := v1.Group("/accounts") { accounts.GET(":id", c.ShowAccount) accounts.GET("", c.ListAccounts) accounts.POST("", c.AddAccount) } //... } } ``` Then the schema level descriptions are mixed in with the code responsible for outputting resources like this: ``` type Account struct { ID int `json:"id" example:"1"` Name string `json:"name" example:"account name"` } ``` Once the API, endpoints, and resources have all the appropriate annotations there is usually some sort of command you can run to get an OpenAPI document out of it, and that machine-readable document can be used to deploy documentation to [Bump.sh](https://bump.sh/?ref=apisyouwonthate.com) or wherever your API documentation lives. ``` swag init --outputTypes yaml bump deploy swagger.yaml \ --doc my-documentation-name \ --token my-documentation-token ``` This approach has been popular for years, with the main selling point being the idea that keeping OpenAPI metadata near the code will hopefully mean developers keep it up to date as they work on the code. This is not always the case, which is one of a few reasons this practice is dying out. The other is that many of the annotation tools are stuck on older less useful versions of OpenAPI, namely v2.0 instead of v3.0, or the latest and greatest: v3.1. - **Go** - [swag](https://github.com/swaggo/swag?ref=apisyouwonthate.com) (OAS 2.0) - [go-swagger](https://github.com/go-swagger/go-swagger?ref=apisyouwonthate.com) (OAS 2.0) - **PHP** - [Swagger PHP](https://zircote.github.io/swagger-php/?ref=apisyouwonthate.com) (OAS 3.1 & 3.0) - **‌Java/Scala** - [Swagger Core](https://github.com/swagger-api/swagger-core?ref=apisyouwonthate.com) (OAS 3.1 & 3.0) - **C#** - [NSwag](https://github.com/RicoSuter/NSwag?ref=apisyouwonthate.com) (OAS 3.0 & 2.0) - [Swashbuckle](https://github.com/domaindrivendev/Swashbuckle.WebApi?ref=apisyouwonthate.com) (OAS 2.0) - **Node.JS** - [swagger-jsdoc](https://github.com/Surnet/swagger-jsdoc?ref=apisyouwonthate.com) (OAS 3.1, 3.0, & 2.0) - **Ruby on Rails** - [rswag](https://github.com/rswag/rswag/?ref=apisyouwonthate.com) (OAS 3.0 & 2.0) - [OpenAPI-Rails](https://github.com/slate-studio/openapi-rails?ref=apisyouwonthate.com) (OAS 2.0) - **Python** - [drf-spectacular](https://github.com/tfranzel/drf-spectacular?ref=apisyouwonthate.com) (OAS 3.1 & 3.0) - Django-REST-Swagger (OAS 2.0, abandoned) - Flask-RESTplus (OAS 2.0, abandoned) - **Spring** - [SpringFox](https://github.com/springfox/springfox?ref=apisyouwonthate.com) (OAS 3.0) Depending on your language and framework choices you may or may not have an option for working with modern OpenAPI, but the lack of modern tooling has been a driving force in people giving up on this approach and looking for alternative workflows. Let's have a look at some others. ## OpenAPI-aware Frameworks There's a new breed of API-centric backend application frameworks popping up which take an exciting approach. Instead of asking you to tack the annotations in around the existing codebase, the frameworks simply produce OpenAPI for you from the actual code you're writing. Your application is already declaring routes, defining parameters and incoming validation logic, and helping serialize output. It makes a lot of sense for the framework to help produce this machine readable format for you, from the code you're already writing. There are not as many tools that work this way, but this is likely to be a trend that continues as OpenAPI becomes the dominant API description format. - **Go** \- [Huma](https://huma.rocks/?ref=apisyouwonthate.com) - **PHP** \- [API Platform](https://api-platform.com/?ref=apisyouwonthate.com) - **Python** \- [FastAPI](https://fastapi.tiangolo.com/?ref=apisyouwonthate.com) Just like annotations you can usually run a command to extract the OpenAPI document, or you can run the web server and pull it down over HTTP. ``` $ go run . $ bump deploy http://127.0.0.1:8888/openapi.yaml \ --doc my-documentation-name \ --token my-documentation-token ``` ## Traffic Sniffing If there's no annotations approach, and you have an existing codebase which cannot be rebuilt with one of these OpenAPI-aware application frameworks, there is another powerful option: sniffing web traffic. There's [a whole category of tools](https://openapi.tools/?ref=apisyouwonthate.com#learning) popping up, which refer to this functionality as "Recording" or "Learning". Basically you run an instance of your API somewhere (could be local, test, staging, or even production) and put as much web traffic through it as possible. It will then learn how all the requests and responses look, and produce the best composite OpenAPI that it possibly can. [Turn HTTP Traffic into OpenAPI with OpticCapture real HTTP traffic from production or anywhere else, and create OpenAPI from it, for documentation, mocks, SDKs, or contract testing.![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/size/w256h256/2023/03/apis-logo-square.png)APIs You Won't HatePhil Sturgeon![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2023/03/optic--1-.webp)](https://apisyouwonthate.com/blog/turn-http-traffic-into-openapi-with-optic/) You can usually run some sort of proxy to do this, but there are other ways, like using [rspec-openapi](https://rubygems.org/gems/rspec-openapi/?ref=apisyouwonthate.com) for Ruby on Rails users. This tool will sniff all the integration tests coming through the test suite to the API and use that to construct a rudimentary OpenAPI description. ## Code-first usually needs enhancing Whether you're generating from annotations, the framework, or HTTP traffic, there's a strong chance that you'll need to put some work in to improve the quality of that OpenAPI. It's going to be missing long form descriptions, the sort of content that tech writers often produce, and depending on the tool used it's probably going to be missing examples too. In order to improve this you can use OpenAPI Overlays to enrich the generated OpenAPI with your own logic, and avoid it being overridden the next time OpenAPI is generated. ## Move to API Design-first With so many of the annotations approaches being outdated, and people usually unable to rebuild an entire codebase to use a framework that happens to emit OpenAPI, a lot of people have given up on the whole code-first approach. This is not just an opinion. Searching around the Go community for code-first tooling, most of the "How do I do Code-first in Go" search results show people talking about how they moved to API Design-first and massively prefer the approach. The main idea is that instead of writing loads of code and sprinkling in some annotations later to create docs, you create the OpenAPI before writing any code at all. This is usually in the form of JSON/YAML, but that does not need to be written by hand. There are lots of visual editors to help you build this all up through buttons and forms, with an increasing amount of intelligence to create things. If you're using VS Code then Copilot is actually incredibly good. Once you have the OpenAPI document you can leverage it at every step of the API lifecycle, producing mock APIs for clients to test assumptions with, produce client libraries without writing any code, make really effective contract testing, even generate backend code to get the application teams started once the contract is all signed off. API Design-first is a bit more work up front with a massive payoff in productivity going forwards and forever. The main concern with API Design-first is "drift", where the code and schema diverge over time. The annotations approach only pretended to solve this problem, confusing proximity with accuracy. The comments above code could still completely fail to accurately describe the code below, but nobody would ever notice until a user complained about it. The OpenAPI-aware Framework approach does solve this by making the code a single source of truth, but the Design-first approach can be used to make any framework OpenAPI aware, with a source of truth that exists before the code, and continues to be useful after the code is built. Server-side validation with OpenAPI can avoid the need to write lots of request validation, using middleware to compare incoming HTTP requests against the contract defined in the OpenAPI description and automatically return validation errors instead of having to build that all our yourself. - **Java** - [openapi-request-response-validation](https://github.com/gcatanese/openapi-request-response-validation?ref=apisyouwonthate.com) - **JavaScript** - [express-openapi-validator](https://www.npmjs.com/package/express-openapi-validator?ref=apisyouwonthate.com) - [fastify-openapi-glue](https://www.npmjs.com/package/fastify-openapi-glue?ref=apisyouwonthate.com) - [openapi-enforcer](https://www.npmjs.com/package/openapi-enforcer?ref=apisyouwonthate.com) - [openapi-validator-middleware](https://www.npmjs.com/package/openapi-validator-middleware?ref=apisyouwonthate.com) - **Perl** - [JSONSchema::Validator](https://metacpan.org/pod/JSONSchema::Validator?ref=apisyouwonthate.com) - **PHP** - [openapi-psr7-validator](https://github.com/thephpleague/openapi-psr7-validator?ref=apisyouwonthate.com) - **Ruby** - [openapi\_first](https://github.com/ahx/openapi%5Ffirst?ref=apisyouwonthate.com) - [committee](https://rubygems.org/gems/committee?ref=apisyouwonthate.com) Responses can be validated using any existing test suite, with all popular testing frameworks supporting OpenAPI by extension. - **JavaScript** - [chai-openapi-response-validator](https://www.npmjs.com/package/chai-openapi-response-validator?ref=apisyouwonthate.com) - [jest-openapi](https://www.npmjs.com/package/jest-openapi?ref=apisyouwonthate.com) - **PHP** - [Spectator](https://github.com/hotmeteor/spectator?ref=apisyouwonthate.com) - **Ruby** - [openapi\_contracts](https://github.com/mkon/openapi%5Fcontracts/?ref=apisyouwonthate.com) Years ago the API Design-first workflow was a rough approach, but thankfully a whole bunch of tooling developers spent those years making things excellent, and now it's easier than ever. Bump.sh adds to that legacy by adding amazing [change detection](https://bump.sh/help/changes-management/?ref=apisyouwonthate.com), helping check the OpenAPI in your git repository for changes that would be breaking for end users, [letting you know in the pull request](https://bump.sh/help/continuous-integration/github-actions/?ref=apisyouwonthate.com) when there's a problem, providing beyond a shadow of a doubt that having your OpenAPI as a source of truth in a git repository along with your source code is not only handy, but probably the best way to go for many teams. ## Support APIs You Won't Hate When you become an member, you'll get access to members-only content while directly supporting our work. Your support helps us to keep making resources for the API community. [Become a member today](https://apisyouwonthate.com/#/portal) ### Use OpenAPI Overlays Today URL: https://apisyouwonthate.com/blog/use-openapi-overlays-today/ Last updated: 2025-01-13T18:38:41.000Z It's been a while since OpenAPI v3.1 came out, an [Project Moonwalk](https://github.com/OAI/sig-moonwalk?ref=apisyouwonthate.com) (OAS v4.0) is still in discussion and design phase, but that doesn't mean there's nothing new in the world of OpenAPI. [Overlays](https://github.com/OAI/Overlay-Specification?ref=apisyouwonthate.com) are one of the OpenAPI Initiatives newest offerings, and I've already been using them to solve problems. Overlays are a way to "patch" an OpenAPI document, by pointing to a specific part of it with a JSONPath, then using "actions" to update or remove parts of the document. Why would anyone want to do this? ## Technical Writers Perhaps you're a technical writer trying to improve the OpenAPI-based documentation, but the OpenAPI is being generated from source code, and you don't fancy learning 5 different programming languages, web frameworks, and OpenAPI annotation frameworks. Fair enough! Instead of having your changes overridden, you can put all of your expanded guides and descriptions into an overlay, and have CI patch them together before deploying to your documentation hub. ``` # Export OpenAPI from code (this is how Huma does it) go run . openapi > openapi.yaml ``` Then when you've got the `openapi.yaml`, you can create a second document called `overlays.yaml`, which contains all the changes you'd like to apply. ``` # overlays.yaml overlay: 1.0.0 info: title: Add descriptions to tags version: 0.0.1 actions: - target: '$.tags[?(@.name=="Order")]' description: Provide more information for Order tag. update: description: > The Order resource represents a single order for trees, which can be fulfilled by one or more deliveries. Orders are created by the [Protect Earth team](https://protect.earth/contact) and are used to track the progress of your order from creation to delivery. - target: '$.tags[?(@.name=="Organization")]' description: Provide more information for Organization tag. update: description: > The Organization resource represents a single organization, which can be a charity, business, or other entity. Organizations are created by the [Protect Earth team](https://protect.earth/contact) and are connected to each of your Orders. ``` I wrote more about [how revolutionary overlays can be for technical writers](https://docs.bump.sh/guides/technical-writing/efficient-tech-writing-process/?ref=apisyouwonthate.com) for [Bump.sh](https://bump.sh/?ref=apisyouwonthate.com), and I suggest you check that out if you're more interested in some of the ideas for what you could overlay, e.g. visual improvements for various docs tools, adding code samples, etc. [Efficient Technical Writing Processes for API DocumentationHow can engineering and technical writing teams worth together on creating fantastic API documentation without stepping on each others toes?![](https://docs.bump.sh/images/favicon.ico)Bump.sh Docs & GuidesPhil Sturgeon API Design and Reforestation![](https://docs.bump.sh/images/bump-logo.svg)](https://docs.bump.sh/guides/technical-writing/efficient-tech-writing-process/?ref=apisyouwonthate.com) ## White Label APIs My [green tech](https://philsturgeon.com/what-is-green-software/?ref=apisyouwonthate.com) company [Green Turtle](https://www.greenturtle.io/?ref=apisyouwonthate.com) produces a back-office system for funding tree planting and other biodiversity work, which has an API for integrating with funding partners like [Ecologi](https://ecologi.com/?ref=apisyouwonthate.com). Multiple clients use this, including my own charity [Protect Earth](https://www.protect.earth/?ref=apisyouwonthate.com), but other clients too. At first I was deploying the [generic API documentation](https://bump.sh/green-turtle/doc/tree-tracker-api/?ref=apisyouwonthate.com), but increasingly this was problematic as the contact details, introductions, and server URLs were all generic rubbish. I wanted to customize it all to be more specific to the client, and overlays worked perfectly for this. ``` overlay: 1.0.0 info: title: Overlay to customise API for Protect Earth version: 0.0.1 actions: - target: '$.info' description: Update description and contact for our audience. update: description: > Protect Earth's Tree Tracker API will let you see what we've been planting and restoring all around the UK, and help support our work by directly funding the trees we plant or the sites we restore. To get involved [contact us and ask for an access token](https://protect.earth/contact) then [check out the API documentation](https://protect.earth/api). contact: name: Protect Earth Support url: https://protect.earth/contact email: help@protect.earth - target: '$.servers.*' description: Remove all other servers so we can add our own. remove: true - target: '$.servers' description: Pop our server into the empty server array. update: - description: Production url: https://api.protect.earth/ ``` Now I can deploy [client-specific API documentation](https://bump.sh/protect-earth/doc/partner-api/?ref=apisyouwonthate.com), so they can share that with their partners without having to explain what a Green Turtle is. ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2024/04/image.png) A preview of the customized OpenAPI document, deployed on its own documentation via Bump.sh, without any of the old generic nonsense from the source OpenAPI. ## What tools support OpenAPI Overlays? Overlays is still marked as "experimental", but considered stable enough for tools to start implementing. And implement they have! ### openapi-overlays-js As always Lorna Mitchell was there doing amazing things, and she's put together a handy JavaScript library [openapi-overlays-js](https://www.npmjs.com/package/openapi-overlays-js?ref=apisyouwonthate.com) which handles overlays brilliantly. At first this seemed like a bit of a hackathon project, but it's being actively maintained and has a few contributors. So far it doesn't have a CLI, but the [Bump.sh](https://bump.sh/?ref=apisyouwonthate.com) folks have helped out there! ### Bump CLI The [bump-cli](https://www.npmjs.com/package/bump-cli?ref=apisyouwonthate.com) wraps this up to help their users use the functionality when deploying their documentation, but anyone can use the cli for any reason. ``` npm install -g bump-cli # apply the overlay and create a new file bump overlay openapi.yaml overlays.yaml > openapi.public.yaml # or conviently deploy docs with overlays applied bump deploy openapi.yaml --doc my-doc --token my-token --overlay overlays.yaml ``` Instead of just lifting it with quiet thanks, the team have been sending fixes/functionality back, which is what you want to see. ### Speakeasy The [Speakeasy](https://www.speakeasyapi.dev/?ref=apisyouwonthate.com) team have also built overlay support into their CLI. The [Speakeasy CLI](https://www.speakeasyapi.dev/docs/speakeasy-cli/getting-started?ref=apisyouwonthate.com) is written in Go which will excite some people, but that makes it a little tricker to install in GitHub Actions which is what I was looking for. ``` # macOS or linux brew install speakeasy-api/homebrew-tap/speakeasy # Windows choco install speakeasy # Apply Overlay changes and make a new file speakeasy overlay apply -s openapi.yaml -o overlays.yaml > openapi.public.yaml ``` They've gone for a really advanced implementation here, offering a few commands: - `speakeasy overlay apply` \- Given an overlay, construct a new specification by extending a specification and applying the overlay, and output it to stdout. - `speakeasy overlay compare` \- Given two specs, output an overlay that describes the differences between them. - `speakeasy overlay validate` \- Given an overlay, validate it according to the OpenAPI Overlay specification I managed to get this working on GitHub Actions with the following workflow file: ``` # .github/workflows/deploy.yml name: Deploy API documentation on: push: branches: - main jobs: deploy-doc: if: ${{ github.event_name == 'push' }} name: Deploy API documentation runs-on: ubuntu-latest steps: - name: Checkout uses: actions/checkout@v3 - name: Set up Homebrew id: set-up-homebrew uses: Homebrew/actions/setup-homebrew@master - name: Install Overalls dependencies run: | brew install speakeasy-api/homebrew-tap/speakeasy - name: Apply Overlays to customise OpenAPI working-directory: ./api run: | speakeasy overlay apply -s openapi.yaml -o overlays.yaml > openapi.public.yaml - name: Deploy API documentation run: ``` If you'd like to play with Speakeasy's fantastic SDK generation then you can get two for one jumping through these hoops, but if you find working with NPM easier I'd definitely go for the [Bump.sh CLI](https://github.com/bump-sh/cli?ref=apisyouwonthate.com) approach. ### Overlay Playground Speakeasy have released [Overlay Playground](https://overlay.speakeasy.com/?ref=apisyouwonthate.com), an online tool to help experiment with overlays without the need to install anything. [OpenAPI Overlay Playground | SpeakeasyA tool to visualize and generate OpenAPI Overlays![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/icon/favicon-8.ico)![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/thumbnail/og-image-600x900.png)](https://overlay.speakeasy.com/?ref=apisyouwonthate.com) ## Do we want more JSONPath in our lives? If you're new to JSONPath, this might seem like a lot to take on, and if you are familiar with JSONPath you might be groaning at the thought of working with it more. JSONPath is a query language that can be used to extract data from JSON documents, and readers might be familiar witht Spectral which uses it to help identify sections of JSON/YAML documents (most commonly OpenAPI and AsyncAPI) to apply linting rules to. Much like Markdown in the days before CommonMark, there are a few different JSONPath definitions knocking around which are similar but different in problematic ways. 1. [JSONPath "The Blog Post"](https://goessner.net/articles/JsonPath/?ref=apisyouwonthate.com) \- Written by Stefan Gössner in 2007. 2. [jsonpath.com](https://jsonpath.com/?ref=apisyouwonthate.com) \- An online evaluator which as far as I can tell matches the blog post. 3. [JSONPath-Plus](https://www.npmjs.com/package/jsonpath-plus?ref=apisyouwonthate.com) \- A popular (but now abandoned) fork which expands on the original specification to add some additional operators. 4. [Nimma](https://www.npmjs.com/package/nimma?ref=apisyouwonthate.com) \- A fork of JSONPath Plus created by the Stoplight team for Spectral to handle more advanced use cases. A list of caveats can be found [here](https://www.npmjs.com/package/nimma?ref=apisyouwonthate.com#caveats). 5. [RFC 9535: JSONPath](https://www.rfc-editor.org/rfc/rfc9535?ref=apisyouwonthate.com) \- An IETF proposed standard that attempts to solve this whole mess by defining one implementation, confusing people like me who are used to JSONPath-Plus and Nimma. If you are building tooling that uses low-level JSONPath tooling, make sure that tooling is based on RFC 9535, like these tools: - [JsonPath.Net](https://github.com/gregsdennis/json-everything/tree/master/JsonPath?ref=apisyouwonthate.com) in C# - [jpt](https://rubygems.org/gems/jpt?ref=apisyouwonthate.com) in Ruby - [serde\_json\_path](https://github.com/hiltontj/serde%5Fjson%5Fpath?ref=apisyouwonthate.com) in Rust Or at least "plans on" aligning with it [as Nimma does](https://www.npmjs.com/package/nimma?ref=apisyouwonthate.com#caveats). Assuming everything can align on the standard, is JSONPath "good" or "bad"? Well, it can take a bit of getting used to, but it's incredibly powerful. I accept that it's not for non-technically minded people, but I already know of a lot of excited technical writers. For example, a slightly bonkers JSONPath, but I really enjoyed this generic action which would look for specific server names to remove: ``` # overlays.yaml overlay: 1.0.0 info: title: Overlay to customise API for Protect Earth version: 0.0.1 actions: - target: '$.servers[?(@.description=="Development" || @.description=="Staging")]' description: Remove Development and Staging servers but leave anything else. remove: true ``` The whole approach of trying to simply name all the objects in OpenAPI falls a bit flat when you are trying to filter and query like this, and you'll only end up inventing a DSL if you try and avoid it, so I say lets stick to a standard instead of making stuff up, and if that standard is JSONPath then that's fine enough with me. ## The Future of Overlays It seems like the specification is settling down and tooling vendors are happily starting to adopt it, but there is one change being discussed which I think could make it a lot better. Currently you can only `update` or `remove`, but `update` is unclear in its intention when it comes to trying to add things. It's like the difference between [PUT and PATCH](https://apisyouwonthate.com/blog/put-vs-patch-vs-json-patch/), should this be a total replacement, where missing things are removed, or should it be a merge, where provided values are mixed in to what is already there? There is [a proposal](https://github.com/OAI/Overlay-Specification/pull/32?ref=apisyouwonthate.com) to ditch `update`, and define two new operations `add` and `replace`. These two new actions have their semantics lifted from [RFC 6902: JSON Patch](https://www.rfc-editor.org/rfc/rfc6902.html?ref=apisyouwonthate.com), and would help end users and tooling vendors have a lot more clarity in what they're doing. [Switch from add to replace/update properties by lornajane · Pull Request #32 · OAI/Overlay-SpecificationI’m proposing this change as a response to the ongoing discussion in #30 . Using “add” can be confusing and leaves us without being able to support some use cases, such as setting an array to a par…![](https://github.githubassets.com/assets/pinned-octocat-093da3e6fa40.svg)GitHubOAI![](https://opengraph.githubassets.com/35e4486d8d41d6e7af246ba876114b49bccecd84d3f06eb8408f19dcb811f410/OAI/Overlay-Specification/pull/32)](https://github.com/OAI/Overlay-Specification/pull/32?ref=apisyouwonthate.com) Still, don't wait for everything to be perfect. Get out there and start using this. Build tools for this. Give feedback and bug reports for anything that doesn't work, and comment back here to let everyone know what you've using overlays for! ## Further Reading - [*Augmenting OpenAPI with Filters & Overlays*](https://docs.bump.sh/guides/openapi/augmenting-generated-openapi/?ref=apisyouwonthate.com) by me for [Bump.sh](https://bump.sh/?ref=apisyouwonthate.com) - [*Efficient Technical Writing Processes for API Documentation*](https://docs.bump.sh/guides/technical-writing/efficient-tech-writing-process/?ref=apisyouwonthate.com) by me for [Bump.sh](https://bump.sh/?ref=apisyouwonthate.com) - [*Improve Existing OpenAPI Descriptions*](https://lornajane.net/posts/2023/overlay-improvements-to-openapi?ref=apisyouwonthate.com) by Lorna Mitchell ## Support APIs You Won't Hate When you become an member, you'll get access to members-only content while directly supporting our work. Your support helps us to keep making resources for the API community. [Become a member today](https://apisyouwonthate.com/#/portal) ### APIs for websites that don't have APIs, with Suchintan Singh from Skyvern URL: https://apisyouwonthate.com/podcast/apis-for-websites-that-dont-have-apis-with-suchintan-singh-from-skyvern/ Last updated: 2024-04-05T11:37:31.000Z ## Show notes - Skyvern [https://www.skyvern.com/](https://www.skyvern.com/?ref=apisyouwonthate.com) - Skyvern is Open Source! [https://github.com/Skyvern-AI/skyvern](https://github.com/Skyvern-AI/skyvern?ref=apisyouwonthate.com) - Launch on Y Combinator's site: [https://www.ycombinator.com/companies/skyvern](https://www.ycombinator.com/companies/skyvern?ref=apisyouwonthate.com) - [Suchintan Singh (YC S23) on LinkedIn](https://www.linkedin.com/in/suchintansingh?ref=apisyouwonthate.com) #### Transcript \[00:00:00\] ****Mike Bifulco:** Hello and welcome back to APIs you won't hate. My name is [Mike Bifulco](https://mikebifulco.com/?ref=apisyouwonthate.com). And today I'm sitting down for a conversation with a friend of mine from actually last summer, which I guess we'll get into in, in a little while here. But talking a little bit about Sky Verne with Ton Singh from the founding team of Sky Verson. How are you today? \[00:00:19\] ****Suchintan Singh (Skyvern S23):** I'm good. I'm good. You know, it's bright and early here. \[00:00:23\] ****Mike Bifulco:** Yeah, that's it. Well, are are you on the West Coast? \[00:00:25\] ****Suchintan Singh (Skyvern S23):** I'm actually in the east coast here. Yeah. But it's I live in Canada and we're a nice, like sunny day today, so I'm pretty, pretty excited about \[00:00:31\] ****Mike Bifulco:** oh, nice. Very cool. Yeah, I'm a fellow East coaster, so at least we're on a similar time zone. I was worried I made you get up at the crack of dawn for this. Yeah. Cool. Thank you. So why don't we start here. Tell me a little bit about about yourself and about Sky Verne. So let, let's start with Sky Verne. \[00:00:44\] What's the elevator pitch for what you're building? \[00:00:46\] ****Suchintan Singh (Skyvern S23):** So Sky is a a browser based automation tool and we use AI to help like companies automate workflows in the browser. Specifically what we do is we help companies automate, like really boring things like form filling particularly interacting with \[00:01:00\] government websites or generating insurance quotes online. \[00:01:02\] So really targeting like the back office companies, things that people don't even know happen in a company, which is kind of why we get, you know, excited about it. \[00:01:12\] ****Mike Bifulco:** Yeah, the classic startup of of making a boring problem into a business. \[00:01:17\] ****Suchintan Singh (Skyvern S23):** that's \[00:01:17\] ****Mike Bifulco:** cool. So I definitely wanna talk about that. Want, want to dig into it and learn more about how you got there and how like, like many of us, you fixated on an exciting, boring problem. But before we do that, tell me a little bit about yourself. \[00:01:29\] What was your background sort of before you. Started building Sky Verne, what did your career kinda look like? And, and yeah, I don't know if, hit me with the the exciting, interesting points. \[00:01:38\] ****Suchintan Singh (Skyvern S23):** Yeah. So I I was I've been programming for a long time. I started programming when I was like a, a kid. My dad tried, actually taught me c plus plus when I was in grade seven because he was like, this is gonna be the future. \[00:01:47\] And I was like, okay, sure. Whatever. I don't care. Yeah. And we made like some really dumb programs back then. \[00:01:51\] And then when I was in high school, I was really into this game called Scape, which many, I'm sure many of the listeners are into or into in the past. And \[00:02:00\] I think I really hit my stride because I started using, I started writing bots for Inscape, using some like client libraries that were exposed. And that really, like, I think some of the source codes still open, open source on my GitHub. \[00:02:12\] I don't, I don't think it runs anymore, but it's, it's still there. And I always think back to it is like a really fond memory of when like, I really like embrace programming and I always loved like solving problems with it. And, um. Coding in general, and it just happened that it could also turn it into a career, which is kind of lucky. \[00:02:28\] Lucky happenstance, right? And so I went through university here in Canada and kind of just fell into programming as a career. And spent a few years working at a few companies. And my career really, I would say, took a shape when I started working at this company called Fair. Here, here in Canada and I got to own their machine learning platform there. \[00:02:48\] I got to build it up from ground up having no machine learning background at all at the time, and it really made me fall in love with what turned out to be the next wave of computing. I didn't, I didn't know that back then either, but it turned out to be the next wave of computing \[00:03:00\] and really helped shape my career. \[00:03:02\] So I got the opportunity to, to build their search and discovery system from ground up and make, you know, millions of dollars of impact there. And I took that learning to another company afterwards, called goPuff and did the same thing there. And after that decided to start my own company. And here, here we're today, \[00:03:17\] ****Mike Bifulco:** Got it. Wow. Yeah. So it sounds like based on that timeline, you were probably building ml things like before people were even breathing the word GPT into the air. Right. How, how long ago did you start working \[00:03:27\] with ML stuff? \[00:03:28\] ****Suchintan Singh (Skyvern S23):** I. It was like, it was around the time that GPT two was out, but GP three wasn't out yet. And so people didn't really know what it was at the time. Yeah. And ML was still, you know, like more, less neural networks and more like other algorithms. Today, you know, people don't really like the other ones that much anymore. \[00:03:44\] So yeah, things have changed quite a bit. \[00:03:45\] ****Mike Bifulco:** Yeah. Cool. Yeah. Oh, that's really interesting. So you built an ML platforms at, at two companies and then you kind of made the leap into something a little more entrepreneurial and, and doing it for yourself from the sounds of \[00:03:56\] it. And so what, what was, what's Sky Run's story like what, what \[00:04:00\] was the first I don't know, attempt at a product? \[00:04:02\] How did you decide that you were gonna do \[00:04:04\] ****Suchintan Singh (Skyvern S23):** So Sky's actually our third pivot, believe it or not. So we've been building startups for a while. My buddy, my buddy, my co-founder Shu and I, we decided to start a company together on like one random trip. We're like going surfing on the car drive. We're like, Hey, we should start a company and, you know, like, any good stories so that, that we ended up materializing that. \[00:04:21\] And the first product we built was actually a tool to help onboard software engineers because I had recently become a manager at a company, and I went through the pain that was poor onboarding. I later learned that that's a very common target ideas for engineers to build work on. But we went through the process of building a startup. \[00:04:35\] We made every single mistake you could imagine. I. I think Y Combinator has been very helpful in our career, in our career because they put out a lot of content online talking about how to build startups, and we later watched all those videos and they recanted all the mistakes we'd made. We'd already made up to that point. \[00:04:49\] So we built a product for us without talking to any users. We're like, we know exactly what people need to solve this problem. So we built a solution and then we started trying to find people who'd buy it and turns like nobody wanted to buy it because we were not \[00:05:00\] really solving a problem the way that people were having the problem. \[00:05:03\] And it was, it was not obvious at the, at the outset that that was that, that, that that's where we're going down. But I'm, I'm still glad we did it because it taught us, you know, how to ship code fast. We're working part-time. Like, you know, we had, we had our full-time jobs. We were just coding on the side, trying to get a product out. \[00:05:18\] And all the lessons we learned after we built the product, like how to do sales, how to talk to people, how to make sure that the problem we're solving is aligned with the problem people are actually having. Were, were things that we wouldn't have learned had we not gone through that journey. \[00:05:29\] ****Mike Bifulco:** Yeah, definitely. I would love to \[00:05:31\] see, I'm sure someone somewhere has a chart on correlation between startup founders and having had a, a, a failed experience in the past, or at least something that didn't go according to plan. I'm definitely one of those, the, the first couple of attempts at building a company for me were not not, not successful experiences in the traditional sense, but definitely in the sense that I took away lots of learnings from it and like. Often harken back to the things I did poorly in the past as I'm making decisions now. Yeah. Okay. So, so you said you're in your third pivot at this point? \[00:06:00\] Yeah. \[00:06:00\] ****Suchintan Singh (Skyvern S23):** That's right. Yeah, that's right. So after that, we pivoted into something that was much more in our domain, which was since I've helped two companies build their machine learning platform for, particularly for search and discovery experiences, to help like marketplaces increase their revenue I had a, we basically found a company that wanted, was interested in buying that product. \[00:06:18\] And so we pivoted so we could build it and sell it to that company. And we got a commitment very early. We, we, we actually changed how we did software development. In this case. We did sales first. Made sure the shape of the product made sense. We were a lot of documents to align with the people who wanted to buy it. \[00:06:34\] Talked to many other companies along the way. And then we started building the product. And so we, we did a hard pivot at that point because we found that there was actually, this, this problem I had solved at two companies was actually a problem that many other companies had. \[00:06:47\] Started exploring the general space of like search and discovery. Went down a few lanes that other companies had also gone down. So one very common idea that people who have worked in, in this field go into is they try and build search as a \[00:07:00\] service. Like they build like a plugin for Shopify where people can plug, put in a search engine and make more money that way. \[00:07:06\] We went down that path, we went down doing the same thing for recommendations, and it turns out both those ideas are very tough to execute on because many people have tried before and the value of the product isn't there. Yet isn't like the shops are too early to realize the value of the product. \[00:07:19\] And so this, this is I think another area of possible tar ideas. It's unclear whether it's actually a tar idea today, but a tar idea being an idea that's seems attractive from the outset, but it's not has lots of people who've tried it before and all failed. And so we talked to a bunch of founders who had tried it and had failed with those and kind of had had some learnings. \[00:07:36\] So we went down this path of building a different platform. Yeah. \[00:07:40\] ****Mike Bifulco:** so why don't you tell me a little bit about the first version of what has now become Sky Verne and how you decided on that pivot. \[00:07:47\] ****Suchintan Singh (Skyvern S23):** Yeah. So we were just to quick, quickly finish the loop on Vern, we built a product and what we learned was that we actually learned during sales. We got about three commitments from companies and. We learned that the companies were \[00:08:00\] excited about our product. All had a division between their data science team and engineering teams, which made sales very challenging. \[00:08:06\] Particularly if a company had that division. They were really excited about our product and if they didn't have that division, they were not excited about our product at all, which made selling the product very challenging. And so we again, decided to pivot the third time. This one was way more deliberate. \[00:08:17\] We had revenue. We were like, had some some traction, but we decided to pivot again and we wanted to build something with AI agents and we didn't really understand. We knew from a technical perspective, okay, we're like, this is cool stuff. We wanted to build cool stuff with this, you know, as engineers do. \[00:08:31\] But we didn't really understand how users wanna use this product. So what we started doing was reaching out to a lot of people to see how people would wanna use AI agents. And the thing that kept coming up over and over again was that people were really excited about the possibility of automating, like basically controlling the computer to do something in a company. \[00:08:48\] Okay. And that something was a big, big question mark. So we talked, talked to a few companies that actually building products in the space, but we also talked to a bunch of people who were very excited about, like excited about this by reaching out to random people on the internet, like looking at \[00:09:00\] forums, looking at Discords and so on. \[00:09:01\] And we found a few companies that were really, really excited to automate workflows in their background. And the first one we ended up working with was a company who wanted to automate insurance code generation using AI agents. You can imagine that's really boring. But the product they, they sell is, they're an insurance reseller. \[00:09:16\] You go to their website, you fill out their workflow and then they basically fan out to a bunch of insurance providers like State Farm, Geico, Allstate, and generate quotes for you and that stuff. If anybody, if any of your listeners have experience building scrapers before, that stuff breaks all the time because the layout changes a little bit. \[00:09:33\] They have like all these nuances, and so the question was, could you make that better using ai? Could you make that better? Using computer vision and some form of like GPT-4 S things. And so that's, that's the, we took, tried to take a stab at that. So at first we asked, the few people we ran into we're like, Hey, what have you tried to solve this problem today? \[00:09:51\] They're like, oh, we talked to these companies, nobody solved it for us yet. And we're like, okay, let's try it out. And that's kind of how um, was born. So \[00:10:00\] we ended up working with a handful of these companies. Just to begin with, just to get the shape of our product. Right. And then now we're starting to scale out to more and more companies. \[00:10:06\] ****Mike Bifulco:** I would imagine that many people listening to this podcast have tried to write a screen scrapper in the past, either for like a assignment for a, you know, a course or a code school or whatever, or for the, you know, some real business reason that they wanted to have something automated. And I think having done it once is enough to tell you how painful it can be when things break, and how hard it is to build something \[00:10:26\] ****Suchintan Singh (Skyvern S23):** And it's funny, I've done it too, you know, like when I was kid, I was writing, it's the same idea except now using way better technology, you know, it's, it's great. \[00:10:36\] ****Mike Bifulco:** Yeah, exactly. And so you, you pivoted to, to this sort of third place where you're starting to, to build for automation. If I remember correctly, you had applied to and gotten into yc. So this is how we met. I sort of alluded to this in the intro. We both went through Y Combinator's summer program last year. \[00:10:52\] We were in the the S 23 batch. And at the time you were y Vern, right? So you were in, in, \[00:10:57\] your second pivot building marketplace \[00:11:00\] tooling and, and pivoted there. Can you tell me a little bit about what the YY Combinator experience was like for you and, and some of the things you might have gotten outta that? \[00:11:06\] ****Suchintan Singh (Skyvern S23):** Yeah, so the Wes experience is pretty cool. So I, I, I feel like Wesley's community. It was just like a bunch of really technical people who want to build something cool. And I think that I, I don't really think I real recognize that, but when we applied, we applied because the videos are very, very helpful to us in our first pivot journey into our second pivot. \[00:11:27\] But the experience itself is very cool because you're in this like concentrated group of highly technical people working on really interesting problems and that that is like nothing short of inspirational. Because anytime you meet up with somebody, like even even me and you, right? Every, when we had our convers, when we chatted, I would just be inspired to work hard and solve problems and you know, you could talk with people at a higher level and I think that's something that YC unlocked for everybody. \[00:11:53\] In terms of my experience specifically, my experience was like a little bit unique because I actually had a kid during the YC batch, which I \[00:12:00\] learned I wasn't the only one first of all. But I also don't recommend if anybody's listening and thinking about that. I don't recommend going through I can't imagine. Yeah. \[00:12:08\] Yeah. But \[00:12:10\] ****Mike Bifulco:** So that must have complicated thing. So your kid was born during the batch. \[00:12:13\] ****Suchintan Singh (Skyvern S23):** it was, yeah, she was born during the batch. She was born actually exactly halfway through the batch. \[00:12:16\] ****Mike Bifulco:** Wow. Well, first of all, \[00:12:18\] congratulations. \[00:12:20\] ****Suchintan Singh (Skyvern S23):** I didn't get, I didn't necessarily get the full experience of yc, but it was also kind of extra focusing 'cause we really had time for nothing else. Like you really had time for nothing beyond startup and, and life. \[00:12:28\] And that's it. \[00:12:29\] I think, I think the month after she was born, I lost like 10 pounds because I was sleeping like three hours a night and just like grinding during the day and like, you know, baby taking care of the baby at night. \[00:12:38\] ****Mike Bifulco:** What a wild motivator. That's really interesting. I feel like few people in the world can relate to a desire for things to be automated, like parents of a newborn. \[00:12:46\] Uh, you know, like if, if everything was just a little bit easier, your life \[00:12:49\] ****Suchintan Singh (Skyvern S23):** Well, it's funny because like there, there was a whole class of ideas we were exploring that were like born of that pain, right? Because I was like, okay, so I'm taking care of baby while programming or while doing sales, how can I control my computer with my \[00:13:00\] voice? There was like a whole subset of whole subset of like applications we were exploring, like voice of text, like, you know, computer automation, all based on this huge problem where I was just like, my hands are busy, but I still wanted to, to be productive, right? \[00:13:12\] And so we ended up not, not going down those paths, but something similar I guess is what we landed on. \[00:13:18\] ****Mike Bifulco:** Sure. Yeah. Wow. Well it, it's really interesting. So you, you definitely had a non-traditional path in yc, right? Like probably in the per percent of a percent of people that who've ever been through YC have probably had a kid during the batch. \[00:13:30\] ****Suchintan Singh (Skyvern S23):** Actually it's more common than you think. It's more common than you think. That's what I learned during the batch. It's when you are in a situation, you find out other people are also in that situation. And I learned that it actually, maybe a percent of the batch, I would say is \[00:13:41\] like that. \[00:13:43\] ****Mike Bifulco:** That's fascinating. Yeah, I definitely know some, some friends who I met through IC who got married before, during, or after the batch, and that's a, a similarly like life shifting experience. Usually something that's hard to explain to, you know, a, a potential partner that like, Hey, I know we're getting married like in a month, but I'm \[00:13:59\] have to move to \[00:14:00\] San Francisco now. \[00:14:01\] It's a hard thing to deal with. like you, I had an uncommon YC experience in as much as my, my company craft work is very, very geographically focused. Right. We are in Charlotte, in North Carolina, and so I spent all summer flying back and forth to San Francisco for for meetings because our business is so importantly here, \[00:14:17\] um, I can definitely relate to like, not fitting the pattern. \[00:14:20\] Exactly. Okay. So, so you made it through yc, your pivot into skiver. Then was was also during yc or was that after the batch ended? \[00:14:28\] ****Suchintan Singh (Skyvern S23):** It was after, so it was about a month after. So what happened during the Bachelors were, were. Basically one thing YC focuses you a lot on is like picking one metric and growing that metric only. And so the metric we we had picked was revenue. And so we're like singularly focused on growing, growing that metric. \[00:14:42\] And what happened during the batch was we had one customer that was paying us like some amount of money and we had actually. One commitment from another customer for a bigger amount of money. So we're very excited. Things are going well, and after the batch, we got a third commitment. So things are going amazingly well, right? \[00:14:58\] And we're like, yeah, this is gonna be a big product. It's gonna \[00:15:00\] be huge, you know, amazing. But what happened was right after that, we started looking into our sales data. See, okay, so now that we have three commitments, like what is repeatable about this process? Like what is common in these customers? \[00:15:11\] What is un uncommon in the cus common in the customers that rejected us? So we can start like selecting earlier, if that makes sense. And that was about, like, that was about three weeks after the batch. And we, in digging into that data, that's when we realized that commonality where the companies that were excited about our product were ones that had like their data teams and engineering team structure in like a very specific way where they didn't communicate effectively. \[00:15:32\] And if you look at the total number of companies like that in the world, one, there aren't that many. And two, even if there are many, it's, it, it, it becomes a much less exciting problem to solve when you're not able to help many companies, you know? Be more effective. And so that's why we actually \[00:15:46\] pivoted. Yeah. So it was \[00:15:48\] after the batch, it was \[00:15:49\] about a month after the batch. \[00:15:51\] ****Mike Bifulco:** the scale story is, is dramatically different if you're targeting, you know, half a dozen companies or less, \[00:15:55\] uh, you may still be able to reach an exciting revenue number, but harder. \[00:15:59\] ****Suchintan Singh (Skyvern S23):** Yeah. And then when \[00:16:00\] you're pivoting, you start asking yourself questions. Right? You're like, okay, well is there a way, is there a shape of the product that I can build that maybe appeals to more people? Right. Or is there, is there, or is can we work on this problem somehow? \[00:16:11\] But I think the question that we always ask ourselves when we're pivoting is, do we wanna still go after this market? Do we still want to go after this thing? If we are thinking right, do we take one step back or do we wanna take this full step back? I take the full picture and this entire time, you know, we saw the, the evolution of language models we're getting more and more excited by it and had a lot of experience with it because I'd worked in search and language models have been part of search for, for pretty long time. \[00:16:34\] I would say transformers was, you know. Created to help with natural language processing, which is like core to Google search algorithm. And so that's why we decided to take a leap forward and and do the full pivot instead. \[00:16:47\] ****Mike Bifulco:** Yeah. Okay. And so now we're in the world of Sky Verne and you're automating things with AI agents. , \[00:16:52\] from what I can tell, the point you're at right now is sort of early on in the product building experience. And so can you tell me a little bit about what, the \[00:17:00\] current version of HelloWorld looks like for someone using Sky Verne? \[00:17:02\] \[00:17:03\] ****Suchintan Singh (Skyvern S23):** So we're at, we're at a stage where we're actually manually onboarding customers because the user interface is something we're still trying to get, like correct. We're looking in the process of onboarding. We're learning about how people would want to interact with our product. But today, sky Ver is an API that you call. \[00:17:17\] It's a pure API product. And what you pass it is basically a URL you want to go to. So we have four customers that are live. They all pass us to different URL. The insurance reseller passes us something like echo.com. Other customers pass their own URLs and and then they pass us what we call a navigation goal. \[00:17:35\] And that basically tells our Charles Skiver to go do something. In the case of generating an insurance quote, it's like, Hey, go, go generate an auto insur or auto insurance quote. Don't generate a home insurance quote. You're done when you have gotten to the quote page. And then what? What we're able to do is take that instruction and basically start@geico.com and just keep going until it hits that goal. \[00:17:56\] So it just takes a look. Look at every action that's possible on a screen and \[00:18:00\] just decides, hey, this is the most likely thing to get us towards this goal. Over and over and over again until it gets there. So it'll click on auto, it'll, you know, fill in, fill in information. And then what we also ask our customers to pass in is basically some information that's necessary to complete the goal. \[00:18:14\] So in the insurance case, all of your information basically. And then we just keep using, mapping that information to whatever's on the screen in real time and just like. Boom, boom, boom, until it gets to the goal. So to go back to your original question, which is where are we in the development lifecycle? \[00:18:28\] So we have four customers we're live with today, and we're looking, we're, we're onboarding about three a month at, I would say, at this, at this point. \[00:18:35\] ****Mike Bifulco:** Are there so that's a reasonably small sample set, but in terms of the customers you're courting right now or talking to, are there commonalities between them? \[00:18:43\] ****Suchintan Singh (Skyvern S23):** No, actually, and that's by design. That's actually by design. So whenever you build a product that has like too many applications, you always hit this like decision point where you wanna have you wanna onboard people, like you wanna solve one problem well and onboard lots of people in that, that are in that vertical. \[00:18:58\] Or do you wanna solve \[00:19:00\] a lot of people's problems? Kind of all, and we're actually in the second, we're doing the second one on purpose. And the reason for that is as you can imagine, interacting with the web is very messy. Every single website is like, designed differently. Some, like we talked to this one customer where their workflow in involved interacting with this like website in in, in India, like the state, state government website in India. \[00:19:20\] And halfway through the workflow to refresh the page five times to get a button to show up. Like, how do you, how do you instruct an agent to do that? You gotta like, you know what I mean? Like how do you instruct, how do you instruct any AI to know that, okay, you gotta to this page, you don't see a button you need to click on, you gotta refresh the page. \[00:19:35\] Like, wow. But \[00:19:36\] the reason, the reason we're going horizontal though, is to, to kind of learn about cases like that. And what we found with our product is every time we onboard a new case, we actually end up solving like 10 other cases we didn't know about. And so as we onboard more and more websites. Our coverage increases, and that's kind of the thing we're going after right now. \[00:19:53\] And there'll be a tipping point, whereas coverage will be, you know, maybe 50% of the web or 70% of the web, where then we can start \[00:20:00\] onboarding customers extremely fast. We can open it to everybody. They can have a recent, like a moderately good experience that you would expect, and that's kind of what we're pushing towards today. \[00:20:09\] ****Mike Bifulco:** Oh, that makes a lot of sense. I, I think you, you're more likely to get more broad feedback than to focus on, you know, just solving problems for the, the insurance adopters or whatever it may be in the world. Okay. So let, let's talk about this from the perspective of, of an API developer knowing that our audience for the podcast is generally people who are building and designing APIs. \[00:20:27\] I'm kind of interested to hear, two, two things from you. One is like, what's an interesting use case for for using Sky ver that might stand out to API developers? And then I, I want to ask essentially the same question, but about building Sky ver things you've learned about APIs, but, we'll, we'll do that next. \[00:20:42\] Tell me about like what's an interesting use case for an API developer that they might use Sky ver for \[00:20:47\] ****Suchintan Singh (Skyvern S23):** Yeah, so. I would, I'm gonna answer this question from, from the perspective of an API user. So I would say an API developer would be looking to, you know, build APIs on things that don't already have, have APIs. Right? And actually, \[00:21:00\] if we were to distill why like Sky Run's goal down to what it's actually trying to do, we're trying to solve that exact problem, which is how can we build an API for websites that don't have APIs and might never \[00:21:10\] have APIs. \[00:21:11\] And might never have APIs. And then there's two categories of websites that are like that, right? One is, maybe, there might be more, but two, two big ones that I think about. One is websites that don't want you to have an API. And then there's obviously like, like you could, you could put like insurance code generation in that, right? \[00:21:28\] Geico probably doesn't want to expose an API because even though they have to make the algorithm, they, they used to generate quotes public. They obfuscate it because they don't want people to replicate it. Right. So they, so there's a lot of websites that don't want to expose an API, LinkedIn might be this, you know, Twitter's recently become this with their, the way they charge for APIs and, and so on. \[00:21:47\] So we can help companies interact with websites that don't want you to have an API. Obviously, we don't wanna violate any terms of services or anything like that. So we, we make sure that the, the use cases are reasonable. And then the, and the second type is \[00:22:00\] the websites that. Can't have an API and, and that's like generally I would consider government websites that fall in that category, right? \[00:22:07\] Like legacy websites that are relatively unmaintained. We, we've been talking to like a bunch of companies that are like dealing with old school, like supply chain procurement companies, right? And they have like these content management systems they were using for their purchase orders that were built in like the nineties or the two thousands that are unmaintained, but. \[00:22:28\] You know, or the definition of a moat where your data is so far in it that it's so hard to get it out that nobody's gonna do it, which is like the textbook definition of what a good software moat could be like. Right? So those are websites that would never have an API. And so we really are trying to solve the problem of building an a p on websites that don't have one. \[00:22:45\] So from an API developer's perspective, it's like how do you develop an API that is the API to any website that doesn't have one? And that's a challenging problem, right? That's a really challenging problem. And how can you like build an interface that. Conforms to the, the \[00:23:00\] peculiarities of every website. \[00:23:02\] And what we landed on was actually we just have four fields that kind of, and we use language models to do everything else. So we have one field that's the URL, where, where do you wanna start? Second field, which is what we call navigation goal, which is what do you wanna do on the website? So what navigation actions do you wanna take? \[00:23:15\] And then third is what do you data extraction goal, which is what do you wanna extract? Then fourth is what we call payload which is what data do I need to do everything that you asked me to do? And so since, since the last three fields are unstructured, they are like, like natural language fields, we can then use language models on the backend to map whatever they pass in to things that need to be done. \[00:23:40\] And so the API from from the user perspective is actually very simple. All of our customers use the same one. The things that passed in are totally different, and then we kind of map it back and that creates its own challenges, of course, but I'm happy to get into that as \[00:23:51\] well. \[00:23:52\] ****Mike Bifulco:** yes. I think I definitely would like to, this is, this is very juicy and I think you've just given me the title for this episode of you're building the API for websites that don't \[00:24:00\] have APIs. That's like very, very juicy and very good. Okay. So that, that's a perfect dovetail actually into the next step then. \[00:24:05\] So like you, you have a, a wildly simple at least shape of your API, but three of the four fields are NLP fields. What is \[00:24:13\] it like to build something where you're asking someone for a very abstract input and trying to map that to a real goal? Like how, how do you build around that? \[00:24:20\] ****Suchintan Singh (Skyvern S23):** This is something we're actively iterating on because the problem with the, the benefit of abstract inputs is that the demos are very, very attractive, right? You, You, can make, you can make, \[00:24:28\] something work pretty effectively, but how do you make it work a hundred percent. And we found that the more precise you are in the goals, it's same as when you talk to like chat pd. \[00:24:36\] The more precise you are with what you want to get done, the more accurate it is that being able to get that done and then you can start abstracting that as concepts a way to like more deterministic things like I would say the most of our time is not spent iterating on how we understand the logic coming in from, from our customers, but it's, it's. \[00:24:55\] How do we make sure everything that's happening with this thing that is not \[00:25:00\] deterministic, every binding to it is deterministic, and how can we conform it to be more deterministic? So all of our time is actually spent in like massaging, I guess the, the, the HTML, the, the JavaScript that you see on a, on a browser that, that runs in a browser. \[00:25:15\] To conform to what the language model can interact with and how it relates all the way back to a goal. And so we spend a lot of time helping our customers, one, create the prompts and create the payloads they wanna send to us, but two, finessing the language model to output a specific way. And actually, if you look at how like GitHub copilot is made, they do the same thing. \[00:25:32\] Like most of their remote, I would say is not really the language model. I mean, of course they benefit from improvements in it, but it's how do you force the language model and, and have all these like edge guards in place. So that the output is productive, \[00:25:46\] the output is very productive. Right. It's like extraordinarily productive. \[00:25:50\] Yeah. \[00:25:51\] ****Mike Bifulco:** I, I love this sort of problem for, for so many reasons that like, this is, to me, the core problem of dealing with large language models right now is the \[00:26:00\] non-deterministic angle of it. And I'm sure you've seen articles that have come out recently about I think it was Ford who had a customer ask for a car for free or something like that, and they were like legally obligated to give away a car because their, their \[00:26:11\] chat bot on their site said they could do that. \[00:26:13\] And I think. If there's maybe an airline that did something similar recently and, and all of these things that are coming up that is like the real consequence of using something non-deterministic is that you can offer up a result to someone that is completely not based in reality and, and be legally bound to it. \[00:26:27\] And there's a lot of really challenging things that come along with that. I was talking to another founder recently who, who was musing about LLMs and, and was saying something like you know, trying to imagine themselves as a, as a kid picking up their first program and trying to imagine how they got from like adding up the perimeter of a square to you know, something where it's like, ask an abstract question. \[00:26:45\] I'll do anything you want is, is a pretty wild response. And it's some of the magic of where we're getting to today, but definitely like, involves a, a reasonable amount of, of risk and sort of uncertainty. And maybe that's why someone building a, a startup like you are is isn't a good place to tackle this. \[00:26:59\] That you can kind of deal with \[00:27:00\] that uncertainty and take chances that probably like, I don't know, Delta Airlines or whoever it was, probably shouldn't be taking right now. \[00:27:07\] ****Suchintan Singh (Skyvern S23):** Yeah. . Just to, just to touch on that a little bit, the other side of risk management though, is like also thinking about how the thing is used, right? \[00:27:13\] ****Mike Bifulco:** Mm \[00:27:14\] ****Suchintan Singh (Skyvern S23):** Like, and, and that's the part that we, we think about a lot internally is like, you know, the cost of getting an insurance quote wrong is low. \[00:27:23\] It's like, it's not, it is not like you're processing a transac transaction. It's not like you are, you know submitting like a birth certificate where the cost of an error is like catastrophic, right? And so we generally steer ourselves and ourselves and of course our customers as well to handling use cases where the cost of an error is low and the air error error is not necessarily in like the particular text you input, but the, the result of the combination of effects. \[00:27:47\] And so one thing we don't do, for example, is we don't do deal with any transaction events. Like if you wanted to, for, if you wanted to use our product to go buy, automate some procurement pipelines on websites that don't have APIs, and you wanted to order hundreds of \[00:28:00\] thousands of dollars for the product, we would say we'll build a card for you. \[00:28:03\] We won't check out, you know, we won't check out. And, and that's like a reasonable safeguard to put into place because then the value still generated for you where you don't, you don't have to go through the effort of building a cart for your procurement pipeline, but, but the risk, you can still have a human in the loop to do the final validation. \[00:28:18\] Which is something that is necessary today. Maybe, maybe in five years it might not not be necessary, but today certainly is. \[00:28:25\] ****Mike Bifulco:** Yeah. \[00:28:26\] Brilliant. That, that is a great way to handle risk and that's actually a lot of, of how I talk to. So in a past life I worked at Stripe as a developer advocate. And this was the, the conversation I would have with a lot of people building early stage products is that essentially around credit card information specifically, like you don't ever wanna store a credit card number in your database, and Stripe has a massive team of very qualified people worried about data privacy, and that's why you should offload it there and like trust that stripes the holding of your customer's credit card information is better than you could ever do it. You're almost the inverse of that, where it's kinda like, well, we just wanna do the things , that we are relatively sure \[00:29:00\] the consequence of it is, is reasonable both for, you know, sky ver and for your business, but for your customer's business too. Yeah, I love that that's, that's nuanced and valuable perspective to give to your customers along the way. In terms of use cases for skyr, so you've got a small amount of customers onboarded. Are there use cases that are interesting to you, or ones that you would like to see customers jump on board with that you're maybe particularly excited about or you think are creative or uses for skyr that, that are maybe non-intuitive at first? \[00:29:25\] ****Suchintan Singh (Skyvern S23):** Yeah, I think we've found a bunch of use cases. The one that we're actually actively focused on right now is in the general field of interacting with government websites. So you're a founder when you were, when you incorporated, you can imagine the delusion paperwork you had to fill out with the state of Delaware with, you know, state of California. \[00:29:42\] And we found that there's like a bunch of companies like accounting firms, law firms. Banks even that ha, that have this like back office that either has people doing it or they have like really haphazard pipelines that automate it, where they just interact with those government websites. So something that is probably on top of many founders' minds right now is trialing \[00:30:00\] for Delaware franchise taxes. \[00:30:02\] It is due in seven days. I haven't done it. I don't know if you, your company's done it yet, but you know, it's on my to-do list, but it's something you have to do, and if you do it wrong, you're gonna end up paying like tens of thousands of dollars worth of taxes that are not necessary or get fined. \[00:30:15\] ****Mike Bifulco:** Right, \[00:30:16\] right. \[00:30:17\] ****Suchintan Singh (Skyvern S23):** people have to do manually. \[00:30:17\] Right? And, and the question, is it, does that have to be something that people do manually or can it be automated in the future? And so that, that whole area of use case is something that we were very excited about very excited about automating, which is kind of funny because like, who would be excited about that? \[00:30:30\] Right? I, I certainly wasn't from like a user perspective, but from a a business perspective, I'm very excited about automating that stuff. Yeah. \[00:30:37\] ****Mike Bifulco:** That's, that's the thing. Those are the automations that, that are most valuable is the ones that people want to do the least or that are the most painful for them to do. Super cool. That makes a ton of sense. I, the thing I always I. Tell. Well, so the, the first thing that came to mind for me when you said automating government websites is something that I've had to do a few times, which is baffling when you have to do it is looking up an EIN for a company you own, but you've misplaced and \[00:31:00\] that is so painful to do and such a nightmare, and you can understand all the red tape involved in all this stupid government websites you have to navigate. But I've certainly lost hours of my life to just looking up a eight digit code or whatever that is. Yeah. \[00:31:13\] that's a great use case. I'm super into that. , and looking forward to seeing, you know, the realities of what you're building there. , can you tell me about your team? So like, how big is the company and what is your stack, what's your architecture look like? \[00:31:25\] ****Suchintan Singh (Skyvern S23):** Yeah, so we at the end of yc, we did raise like a small pre-seed round for Wyvern before we pivoted. We ended up stopping the fundraising after that. But we are a team of three people right now. All three, three of us are co-founders. We are all technical people, so we all write code every day. Which is, which is cool. Obviously the, the responsibilities are not equally distributed. Like I, I also write code, but I tend to write the least amount of code because I spend most of my time doing sales as well. \[00:31:49\] ****Mike Bifulco:** Yeah. Okay. \[00:31:50\] ****Suchintan Singh (Skyvern S23):** yeah, from an architecture perspective. So we decided to keep our product relatively lean as, as it is today. \[00:31:55\] This might change. This will definitely change in the future. And so our product is entirely in the, in Python. \[00:32:00\] It's an API product. We actually don't have a user interface. I think this is the truth about startups that people don't necessarily talk about is many companies don't invest in things that aren't necessary. \[00:32:09\] Like we haven't had a user interface for four months now. We've been onboarding customers and it turns out there's a subset of customers out there that are okay with that, that are actually totally okay with not having that. And so we didn't build it yet. We'll build in the future when we have more time and more, more, you know al time allocation to build it. \[00:32:27\] But our stack is fully in Python today. We use like Fast API as our primary API driver. And under the hood we use Postgres just as our database, super base, as our database data store. \[00:32:38\] We run everything on AWS Nothing too \[00:32:40\] crazy. \[00:32:42\] ****Mike Bifulco:** Certainly familiar shape and clever application of, of the tools that you're using to build towards something that makes a lot of sense. I. Okay. And so you're at the point, you've, you've onboarding customers, you're starting to ramp that up. \[00:32:53\] So tell me about what's next. What is the next milestone for Sky Ver that you're looking towards? \[00:32:58\] ****Suchintan Singh (Skyvern S23):** Yeah. So something we've been thinking \[00:33:00\] a lot about internally is how can we solve a few very specific problems that actually have come up in, in our sales conversations. And so there's two, two problems that are pretty, that stand out a lot. So one is. I talk about interacting with government websites a lot as like a use case, but one that I didn't touch on is a vertical that has another set of problems, which is the healthcare vertical. \[00:33:20\] Particularly healthcare and health tech companies do a lot of boring form filling as well, largely because of like compliance requirements. So they have like disparate systems that each store data, and because of compliance, they can't actually like, share, share data. So they have like processes that involve taking data from system A and transcribing the system B. \[00:33:37\] And the problem with these systems is that they require. Every vendor to be compliant, like in some way, whether it's HIPAA compliance, whether it's SOC two compliance many actually even require a self hosted solution. And so an idea that we've been going pretty far down is actually open sourcing a product to one, solve that problem. \[00:33:53\] But two, also just share the cool stuff that we're building with the world. You know, as a developer, playing around with a tool that can interact with the web \[00:34:00\] would be pretty awesome. I mean, there's some websites that certainly wouldn't work for, like if you try to use it on LinkedIn, it just doesn't work. \[00:34:05\] We don't, we don't allow that. But, but there's many websites where it would work for and very cool. So one, it shares the cool of the product with the world and gets developers pretty excited about what we're building. And two, it actually solves a real problem that we've experienced talking to our customers is how do we create a self hosted, reliable version of our product where our customers actually have full transparency on how it works, and open source actually solves that problem completely. \[00:34:29\] And so we're pretty excited about launching an open source version of our product. \[00:34:32\] ****Mike Bifulco:** Wow. That's, that's really cool. That is super exciting as a not only a founder of a startup, but I tend to get into like dangerous side projects on the weekends when I have a silly idea. And there's probably half a dozen things that have floated across my brain as you and I have been chatting. \[00:34:45\] That'd be like, man, I would love to have an automation solution for this thing that I is a, a part of my life for whatever reason. I think open source is a really interesting angle for this too. 'cause no doubt you'll see people with creative ideas that you haven't even thought of yet for feature additions or, you know, tweaks to the API and \[00:35:00\] functionality or shape or whatever the case may be. \[00:35:01\] ****Suchintan Singh (Skyvern S23):** One thing we're really excited about open sourcing specifically is as I mentioned before, like one of the problems we have is the coverage of, of the web with our product is increasing over time. And the truth is we are limited, right? We have three people working. We can only talk to so many people and experience so many websites. \[00:35:16\] One thing we are excited about open sourcing is that maybe that you will try it out for this idea that you had over the weekend. And you'll find an issue with the website and maybe you'll fix it. \[00:35:25\] Right. And that then, then we can really like harness power of the open source community to build a really cool product that can interact with the entirety of the web. \[00:35:33\] Right. That is like the dream we're going towards. And, that could be possible, right? With the, with the open source angle. \[00:35:38\] Yeah. \[00:35:38\] ****Mike Bifulco:** You're enabling a feedback loop for more of the intranet to, to make things more broadly applicable. I would love to be able to, to poke with that. And so you're, you're opening up some open source projects. Presumably those are gonna be based on similar stacks, so Python, \[00:35:52\] uh, fast, API and Postgres in whatever shape. That's really great. And so what about in terms of your team? Are, are you thinking about hiring? Is that something that's gonna come \[00:36:00\] into focus for you anytime soon? \[00:36:02\] ****Suchintan Singh (Skyvern S23):** I, I think probably in six months we'd be looking to hire some people, but for now we're trying to stay lean. \[00:36:06\] ****Mike Bifulco:** Sure. \[00:36:06\] ****Suchintan Singh (Skyvern S23):** we're actually like optimizing for high, high frequency communication and like just when you have a small team, you have only a small number of projects you never work on. And, and that \[00:36:16\] small number of projects actually for startups can be a benefit because it really makes you focus on what is good or bad. \[00:36:24\] And that's something that people don't necessarily always talk about is like the, the real advantage of the small team, which is high, high amount of ruthless prioritization is needed. Otherwise you end up working on things that don't matter. And if you work on things that don't matter to startup, then you launch things next week instead of launching things this week. \[00:36:38\] And that effect compounds negatively. And so we are actually quite a. Happy with how lean our team is today. Obviously it won't be sustainable as we ramp up sales and acquire more customers. You know, there's some use cases we're solving for right now that are one of the companies we're helping with. \[00:36:53\] We help 'em apply to like, job applications online and, and at night we'll wake up and we, and some jobs will fail and we have to manually resubmit \[00:37:00\] them, you know, and so as we acquire more customers, we won't be able to scale that side of it forever. So we'll definitely need to hire for some help, but that's not something we're planning any time in, in the short term. \[00:37:09\] ****Mike Bifulco:** that's really good perspective too. A lot of the successful founders I've spoken to are really good at that prioritization and things like cutting scope dramatically is like a real skill. And just like you said, shipping this week is always more important than shipping next week, especially if you're able to learn from it and sort of iterate more quickly as a result. \[00:37:25\] That's great. So for folks listening to the, the show right now where's the best place to go to get started with Sky Ver. \[00:37:31\] ****Suchintan Singh (Skyvern S23):** So if you go to sky.com, you'll see a link to our GitHub. I would recommend checking it out and you know, if you could clone the repository, play around with it, try a couple websites, maybe open a bug report, that would be super helpful for us. Also, any stars are of course, always appreciated. \[00:37:46\] ****Mike Bifulco:** , for people who may be interested in sort of like. The customer side of things. So if, if people are working on teams that they feel like might be a candidate for someone who would be a good customer for you is there a place to reach out on, on the website there as well?\[00:38:00\] \[00:38:00\] ****Suchintan Singh (Skyvern S23):** Yeah, so if you go to our landing page, there should be a button where you can actually book a meeting directly with me. Feel free to feel free to do that anytime, even if you don't have any customer use, 'cause I'm always happy to chat and share and talk. But if you also wanna get in touch with me, you can always email me at ton at we or@sc.com, so@sky.com and I'll be happy to respond. \[00:38:19\] ****Mike Bifulco:** Okay. Well switching to thank you so much for joining today. It's been a pleasure talking to you. I'm really, really excited to see that you're opening up some open source angles for, for the community to tackle here. And I wish you the best. Please feel free to, to come back and join us anytime if you have other things you wanna chat about or if you have launches upcoming. \[00:38:34\] And tha thanks a lot for joining me today. I appreciate it. \[00:38:36\] ****Suchintan Singh (Skyvern S23):** Oh no, it was . Great. \[00:38:39\] ****Mike Bifulco:** Sure. Talk to you soon. ## Sign up for APIs You Won't Hate The largest community for API Devs on the web. Subscribe Email sent! Check your inbox to complete your signup. No spam. Unsubscribe anytime. ### Revamping OpenAPI.tools URL: https://apisyouwonthate.com/newsletter/revamping-open-api-tools/ Last updated: 2024-04-01T11:54:39.000Z Mike from APIs You Won't Hate has been upgrading a long standing APIs You Won't Hate project, [OpenAPI.tools](https://openapi.tools/?ref=apisyouwonthate.com). If you have no idea what I am talking about, maybe it is time to question whether you even know what an API is 😉. OpenAPI.tools is a collection of tools and resources for working with OpenAPI. It's also community-driven: tools features on the site are [submitted via GitHub](https://github.com/apisyouwonthate/openapi.tools?ref=apisyouwonthate.com). If this is the first you've heard about it, go check it out. You can check out the preview of the [new version here](https://openapi-tools.vercel.app/?ref=apisyouwonthate.com), and have features you'd like to see added to the site, check out the [GitHub project](https://github.com/orgs/apisyouwonthate/projects/5?ref=apisyouwonthate.com). Mike is building the latest version in Astro and did a little write-up on why [new frameworks make us better developers](https://mikebifulco.com/newsletter/why-new-frameworks-make-better-developers?ref=apisyouwonthate.com). I personally love Svelte, but I keep rocking React as it pays for my mortgage 😂😜. \-- Alexander ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2024/03/vscode-ad-100--1-.jpg) ## Introducing a new VS Code Extension for API Insights! Streamline your API development with quality and security assessments directly in VS Code. Enhance your work effortlessly and stay ahead in the industry. Try it now and elevate your APIs to A-grade. [Download the VS Code Extension ](https://apiinsights.io/?utm%5Fsource=newsletter&utm%5Fmedium=apis-hate-sponsorship&utm%5Fcampaign=vs%5Fcode) ## The API Round-Up Our monthly round-up of articles, news and updates from APIs You Won't Hate and around the web. ### [Spectral OWASP API Security Ruleset](https://blog.stoplight.io/spectral-owasp-api-2023-security-ruleset?ref=apisyouwonthate.com) Explore the updated Spectral OWASP API Security Ruleset for 2023: a comprehensive tool integrating the latest API security top ten risks with new rules and enhancements to improve API security. Thank you, Phil, for talking us through the changes. ### [Membrane-Laravel](https://github.com/membrane-php/membrane-laravel?ref=apisyouwonthate.com) Not enough noise is being made about this incredible PHP validator. It works amazingly as Laravel middleware to validate your requests and returns failures, so you don't have to. It is a relatively new project, so check it out and show some love. ### [Webflow OpenAPI Specs](https://github.com/webflow/openapi-spec?ref=apisyouwonthate.com) Webflow is a tool for building websites; think of Wix or Squarespace on steroids. They just dropped the OpenAPI spec for their API. Webflow does a lot to allow developers and non-devs to work together to build a website, but now, with the spec, integrations will be much easier. ### [OpenAPI AutoSpec](https://github.com/Adawg4/openapi-autospec?ref=apisyouwonthate.com) Need last-minute API docs? Check out this project. OpenAPI AutoSpec sets up a local proxy server, listens to traffic, and turns it into an OpenAPI spec. It will capture request bodies, headers, parameters, and responses. Giving you specs you can view in your terminal or download. ### [Why Ryan Dahl made Deno](https://stackoverflow.blog/2024/03/19/why-the-creator-of-node-js-r-created-a-new-javascript-runtime/?ref=apisyouwonthate.com) Ryan covers why he moved on from Node to make Deno—talking about Edge Functions and the new [JavaScript Registry JSR](https://jsr.io/?ref=apisyouwonthate.com). A lot is happening in the JS space, and building APIs can now be built and deployed quickly, thanks to Deno and Deno Deploy. ### [Node.js: The Documentary](https://www.youtube.com/watch?v=LB8KwiiUGy0&ref=apisyouwonthate.com) At some point, you will all realize I am a JS developer and have never touched PHP, and you will get rid of me. Apologies for some JS-heavy content this week, but the Node.js documentary dropped, covering how Ryan Dahl and V8 came together to make the hugely popular Node.js. Node's website also got a [makeover recently](https://nodejs.org/en?ref=apisyouwonthate.com). --- ## APIs You Won't Hate The latest from the team at API's You Won't Hate. ### [🎙️ Building a unified API on the shoulders of OSS with Robin Guldener from Nango](https://apisyouwonthate.com/podcast/building-a-unified-api-on-the-shoulders-of-oss-with-robin) Robin Guldener from Nango talks to Mike about building an open, unified API, the value of building on top of Open Source products, and building a growing product team on this episode of the podcast. --- ### From Our Community Articles written and shared in our free [Slack community](https://slack.apisyouwonthate.com/?ref=apisyouwonthate.com). ### [Rolling versions](https://getconvoy.io/blog/rolling-versions?ref=apisyouwonthate.com) Oluwalana shared this article in our Slack about the trend of rolling versions for API versioning. Rolling versions offer dynamic, backwards-compatible updates. The blog covers what makes it a great choice over other approaches. If you would like more details, check out this [blog from Stripe](https://stripe.com/blog/api-versioning?ref=apisyouwonthate.com) about how they use this approach. ## Support APIs You Won't Hate When you become an member, you'll get access to members-only content while directly supporting our work. Your support helps us to keep making resources for the API community. [Become a member today](https://apisyouwonthate.com/#/portal) Thanks so much to our members for supporting APIs You Won't Hate: Nolan S., Rüdiger S., Alex R, Vedran C, Danny S, Juxt, Kin L., and Olekshii T. Your support means the world to us! Until next time, Alexander, Phil & Mike ### Building a unified API on the shoulders of OSS with Robin Guldener from Nango URL: https://apisyouwonthate.com/podcast/building-a-unified-api-on-the-shoulders-of-oss-with-robin/ Last updated: 2024-04-01T11:00:46.000Z ## Show notes - [Nango.dev](https://www.nango.dev/?ref=apisyouwonthate.com) \- Open, Unified API - [Nango on GitHub](https://github.com/NangoHQ/nango?ref=apisyouwonthate.com) - [Robin Guldener on LinkedIn](https://www.linkedin.com/in/rguldener/?ref=apisyouwonthate.com) [Nango - Open Unified APIShip any integration, fast. Leverage pre-built integrations and limitless customization for 150+ APIs on a single platform.![](https://assets-global.website-files.com/63c092e946f9b71ff6874169/63d398f7a6b96f55cce65f41_icon-256px.png)Open Unified API![](https://assets-global.website-files.com/63c092e946f9b71ff6874169/653a6fc9084dbed7d7ef0b63_open-graph-preview.png)](https://www.nango.dev/?ref=apisyouwonthate.com) #### Transcript \[00:00:00\] ****Robin:** Hi. \[00:00:40\] \[00:00:40\] ****Robin:** Awesome. Sounds good. Yeah, so I started coding when I was 12\. I think like sort of, you know, got out of curiosity and trying to start up when I was 16\. I think I've been sort of in the, you know, startup and B2B SaaS space in a way ever since. I was, you know, writing iOS apps for a few years. \[00:00:54\] Like I. Like built a little agency around that and got bored with like the repetitiveness and, and knew wanted to build more \[00:01:00\] product stuff. So right out of college I started a company called a with a couple of friends and it was basically like a fleet management software for B2B, a B2B for fleet management. \[00:01:10\] And as part of this, we had to build a ton of integrations. So kind of, I would say like, you know, a third maybe of the product in one sense or the other was like, kind of integrations basically. And so we had like about 50 different integrations or, or different providers that we had integrated with. \[00:01:25\] And it took up, you know, sort of a good chunk, I would say like 15 to 20% of our engineering resources for a good of the life of the company. And I was the head of product there. And I always thought like, that's crazy, you know, that we're spending this much time on sort of. Building and maintaining what was essentially kind of data pipelines. \[00:01:39\] And at the same time, my CEO, you know, my co-founder, CEO, he was very annoyed that we were not, did not have more integrations because he sort of felt like, you know, a large chunk of deals sort of would not go through because we were missing integrations. And that's kind of how I realized that like. \[00:01:53\] Integrations are just like, you know, super critical function inside of the company. Super critical function inside of the product that you know, it's kinda like \[00:02:00\] never done. And it's super annoying to maintain. It's actually like, takes a lot more infrastructure than, than you first expect. And so when I left that company, I, you know, knew I wanted to start something again. \[00:02:09\] And I ideas for about six months. \[00:02:12\] On, on this integration idea and realized that it's actually still an, an unsolved problem for a large chunk of. \[00:02:27\] Yeah, I mean, it was a side job, right? You know, I was in high school at that point in time. I kind of started working for a few hours a week next to school for, for, for the startup and like you know, didn't, I didn't pay me a whole lot of money, but like, you know, it was a great learning experience for me. \[00:02:39\] And I. Yeah, like a, a year or so in you know, the, the iPhone came out. It was like the first iPhone you could program for, like, it was like, I think the iPhone 3G or three s or something like that. And I asked them like, you know, could, like, could you like maybe buy me an iPhone as part of like sort of my salary and birthday present in one? \[00:02:54\] And that's how I, you know, got to have an iPhone as 17-year-old and I never could have afforded this myself. And that \[00:03:00\] got me into sort of like iOS development, which led to me doing some. Did that for a few years at about, you know, 20, 25 apps in the app store. Was making decent money for kinda like a high school student, you know, side project. \[00:03:13\] But I just realized it's like not, I guess a repetitiveness of, of the way I was doing it was not something that I really appreciated. It was a good lesson sort of in, in business and, and some finance, but like, I realized that I think like what I really enjoyed at that startup was like building product, having an impact on people's lives and being able to grow something from zero to to one and beyond. \[00:03:34\] And I think that like, really, I guess it's like kind of the thread that goes through the rest of my you know, career at this point is like, sort of starting things and, and building things from \[00:04:00\] scratch. \[00:04:03\] \[00:04:03\] ****Robin:** Yeah, I mean at NGO we really build a single API for all your integrations. And I know that's, you know, there's a bunch of companies out there I think that. They're claiming to, to give you that. I think the thing that makes us different is that we, instead of trying to pre-built as many different integrations as possible, we've sort of built a really great developer experience to build integrations. \[00:04:24\] And that, you know, has a, a bunch of interesting trade-offs. Like it gives you a lot more flexibility on the kind of integration and control over integration, so you can really build exactly the integration that your customers need and, and never be constrained by the platform. But at the same time, we aim to give you a lot of the sort of you know, ship integrations fast and, and low maintenance aspects that you get from sort of like more prebuilt solutions. \[00:04:44\] And I think the secret sauce is sort of in like how we've chosen the abstractions and how we've made those trade offs to give you that flexibility while still providing a lot of support in the areas where it truly matters and, and you're not having to like, sort of learn a. \[00:04:58\] Me and my co-founder started on this \[00:05:00\] about, you know, two years ago almost we, we basically, you know, an exec, so many, you know, teams. I guess we started with like, sort of an idea. And like started talking to like, so I saw this problem right at my last company and, and we started just talking to other, you know, engineering teams and like, so how do you guys build integrations? \[00:05:15\] Right? It's like, you know, it's 2022\. It was at that point in time and, and like, you know, sort of unified APIs that kind of like, you know, second you know, coming with like, you know, the rise of merch and, and fi and some of those other solutions. And so we talked to a lot of people and what we saw is this really, like the vast majority was still like building in-house. \[00:05:32\] And like I. Was usually aware of sort of the unified API solutions, but it just didn't really feel like it fit, you know, with what they were doing. And on the other hand, you have like these embedded IPIS solutions that are more kind of like low-code, no-code tools similar to sier for kind of embedded integrations. \[00:05:47\] But I'd also like, I think to a lot of developers just didn't feel like the right way forward. Like they. End up writing code sort of in a low code, no code tool to like run API requests and something that's like kind of a visual workflow builder \[00:06:00\] when really what we're they we're building right? Is like sort of a, a key feature of the product and the key infrastructure really. \[00:06:05\] Yeah. And so we, we started iterating on like, how could we solve that problem? And honestly think it took us almost a year to sort of find the right approach and find the right angle. And there have been some, you know, interesting detours on along the way, but, you know, I'll save those for conversation. \[00:06:43\] \[00:06:43\] ****Robin:** Yeah, it's a good question. I. We were following what we were hearing from developers and then we tried different approaches sort of to solve the problem. And so like I mentioned that like it took us, you know, almost a year to find the right entry point and like. It speaks to how complex the \[00:07:00\] problem is and I think like how, you know, nuance sort of the solution has to be to be truly helpful. \[00:07:04\] So I think the part that we had to get right was like sort of find the right level of abstraction. Like we knew from the beginning that like what people truly want is control, right? Like they never wanna be constrained. By what they can build with their integrations. They like need full control over their product experience. \[00:07:19\] This is a key feature your customers are asking for. Like if an enterprise customer wants and wants a specific thing and a specific Salesforce integration, like you can't be like, you know, I'm using this framework that makes things really easy for me. But unfortunately that means, you know, I can't like, you know, give you what you want in this integration, right? \[00:07:34\] Like this is not really an option. And so I think what we had to really nail is like sort of these abstractions, and I think we started with like, eventually we figured out what we were gonna do, like sort of start really small. And so we ended up focusing on. For about a hundred APIs, it was kinda like, or 30 APIs I think was like the first version of this product. \[00:07:52\] And so we, we took sort of everything that we had built and like threw everything out that was not o and just focused on making oof a really smooth and really \[00:08:00\] simple experience. That I think sort of got us, you know, the first, you know, foothold in, in foot in the door with, with a, with a number of you know, potential users. \[00:08:07\] And then from there we kind of went back and we're starting to look at all the other things you have to do, like sort of two-way data, things at WaPo handling and, you know, data caching and, and deduplication and, and all those sorts of things that were sort of like added on back in. \[00:08:25\] We could really start untangling the problem for people so that even with a sort of minimalist version, the product could already do things that were meaningfully better than if they were to build in-house, \[00:08:49\] right?\[00:09:00\] \[00:09:08\] Oh yeah. Yeah, I mean, you know, there's two of 2.1 coming up, right? At least or has been coming up for two years or something. And I think it's like, but think the hard part with all, with o what you're alluding to, it think is the problem that we see in all parts of APIs, right? It's like there's sort of semi standards and like sort of conventions on how you do things and that can give you the illusion of things being similar when actually they're not always similar and they behave differently in different edge cases. \[00:09:35\] And I think that's kinda like what we're trying to. To do it. A platform of what we've done with oof is like sort, if we pre implemented oof 450 APIs, right? And still today, every time we implement a new off version, there's like a bunch of like little quirks that we, that we find that we sort of have to like, you know, work around. \[00:09:50\] What we see is like, sort of, there's repeatability in those quirks. And so with the platform that we're building, really we hope to sort of give you the right tools so that it's kinda like, you know, you just need to find the right screwdriver inside of mango \[00:10:00\] and then like it will fit nicely and the screw will actually turn. \[00:10:03\] But if you've gotta like build the screwdriver every time from scratch, it's just very annoying. \[00:10:14\] Oh yeah, for sure. So today we're a full like, you know, solution for, for two-Way data syns, right? Like, that's what most of our customers use us for these days. So if you wanted to say like you know, integrate with A CRM, you wanna maybe create a contact there, but then if you want to create a contact, you typically have to know which contacts already exist. \[00:10:30\] ****Robin:** So we first like sort of sync out all the contexts. You can have a version of those inside of your own, you know, products database. And that's, you know, really useful because now you, you can do much, you know, quicker search. You can enrich the context sort of from the CRM with other data from your system. \[00:10:44\] And then we let you have those full like sort of two-way sync experience. And then also webhook. So we support like, sort of webhooks on, on both sides of the equation. And so you can have like realtime data. There's two way data can also be real time basically with the external a p. And we have like, you know, sort of \[00:11:00\] support for that for 150 different APIs. \[00:11:02\] And I think the, the part that we do differently is, as I said, you know, we sort of not, instead of like, we're not gonna go give you the full prebuilt, like how to create a contact in every CRM that we support. We have some, you know, templates that are prebuilt so that you can get started quickly. But then the goal is really that we have the building blocks. \[00:11:17\] So you can have exactly the way to create a contact in A CRM that fits your app. And so you have full flexibility on the business logic and how exactly you integrate with the app. \[00:12:00\] I imagine. \[00:12:17\] Mm-hmm. \[00:12:42\] Yeah, I think it's a good question. I think it's you know, definitely B two, like we're exclusively doing this for sort of like, you know, integrations inside of B2B SaaS, right? So all our customers are, are B2B SaaS companies where integrations are a core part of their product typically. So, you know, with the, with the rise of AI use cases, you know, obviously almost any AI product, \[00:13:00\] right? \[00:13:00\] Like needs to interact with other. You know, sort of services around it. And so, you know, that's a, a big sort of use case. But then I think another one is, is really sort of like any B2B SaaS that, you know, lives inside of an ecosystem. So like, almost no product today kind of is its own island, right? \[00:13:15\] Like everybody, like your customers basically always have just like, you know. Bundle of other SaaS products that they use around you, and they expect you to sort of seamlessly integrated their workflows. So I think popular categories for us as sort of like, you know, CRMs, accounting systems also ticketing systems are pretty popular. \[00:13:31\] Productivity. But then also like sort of, you know documentation, like kind of notion confluence file storage systems like Google Drive. We support 28, you know, different categories of APIs. And we see that that's really needed because. The way you basically build an integration and the way you integrate with those other APIs always depends heavily on your product. \[00:13:51\] Like it's really, you know, goes to the core of like the value you provide to your customers. And that is gonna shape like what kind of data you need from the external system, how \[00:14:00\] the interaction should be with the external system and what the experience should be for the end customer. And so it's typically I think like, often people will actually sort of like, you know, use some part of CRMs, but then also, I don't know, have some, you know, task tracking systems, which at first you wouldn't think, like, sort of go together in the same, you know, product. But like it, it, it gets like pretty diverse. \[00:14:39\] Right. \[00:14:44\] Yeah, I think they ended up finding today they mostly find us. And I think initially we, we kind of, sort of went to. I think when with the first versions, basically we, we, we posted it in, you know, a couple of slack communities where they're like, sort of were were other startups \[00:15:00\] and, and then the entrepreneurs we wrote some blog posts about our journey on like, you know, how what we were doing and how we were building, like what we were learning. \[00:15:07\] Those got some traction and we brought actually a blog post that was. Got a lot of attention, but like almost, you know, seemed like to, to get the wrong audience in a way. Like, we wrote a blog post about o and like, you know, why Oof is still hard in 2023, and this was published in that year. And it, you know, we came, was very successful I think in, in the sense of like, you know, it really resonated with a lot of people. \[00:15:28\] What we realized is like almost everybody who read it apparently used oof, but used it mostly in sort of context like logging into an. So it didn't quite give us, you know, the users that we were hoping we would be getting out of it. But I think it did help sort of, you know, put the, the name on the, on, on, on the map. \[00:15:45\] I think it's just honestly early on it's just like being creative and, and like sort of you know, I think the way we're thinking about, it's kinda like if, if even just 10% of what we do like ends up, you know, having some effect That's great. So like we just gotta do a lot and try different things and what \[00:16:00\] works. \[00:16:01\] I think another interesting aspect to this is sort of how we. And NGO is fully open source. I think we never even talked about this so far. But like, you know, all the code is on on GitHub, right? Like you can, you can inspect everything. And that I think also helped sort of with, you know, early interest in adoption. \[00:16:16\] But it's a funny backstory to this where we initially built like sort of a data syncing product first and that like, sort of didn't really resonate as much, but we needed off as part of the data syncing product to like basically, you know, be able to offer access token. And when we were looking into that, there was this company called Bearer that had built an oof library called Peasley. \[00:16:37\] That was essentially a tool for, for doing oof with a lot of different APIs. And they had built this a couple of years earlier, had sort of like launched it and, and then unfortunately didn't have the resources to maintain it. So by the time we found Peasley was like, you know, a week or so after they had officially like archived the repository into maintenance mode. \[00:16:52\] Onth, you know. A dozen or so users, I guess, and, and a couple hundred stars. And so we reached out to them and asked like, Hey, \[00:17:00\] you know, this specific thing, like is actually really what we've been thinking about building forgo. Like, you know, would you be okay if we like you know, maybe take this over and maintain it and, and they. \[00:17:09\] Super grateful and, you know, sort of gave us the, the, the, the repo. And that I think has been a boost for us in terms of visibility, but also I hope that, you know, we could give something back to the community. And to this day, actually, we, we keep the WAF part entirely free, open source and, and, and like, you know, free for everybody to use like in, in part for the spirit of like, you know. \[00:17:29\] Making sure that can live on its original mission. And, and a thank you for the gift of sort of free code that we were allowed to take over from. \[00:17:51\] Super lucky. Yeah. Yeah, definitely. And I'm really glad also that, you know, the team had the you know, really like, was, was so \[00:18:00\] welcoming to us taking this over. Like, you know, they, they could have been like, no, this is something that we've built. And, you know, like but I think they really left the spirit of open source and, and made sure that, you know, the project can succeed. \[00:18:09\] No matter how that, what, what format takes. \[00:18:25\] \[00:18:25\] ****Robin:** I mean, I think we integrate with all the public APIs that are out there. So NGO is built to work with any, you know, public API that exists. I think the teams that are building public APIs have a lot of respect for them because I think it's a really hard task. Because you have, you know, such, if I look at like the use cases, right? \[00:18:40\] That like people come towards you and are like, look, here's what I want. Build on top of your API. And it like, you know, ranges so widely. I think it's really hard to do. I think it's like the best you can do is Do your best job in a way, and, and, and like, you know, make sure that you know what you publish is accurate. \[00:18:54\] I guess. Like it's the usual stuff, right? Like if the docs are up to date, that's amazing. Like, you make everybody's life easier. I \[00:19:00\] think one thing that we see is really helpful is like if there's, a P key off. And o that's usually seems to be like sort of a, a best off breach situation. Like O is great for like, you know, integrating when with third party APIs. \[00:19:12\] API key makes it very easy for your customers to still use the API without going through a lot of hassle. Yeah. And then, beyond that, I think it's, you know, it like how you design the API, I guess like, you know, really comes down to your business and what you think is right for your company. And I think like most teams that we see there do a good job at this because they know their product really well. \[00:19:29\] So I think that's you know good that they're doing that. \[00:19:46\] Yeah, that's a good question. So we sort of have like two layers you can think of in, in ngo. One is like sort of the access layer to the API. So this is really about like implementing the authentication method, like o or API key. Along with that, we also do things like, you \[00:20:00\] know, figuring out what other, it's other like rate limits that we should respect. \[00:20:02\] So like, you know, we, we pre-configure those in the platform. Then you're like, how should the retry handling work? Sometimes there's ation that's sort of globally applied to the API. All of these things we sort of like pre configuring. Angle. And then the second part is basically like the, what we call sort of the building blocks of your actual integration. \[00:20:18\] So like the two-way data sinks in a way. Like which data do you wanna read from those external APIs and which data do you wanna write back? And those we don't pre-built. So we usually build a few examples, but then we don't really pre-built like sort of entire integrations there. Because that's precisely where we give our customers control integration. \[00:20:36\] So what gives you basically is like this framework. Right, your actual business logic that interacts with the external API. And for this, actually we're using a lot of LLMs these days. So what we figured out is like, basically because NGO brings you, gives you all the infrastructure and you really just have to decide sort of, kind of like, which data do I want to fetch from the external API? \[00:20:58\] What does that look like? And, you know, how do I \[00:21:00\] transform it into my worldview? \[00:21:04\] That's something that like LLMs know really well once they know the documentation of the external API. So we either pointed at that. And then we have a, you know, A GPT for instance, that we, we fed with, with some examples from Mango Scripts. And we usually get like 80, 90% correct answers. It figures out the endpoints it should be calling. \[00:21:20\] It usually figures out the parameters. It figures out a lot of the data mapping. There's still some editing required, but it makes us at least two to three times more productive when actually building integrations. And we're increasingly, you know, giving the same, you know, option obviously to our clients. \[00:21:34\] And I think they're finding similar results that like they can have something that's custom basically at the speed almost of prebuilt, because thanks to all s it does all the heavy lifting for them of figuring out what to call and how.\[00:22:00\] \[00:22:05\] Right. \[00:22:44\] Yeah, I think like, you know, copilot came up pretty much with us starting mango or, or a little bit before. I don't remember the exact timeline, but I think it's like sort of, you know, I think we were using. Co-pilot and, and LMS to some degree to build, you know, NGO from, from day one. Right. And so I think it \[00:23:00\] naturally extended to us that, you know, when with the rise of Jet GPT, basically we would start to ask Jet G PT to like, you know, it's great at other coding tasks. \[00:23:06\] Like why shouldn't it be great at like helping us write, you know, integrations. And I think it's like, you know, you mentioned like you, we don't, we don't see LMS as kinda like a black box solution, at least not at this stage for building integrations. And I think, you know, our, you know, one of the philosophical principles in a way of mango is that like we put you, the customer and the engineer who builds the integration into full control, right? \[00:23:29\] Like you always have the last word on what goes out and you can always build whatever you want with the platform. We're is there to make your life easier. And I think, you know, that LMS in, in deploying them on Mango is a natural extension of that. So. Yeah, I think it was always clear to us that like it would be more of a productivity booster and then eventually if it can do like 99% of the job for you and you're kind of just like, oh yeah, that looks great, you know, awesome. But I think the other thing is also with LLMs and, you know, with prompts in general, right? It's like it's natural language and like we, even as humans, like when we're talking to each other on what we're \[00:24:00\] gonna build, have like, you know, misunderstandings of what we were communicating with like sort of natural language and describing what we wanna have built. And so I, you know, not sure that like sort of. Ever be able to kind of like read our minds? Because I think sometimes when we write out what we want, we don't even know yet entirely ourselves. Yeah. So I think it will be interesting to see how that, you know develops. But I definitely have been a big productivity. \[00:24:22\] boost for everybody in our customer base that has been, you know, using them to, to build on top of \[00:24:27\] ngo. \[00:24:27\] ****Mike Bifulco-1:** like you had really good timing for starting to build the company as well, and in as much as copilot, you know, sort of started to coexist with you at the same time there too. Can you tell me a little bit about how pricing works for Mango right now? \[00:24:39\] ****Robin:** Sure. We have a free tier, you know, to get started for like smaller you know, early stage startups. And then we have like sort of startup friendly, you know, sort of early category, like early pricing. So that, you know, you can really focus on getting to product market fit and like sort of, you know leverage language for that. \[00:24:53\] And, and we see that people also there each trade a lot. And then, you know, we have basically a scale plan that's really meant for you to scale. \[00:25:00\] Thing that is very important for us is that like, you know, we try to align our, our pricing with like sort of the value integrations that driving for your business and, and sort of the usage that you have on the platform. So we're not big fans of like sort of you know. Prohibiting. I mean, we definitely wanna be prohibitively expensive, right? And like, sort of, we believe, basically we wanna power all your integrations, right? We understand that means that like we need to be on the same boat of like, if you are really successful building a successful business, having a lot of customers use those integrations, like, you know, we should get like a fair share of that success and, and, you know, we should be able to, to cover our costs. \[00:25:34\] But we also believe that, you know, you should be able to have a very profitable business. I think that's, you know what, yeah, with integrations basically, I think that's not always easy because like they are so custom to the product and so I think a lot of our pricing packages end up reflecting that and end up sort of, you know, being really built in collaboration with the customer to figure out like what you know, makes sense for their business and for their specific use case of \[00:25:57\] integrations. \[00:25:59\] ****Mike Bifulco-1:** to have some flexibility \[00:26:00\] there too, and hopefully. When everyone wins, happier. And, uh, you're all, you know, building together in the direction towards, uh, a future that makes not only their tools better in their company and product, but also your product can continue to support, you know, whatever's pipe. \[00:26:12\] Um, Especially as new versions of the open API spec and I'll \[00:26:16\] things, uh, start to be \[00:26:18\] ****Robin:** No, I think we, we definitely see it as a partnership, right? Like with, with our customers. Like it's, you know, integrations are never done. They always keep evolving. They always keep changing. And, you know, as you said, like, you know, the environment keeps changing because you've integrated with third party APIs. \[00:26:30\] Those keep changing. And so it's like you know, we definitely see it as a, as a long term partnership, more than like, you know a quick, like build it once and then like you're done \[00:26:38\] forever. \[00:26:40\] ****Mike Bifulco-1:** the right strategy to have. , Uh, everyone, what's the phrase? Um, uh, A rising tide raises all ships or whatever. Uh, The, the old trope is \[00:26:48\] ****Robin:** Yeah. \[00:26:48\] ****Mike Bifulco-1:** Uh, So we've talked a little bit about what, what Mango does and, um, how teams are adopting it, why teams are adopting it. I'm curious if you can maybe describe, what sort of like the typical first \[00:27:00\] experience with Mango is. \[00:27:01\] So what's HelloWorld? Uh, When someone first jumps in to give the tools a try. \[00:27:05\] ****Robin:** Yeah, good, good question. So actually when you sign up for the, for the, for the cloud version of the product, like we immediately have getting started. That lets you connect to GitHub and sink in issues from GitHub. So you can see how that works. And then you're actually gonna be able to create an issue on a public repo as well. \[00:27:21\] And so like we have like sort of, you know, test that, that we use for this. But you see basically like what. The end user experience flow is going to be, so if a customer goes and authenticate, you know, an integration of your product, what, what will they see? Like that experience is fully wide label. \[00:27:34\] Like you can customize it entirely. Like we're literally just the JavaScript SDK that you call. It's nice, you know, to, to see that for people. And then you get to experience the Nway APIs as well. So you see like how the two day data syn work, you see how you know. How you're going to interact from your app with the platform. And like sort of, you know, the flexibility also of this. And I think we see that that's like a, a great way for people to imagine, okay, like, is this gonna like work for my use case? \[00:28:00\] And start to like sort of architect in their mind on like, how will I integrate my integrations here And like, how could that work for my product? \[00:28:07\] ****Mike Bifulco-1:** Yeah, I actually don't think we talked about this early on, but Mango, um, supports, uh, is it TypeScript at the moment? Are there other languages that you're supporting or planning to support? \[00:28:16\] ****Robin:** Yeah, so like the language itself is, is sort of language agnostic. \[00:28:19\] So like, you know, any, any backend, you know, language that you use like is fine. Like we have a rest, API for you to interact with. We have SDKs for various different languages. And then there's a. Parts of the, the integrations yourself that you write, they're, they're written in code, they live inside of your gire, they're version controlled. You deploy them to NGO with a CLI and you can think of those as like, you know, very small sort of TypeScript Lambdas that just do some interaction with the external API. And they run in the context of sort of the NGO frameworks. So those are written in TypeScript. But as I mentioned, you know, there're usually sort of 20 to maybe 50 lines of code. \[00:28:50\] Pretty simple types code. And, and that needs to be just because it runs on the platform itself, but is. \[00:28:56\] ****Mike Bifulco-1:** That makes a lot of sense. Yeah. A, a very interesting developer story. I \[00:29:00\] like that it's, um, uh, source controlled as well. Uh, The, the having that sort of history of how the integration plugs itself in is really helpful, I'd imagine when it comes to debugging and also environments and things like that as well. \[00:29:12\] ****Robin:** Yeah, exactly. So I think that's a big reason for us, right, is like, sort of a lot of our customers basically will deploy their NGO integrations as part of their ci slash cd process. So because it's just another like folder in your repo, basically it follows, you know, your regular sort of coding, deployment flow, whatever, you know, that might be. The only thing you gotta make sure is like when you deploy the production, you, you have the latest version on Mango, and that's as simple as like running a single CLI \[00:29:35\] command. \[00:29:36\] ****Mike Bifulco-1:** Okay, a couple more questions for you. Uh, and then, then we will, uh, head on our ways here. what are you thinking about next? What are like the problems you're interested in solving or the, the big features you're looking forward to, uh, building next with Mango? \[00:29:46\] ****Robin:** Yeah, good question. We're building out more on the, on the webhook side. Like we have support for webhooks for a lot of the APIs already. But, you know, definitely building out more there. I'm very excited about our initiatives to leverage LMS more, to build integrations more quickly on the platform. \[00:30:00\] That I think is something that I'm, I'm, I'm very excited about and honestly, but I still see that we still have a lot of potential. And then the other parts is sort of like, I think you know, one thing that we. We already let you customize integrations for individual customers. So what this means is basically like if you, if you have a customer that has Salesforce, right? \[00:30:16\] Like, I mean, almost every customer customizes their Salesforce. And so we see that like, sort of those differences in Salesforce can make it tricky to build integrations against those kind of you know, more complex systems. We have some helpers already there for you today, but like we have a couple of good ideas on like how we can make that a lot less painful in the future. And that's something I'm really excited about because I think the world of SaaS is in a way moving more and more towards customization and more complex systems as like everybody has all of their data and software. They want their software to be more flexible, to really fit their needs. And so I think we're gonna see much more customizability of different, you know, systems and APIs in the future. And so I think supporting our customers in supporting their customers you know, with this, I think is a, is an important part \[00:31:00\] of our mission. \[00:31:01\] ****Mike Bifulco-1:** Yeah. And again, that's another great place where you can grow together and, and sort of hopefully support features that help your, your customers do better cool. Robin, what about, uh, your now? Are you currently hiring, \[00:31:11\] ****Robin:** Yeah, we're always growing. We have just had a, you know, good, growth spurt, but we are always looking for, you know, talented people. Especially on the backend engineering side. If you are, you know, excited about APIs maybe have experience building integrations, would love to build an open source dev tool for other developers. \[00:31:25\] We'd, you know, love to hear from you. \[00:31:27\] ****Mike Bifulco-1:** what's the URL? Where can people go to find mango? \[00:31:30\] ****Robin:** Sure that's www ngo. Or just Google Mango. That's like mango? \[00:31:35\] with an N and you're \[00:31:36\] gonna find us \[00:31:38\] ****Mike Bifulco-1:** I'll make sure I have some links in the show notes as well, both to your website and your, uh, repos on GitHub. \[00:31:43\] and one last question for you, Robin. Where's the best place to find you online? If people wanna shout about integrations and, uh, APIs for, for integrating and things like that, where are you most active? \[00:31:52\] I. \[00:31:53\] ****Robin:** You can always find me on LinkedIn. If you look for Robin Mango, you'll find me there. You know, when you sent me \[00:32:00\] a connection request, maybe, you know, call out the show here. That I. \[00:32:03\] know you know where you're coming from and I'll be happy to accept and you know, we can message there. Otherwise, I'm also active on the NGO Slack community. \[00:32:10\] If you are looking into ngo, you're probably gonna end up there and I'm always \[00:32:13\] active there. \[00:32:15\] ****Mike Bifulco-1:** Well, Robin, thanks so much for joining today. It's been, uh, really interesting to talk to you and, and to hear about Mango and how you got to where you're at. Um, Feel free to come back anytime. Uh, If you've, you've got launches in the works and things like that, we'd love to hear from you again. \[00:32:26\] Thanks so much for joining. It's been a pleasure. \[00:32:28\] ****Robin:** Thanks a lot for having me, \[00:32:29\] Mike. \[00:32:30\] ****Mike Bifulco-1:** Take care. ## Sign up for APIs You Won't Hate The largest community for API Devs on the web. Subscribe Email sent! Check your inbox to complete your signup. No spam. Unsubscribe anytime. ### So Long, OpenAPI PetStore URL: https://apisyouwonthate.com/newsletter/so-long-openapi-petstore/ Last updated: 2024-03-15T15:50:04.000Z We all appreciate a great example, and the longstanding go-to for OpenAPI specs has been the Pet Store API, which now feels quite outdated and fails to showcase current best practices. Thankfully, Phil and the talented team at Bump have introduced a fresh, modern example: the [Train Travel API](https://bump.sh/bump-examples/doc/train-travel-api?ref=apisyouwonthate.com). Additionally, the [Museum API](https://github.com/Redocly/museum-openapi-example/?ref=apisyouwonthate.com) serves as another outstanding reference. Both examples are instrumental in guiding you towards crafting excellent OpenAPI specifications. I am currently adding a task to Jira to ensure my team's API follows these guides. – Alexander ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2024/03/vscode-ad-100--1-.jpg) ## Introducing a new VS Code Extension for API Insights! Streamline your API development with quality and security assessments directly in VS Code. Enhance your work effortlessly and stay ahead in the industry. Try it now and elevate your APIs to A-grade. [Download the VS Code Extension ](https://apiinsights.io/?utm%5Fsource=newsletter&utm%5Fmedium=apis-hate-sponsorship&utm%5Fcampaign=vs%5Fcode) ## The API Round-Up Our monthly round-up of articles, news and updates from APIs You Won't Hate and around the web. ### [Train Travel API: A Modern OpenAPI PetStore Replacement](https://bump.sh/blog/modern-openapi-petstore-replacement?utm%5Fsource=twitter&utm%5Fmedium=blog&utm%5Fcampaign=train-travel) If you want a deeper dive into the new Train Travel API, Phil has written this excellent blog for Bump. He breaks down some of the components of the new example, the motivation behind it and why it's a lot better than the Pet Store example. ### [Add OpenAPI tags for next-level API descriptions](https://redocly.com/blog/tags-are-great/?utm%5Fcontent=286053476&utm%5Fmedium=social&utm%5Fsource=twitter&hss%5Fchannel=tw-983968312697806848) Explore the magic of OpenAPI tags with Lorna Mitchell's insightful piece on how they revolutionize API documentation. See how tags enhance navigation, improve endpoint discovery, and bring structure to your API descriptions. Get inspired by practical examples like the Museum API to see tags in action. ### [Testing Laravel Nova Actions](https://philsturgeon.com/testing-laravel-nova-actions/?ref=apisyouwonthate.com) Phil guides you through testing Laravel Nova Actions effectively, sidestepping the need for full browser testing. Using the Tree Tracker project, he provides a step-by-step approach to feature testing Nova Actions directly. This is a good read that will help you simplify the complexities of automated testing. ### [Express is Back](https://github.com/expressjs/express/releases/tag/4.18.3?ref=apisyouwonthate.com) That's right. The beloved and most used Node web framework is finally getting updates. A massive chunk of the JS ecosystem has been building all its web APIs on a framework without meaningful updates for years. However, this is a thing of the past. Express is back with its first update and plans for the [following three versions](https://github.com/expressjs/discussions/issues/160?ref=apisyouwonthate.com). ### [Terminal UI For OpenAPI](https://github.com/zaghaghi/openapi-tui?ref=apisyouwonthate.com) A fun Terminal UI is built in Rust for browsing an OpenAPI Spec inside your terminal. The demo gif is fantastic, so I am linking it here. ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2024/03/demo-1.gif) --- ## APIs You Won't Hate The latest from the team at API's You Won't Hate. ### 🎙️ [Building a developer brand and simple API for email, with Zeno Rocha from Resend](https://apisyouwonthate.com/podcast/building-a-simple-api-for-email-with-zeno-rocha-from-resend/) Zeno Rocha from Resend shares the story of his Y Combinator backed company, and what it's like to build great open source tools, focused & minimal APIs, and a great engineering team. --- ### From Our Community Articles written and shared in our free [Slack community](https://slack.apisyouwonthate.com/?ref=apisyouwonthate.com). ### [Five more Burp extensions for API hacking](https://danaepp.com/5-more-burp-extensions-for-api-hacking?ref=apisyouwonthate.com) Once again, Dana shares another blog post on API hacking to help with API security testing. His latest blog runs you through five amazing Burp extensions to make the job easier. ## Support APIs You Won't Hate When you become an member, you'll get access to members-only content while directly supporting our work. Your support helps us to keep making resources for the API community. [Become a member today](https://apisyouwonthate.com/#/portal) Thanks so much to our members for supporting APIs You Won't Hate: Nolan S., Rüdiger S., Alex R, Vedran C, Danny S, Juxt, Kin L., and Olekshii T. Your support means the world to us! Until next time, Alexander, Phil & Mike ### Building a developer brand and simple API for email, with Zeno Rocha from Resend URL: https://apisyouwonthate.com/podcast/building-a-simple-api-for-email-with-zeno-rocha-from-resend/ Last updated: 2024-03-15T15:34:06.000Z ## Show Notes - Zeno's flash->javascript demo with HTML5, Wormz - [https://zenorocha.github.io/wormz/](https://zenorocha.github.io/wormz/?ref=apisyouwonthate.com) - [Zeno Rocha](https://zenorocha.com/?ref=apisyouwonthate.com) - [@ZenoRocha](https://twitter.com/zenorocha?ref=apisyouwonthate.com) on Twitter - [@zenorocha](https://www.threads.net/@zenorocha?ref=apisyouwonthate.com) on Threads - [Dracula theme](https://draculatheme.com/?ref=apisyouwonthate.com) - [React email](https://react.email/?ref=apisyouwonthate.com) - [Resend.com](https://resend.com/?ref=apisyouwonthate.com) - [Resend Forward](https://resend.com/forward?ref=apisyouwonthate.com) - Broadcasts - marketing emails - Batch email sending - send 100 emails with one API call #### Transcript ## Support APIs You Won't Hate When you become an member, you'll get access to members-only content while directly supporting our work. Your support helps us to keep making resources for the API community. [Become a member today](https://apisyouwonthate.com/#/portal) ### Frontend with OpenAPI URL: https://apisyouwonthate.com/newsletter/openapi-to-frontend/ Last updated: 2024-03-06T18:21:54.000Z My day job is all things backend development; well, mostly. I also love all things frontend. React, Vite and Remix have become my go-to tools for building websites and web apps. That's right, I love JavaScript, and I won't stop pushing it till we all embrace the one true language and use nothing else 🤣😉. Anyway, jokes aside, we can agree on the importance of the API Design First approach. Can we reuse this effort that we are putting into our OpenAPI spec? Well, you're in luck. I have put some great tools together to help generate queries, hooks, validations, and more in your front end with your OpenAPI specs. \-- Alexander Thanks so much to our sponsor: ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2024/03/vscode-ad-100--1-.jpg) ## Introducing a new VS Code Extension for API Insights! Streamline your API development with quality and security assessments directly in VS Code. Enhance your work effortlessly and stay ahead in the industry. Try it now and elevate your APIs to A-grade. [Download the VS Code Extension ](https://apiinsights.io/?utm%5Fsource=newsletter&utm%5Fmedium=apis-hate-sponsorship&utm%5Fcampaign=vs%5Fcode) ## The API Round-Up Our monthly round-up of articles, news and updates from APIs You Won't Hate and around the web. ### [Orval](https://orval.dev/?ref=apisyouwonthate.com) Dive into the fun of coding with Orval, where turning OpenAPI specs into TypeScript magic is just a click away! It's not just about TypeScript; you can craft query code effortlessly for React, Vue, and Svelte using Tanstack Query. This combo makes building and maintaining robust web applications easier, keeping them in line with your OpenAPI spec. ### [Quicktype](https://quicktype.io/?ref=apisyouwonthate.com) Standing out from the other tools/libraries I recommended this week, Quicktype can work with JSON or GraphQL queries. This lets you generate types and API clients in Kotlin, Swift, TypeScript, Go, Java, and more. It is a fantastic tool for all languages. ### [OpenAPI Qraft](https://github.com/OpenAPI-Qraft/openapi-qraft?tab=readme-ov-file&ref=apisyouwonthate.com) Unleash the power of Tanstack Query with OpenAPI-Qraft react client —a game-changing toolkit for React apps that simplifies API interactions. Make your development journey smooth and efficient by generating type-safe API queries from your OpenAPI spec. Dive deeper into their toolkit with the CLI package to enhance your project's automation and efficiency. ### [Kubb](https://www.kubb.dev/?ref=apisyouwonthate.com) Take things further with Kubb generating not just Tanstack Query but SWR (another popular networking library), Zod for validation, MSW for mocking and more. Kubb can build half of your front end using your OpenAPI spec —another reason to be API design first. --- ## APIs You Won't Hate The latest from the team at API's You Won't Hate. ### [How to Build a Perfect API Description](https://apisyouwonthate.com/blog/how-to-build-a-perfect-api-description/) Sometimes, choosing tools and applying best practices is tricky because there are so many. Thankfully, Phil cuts through the noise and gives you the rundown on how to build the best quality OpenAPI documentation. ### 🎙️ [Chatting with Quobix about Vacuum, WireTap, and libopenapi](https://apisyouwonthate.com/podcast/vacuum-wiretap-quobix/) On this episode of the podcast, Phil has a chat with Quobix about his tools for API linting, testing, and compliance. We hear the story behind Open Source tools Vacuum and Wiretap, and Quobix's journey building highly performant devtools with golang. --- ### From Our Community Articles written and shared in our free [Slack community](https://slack.apisyouwonthate.com/?ref=apisyouwonthate.com). ### [Documenting Ruby on Rails APIs with OpenAPI](https://bump.sh/blog/documenting-ruby-on-rails-apis-openapi?ref=apisyouwonthate.com) Bump has spent much time with Phil and wrote a blog for a concise guide on documenting Ruby on Rails APIs with OpenAPI. This post offers essential insights for Rails developers aiming to enhance their API documentation, ensuring clarity and improving usability. A valuable resource for those dedicated to refining their development practices ### [Writing API exploits in Python](https://danaepp.com/writing-api-exploits-in-python?ref=apisyouwonthate.com) Explore the art of securing APIs with Dana Epp's enlightening blog post, focusing on crafting API exploits using Python. It's a treasure trove for developers and security buffs eager to navigate the nuances of API vulnerabilities. Dana, thank you for sharing. ## Support APIs You Won't Hate When you become an member, you'll get access to members-only content while directly supporting our work. Your support helps us to keep making resources for the API community. [Become a member today](https://apisyouwonthate.com/#/portal) Thanks so much to our members for supporting APIs You Won't Hate: Nolan S., Rüdiger S., Alex R, Vedran C, Danny S, Juxt, Kin L., and Olekshii T. Your support means the world to us! Until next time, Alexander, Phil & Mike ### Princess Beef Heavy Industries, Vacuum, Wiretap: API linting & Compliance with Quobix URL: https://apisyouwonthate.com/podcast/vacuum-wiretap-quobix/ Last updated: 2024-03-01T13:29:21.000Z ## Show Notes - [Quobix](https://quobix.com/?ref=apisyouwonthate.com) \- Dave Shanley's site. Code is art - [daveshanley/vacuum](https://github.com/daveshanley/vacuum?ref=apisyouwonthate.com) on GitHub - the world's fastest OpenAPI 3, OpenAPI 2 / Swagger linter - [wework/speccy](https://github.com/wework/speccy?ref=apisyouwonthate.com) on GitHub - Well Spectually 🤓 Enforce quality rules on your OpenAPI 3.0.x specifications. - [Wire⚡️Tap](https://quobix.com/articles/introducing-wiretap/?ref=apisyouwonthate.com) \- is the world’s coolest OpenAPI compliance and testing tool, from Quobix - [Battle of the API Ratings](https://apisyouwonthate.com/blog/api-insights-vs-rate-my-api/) from APIs You Won't Hate - Dive into a detailed comparison of Trebble's API Insights and Zuplo's Rate my API, exploring their unique approaches to evaluating API design, performance, and security - [Openapi.tools](https://openapi.tools/?ref=apisyouwonthate.com) \- an Open Source and community-driven list of high-quality, modern tools for OpenAPI from APIs You Won't Hate. ## #### Transcript \[00:00:00\] ****Phil Sturgeon:** Get my sexy radio voice on. Ah, no, I'm \[00:00:03\] ****Quobix:** Radio dj. \[00:00:05\] ****Phil Sturgeon:** that was a bad joke and everyone's upset. Hello everybody. Welcome to another episode of APIs. You won't hate the podcast version. And yeah, I'm excited once again. I'm running the show. No, Mike to help me out, so bear with us. I think it's recording. I guess you'll find out. I, today I am joined by David Shanley who you may have heard all over the place. \[00:00:30\] He's popping up. I, I was talking to him at API days. In Paris recently and he's made a whole bunch of really cool tools. So David, do you wanna tell everyone a bit about yourself? \[00:00:42\] ****Quobix:** Thank you for having me, Phil. Yeah, so my name is Dave Shanley. I go by the name of Quos. You know, but, you know that's just, you know, a stupid internet thing. But anyway Dave, Dave Shanley. I, \[00:00:52\] ****Phil Sturgeon:** I, I called you David. Sorry. I, I, I go, I, I'm, I'm posh Now I go too formal. That's upsetting. Just Dave, \[00:00:59\] ****Quobix:** \[00:01:00\] No problem, no problem at all. So I'm the founder of Princess Beef Heavy Industries or PB three, three F. You know, it's, it's, it's, it's shorter to write, but it's just longer to say in all, in all of all forms. \[00:01:12\] ****Phil Sturgeon:** What is that? name about? Tell us some more about that before you get back to the intro. \[00:01:16\] ****Quobix:** Yeah, so actually I was, I was thinking of, you know, what do I call, what do I call, what would I call myself or would I call my company? And it was actually my 2-year-old daughter, which she was two at the time. And she came up, she used to watch this show. It was blaze in the Monster Machines, which is a show about monster trucks. \[00:01:31\] And she said, I wanna watch Princess Beef. I'm like. What Princess? Princess Beef, there's no mentions of princesses or beef or anything like that in the show. She just called it Princess Beef. So then, you know, actually it was a little while longer, she said she was playing the drums, like toy drums and she called them the honey drums and I put the two together and thought Princess Beef and the honey drums. \[00:01:52\] Isn't that a great name for a band? If I was younger? You \[00:01:55\] know, I thought, I thought, yeah, may maybe, maybe. But then I thought, you know, it's bit too much for \[00:02:00\] company name, but Princess beef and then. I looked it up and there's already a company called Princess Beef. They're a ranch in Colorado, I think. \[00:02:08\] And they, they, they raise cattle. \[00:02:10\] So I thought, well, okay, well how do I differentiate it? And I was thinking back to, you know, my, my, my nineties roots and there was a wonderful company called Loft Heavy Industries, or. 10 PHT depends on how you, you would say the acronym. It was a hacker think group hack, hacker tank think tank type thing. \[00:02:27\] And I just love that heavy industries element. And there was like a bunch of skater companies that had heavy industries on there, you know, postpay it to the name. I thought, why not? Let's, let's go right back to the nineties and re, you know, relive all of this, this joy. Anyway, so that's the, that's the reason why it's called that. \[00:02:42\] No, nothing else other than I just, I love \[00:02:45\] ****Phil Sturgeon:** software. I like the approach. I'm, I'm, I'm apparently trying to do like the punk version of the Woodland Trust, so I like the, the punk version of software. Let's, let's do it. Keep it interesting. Alright, brilliant. So I interrupted you, but you were telling us about a bit more about what yourself and what you get up to.\[00:03:00\] \[00:03:00\] ****Quobix:** Yes. So I'm the f founder of Princess Beef. I'm also, you know, that's, that's what I do by, by night. But I also have a full-time job as a distinguished engineer at a company called Splunk. If you haven't heard of it, it's it's like we do. Big data. We collect all the logs, all the information, put it into our big index and allows you to make sense of it. \[00:03:19\] And we build premium applications on top of that platform. So yeah, that's what I do during the day \[00:03:23\] and \[00:03:24\] ****Phil Sturgeon:** have heard of them. The, that's, it's like a pretty big deal, right? It's like a pretty big company or how many? A few are \[00:03:31\] ****Quobix:** Yeah, it's, but there's, there's, there's about 8,000 people in the company, maybe 9,000 now. It's pretty large. It's actually just been bought, acquired, well, it hasn't closed yet, but Cisco has just purchased it for 27 billion, which \[00:03:43\] ****Phil Sturgeon:** Good lord. \[00:03:44\] ****Quobix:** largest. Yeah, it's a larger, large amount of money. So yeah, we'll see what hap what happens with that. \[00:03:48\] But \[00:03:48\] ****Phil Sturgeon:** much of that do you get? \[00:03:51\] ****Quobix:** 0.0001% \[00:03:54\] ****Phil Sturgeon:** But yeah, there, there was a number at the end of all those zeroes though. So there is something, there is \[00:03:59\] ****Quobix:** \[00:04:00\] There is some, there is definitely something. But yeah, definitely not what the CEO is gonna be walking away with for sure. \[00:04:05\] ****Phil Sturgeon:** now. Fair. Alright, well that's, that's, that's news. It's not, it's not bad news, it's news. Fantastic. And so on the side you are working on a bunch of really cool stuff and so. I, we, we spoke a bit. You've been helping out with open API tools, the website that lists, guess what? A bunch of open API I tools. I dunno why I expanded on that. \[00:04:26\] It's in the name. It does what it says on the tin. But we'd been a little bit snowed under and you were helping with some pull requests and I appreciated that. And then, yeah, I got to see you give a talk at API days in Paris. I went over there, was running about. Who needs some work? Do you need some work? \[00:04:41\] Can I write for you? Oh God. Work. You know, work, work work is hard when you don't have a regular, regular job. I just have to go and \[00:04:48\] ****Quobix:** It is, yeah. You know, trying and lots of talking to people \[00:04:51\] ****Phil Sturgeon:** Yeah, it was the first time I've actually had to do networking, right? Like, normally I just turn up at a conference and I'm like, the company I'm working for paid me to be here. \[00:04:58\] Here's my talk. I'll be at the \[00:05:00\] pub from here on out. But this time it was very much like I actually need to go to every booth and shake every hand and talk to everyone and see if anyone has any paid work for me anyway. Seeing your talk was, was a, a nice refreshing break from a lot of, like, AI will save the world and, if you don't need to build an API, you don't need to build an API Ever 'cause you can just get AI to do it. Although you have to write a specific API for the AI to work. Just a lot of very silly talk. Brilliant talk. We won't, we won't get onto that tangent. Good talk from you. You were, it, it was, I really enjoyed the talk considering you spent the a good half of the talk, making fun of spectral a tool that I helped to build. \[00:05:36\] So I did what any rational human being would do and we all, we went to the pub and shot the shit and it was great. You built, you built vacuum, which is kind of like a go equivalent of, of spectral. And to save me butcher in the introduction, like, tell me, tell, tell our listeners a little bit more about vacuum. \[00:05:52\] ****Quobix:** Yeah, so, so it's, it's it's inspired by spectral it's, it's it's based on the same concept, so it's compatible with \[00:06:00\] spectral rules and core functions and things like that. So you. In theory, you should be able to take a spectral rule set, plug it into vacuum, and it'll give you the same results. \[00:06:09\] There's, there's, there's different logic and it, it's, it's built differently. Completely different. It's different architecture, but the reason why it exists and now understand I was making fun of some of the speed was, was, was because of the, was because of the speed. So I'll take, I take you back to the, the kind of origins of it. \[00:06:25\] It was around like 2019 when I was working at a company called VMware. It doesn't exist anymore. It's been bought by Broadcom. But yeah, but the, the, the, the big thing there was, you know, there was a huge, there's thousands and thousands of open API specs, and there was, some were generated, some were handwritten, some were rubbish, some were just, didn't even work. \[00:06:45\] But these were all being just. Pushed out as documentation to customers, and they were just rubbish. They were just terrible. They didn't work in co generators. They, the, the documentation was awful. It, it just didn't load. It was, there was no SEO, you couldn't discover it. All, all kinds of \[00:07:00\] problems anyway. \[00:07:01\] So I got tasked with this problems like, can you start figuring this out? Can you start figuring this mess out? And the first thing I did was, okay, let's go and grab some tools from the internet. So I started grabbing documentation tools and linting grab spectral started running it, and it was doing fine for the smaller specs, you know, the smaller SaaS services. \[00:07:20\] But when I fed in, there's a product of VMware called vSphere. This thing was built in 2006, and it has. 10,000\. 20,000 a Ps, an ungodly amount of APIs, and all these models were object oriented, fed some of those specs in, and they were like 70, 80 megabytes a piece. spect, which did give us a response back. \[00:07:43\] It just, it just never came back. It went off, never came back. It was like, okay, what do we do about that? So started, we actually tried to build a tool internally to replicate spectral, and failed absolutely miserably. We failed. Couldn't do it just because spectral was, it's so complicated. Did such a good \[00:07:58\] job and it was, \[00:08:00\] you know, \[00:08:00\] ****Phil Sturgeon:** does a lot of different things as well. Like the way, so one thing that people always kind of forget about with linter is like when you've got a code-based linter, you can usually look at that file to find out if that syntax is correct, or you can, you can kind of use reflection APIs in PP to use the PHP term. \[00:08:15\] You can look at the reflection. API. To load it up and be like, does the name of this function look right to you? But it, but you, you know what line that that file is on or that function is on and you, and it's, it's a lot easier to, to work with individual files, which is therefore quicker than it, than with open API because you have to. \[00:08:32\] Build all of the structure into memory, like you have to follow every single ref of which there might be thousands. And then build kind of like a, an actual map, a fully referenced map of that, which gets quite chunky. And then you have to like find out if something is a problem in every single location. \[00:08:47\] What it is used like the. The issue might not be in the file you are looking at, it's when that file is, is reved into a place and now it's a problem. And then, and then a lot of tools have done that. Like Specky, the thing that kind of inspired Spectral did \[00:09:00\] that, but then it couldn't tell you which file the problem was in. \[00:09:03\] So you have to kind of go up and down into these different formats and like map everything to a different place and it's bloody confusing to do so. I can only imagine, like I've, I've played with the Spectral Source code quite a lot. I didn't write it, but I was like the product manager there for a while. \[00:09:16\] And yeah, every time I look in there, I just go, Hmm, someone else can work on that. I'm not gonna, I'm not, not gonna even try. \[00:09:24\] ****Quobix:** It, you know, the, so and I discovered that when, after, long story short we were trying to run spectral with some of this tooling, couldn't get it to run at the speed that we needed it to. So I said, look, let's, let's rebuild it. And VMware said no. And as I got all these other tools that we built, said, let's open source them. \[00:09:40\] And they're like, no, we're not gonna do any of that. So I was like, okay, I want to, I want to give back to the community. I want to work on this stuff. I want to build this link. You won't, you don't wanna do it. I don't have a future here if you don't wanna work on it. So I'm out. So when I, and I basically quit VMware and started building it from scratch, so. \[00:09:56\] Th going through that process of day zero, blank \[00:10:00\] slate. Okay, I'm gonna build my own version. I'm gonna take spectral as a template or as a, as a, as an inspiration. Look at the source code and then reimagine that and go. And I spent two months rewriting that. Reference, look up, trying to figure that out. \[00:10:16\] Recursive logic, months, months of scratch scratching my head and I failed multiple times. I couldn't get it to work and I couldn't get it to work fast. So trying to make that work, you know, the work that was done on Spectra was, was phenomenal. It to be months and months just to wrap my head around how to get the logic together. \[00:10:33\] Eventually I pulled it, pulled it, pulled it together, but I was having like. Recursive dreams. Like I was waking up, you know, in the same like loop in my head, like trying to program my way through a dream, you know, in this recursive stepping in way. Anyway, it was driving me nuts, but yeah, got there in the end and, and the whole purpose was to, I want to be able to generate a spectral report or a report but I want it to run. \[00:10:56\] Fast and I need a, a language that's more suited \[00:11:00\] to do this type of stuff. You know, ideally it would've been in Rust or done it in c plus plus or something, but you know, I was familiar with Go Loved Go, been using it for years and it's almost as fast, you know, you know, \[00:11:10\] it's not as, not as fast, but yeah, almost as fast. \[00:11:14\] So it would, it would do the job. Plus there's a, there's a pretty large community for Go, so yeah, it's spent like four months, five months building it, you know, just every day. Chop away. Chop away. Chop away. Threw it away. Start again. Start again. Eventually I got a design that actually worked and worked well and was something I could then add onto to and scale up and open. \[00:11:35\] Sourced it and said, you know, what do you think? \[00:11:39\] No. First few months there was zero. It was like, no, it was cricket, but yeah. Started to take off. \[00:11:44\] ****Phil Sturgeon:** that's wild. I mean, yeah. The, the entire API linting when was vacuum released a year or two ago, \[00:11:51\] ****Quobix:** Yeah, it was 2022, like April or May, \[00:11:54\] ****Phil Sturgeon:** Yeah, I gotcha. So that's, that's like linting is a thing now by 2022, right? Like that, \[00:12:00\] that's it still kind of early days, but in it is been the last year or two where it's really taken off all of the tooling vendors. \[00:12:06\] Are working on it. They've either, you know, they've either baked, spectral in under the hood like Postman did, or e even SmartBear have kind of replaced their, their own kind of mostly red checks or whatever based linting with spectral under the hood. Now they've acquired stoplight or they're working on their own kind of thing. \[00:12:23\] Or like everyone's, everyone's into it now. It's a thing, like it didn't exist as a, as a. It didn't exist more than conceptually in bullshit hacks until, until about then. So yeah. Now, now we're onto the, the good uptake the good part of the hype cycle where it's not just early adopters, it's loads more people and people are looking for different tools and they might not want something that's JavaScript. \[00:12:42\] They might not want something that's slow and that, and they might want something, you know, some people want things that are CLI based or like other different types of interface. We've got the rate my open API app. That's turned up that you've been helping out with. So that's by Zulo, but you know, they, they're running on, on vacuum. \[00:12:58\] And then there's I did a review of these two \[00:13:00\] API insights as a \[00:13:01\] ****Quobix:** by treble though, right? \[00:13:02\] ****Phil Sturgeon:** Trouble. Yeah. And so them on the podcast recently, and I've done a review on APIs, you went hate blog about those two tools, but they're both more visuals instead of CLI based. So we, we've gone from a world where like these tools didn't exist to now where like these tools are aimed at different types of user and different types of use case, which is a brilliant place to be. \[00:13:20\] 'cause now we're all sharing ideas and. Powering each other and improving things. And there is a bit of a, in, in my head, coming from like a, a background with standards bodies. I'm like, why are there five different formats? You know, there's like, you thankfully have gone with a spectral compatible thing and maybe that will change in the future. \[00:13:36\] But then there's kind of the redoc le into, which is its own more simple DSL. Simpler, but somewhat less powerful. And then potentially, and then there's the open, there's the optic CLI lint GPT, which is like, you can just send it a string of human words. And if you, you know, ask the wizard nicely, it will, it will give you the thing you want. \[00:13:56\] And review of that on the blog recently as well. So we're in this amazing \[00:14:00\] place where there's loads of linting going on and, and. I'm into it. I think vacuum, from what I've seen is, is blooming brilliant and I think your choice of langu in go is, is probably isn't gonna be a limiting factor to the speed. \[00:14:13\] Right. I think a, a big benefit that you've got there is the way that it was architected. I think spectral is kind of tipping it hand as being in initially. A utility of stoplight studio where you are editing a bunch of files and whether they're local or on the cloud, it has all the files kind of local. \[00:14:31\] Even the, even the cloud version would clone the entire repo into your browser and have all of those files like there. Yeah, clone it somewhere. It'll clone it into local storage. And then you could kind of access it from there. And basically the idea was, you'd kind of you'd run it and it would get an idea of all of your open API and then you'd, you'd edit one model and it would go, alright, let's just update that one model in, in the storage and it would run it from there. \[00:14:54\] So it's more meant to lint while you edit, which is quite fast. 'cause it \[00:15:00\] does the initial upfront load and then it's just like, oh, you change your speech, change your speech, change this bit. But yeah, like when you've got, you know what's the, you, you were using GitHub. The GitHub example, which is Monstrous and Stripe as well. \[00:15:11\] They're both huge. And yeah, when you just say, go and figure that whole thing out, it just kind of weeps and just can't handle it. So yeah, it's great to see these different use cases being covered by by different approaches and I can only imagine how hard that was to, to build from scratch. Yeah. \[00:15:26\] ****Quobix:** was, it was an adventure. It was, it was, you know, it was definitely putting my imposter syndrome on full tilt, you know, standing and think, why can't I figure this out? You know, why can't I get this algorithm to work? Why is it getting stuck? Why is it spinning out? Why are these threads pauses? Why is this lock? \[00:15:43\] I mean, it was just every, every time I tried to do it. So, you know, hats off to the engineers working on spectral because solving that problem, you know, and doing it in a way that works consistently. Very, very, very hard. Very, very hard. \[00:15:57\] ****Phil Sturgeon:** Do you have you mentioned there wasn't that much adoption \[00:16:00\] at first, is have things been picking up? That's where I started to go with my like hype cycle, ramble. \[00:16:05\] ****Quobix:** Yeah, there's, so it, it kind of, it, it's, it, it goes through little waves, depends on whether there's a holiday or not. Like a, the traffic to the docks, it's weekends, ghost Town, Monday morning, straight again spikes up. So it's very much following that kind of business you know, peaks and troughs. So, over Christmas it kind of dipped a bit, but then it was, you know, it's up to about, between 11, 12,000 downloads a month via MPM, it's had nearly half a million downloads, so it's. \[00:16:31\] ****Phil Sturgeon:** Wow. Yeah, yeah, yeah. \[00:16:33\] ****Quobix:** it's moving. It's not where I'd want it to be, obviously, but it's definitely moving, you know, \[00:16:37\] ****Phil Sturgeon:** Oh yeah, and I saw that. So it can be installed by NPM as well. \[00:16:40\] ****Quobix:** Yes. \[00:16:41\] ****Phil Sturgeon:** That, that's, that's pretty interesting. Yeah. 'cause things I, I've never, I've never tried running go in like GitHub actions or whatever. I'm sure it's fine. But whenever you ask people, like most open API tooling is. In JavaScript. \[00:16:53\] It seems to be like a really node heavy community and not entirely sure why that is, but it is. And it's, so, it's, it is always good to see \[00:17:00\] other languages kind of getting involved. But then because so many people are used to NPM, it's good to have that covered as well. 'cause if you are, if you, if you're currently running spectral, you could just run vacuum and, and see if it works \[00:17:10\] ****Quobix:** I mean, that's where the majority of the downloads come from is from MPM use. Whenever I speak to people on the forums or they come on the Discord, you know, they're. They're running it or installing it via MPM or Docker, those two, the, the most popular, which is really surprising for me. You know, I, I thought, you know, most people would want to be installing it, you know, as a CLI tool and have it as a binary, just, you know, using Home Brew or via Shell or something. \[00:17:32\] But no, it's Docker. Yeah, Docker and MPM most \[00:17:34\] popular. It was, yeah, very surprising. \[00:17:38\] ****Phil Sturgeon:** Nice. Well yeah, great to give people a bunch of different installation options and where. Where's it at and where's it going? What's the plan for vacuum? \[00:17:46\] ****Quobix:** Yes. That's a great question. So the, the, the kind of the purpose of, of what I've been working recently, or up until this point, was to be able to get it to a state where it's consistently operated. The, the bugs that I can fix, I can solve are mostly, mostly \[00:18:00\] cleared. And to be honest, the majority of them live outside of vacuum is actually in lib open API, which we can talk about a bit. \[00:18:06\] And that getting that, that's, that's kind of the beating heart of everything and getting that to a point where. It's working for all the use cases of all the variations from the very, very complicated to the very, very simple. Basically, I wanted it to stop blowing up because I actually run it as a demo in the pipeline and when I first put it up, you know, every day. \[00:18:26\] It just panic, panic, panic. And it would reset itself. You know, it's like a tiny little container, but I was just watching this just blow up over and over and over again, and I'd fix those bugs and then I'd wait, wait for it to spin outta control. And that stopped happening now. So now it's, it's not blowing up anymore. \[00:18:42\] It's not spinning outta control. It's consistent. The error reports I'm seeing are very, very limited or things I already know about. So that's been the goal is to harden it, get it to the point where it's. I'm happy to say it's gonna run, you know, consistently now, wherever it is, the next piece is to start making it \[00:19:00\] more usable. \[00:19:00\] So for example. Like you, like you were talking about with rule sets and rules and the syntax of the DSL and doing J path queries and figuring out functions and function options, it can get a bit hairy particularly with, you know, you know, some really complicated rules and then you ha adding in custom JavaScript functions, all of that kind of, is it working? \[00:19:20\] Can I test it? How do I know it's working? You need some kind of, you know. Tool to be able to test your rule sets, test your rules, test your path, test your functions, test your logic. That doesn't exist right now. So that's the next step is to, to allow you to see your spec, see the linting, so, and label it to work with a language server so we can plug it straight into. \[00:19:42\] Vs code and, you know, make a VS code extension and then bring that even further. So you can have a rule set editor and a rule editor and a, you know, try out your JavaScript function, see if it works. You know, does it, what does it pick up? What does this pick up? What happens? What's the outcome? So it's like a, you know, like a sound bit or a sandbox for rules and rule \[00:20:00\] sets. \[00:20:00\] That's where I'm gonna go next with it and improve the, the rendering. Right now there's an H TM L report, which I think is pretty good, but I could do better, do much \[00:20:08\] better. \[00:20:09\] ****Phil Sturgeon:** Well, it's exciting to hear that 'cause I think the, the dev experience, like anyone. Anyone who's tried writing a bunch of spectral rule sets will know that it can be very hard. I'm currently updating the spectral ow wasp security rule set for security. What the top 10, 20, 23? 'cause the previous report was 2019. \[00:20:26\] And yeah, like I'm, I'm doing zany stuff and I, it's hard work and I think. It was always discussed at, sort like a long time ago that the plan was to build some sort of dev experience to help people do that. But it, it didn't get around to materializing 'cause the the effort was put into like making hosted rule sets where you could, instead of editing a YAML file on your computer, you've got a nice gooey for working on them. \[00:20:49\] But that gooey was replacing the YAML file. It wasn't replacing the, the, the. The guts or the experience of working out what rules to put in. And so e even like a RegX power.com equivalent of like, \[00:21:00\] here I am typing in a rule here you can see it's selecting the right bit of open API Now what would you like to do with it? \[00:21:06\] Anything along those lines would really help. It's why I build them as NPM test suites because I could just type like NPM test, NPM test, NP test, and PM test and just like. Randomly bullshit my way towards getting to the correct solution. But that, that's the key thing because this is why we've seen people making wizard based approaches or like nicer DSL based approaches because the J path thing is so complicated and, and custom functions can be so difficult. \[00:21:29\] But I don't feel like you can just avoid that problem with the DSL. Like with it, there's, there needs to be something in the middle or just a DSL. You can only do what the DSL does. But leaning forward into like, I'm gonna combine JSON path plus RegX based with RegX in it and like have no way of knowing if that's working well or not, apart from like trying it and seeing what happens. \[00:21:52\] Neither of those are particularly good experiences. \[00:21:56\] ****Quobix:** Yes, exactly, exactly that. It's like you just, you just keep hitting it with a \[00:22:00\] hammer over and over and over again until you see something. Having some kind of guide, some like feedback. It's, you're doing it wrong, you're doing it wrong. I've tried to capture that with some of the linting rules that. \[00:22:12\] One thing you don't get with, I haven't seen anyway, with spectral is like, here's how to fix this problem. It's like, here's your problem, but here's what you go and do to go and remedy it. And here's some docs of a good example and a bad example. So doing that same type of thing, but with, you know, the dev experience, you know your rules wrong. \[00:22:28\] This is not finding anything. This is what it's picking up. This is, this is what you think you are getting, this is what you're actually getting, type thing. \[00:22:33\] ****Phil Sturgeon:** Yeah, I gotcha. Alright. We could talk about Linton stuff for ages, and that's why I wanted to get this one done first. But you've built loads of other tools as well, and so wiretap is really cool. A a, a mocking server aiming to do some of what Prism was meant to do, but Prism kind of got left behind in prioritization a little bit and hasn't really done many of the things that it wanted to do. \[00:22:55\] When it, when it started off? Unfortunately. So tell me a little bit about wire itap \[00:23:00\] and \[00:23:00\] ****Quobix:** Yeah. Great. Thank you. So wiretap, it actually started this, there's this, there's something that's a, like a real u real life use case that I have at Splunk. There's two things. First the way that our, that we build our premium applications as Splunk we were using Webpac dev server for all of our front end stuff. \[00:23:16\] So, you know, we, we load up our front end and it'll make all these API calls at the back end. Problem is, is that, you know, when we do. Running this thing for know, a few hours, it would start to choke, right? The, the whole thing would just start, you know, API calls are failing, weird errors are coming back. It was caching some weird stuff. \[00:23:34\] It was just, just didn't work. You know? It didn't work at scale. Great for smaller stuff, and when you start using it, let scale didn't work. So we had that problem there. The next problem is. We have all these front ends and backends, and some of them have a open API spec. Some of them don't, and we don't actually know if any of them are compliant with each other. \[00:23:52\] We don't have the backends been implemented properly. We don't have the front ends been implemented properly. And we don't, well, obviously the API open API spec is in in \[00:24:00\] one condition as well. So problem A is we're struggling with how to build our UIs in the front end Problem B is. We we can't tell if our API traffic is compliant, frontend or backend. \[00:24:12\] And there's the third problem, which is mocking and frontend. Mocking. Obviously you've been in software development, you know, the front end moves at different speed from the backend. So we get a bunch of requirements from PMs and the backend teams start moving at this, you know, a glacial speed. And the front end team, we're all like mocked up and ready to go with the ux, but there's no backend. \[00:24:30\] What do we do? How do we move forward? So with API first, with an API first methodology, working on the contract first, we can then take something like wiretap and, and mock it out so we can have a front end working with the contract whilst the backend does, you know, their implementation. So it solves those, those three problems that we, that we have. \[00:24:48\] And, you know, so. The, the way that we use it today is it powers our, our call local dev suite. So all the UI developers, they, when they run NPM start, it actually \[00:25:00\] spins up wiretap, it's not spinning up webpac. And we can run a, we run web pack as a watcher, so it's recompiling all the JS and the CSS in the background. \[00:25:07\] So it spins that up and then we start capturing all the, all the API traffic. So the other thing that it does was its primary goal. The, the, that's what I wanted to build it for. But I also saw these other use cases as well. Primary goals was mocking and API open API validation. So whilst it's serving all the static content, any APIs, it can, you read them in and you can configure them with, you know allow lists and denial lists and things like that. \[00:25:31\] You can do path rewriting. A lot of the stuff that you get from like, like Webpac dev server, it's actually a drop in replacement for a lot of those features like, you know, variables and authentication tokens and all that kind of stuff. Anyway, it act as a proxy, a, a, a complete transparent proxy for all of your APIs and remap 'em if they need to, but it captures all the API traffic and allows you to, you know, validate the, the requests and responses and sch. \[00:25:57\] Effectively looks at the whole open API contract in \[00:26:00\] detail, and then shows you which of your requests are invalid. You know, how, how compliant are you? And then you can see where those, those, those violations are. And it gives you numbers to straight to the spec. That's something that's missing from a number of tools, is the ability to get to. \[00:26:16\] Here's your problem. Here's why it's wrong here it is in the spec. Here's where it is in those line numbers and in go that functionality was missing from, from, you know, another of other libraries that are out there. So \[00:26:28\] ****Phil Sturgeon:** gotcha. That's pretty handy. 'cause the, that, that's always been, that's been the goal of so many tools that. Kind of the contract validation stuff I think is really important. So when we talk about validation, there's two types. And so vacuum tools like that, the linters, they are schema validators, like is my schema correct? \[00:26:45\] To whatever roles I may have defined to the spec and to my preferences. But then there's the, I call it data validator which I've never said out loud and is annoying. I need a different term now. But data validation doesn't sound so bad. But yeah, the goal of that is like, and is the \[00:27:00\] data, is this, is this instance of Jason or whatever valid to what the schema said it should be. \[00:27:06\] And, and that's pretty bloody key when people are. If you're doing the a PA design first, then you can, your, whatever, your teams in general will agree on a contract, and then everyone has to build to that contract. And, and your front end teams wanna build to it, and your backend teams wanna build to it. \[00:27:21\] And if the, if the clients are sending things that don't conform to the rules. Someone should tell them. And if the backend team is also like emitting stuff that isn't right, someone should tell them too. And I think we kind of aim to do that with Prism. With you, you, you can build a mock server, so if the backend doesn't exist, you've got the mock server and you can just talk to the mock and it all let you know if you're doing it wrong. \[00:27:42\] And then once you've actually got the once you've actually got the API, you switch that mock to a proxy and it will still put the same traffic through pretty invisible. Just there's a bit more business logic happening. Tax rules might actually be applied instead of just static examples coming back. \[00:27:54\] That was kinda the plan for Prism. Sounds like y'all do the same thing too, but what I really liked about it was you, you've got a \[00:28:00\] built in, like a Go terminal, like a ui thing. So you can actually see the requests that have gone through, see which one's passed, see which one, which one's failed. And if you wanted that with, with Prism, you didn't really get it like. \[00:28:10\] You could sniff on some headers. I think we had like a Xen something or other responses validation errors kind of Jason that you then have to like unfurl somehow. But yeah, having it in that terminal built in, otherwise you've gotta go and have another proxy installed. Like go and set up Charles proxy or go and \[00:28:26\] ****Quobix:** Right, right. Yeah. \[00:28:28\] ****Phil Sturgeon:** Some sort of sniffing thing, which is a bit of a pain in the backside. So yeah, having that all in one place is, is pretty handy. And especially if that's one command that your front end people can write. Maybe the front end people are, aren't, aren't the ones that wanna set up Charles' proxy. So yeah, that's, that's really handy. \[00:28:44\] ****Quobix:** Yeah, so it, it was, it was that, it's the same thing with vacuum. Like there's a console, CLI, there's a sort, the terminal CLI, there's A-H-T-M-L report. So with Wire it's got the same thing that, you know, you can run it, run it as a service. It runs, it actually powers some other stuff as well. But it runs as service or it runs as \[00:29:00\] a, you know, as a, as a, a local, local ui, local GUI to be able to, to, to view it. \[00:29:06\] And \[00:29:07\] part of the, you know, the, the, \[00:29:09\] ****Phil Sturgeon:** Sorry. I was just gonna say it, it, it can be a fia when I, I've said in the past some teams, some front end teams, like I, I don't, front end teams are perfectly smart and the front end stack is a lot more complicated than the backend in so many ways. But yes, when you give 'em a new tool, if I've tried to say like, oh yeah, just install like Prism and point it at this open API file and then like pass these options or whatever, and they're like, I don't really want to do that. \[00:29:32\] 'cause then they've got, you know, trying to get the. The proxy information out's a bit of a faf, but when I, if you can host it, then you can just like change an environment variable and say like, just point your front end at this instead of pointing it at that, or it's completely seamless to them. Like just point it at the API and you don't even need to know that it's running the proxy right now. \[00:29:49\] We've just squished that into place. That, that makes it a lot more useful for those teams that don't wanna figure out a new tool right now. \[00:29:55\] ****Quobix:** That's, it was exactly that use case is how, how, how do I make this completely \[00:30:00\] transparently replace the thing that's giving us our dev experience right now and serving all the content and making all the API calls and proxying them to our staging servers. How do I replace that with, you know, so it's completely transparent so that even the MPM commands are the same in the script, is just, it's, you know, they don't have to think about it. \[00:30:18\] That's what obviously makes the adoption so much easier because, you know, it's just. Just all of a, all of a sudden one day, it's just, it's not breaking anymore. \[00:30:26\] ****Phil Sturgeon:** Yeah. Brilliant. Alright again, loads of other stuff I could talk to you about this, about. This is mostly just a chance for. For me to nerd out with someone who's working on the exact same problems that I spent years working on. So hopefully, hopefully some other listeners like this, but it's fun for me at least. \[00:30:41\] Anyway, the next one, otherwise a massive waste of time. Hello. Is this thing on the, another thing you've been working on, which I always wanted to get involved with, but never had the chance is kind of changes and change detection. So this is a new, a new area that's kind of popping up for people is we've now, we've now \[00:31:00\] spent enough time with the API design, first lifecycle, or whatever is going on. \[00:31:04\] Enough people are making open API and describing their APIs well. They've learned how to, you know, describe it well with, with good standards. And they've learned how to make sure it's true by using contract testing in their test suite or prison contract testing style proxies. That stuff's done great. \[00:31:23\] Now it's in production and people are making changes and shit's breaking 'cause people love making breaking changes and. There's an increasing number of tools that are coming out to, I, I haven't looked at how open API changes actually works, but I, there's optic Diff and Bump have got like breaking change detection on their, on their documentation deployment, which I'm, I've been doing some writing for them recently and it's quite a cool, and where you just kind like deploy your docs, keep on deploying your docs, put it in the. \[00:31:51\] Put it in the pull request and just keep on deploying your docs when they go to master. And then on your pull requests it can go, whoa, there, there's a break and change in here. Is that something that you wanted to do? \[00:32:00\] Or like, this has been removed, that's been added. And it's really nice to see more people getting into that. \[00:32:04\] 'cause that's one of the hardest problems in APIs is if y'all break, if you, if you change stuff, things are gonna break for your consumers. So how have you gone about stepping into the world of, of API changes and what problems are you trying to solve? I. \[00:32:17\] ****Quobix:** Yeah, it's a great question. So the, it's, it's exactly that is how do we know, and this is another thing, there's another problem that, again, it's with my day job. I've always wanted to build this, I've actually built it before with VMware, but this is a very real problem, is devs aren't, because there's no testing, there's no compliance validation, there's no quality assurance on, on his open API contracts, they're making changes or they changed automatically through code changes if it's code first and it gets deployed. \[00:32:42\] There's no testing and a customer gets broken. It's like, well, well we, we read the contract and it's not working, and we're breaking each other, like we're breaking product A to product B. The integration points between them, because the teams just aren't. They don't know, they, they're not following the, the dots aren't connected.\[00:33:00\] \[00:33:00\] So first, you know, you, the, the change detection only works if you are using the open a PA contract as that source of truth. And when you are, you know, you want to know exactly what has changed. So there's, there's a, there's other tools out there, like like it's great tools like OASD that does, you know, same thing, you put rules in there. \[00:33:16\] The, the thing that I noticed though with a lot of these diff tools is. I, I, I'm, I'm getting like a, like you would get with vacuum, you get a straight. A hot list of all of the violations that have occurred. And it's, it's when there's, there's a lot of them, it's a okay, where in the model has that changed? \[00:33:33\] Because that property, that name ID, is used 10,000 times and I dunno, which, I dunno where it is. And if I can't see it, I don't have a visual representation of it. I'm struggling to get the gestalt of what's actually changed. So that was the approach that I took with Open API changes is it's actually more of a visual rendering. \[00:33:51\] So it does. Two things. It the way, the way it works, it, it reads in the model. From both A and B, then it does actually \[00:34:00\] every single object as a hash check against this. So it takes the object, hashes it in to says, has it changed? And if it has changed, it looks at every single property, what was added, what was removed, and decide if it was, you know, is there a braking change if you added like a, a non-optional parameter, things like that. \[00:34:14\] And then what it will do is render out those changes as part of the. The model of the open API document. So you can see the past, you can see the parameters were added. You can see where in the structure it was changed and then you can flip it. Instead of just looking at a tree, you can look at a graph like it actually renders out a visual graph. \[00:34:33\] I was really inspired by a tool called jason crack.com and it's allows you to feed in adjacent file and it will visualize it. It broken apart and you can see it. I was like, that's very, very good. That's very useful. I would like to see that, but with something that's not just adjacent file, you know, a, a, a data structure. \[00:34:52\] So I took that same idea. I used all the same light. In fact, I built this all in React just so I could make it work with the tool that they, because I loved it. And yeah, \[00:35:00\] we recreated that same experience, but now instead of rendering out just the J file, it's rendering out the change tree. So you can see the open API document as a visual graph and explore it that way, and you can rotate it, you know, and we'll get all different layouts and stuff like that. \[00:35:14\] But, so it's a visual way to explore and also it then it gives you the ability to go back in time. So you can not only see, you know, the change from where you are today, but you can see where it was previous before that. And then it gives you some graphs and charts of how it has changed over time. So it's, it's like a local tool right now that will give you that, whether, whether you've got a local Git repository, it will look through all of your revisions. \[00:35:37\] It will just do a left or a right, or it can pull straight from GitHub and pull down all the, the revisions for that. You just literally pointed at the, the open file and pull down all the, all the, the commits that you've had, and then do diffs for every single one up to a limit. You can, you can limit it to five or 10 or whatever, but it was a different experience that I was missing. \[00:35:58\] I want to see what's changed \[00:36:00\] over time and I want to visualize it. I don't just want a, a straight list of violations here in like, like a. You know, like vacuum or spectral, when is something more visual? So that's, that's the difference of how it works. It's gets custom built. It doesn't use any other existing libraries other than the open API. \[00:36:18\] And it walks the model as a hierarchy. So it gives you a hierarchical rendering com complete with, you know, line numbers and columns, which is also missing from a number of other tools that don't give you the line numbers and columns. They just tell you the path to where it changed. \[00:36:30\] ****Phil Sturgeon:** Absolutely. Well, yeah, that's, that's back to what I said about some of the pres spectral stuff in it where yeah, people are like, I, I read all of your files into this massive array. Now go figure out where in this massive array array it went wrong. You're like, that's not helpful. I need to fix this. So yeah, I think some of those tools were kind of designed by either. \[00:36:51\] Very small teams that didn't have a chance to like really get into it, or were kind of designed by people that aren't as in the trenches doing open API stuff all the \[00:37:00\] time. So therefore, didn't really realize the importance. Like, Hey, our product manager's happy, but the people that actually need this to solve actual problems aren't, aren't so impressed. \[00:37:08\] So that's, that's pretty handy. I do wonder with some of this stuff, with like the breaking changes in open API, like, I feel there's, there's a little part of my brain that's like, did we create a problem for ourselves? Where originally a lot of the tools that I didn't like so much would keep open API completely separate from the source code. \[00:37:27\] And, you know, maybe some, if you're doing code first and then you just render it, render out some open API and it somewhere that's, whatever, that's a bit different. But as, as well, a lot of people will export the open API from their code and then it goes into the repo. And if you're designed first, it lives in the repo and then you actually use it as the source of truth, maybe for testing, maybe to power some of the code. \[00:37:46\] So either way you go more often, open API lives in the repo so that you know when you make a pull request, any changes that happen will happen to your open API and therefore your docs and your hosted marks and everything else all at \[00:38:00\] once, all along with the code. So it's a bit more atomic. A way that things worked before that was, yeah, you, you've got your code over here and you've got your contract testing suite over there and that meant that if you made a breaking change, then it would probably notice, like the completely separate contract test suite would, would kind of notice 'cause you hadn't updated that. \[00:38:18\] And it was always really annoying if you, if you make an intentional change. Then it breaks your contract test suite and you have to go and change that. So that always kind of annoyed me, but it, it does at least kind of then give you some protection against that happening. But as soon as we've moved it all into the repo, you can easily just like delete something and it breaks for everyone. \[00:38:38\] And it goes, yes, I've contract tested with myself, and I've agreed that I'm correct. So even though I'm not what the clients want and that, and that's where tools like Pact come in, right? 'cause then the, the, the consumer has to set up their own contract testing, which isn't, you know, the, the producer is, hi, my API does what I think it should. \[00:38:55\] And then consumer testing is like, does your API do what I think it should. And I, \[00:39:00\] you know, the consumers can test for the bits they're interested in. But if, if you have change detection running then you can, I dunno if open a PA changes does this, some of the diff tools do where your consumers can actually subscribe to changes and get like an email when changes come out. \[00:39:16\] And I don't \[00:39:16\] ****Quobix:** that's a good idea. I'll run that one down. \[00:39:18\] ****Phil Sturgeon:** automate something that, something the bump does. And I really like that 'cause it, it, it solves both. I've not needed to make this really annoying. Blocker where, you know, your contract testing is totally separate from your code and you constantly have to go and change it in two different places, but you haven't also forced all of the work onto your consumers to find out whether you've broken shit recently or not. \[00:39:38\] They can find out what's changed. They can find out if they care about it. And, and that's a, a powerful thing for change detection, I think. But yeah, that \[00:39:45\] ****Quobix:** Yeah, it is. it it it it very powerful and it's a, it is a big problem, right? Big problem is like, how do, how do, how, how do we, you know, find the balance there. I like that idea of, you know, like a Slack or a message email or something. It don't do any of that stuff because it's, \[00:40:00\] it's just like a local tool. \[00:40:01\] It has no kind of network connect. I mean, it, it, it does some stuff, but yeah, there's no like put your emails, no subscriptions yet. \[00:40:08\] ****Phil Sturgeon:** No, that's cool. Well, we've got more to talk about here 'cause you've got, you've been busy, you've been really busy. We have been talking about three open source tools that you've worked on, and they are powered by lib open API Now do you just enjoy writing code or why, why does that exist? \[00:40:23\] ****Quobix:** So, yeah, so it, again, it started with this, all the tools that I've seen out there, they're missing some of these key pieces that I need to be able to build the tools that I want to build. So I'm still trying to build the tool that I want to build and all the bits that I need to build it haven't, they didn't exist or they didn't exist in the way that I needed them to. \[00:40:40\] And I started with vacuum and lib open API was born inside vacuum. And it was, how do I, how do I pass this open API filing go. So there's already a great library out there, like kin open API. And that was my, just, so why would I reinvent this? Why would I redo this work? And I went to the plug it in. \[00:40:59\] But there's, \[00:41:00\] there's, there's a problem. There's two problems actually. The first problem is it's a fundamental design problem. The way that it, when it passes in the model, it loses all the context of line numbers and columns, which is critical in my opinion, for being able to do change detection and show you where things have changed. \[00:41:18\] And then with the, you know, being able to build, same thing for linting. I need to know where it is to be able to say, here's the problem. And it doesn't have that, and it won't be able to have that because of the, the way the architects put together. So the only way to do it was to literally rebuild it from scratch as a, like a compiler passing the abac. \[00:41:34\] Abstract syntax tree, which is, you know, the raw YAML effectively, or the raw Jason, and then put that into a model that captures all of that low level information. And there's things like anchors and aliases and comments in YAML that all get lost. And you need to know about all of the sources of that and where it was come from, particularly doing like references. \[00:41:55\] Anyway that's why it exists because. F first of all, there was no lineups and columns. The \[00:42:00\] second thing is that KIN doesn't support 3.1 yet, and \[00:42:03\] that \[00:42:04\] ****Phil Sturgeon:** Does it spot 3.0 even? \[00:42:06\] ****Quobix:** it does, yeah, it does support 3.0\. But simply because of it's got its own Jason Schema model. The problem is, is we have exclusive minimum, exclusive maximum, you know, went from Boolean to and things like that. \[00:42:17\] ****Phil Sturgeon:** All the nut \[00:42:18\] ****Quobix:** Unless you've got like, like ability to do dynamic objects, which is what we have in lib open API, you're kind of stuck and you've gotta rebuild the whole thing again. And that's kind of where the project is. It's you've gotta rebuild the model to be able to make this work, which is just, it's just too much for some maintainers. \[00:42:35\] So that's why I, I built it 'cause what I needed didn't exist and we, I wasn't gonna be able to get to 3.1 support. And, you know, it's a useful tool by itself. So live it just inside vacuum. When people don't wanna do linting, they just wanna be able to pass the model or do d diff checking programmatically. \[00:42:52\] So that's, that's why I built it and that's why I exist. And yes, to answer your first question, I love coding. Love coding \[00:42:57\] ****Phil Sturgeon:** Just like you really enjoy writing the really \[00:43:00\] just like difficult under the hood, low level, just like writing some code that in, in like inspects YAML anchors, like most people don't even like working with YAML anchors, but you're like, let me write some code that muck about with this, or \[00:43:14\] ****Quobix:** yeah, yeah. yeah. Part, you know, part of the challenge is can, can I even do this? Do I, am I good enough to do that? And you know, the, then there's the, there's something, there's something really rewarding about, you know, when you're working in these high level experiences and you are, know, rendering, you know, graphs and charts to be able to trace. \[00:43:33\] All the way down to the, the code that literally read those bites in from the spec and know that tree is something very rewarding about that to me. You know, have \[00:43:41\] that, that over at that full, that full view. But anyway, yeah, that's that's a lib open. API. It's the heart of all the tools. It's, you know, it's the ultimate in dog food, if you will. \[00:43:51\] Every single piece relies on this. So when all these things break in these higher levels, it always usually traces back to a problem in live open a \[00:44:00\] p. \[00:44:00\] ****Phil Sturgeon:** Yeah, \[00:44:02\] ****Quobix:** It's pretty robust. There's no real open bug reports. Clinger, there's a few, but maybe it depends on who you look at \[00:44:08\] ****Phil Sturgeon:** there's, there's, there's always some things. \[00:44:10\] ****Quobix:** there's always some stuff in, there's, there's, there's a few things in there, but it's, it's actually the point where, you know, if I wanted to, it would be able to cut a 1.0 release, which is kind of very rare, but Yeah. it's, \[00:44:22\] ****Phil Sturgeon:** Well, that's, that's brilliant. And so you, you're doing this, you're doing this all as a side project and one of the questions someone was asking APIs was like, how sustainable is this? Like, is this just a hobby that you'll get bored of? And so that brings us to the next topic, which is. \[00:44:37\] You're building like a SaaS and a suite to tie this all together. How? How does that work? \[00:44:42\] ****Quobix:** So like that. So all of the individual tools, you know, they're useful in their own purposes, but they're all designed as, they're, they're tools that you download them, you own them, you run them, you modify them. They're, they're yours forever. But then there's the idea of, well, well, all these individual pieces are valuable, but what if it was all. \[00:44:58\] Put together, you know, the, the idea of \[00:45:00\] a suite, that that's, they're all in. They're actually designed to work together. They're all put, built in a way that they'll click together. So it's being able to do quality analysis and have the ability to edit rule sets and, you know, and then be able to visualize your contracts and see your schemas visually. \[00:45:15\] It's another thing that we've kind of lacking on. You know, there's, there's tools out there that will visualize. To a certain degree. But for example, if I'm designing a whole bunch of schemas in an open API, how can I take a look at the, the class hierarchy? Hierarchy, sorry. In, you know, in some form of UML, you know, maybe not ml, I dunno. \[00:45:33\] But you get to the point is how do I visualize it? How do I do change detection? How do I do mocking and testing? How do I do compliance? How do I do validation? Having that as part of. One tool that, you know, works as a desktop app, but also is available as a SaaS platform. So know, so it's got its own full API and then it means it can be introduced into things like GitHub actions and all of those functionalities that you use as the individual tools \[00:46:00\] now become available as a service or as a suite that you can, you can download and then moving that forward and offering, you know, really what I wanna focus on is. \[00:46:09\] The upstream elements of all of open APIs focusing on the contracts, not necessarily the artifacts that are generated from the contracts like SDKs and, you know, docs to some degree, but it's really about maintaining, you know giving you visibility into the contract. A lot of it is inspired by a lot of the work that's stoplight has done, you know \[00:46:29\] ****Phil Sturgeon:** Cool. I mean, yeah, there's, there's a lot of different verticals, right? So there, we talk to a lot of all vendors on this podcast and I'm usually asking 'em which bits they're going for, 'cause. Jason Harmon from Stoplight did a brilliant piece about how you can't do every vertical well, and anyone who tries is delusional or lying. \[00:46:45\] And not, not those exact words, but something along those lines. And it, it definitely, it definitely checks out. Like there's a lot of companies that kind of do a certain vertical as a checkbox activity and, and it's just sad for everyone. But yeah, it seems like, so focusing on. Kind of linting \[00:47:00\] and mocking and, and validation and that, that's, that's already quite and change detection. \[00:47:06\] That's already quite, quite a blooming lot, but it's, it's all areas that aren't massively well served currently. I mean yeah, there's a lot of tools that kind of go for it, but they've all got primary focuses elsewhere. So e even if you kind of do put in some, some docs, I mean, you know. Not, that's not the hardest one. \[00:47:25\] Once you've got some open API, you can render it with a pretty interface. So like everyone does docs and something else. But yeah, that's, it's an interesting series of verticals to go for and it's good that there's a plan to get some money involved to help you work on that. 'cause it's not, I. \[00:47:38\] Something you can do Well if it's purely evenings and weekends, not, not \[00:47:42\] ****Quobix:** Yeah. \[00:47:43\] ****Phil Sturgeon:** not in that sweet sense. Like you can make the tools and give them out to people, but glue in it all together. I, I compare it to like, you know, I want, I want to set a cutlery and and, and I've just got a knife and I'm like, all right, well let me just whittle for the next two days and I'll have that. \[00:47:57\] But like, not everyone wants to do that. They don't want to go, all right, let \[00:48:00\] me plug this into this part of the ci and then that can do these other 12 steps, you know, glue it together for me please. \[00:48:06\] ****Quobix:** Yes, yes. It's the glue. Yeah. Gl I'll, I'll, here's a service that sticks it all together for you and gives you, you know, a lovely set of interfaces, easy APIs to integrate with. So you're right. Yeah. It, it, it can't be something that just stays as evenings and weekends. That's my long term goal is to turn it into, you know, a business that I can work on, you know, full time. That's, that's where, that's where I want to be. And, you know, it depends on the, the reason why I'm doing this right now is validation. Is it, would people even pay, would they be interested in, you know, there's users of open API, there's all these companies that are using it, but are they gonna. \[00:48:41\] Four Cal. And I know that model exists because, you know, the, through the success of companies like Bump or optic or Speakeasy or you know stop Lying, being acquired, definitely a, a market for it. I just wanted to see, first of all, would people be interested in using my tools first? That seems to be, you know, \[00:49:00\] validated to a degree. \[00:49:01\] Now it's, let's, let's make it better. \[00:49:04\] ****Phil Sturgeon:** it's \[00:49:04\] ****Quobix:** the gas. \[00:49:05\] ****Phil Sturgeon:** exactly, I mean that's, I think, I think something you mentioned when I saw you talk was like. If people, if people start paying and you can afford to bring somebody on, then that's, that's something that you know, you'd like to do and start to expand it. And that, that I think is really interesting. \[00:49:18\] Like my, my background when I'm not messing with startups it is mostly bootstrapped, right? Like I, yeah, I, I work for WeWork, definitely not a startup. They like to pretend they are and they have the ping pong tables, but obviously a massive corporation. But before that, loads of loads of bootstrap companies and, and small startups and stuff. \[00:49:33\] And yeah, it's. It's really cool to see you go in the bootstrapped approach and, and look into grow, but also starting off still with so much open source. Because the balance for most of these tooling companies is if you make everything so that people have to pay you a thousand pounds a month just to start using it, you have a really hard time getting people to start using it. \[00:49:55\] 'cause it's, it's 100% like enterprise sales, team driven and you've \[00:50:00\] gotta get those big enterprise contracts and you've gotta have a lot of functionality. So it's a massive chicken and egg. But then if you have open source, you get a lot of developers using it and some of them might start paying for it. \[00:50:09\] So stoplight's approach was, we've got, you know I can't really put numbers on it, I I, but it felt like kind of 20, 25% of our effort would be like entirely open source. Like, you know, issues that only really affect the open source projects or are primarily asked for by open source. And so you're kinda like making that stuff out there in the open that people can use, but then it's also powering your paid products. \[00:50:30\] And not only does that mean that. You know, some people might be contributing bugs that then improve the experience for your paid users, which is cool. But you are also like getting people hooked on the free stuff and then not, you know, then you glue it together for them and that that brings them in. \[00:50:46\] So it is really interesting to see a one man army kind of just go, well, I'll just start off with open source and bloody everything and figure it out later. Like, I really kind of just like the, it's, it's bold. It's a bold choice. But it seems to be working. \[00:50:59\] ****Quobix:** \[00:51:00\] I appreciate that. Yeah, it's, yeah, it's, it's, it's, it's going in the right direction. That, that's for sure. For now, you know, anything, can anything come along and derail anything But it, the, the hunger's there, the, the audience is there. The interest in Open API is growing. It seems there's a friend of mine who, he mentioned, he calls it like an evergreen, it's an evergreen \[00:51:19\] field. You know, in 2016, I remember, you know, there was, we were coming off the Ramel versus swagger wars. And you know, the conversation was like, what, we need to invest in this, this is important. We should care about this. So even then, you know, the same conversations are happening. \[00:51:36\] We really haven't made huge amounts of progress towards actually getting this sold, you know, at at, at the company level. There's lots of individual companies doing this really well, but large enterprises are still mostly absolutely terrible at this. So it's, there's this field where there is evergreen. \[00:51:51\] There's always a need to improve here, and my opinion is the open API is like it. It is the key to rest, \[00:52:00\] like is if, just, just to say that we, if we're gonna do rest, there's open API Sure there's other DSLs out there in other languages, but it's, it's the one that everyone, every company has settled on. \[00:52:09\] I, I don't see anyone talking about any of the \[00:52:11\] ****Phil Sturgeon:** exactly. Doesn't need to like. We don't necessarily need to think it's the best. It's the one that stuck \[00:52:17\] ****Quobix:** Yeah. It's VHS, right? It \[00:52:18\] ****Phil Sturgeon:** Yeah. \[00:52:19\] ****Quobix:** tape. It's, it just, we, \[00:52:20\] ****Phil Sturgeon:** Laser discs were cool. You could use them as giant dangerous Frisbees as well. Yeah, I mean the last the last I think two podcasts we've done, we, we had Daryl Miller and Vincent whose surname I forgot. \[00:52:32\] I'm sorry. I, I, I got really cocky that I knew everyone's names and then, and then went for it. But but Lorna Mitchell as well. And we were talking about moonwalk a little bit, like yeah, there's, there's a V four open, API coming. And I do want to dig in more to see how that is looking. 'cause I'm, you know, that's gonna be a big change for a lot of tooling developers and it's, it's rough to see like this massive. \[00:52:52\] Massive improvement in tools happening recently. Like I set up open APA duct tools because everything, everything was kind of \[00:53:00\] shy, like a lot of the open source space was. We, we were all like, they were all really small tools and everything was like you know, little, little dodgy projects with like three stars. \[00:53:09\] And it was all, we'd all, we were all really just a young ecosystem and everything was coming off of like swagger two and getting ditched and starting over and in open API three 'cause it was so different. And, and it just felt like at that time, so much of the cool flashy works really well, looks really nice, has good marketing, is really battle tested. \[00:53:28\] So much of the good stuff. All of that effort was just going into different communities. It was all going into GraphQL and they were making cool stuff and they were like, screw, screw, rest. The shiny stuff's over here. I was like, you could make, you could do all of that over here. You've just chosen to do it over there. \[00:53:41\] And now we're kind of stuck with nothing. But over the last couple of years, like the, the tooling has, has grown up so much that I like. I have absolutely no envy for those folks over there. And people are starting to come back and, you know, the hype cycle on GraphQL is calming down. And, and, and the ma the tooling here is maturing to a really good spot. \[00:53:57\] So I love to see the competition in, \[00:54:00\] in the different spaces and verticals happening to push everyone forwards. And the focus on, on just like good looking websites and good looking tools and good looking simple things that plug and play together. So, yeah, I'm really excited to, to see where it goes. \[00:54:12\] And I just, I hope that the open a PV four isn't a, a big drop kick between the legs of the progress that we've, we've all been making on that front. So remains, remains to be seen. \[00:54:24\] ****Quobix:** It's quite as, it's quite as significant. W So the moonwalks, from what I've seen from the early earliest specifications, it's, it is gonna, it will be a nuclear bomb being dropped on, on the deaf community, particularly with the existing tools. It's, there's lots of Jason Schema support, which is great. Yeah. \[00:54:41\] But yeah. It will take some time. I don't, I don't even know how close we are to, you know, full adoption of 3.1, yet there's still, you know, a huge amount of 3.0 spec still being created today. But anyway, yeah, it's fun to see where it goes and I'm glad that it's moving. You know, it's, it's the fact that we've got the workflow \[00:55:00\] specification that's almost there, that's, that's really big overlays as well is an interesting thing that's \[00:55:05\] not quite there yet. \[00:55:07\] But like companies like speakeasy, they've already gone ahead and, you know. Adopted some of the, like lib open API, they've already built their own kind of extensions for it to support overlay. I think so anyway, I, I'm guessing they support overlays and they use lib open api, so I'm guessing they've done that, but \[00:55:20\] ****Phil Sturgeon:** Perfect. Yes. No. I was playing around with the speakeasy. Implementation is pretty, it's pretty good. Yeah, so there's loads of cool stuff coming in open API four and the kind of open API connected ecosystem with these working groups, and we've talked about that in previous episodes. I'll have to talk about that more in future ones 'cause. \[00:55:36\] This is getting close to being. Well, this is, we did it. It's a podcast. Thank you so much for coming and talking about all this stuff. We, we could have gone for like two hours, but we'll, we'll, we'll make life easier for the editor and, and, and get outta here, but \[00:55:49\] ****Quobix:** Thank you so much. \[00:55:51\] ****Phil Sturgeon:** pleasure having you. Cheers. \[00:55:53\] ****Quobix:** it. Cheers to the chat. Bye. ### How to Build a Perfect API Description URL: https://apisyouwonthate.com/blog/how-to-build-a-perfect-api-description/ Last updated: 2024-04-01T15:42:53.000Z You're starting out describing your API with OpenAPI, you've read the [OpenAPI Documentation](https://learn.openapis.org/?ref=blog.treblle.com), you've even read the [OpenAPI Specification](https://spec.openapis.org/oas/v3.1.0?ref=blog.treblle.com), now you want to make sure you're doing it well. How can you know? There's a few quick bits of advice to start with. 1. **Use YAML over JSON for your OpenAPI.** Nobody wants to mess with brackets. Convert it automatically somewhere in your pipeline if you need JSON for a specific tool. 2. **Name your main file `openapi.yaml`.** This is recommended by the OpenAPI Specification, and using `.yml` will cause confusion over and over for years. 3. **Use OpenAPI v3.1.0.** It's the best version yet by far. If you find any tools that do not support it, either use another tool, or chip in some sponsorship money to get them there. ## Getting a Jump Start Opening up an empty text editor and trying to build a whole OpenAPI structure is really hard. There are a few ways you can get a jump start. ### Graphical Editors There are a few visual editors out there that can help you click buttons and type into forms, building out the OpenAPI structure as you go. - [ApiBldr](https://apibldr.com/?ref=apisyouwonthate.com) \- Free Online Visual API Designer for OpenAPI and AsyncAPI Specifications - [Stoplight Platform](https://stoplight.io/api-design?ref=apisyouwonthate.com) \- A hosted editing experience that connects to existing Git repos, or real-time collaborative web projects. - [Hackolade](https://hackolade.com/?ref=apisyouwonthate.com) \- Polyglot data modeling for NoSQL databases, storage formats, REST APIs, and more. ### AI ChatGPT and Copilot both understand OpenAPI perfectly, and I use both regularly. ChatGPT is a handy one to get started with. Try writing a prompt like: > Write an OpenAPI description for an API game of tic tac toe where multiple games can be played at once, with endpoints for starting a new game, making a move, and seeing the status of the game. Use OAuth 2 authentication and add errors using the RFC 7807 format. The OpenAPI you get back will not be perfect, but it's a great start. I usually follow up with a few extra requests. > Add schema for 200 responses, make examples pure YAML, and add descriptions to properties. Copy and paste the output into your text editor, and give it a tweak. You might find it's made a few mistakes like putting the OAuth 2 security schemes and schemas into two different `components` keys which will trigger YAML errors, but give it a quick tidy up and you'll be fine. Once I've got the guts of it, I switch to the [GitHub Copilot VS Code extension](https://marketplace.visualstudio.com/items?itemName=GitHub.copilot&ref=blog.treblle.com), which really speeds up text-based OpenAPI editing. I can start writing `oper` and it will know I need to add an operationId, and come up with a good name for it based on the conventions used elsewhere. ![](https://blog.treblle.com/content/images/2024/02/copilot-1.png) The autocomplete is handy, but there's a [GitHub Copilot Chat](https://marketplace.visualstudio.com/items?itemName=GitHub.copilot-chat&ref=blog.treblle.com) too. This lets you ask questions, have it update your document, and generate code as it goes. ![](https://blog.treblle.com/content/images/2024/02/copilot-examples-1.png) I would not trust this or rely on it every day, but I do think it's helpful at rapidly creating big chunks of API description quickly. ## Aspen If you've already built your API, you can produce OpenAPI by hitting it with [Aspen](https://apps.apple.com/hr/app/aspen-your-apis-best-friend/id6475777033?mt=12&ref=blog.treblle.com), the brand new desktop API testing tool from Treblle. Just like other API clients you can enter a URL, headers, and parameters, and make a request, but unlike those other clients, Aspen has a built in AI-butler called Alfred who will offer some handy options. One of those is to produce OpenAPI from the response as seen there. ![](https://blog.treblle.com/content/images/2024/02/aspen-1.png) Learn more about [what Aspen can do](https://blog.treblle.com/meet-aspen-api-testing-tool/?ref=apisyouwonthate.com). [Meet Aspen: Speedier & Smarter API Testing, Powered by AI - TreblleExplore Aspen by Treblle: a revolutionary AI-powered API testing platform for macOS. Experience unparalleled simplicity and privacy in API interactions.![](https://blog.treblle.com/content/images/size/w256h256/format/png/2022/06/LOGO.svg)Treblle BlogAnthony (Andy) Rogers![](https://blog.treblle.com/content/images/size/w1200/2024/01/screenshot1-1.png)](https://blog.treblle.com/meet-aspen-api-testing-tool/?ref=apisyouwonthate.com) ## Validation & Linting Whether you're writing your API descriptions manually, using a GUI, or putting robots to work, you're going to want to make sure the OpenAPI you have is technically valid, and excellent. [Spectral](https://stoplight.io/open-source/spectral?ref=blog.treblle.com) can help you with both. ``` $ npm install -g @stoplight/spectral-cli $ echo 'extends: ["spectral:oas"]' > .spectral.yaml ``` Now you can get feedback on your OpenAPI description, either via the CLI, or extensions like [VS Code Spectral](https://marketplace.visualstudio.com/items?itemName=stoplight.spectral&ref=blog.treblle.com), to get feedback as you type, and if you set up a Git hook you can even make sure your OpenAPI is good before you commit. ``` $ spectral lint openapi.yaml 2:6 warning info-contact Info object must have "contact" object. info 11:9 warning operation-description Operation "description" must be present and non-empty string. paths./stations.get 11:9 warning operation-tags Operation must have non-empty "tags" array. paths./stations.get 20:23 error invalid-ref '#/components/schemas/Stations' does not exist paths./stations.get.responses[200].content.application/json.schema.$ref 33:9 warning operation-description Operation "description" must be present and non-empty string. paths./trips.get 33:9 warning operation-tags Operation must have non-empty "tags" array. paths./trips.get 82:10 warning operation-description Operation "description" must be present and non-empty string. paths./bookings.post 82:10 warning operation-operationId Operation must have "operationId". paths./bookings.post 82:10 warning operation-tags Operation must have non-empty "tags" array. paths./bookings.post 126:13 warning oas3-unused-component Potentially unused component has been detected. components.schemas.Station ``` This is letting me know to add some contact information, and pointing out that I've made some mistakes referencing `#/components/schemas/Stations` when I should have referenced `#/components/schemas/Station`, which is sitting there unused. Once I've fixed all of these problems and made the OpenAPI valid, the next step is to enable some linting rules to spot other issues. ### Security Linting For starters, let's make sure there are no obvious security mistakes showing up in our API design using the [Spectral OWASP ruleset](https://github.com/stoplightio/spectral-owasp-ruleset?ref=blog.treblle.com). ``` $ npm install --save -D @stoplight/spectral-owasp-ruleset ``` To enable it, open up `.spectral.yaml` and add the line to your `extends`. ```yaml # .spectral.yaml extends: - "spectral:oas" - "@stoplight/spectral-owasp-ruleset" ``` Now run Spectral again. ``` $ spectral lint openapi.yaml 15:9 information owasp:api2:2019-protection-global-safe This operation is not protected by any security scheme. paths./stations.get 19:17 warning owasp:api3:2019-define-error-responses-401 Operation is missing responses[401]. paths./stations.get.responses 19:17 warning owasp:api3:2019-define-error-responses-401 Operation is missing responses[401].content. paths./stations.get.responses 19:17 warning owasp:api3:2019-define-error-responses-500 Operation is missing responses[500]. paths./stations.get.responses 19:17 warning owasp:api3:2019-define-error-responses-500 Operation is missing responses[500].content. paths./stations.get.responses 19:17 warning owasp:api3:2019-define-error-validation Missing error response of either 400, 422 or 4XX. paths./stations.get.responses 19:17 warning owasp:api4:2019-rate-limit-responses-429 Operation is missing rate limiting response in responses[429]. paths./stations.get.responses 19:17 warning owasp:api4:2019-rate-limit-responses-429 Operation is missing rate limiting response in responses[429].content. paths./stations.get.responses 20:15 error owasp:api4:2019-rate-limit All 2XX and 4XX responses should define rate limiting headers. paths./stations.get.responses[200] 24:22 error owasp:api4:2019-array-limit Schema of type array must specify maxItems. paths./stations.get.responses[200].content.application/json.schema ``` Quite a few problems detected! Various error responses are missing, which is something [OWASP specifically recommends](https://owasp.org/API-Security/editions/2019/en/0xa3-excessive-data-exposure/?ref=blog.treblle.com) to help with contract testing tools. If it knows what the error is supposed to look like, it is easy to spot if the API implementation is doing something else, like leaking implementation details in a backtrace. It's also reminding me to describe the rate limiting strategy used and define the headers used in responses. If my API does not have rate limiting I should probably add it to avoid my API being taken down by malicious or misconfigured clients. ### API Style Guides Organizations with multiple APIs face a problem: how to keep their APIs consistent. Consistency is key for easy integration for clients, allows for code sharing, and generally looks smarter than random mismatch APIs. Some companies have dedicated API Governance teams who will create an API Style Guide. In days of old this was a large PDF or Wiki which everyone would have to constantly reread to remember it all and spot changes, or it would just be ignored. Modern best practice is to create an [automated API Style Guide](https://apisyouwonthate.com/blog/automated-style-guides-for-rest-graphql-grpc/?ref=blog.treblle.com), defining naming conventions, preferred authentication strategies, which data formats are used (JSON:API, Siren, etc.), and everything else, all of which can be put through Spectral using GitHub Actions or other CI/CD to make sure the OpenAPI matches the style guide before a pull request can be merged. You can build your own automated API style guide as a custom [Spectral ruleset](https://docs.stoplight.io/docs/spectral/e5b9616d6d50c-rulesets?ref=blog.treblle.com), or you can use one of these existing ones: - [Adidas](https://github.com/adidas/api-guidelines/blob/master/.spectral.yml?ref=blog.treblle.com) - [APIs You Won't Hate](https://github.com/apisyouwonthate/style-guide?ref=blog.treblle.com) - [Box](https://github.com/box/box-openapi/blob/main/.spectral.yml?ref=blog.treblle.com) - [Transcom](https://github.com/transcom/mymove/blob/master/swagger-def/.spectral.yml?ref=blog.treblle.com) - [Zalando](https://github.com/baloise-incubator/spectral-ruleset?ref=blog.treblle.com) ### Contract Testing Once you've got your OpenAPI describing a wonderful API with no obvious security issues or style guide violations, the next priority is to make sure your API actually matches that description. Contract testing can be complicated, and in the past it's required using brand new special tooling, but these days its easy to work with regular unit/integration test suites like PHPUnit, RSpec, Jest, making them aware of OpenAPI by pointing them to our `openapi.yaml` and running an assertion to see if the test HTTP response matches the OpenAPI. For example, working with Laravel PHP, you can configure the popular testing tool Pest (or PHPUnit) to contract testing OpenAPI using [Spectator](https://github.com/hotmeteor/spectator?ref=blog.treblle.com). ```php # tests/Pest.php beforeEach(fn () => Spectator::using('openapi.yaml'))->in('Feature'); ``` Once Spectator knows where your OpenAPI lives in the filesystem, it can then use it as the basis for its contract testing assertions. ```php # tests/Feature/WidgetTest.php postJson("/api/widgets", [ 'name' => 'Test Widget', 'description' => 'This is a test widget', ]) ->assertValidResponse(201); }); it('returns a 400 for invalid request', function () { $this ->postJson("/api/widgets", [ 'name' => 'Missing a Description', ]) ->assertValidResponse(400); }); }); describe('GET /widgets/{id}', function () { it('returns 200 for record that exists', function () { $widget = Widget::factory()->create(); $this ->getJson("/api/widgets/{$widget->id}") ->assertValidResponse(200); }); it('returns a 404 for missing record', function () { Widget::factory()->create(); $this ->getJson("/api/widgets/12345") ->assertValidResponse(404); }); }); ``` All the magic is happening in `assertValidResponse()`, where it’s looking at the OpenAPI description, seeing which HTTP method and endpoint being called, then comparing what it sees in the HTTP response coming from postJson against the OpenAPI descriptions response schema. If the status code is not described in OpenAPI, you'll see something like this: ``` FAILED Tests\Feature\WidgetTest > `GET /widgets/{id}` → it returns a 404 for missing record No response object matching returned status code [404]. ``` If there is a mismatch between the schema and the OpenAPI, you'll see something like this: ``` The properties must match schema: data All array items must match schema The required properties (name) are missing object++ <== The properties must match schema: data status*: string data*: array <== All array items must match schema object <== The required properties (name) are missing id*: string name*: string slug: string? ``` Lots of popular language testing tools have something like this. - **JavaScript:** [jest-openapi](https://www.npmjs.com/package/jest-openapi?ref=blog.treblle.com) works with Jest. - **Ruby on Rails:** [openapi\_contracts/](https://github.com/mkon/openapi%5Fcontracts/?ref=blog.treblle.com) works with RSpec. - **Laravel (PHP)**: [Spectator](https://github.com/hotmeteor/spectator?ref=blog.treblle.com) works with PHPUnit/Pest. If you cannot find something to integrate with your existing test suite, consider using [Wiretap](https://github.com/pb33f/wiretap?ref=blog.treblle.com) which can run as a proxy, and handle contract testing of the requests/responses that come through it in testing or staging environments. ## API Insights Once you've got some great OpenAPI, made sure it is valid, made sure its consistent, avoided common security mistakes, and then made certain the API and OpenAPI are fully agreeing with each other, what else is there to do to make your OpenAPI excellent? Treblle has just the tool for you: [API Insights](https://apiinsights.io/?ref=blog.treblle.com). This web/desktop app is API rating tool, which can give you a A-F rating that scores you on all sorts of things you almost certainly forgot about, covering Design, Performance, and Security. ![](https://blog.treblle.com/content/images/2024/02/api-insights-performance-1.png) Spectral was looking out for problems, but API Insights is going to let you know what you've done well. It's better to use this tool when the API has been built, because many of the rules will look at the actual API implementation based on the URLs it finds in the OpenAPI server array, to give a more complete picture of your API quality. ## Summary These tools all help focus on consistency, security, performance, completeness, and generally putting in enough descriptions and examples so that OpenAPI can be used effectively throughout the API lifecycle. Top quality OpenAPI helps with producing excellent documentation, mock servers, testing, SDK generation, server-side validation, API consoles, and developer portals, but as the use-cases are constantly evolving, your quest for excellence will continue to evolve. Let us know what you prioritize in your OpenAPI descriptions, and if we missed any ideas or tools that help you out. ### Web Frameworks & OpenAPI URL: https://apisyouwonthate.com/newsletter/web-framework-api-tools/ Last updated: 2024-02-15T13:08:29.000Z AWS released their new JS runtime called [LLRT](https://github.com/awslabs/llrt?ref=apisyouwonthate.com). Faced with a new level of JavaScript fatigue, not just frameworks but different runtimes that are all different, I decided to learn PHP 😉. I jest a little; we seem to be coming full circle with adopting standards; check out the [WinterCG group](https://wintercg.org/?ref=apisyouwonthate.com), so working with these new runtimes should be easy 🤞. In the last two weeks, I have spent far too much time getting OpenAPI generation to work on a Kotlin project. However, it inspired me to look into some of the most popular OpenAPI tools in some of the best web frameworks. Here is what I found. \-- Alexander ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2024/02/aspen-update-square-1.png) ## Upgrade to Aspen V 1.0.2: API Testing Reimagined with New Functionalities! Aspen V 1.0.2 brings transformative enhancements to API testing: better JSON response handling, an innovative JSON request editor with smart indentations, header count insights, and HTML display corrections. Elevate your testing routine with ease. [Download Now (no account required)](https://getaspen.io/?utm%5Fsource=newsletter&utm%5Fmedium=apis-hate-sponsorship&utm%5Fcampaign=aspen) ## The API Round-Up Our monthly round-up of articles, news and updates from APIs You Won't Hate and around the web. ### [Python: OpenAPI Core](https://openapi-core.readthedocs.io/en/latest/?ref=apisyouwonthate.com) Validate request and response data against an OpenAPI spec. The library openapi-core includes integration for popular frameworks like Django, Flash and Falcon. Make sure your Python API matches the spec with this great library. ### [JS: Zod OpenAPI Hono](https://github.com/honojs/middleware/tree/main/packages/zod-openapi?ref=apisyouwonthate.com) Who uses Express anymore, right 😉? Hono is a speedy web framework for JavaScript. Hono has built an official plugin for Zod to OpenAPI, allowing you to create schemas and routes using Zod, an extremely popular validation library. If you have not tried Hono or Zod yet, what are you waiting for? Start a new side project today. ### [Go: OpenAPI Client and Server Code Generator](https://github.com/deepmap/oapi-codegen?ref=apisyouwonthate.com) Generate GO server boilerplate from your OpenAPI specs. Oapi-codegen offers even more, allowing you to create client boilerplate as well. The package is looking for contributors, so get involved. ### [Java/Kotlin: Spring Doc](https://springdoc.org/?ref=apisyouwonthate.com) Spring Boot is an incredibly popular Java library for building API's. When it comes to documenting the API, look no further than Spring Doc. Use annotations on your controller's routes and classes to generate OpenAPI spec, which in turn can be generated into Swagger UI. ### [Python: Sane and Flexible OpenAPI Schema Generation](https://github.com/tfranzel/drf-spectacular?ref=apisyouwonthate.com) A fantastic plugin for the popular web framework Django drf-spectacular generates as much schema from Django as possible. A fork of Django's own OpenAPI generator, it add a lot of missing features. ### [JS: Fastify](https://fastify.dev/?ref=apisyouwonthate.com) A modern and fast alternative to Epxress.js, Fastify makes building APIs a dream. They take things further with two fantastic OpenAPI plugins. [@fastify/swagger](https://github.com/fastify/fastify-swagger?ref=apisyouwonthate.com) for generating documentation from your Fastify route schemas or OpenAPI schema. If you want to go the opposite way, check out [Fastify OpenAPI Glue](https://github.com/seriousme/fastify-openapi-glue?ref=apisyouwonthate.com) for generating Fastify routes from OpenAPI. ### [Go: Swag](https://github.com/swaggo/swag?ref=apisyouwonthate.com) To create OpenAPI spec from your code, check out swag, which works with the most popular Go web frameworks. Swag has an in-depth guide for supporting the ultra-fast Gin framework to create Swagger UI. The only downside is they don't fully support OpenAPI 3 yet, but they're getting close. You can track progress in this open [pull request](https://github.com/swaggo/swag/pull/1513?ref=apisyouwonthate.com). --- ## APIs You Won't Hate The latest from the team at API's You Won't Hate. 🎙️ ### 🎙️ [Every business is an API business, with Allan Knabe from Apiable](https://apisyouwonthate.com/podcast/allan-knabe-apiable-api-portal/) Apiable is building an API Portal service that helps API teams to create, secure, market, and monetize API products. In this podcast episode, Apiable founder Allan Knabe has a chat with Mike Bifulco about building a great API product. ### [OpenApi.tools](https://openapi.tools/?ref=apisyouwonthate.com) As it turns out, there are *many* great OpenAPI tools that you can get your hands on *right now*. We don't mention it here often, but APIs You Won't Hate maintains an awesome-list style catalog of OpenAPI tools on our website... [openapi.tools](https://openapi.tools/?ref=apisyouwonthate.com)! No matter how obscure your stack, there's probably something great in there for you. ## Support APIs You Won't Hate When you become an member, you'll get access to members-only content while directly supporting our work. Your support helps us to keep making resources for the API community. [Become a member today](https://apisyouwonthate.com/#/portal) That's all for this edition. Thanks so much for supporting APIs You Won't Hate! \-Alexander, Phil, and Mike ### Every business is an API business, with Allan Knabe from Apiable URL: https://apisyouwonthate.com/podcast/allan-knabe-apiable-api-portal/ Last updated: 2024-02-14T13:42:04.000Z ## Show Notes - [Allan Knabe](https://www.linkedin.com/in/allan-knabe/?ref=apisyouwonthate.com) - [Apiable](https://www.apiable.io/?ref=apisyouwonthate.com) \- Every API is a business - [Careers at Apiable](https://www.apiable.io/about-us?ref=apisyouwonthate.com) - [Pirate Weather](https://pirateweather.net/en/latest/?ref=apisyouwonthate.com) #### Transcript \[00:00:00\] \*\*Track 1:\*\* Hi friends, and welcome back to APIs you won't hate. My name is Mike Biko. I'm one of the co-founders of APIs you won't hate. And of course, podcast co-host and occasional friend of Phil's. We get to catch up and talk about API stuff when I can find him wherever he is in the world. But today I'm excited to sit down and chat with a new friend of mine. \[00:00:19\] I'm here talking to Alan Kaba about what he's building with API bull. And really excited to hear your story. Alan, thanks so much for joining today. How are you? \[00:00:28\] \*\*Allan Knabe:\*\* Hi, Mike. I'm doing very well. Thank you. Thanks for having me on the show. First of all. \[00:00:32\] \*\*Track 1:\*\* Of course. Yeah. It's a pleasure to have you here. I would love to to to hear a little bit about you to start. Gimme your, the history of how you got to what you're building today, and then I want to hear all about API \[00:00:42\] \*\*Allan Knabe:\*\* Okay, cool. Yeah, I'll skip the early years 'cause I don't think they'll interest anyone, but \[00:00:47\] \*\*Track 1:\*\* Sure. \[00:00:48\] \*\*Allan Knabe:\*\* I, I guess, you know, the, the origin is like starting at university saying information systems, which is code word for not very good at programming. I think, you know, everyone else were, you know, software engineers, but I \[00:01:00\] would write code that was like one page long or something like this, and then a real software engineer would come along and say, okay, you know, I could write that same thing in one line. \[00:01:08\] Right. And, kind of like the, the, the penny click that I needed to be a little bit more diverse. So information systems where, where I, I kicked it off with my, my computing career. And yeah. From there I, I kind of started off with an Oracle consulting company in Birmingham, which is, you know, basically where I'm from in Birmingham, uk. \[00:01:26\] And great Years had some like really good fun in like these, like very small teams of like 10 people. It's where I really love to, to be, you know, where everyone's working hard and you can see like what you're achieving. So good fun there. But ultimately I left for a bigger company, Fujitsu computers. \[00:01:43\] Mainly kind of. Backend development type tasks and Oracle databases and all this kind of stuff that were, you know, popular in the early two thousands before moving on to being a Siebel consultant. So, I dunno if anyone still knows what Siebel is, but \[00:02:00\] Siebel CRM was really big in the two thousands. \[00:02:03\] And \[00:02:03\] \*\*Track 1:\*\* Sure. Yeah. \[00:02:05\] \*\*Allan Knabe:\*\* yeah, I, I I. \[00:02:05\] \*\*Track 1:\*\* been a while since I've heard that word. Yeah. \[00:02:07\] \*\*Allan Knabe:\*\* Yeah. Yeah. Siebel, it was massive. Right? And then Oracle bought them and kind of, they disappeared a little bit, but there are still some projects out there. But I kind of, I san about 10 years of my life into Siebel, I'll never get back somehow. Right. It's, it's, I don't know for people who don't know where it is, it's, it's basically before you had Salesforce, you had Siebel and. \[00:02:28\] Instead of programming, they gave you like this tools console, which basically just toggle switches and you toggle them on and off. Right? And it would just, you know, activate feature toggles in code that you could never see, right? So like this black box and you had no real idea what was going on. And sometimes you would just kill performance, right? \[00:02:48\] Because you were doing some sort of outer join in behind the scenes. So not nice stuff if we're being honest. So my API story kind of started when I, when I left Germany. So I was in Munich for 10 \[00:03:00\] years. And I went to Swisscom in Switzerland. So Swisscom is like at and t basically, but in Switzerland. \[00:03:06\] And joined the API team. So I, I moved across to, to doing APIs and, well, it was a world of difference, right? The, the API team, you know, was really young, fresh, cool thing to be doing. You know, we were having like hackathons and it was really about that genesis about 10 years ago on now of like when the, the API started to really come onto the scene. \[00:03:27\] It was a really, really cool time to be working in APIs. Really \[00:03:31\] \*\*Track 1:\*\* Yeah, that's an interesting time to be getting into it. There was sort of this like renaissance of people kind of having this realization that they can go in and grab things from services they want and like kind of assemble products from, from whole cloth based on just little bits and bobs of services round and about. \[00:03:46\] And that's de definitely where the modern conversation of how do we build good things and like how, how do I behave properly as someone consuming or building these APIs started? That's really cool. What interesting timing. \[00:03:58\] \*\*Allan Knabe:\*\* Yeah. Yeah, it was really cool. And it was \[00:04:00\] like, you know, I think Apigee pre acquisition by Google at the time, right. So it was still in like startup mode as it were. And, you know, we got to go across to San Jose and, meet with a and go to the, I love APIs conferences and all of this stuff. And it was yeah, it was a nice time, really great time to be getting into, into the APIs. \[00:04:18\] I think nowadays we're maybe a little bit more mainstream, although there are a lot of API companies coming out now, like startups, like my company, a pable, right? We're we're startups and we're entering this space and kind of sitting over the top of what these more traditional players are doing. \[00:04:36\] \*\*Track 1:\*\* Yeah. And you know, that's a perfect segue. So what's the elevator pitch for a pable? \[00:04:42\] \*\*Allan Knabe:\*\* Okay, well the, the elevator pitch for a pable I mean if we, if we think about when we came up with the idea, it was also at Swisscom. So my co co-founder is a, a clever guy called Alexander Ham. And he was working. With me in the, in a squad \[00:05:00\] as it were, we had like this Spotify squad concept going again, it was 10 years ago when it was mega popular. \[00:05:06\] And we led a team of like 10, 11 developers together. And you know, we had a great time doing that. It was amazing fun. But one of the challenges that we had is that we were asked to create new digital products. So API products for Swisscom, so think like payment services and. You know, the precursor to like WhatsApp and things like this. \[00:05:26\] That's what we were trying to do. But we found that the tooling that came out of the box for developer portals was pretty poor. So we were using the one that came from apg, which was this Drew Pal PHP solution that, you know, we didn't really like it, but we used it for a while. And eventually no one kind of patched that solution and security said, okay, it's just too vulnerable. \[00:05:50\] You have to take it down. So a lot of companies did that at the time. And what did they do? They built their own API portals, right? That's what we did. Right. You know, we spent like, I don't know, \[00:06:00\] like two years, like making the API portal and did a really good job. Fantastic. But it was kind of that feeling that the, all this energy from mankind is going into building. \[00:06:12\] API portals there are bespoke. And, and really the amount of features you want to build for an API portal. If you're building it yourself, it's limited, right? You can only get X features in there before someone says that'll do. And it's basically just getting the API key sticking some documentation on it. \[00:06:29\] But we wanted to do a lot more with it, right? And, and that's kind of like where, where we went with API all is saying, okay, well, treating APIs as products right is one of the biggest things that we do. You know, I really believe in API product management. I am an API product manager and see the value in that. \[00:06:46\] So we really try to bake that into API able, and that does go hand in hand with monetization. Not every company wants to monetize, but those that do, can do it with a pable, right? So it's, yeah, it was \[00:07:00\] that kind of like genesis moment of saying, okay, if everybody in the world is building these API pools, why don't we just build one that's like great and provide it as a service. \[00:07:09\] So that's where we got to. And I, I think what we're really noticing as well on the, again, more on the enterprise side, is the amount of organizations that have more than one API portal. It sounds a bit crazy, but if you're as an organization, you've done some acquisitions and you end up with like multiple technologies, so, so imagine a situation where I think you've got MuleSoft and you've got Kong and you've got Apigee, for instance. \[00:07:37\] Sometimes you'll have three portals. My, my previous company, Connor, at one point, I, I'm glad to say it's not the case anymore, but we had, I think Dev. dot and we have develop dot and we have developer dot, right? And you've seen some, and you've seen banks as well. You see this very similar situation as this, like different factions within the organization using different \[00:08:00\] technologies. \[00:08:01\] And the technology assumes that you will use our portal and we are, you know, the only company in the world and, and you know, so there's. Cross usage there. So that's one thing that, that we kind of identified and we said, okay, well we're, we're technology agnostic then, right? So you can have Kong and MuleSoft and Apigee in your organization and just go ahead and connect the gateways in, right? \[00:08:22\] So as a developer, we're making developers life much easier that they don't have to then log into three different API portals. You have like. One unified portal where you can just get everything you need. And the token management should work in the same way across, across all. \[00:08:39\] \*\*Track 1:\*\* Sure. Yeah, I think that's, it speaks to an interesting thing that happens in product development on any scale, but certainly at enterprise level. I would imagine pick, pick, I mean, if you're listening to the show, pick any like, fortune 500 sized company that's not primarily a tech company. Imagine the size of their dev team and the amount of stress they're under to deliver things probably in \[00:09:00\] an underfunded under timelined sort of way. They will put as many resources as, as they're able to, I'm sure, to developing the API itself, right? \[00:09:09\] Make sure it does the things it purports to do, right in the languages that they, they are comfortable with. And when it comes time to make that thing available for usage there's probably a very rushed moment where it's like, ah, okay, now we need to expose this thing to the world. \[00:09:22\] So let's, \[00:09:23\] what's the simplest way to put up a, a. A developer portal where someone can go and now consume our APIs. \[00:09:29\] And I think that's maybe the curse of product development is it's like you spend so much time building it, and then the last bit of pushing it over the line often is where things start to fall apart. \[00:09:37\] And if you've, I, I mean lots of our listeners are probably familiar with this, but if you've tried to consume an API from, like some random company Inc. You know, from, from wherever in the world and you've come across a site that looks like it was built in, you know, visual studio.net in \[00:09:51\] 2007, you've had this feeling. \[00:09:53\] Exactly. \[00:09:54\] So, so that's really interesting. And so, I mean, it's, it's a core problem that, that a lot of these product teams face \[00:10:00\] and a lot of API devs have certainly dealt with. I'm curious, how, how long have you been working on this? You said it started when you were at Swisscom. What's kind of the timeline for that? \[00:10:07\] \*\*Allan Knabe:\*\* Yeah, we kind, we, we carried the idea from there. And it was really when I, when I left Swisscom I, I moved to Finland to Helsinki and started working with an elevator or actually I think they call themselves a people flow company. Right. To, you know, be more modern. Right. I guess, you know, but they're an elevator company. \[00:10:24\] Yeah. And when I started seeing, okay, the same problem there, right. Different technology, but you know, multiple gateways and, you know, not, you know, no one providing a good API portal. That was kind of the, the point for me where I said, okay, I need to quit my day job and do this. Of course, I quit in February, 2020. \[00:10:43\] I mean, the best timing in the world, right. \[00:10:46\] \*\*Track 1:\*\* Yeah. \[00:10:47\] \*\*Allan Knabe:\*\* Yeah, I, I, I, you know, of course, you know, coronavirus was, was around at the time in China and I just thought, okay, you know, that won't be a problem for me. And I quit my day job and by April I was babysitting. 'cause we have three children, so my \[00:11:00\] wife was working and I was just looking after the kids. \[00:11:02\] So that was good fun. But that was kind of like the genesis there in that. In 2020 when I formed the company and basically was looking around for potential customers and we found a great company called bmo. They sell bicycle insurance. And what they're trying to do is embed their APIs within the shopping cart of you know, larger bike companies. \[00:11:24\] So you can go like specialized or con or someone like this. See an insurance offer within the shopping cart process, right? So, so we've helped them with their APIs and getting them exposed so that they can get more partners effectively and get more customers that way, rather than relying upon someone buying a bike and then thinking, oh, I wonder if I should insure this thing, right? \[00:11:45\] Because that never happens, right? But if it's in the shopping cart process, you're like, oh yeah, I can do this 50 bucks a month, whatever. And put it in. So, so yeah, so we, we built like out the first version for bmo, which is kind of like a small and medium sized company which is good 'cause we could \[00:12:00\] do like, you know, the base and work with them to get that done. \[00:12:03\] And so we were building it like basically 2021 22\. We got funding in 23\. We basically perfected the products and then we have a bunch of customers working on it from like smaller startups. We have a startup in Canada that's called pirate weather. And they've got now, I, I guess it's something like 8,000 developers using the portal, which is, is amazing. \[00:12:29\] Yeah. But told us something about scale within the product as well, so that gave us some work to do. So it's pirate weather. Yeah. And \[00:12:38\] \*\*Track 1:\*\* Wow. \[00:12:39\] \*\*Allan Knabe:\*\* so, so we basically we're, we, we've onboarded these guys. And that, that's where it kind of came in and we've been working on it for, yeah, some, some years now. We've probably been working on the product too long. \[00:12:51\] We should have, you know, gone in, out the door sooner. But we, we wanted to make it like really feature complete, so wanna make sure that it has \[00:13:00\] like the, you know, great API products that are monetized. And one of the features we just added, actually, that I really wanna bring out is the API catalog. \[00:13:10\] Which is more like an internal developer thing. 'cause we came across some, quite some customers who have so many APIs, like thousands of APIs, and are cataloging them in an Excel sheet. So it must be a developer working in an organization like this bank we, we were talking to in, in South America. And if you wanna know what APIs the company has and and which one to use for any specific case, you gotta be searching an Excel sheet right. \[00:13:39\] No developer in the world wants to search an Excel sheet, right? Nobody wants to search it, so, so we put together this API catalog, which effectively, because we connect into the API gateways, we can retrieve a list of all the APIs available in that gateway and then list them. Now if you've got like just one API gateway, maybe that's \[00:14:00\] not so useful. \[00:14:00\] But we find that especially in enterprises, they can have like up to a hundred API gateways or more. Right. You know? Especially if they've got lots of regions, again, different technologies coming into Play Kong versus Apigee and so on. But having like this one unified API catalog. Is being really helpful. \[00:14:21\] It's like one of the features that we didn't intend to build, but, you know, it was the most requested and it was actually kind of easy for us to build since we already did all the integration work to the underlying API gateways. \[00:14:35\] \*\*Track 1:\*\* Yeah, I think that fits pretty cleanly into the, like, problems worth solving that a lot of certainly enterprise scale companies will never dedicate time to you know, if, if the thing is working and someone can find it in the Excel spreadsheet, like dust it off, problem solved, we'll call that good enough. \[00:14:49\] But I'm, I'm sure, especially if you've. Been able to build a solution that's reasonably plug and playable from there and it makes life better. There's clear value add, especially if they see, you know, API \[00:15:00\] usage, picking up or whatever monetization, picking up whatever the metric is there. I think there's lots of ways to measure that. So can we talk a little bit about the, I guess the like major features that API will provides today? It sounds like there's obviously the, the gateway product and the catalog. Is there a documentation angle as well? What else are you providing right now? \[00:15:18\] \*\*Allan Knabe:\*\* Oh yeah. Yeah. I mean the, the core developer experience is something that was always very very important to us. And, and so there we're actually using something open source called rapid Doc Web that I, I guess we can get to that. In more detail, but we're using Rapid Doc web to, you know, put the API specifications up. \[00:15:36\] We were going to build our own documentation tool, but then we found this and said, okay, look, it's absolutely fantastic. We'll go ahead and use that. So, so we put that in there for, you know, developer documentation. We do some nifty things like synchronizing the documentation from the API gateway. \[00:15:54\] Which is nice. So you can update it on the documentation on the gateway and you get it synchronized across \[00:16:00\] into your API products. Some of the other things we do as well is we, we really tailor the documentation for the subscription you are using, right? So sometimes you, you see like an API specification, which is, I dunno, 50 pages long. \[00:16:20\] And you have to try and work out as a developer, you know which parts of it are relevant to like your use case that you should be using, right? And so what we said is that if you can say, okay, in this particular API product and in a particular plan, because you might have like a small, medium, and large plan. \[00:16:40\] And in the large plan, you might get more endpoints. More methods, right? And you want the documentation to reflect that, right? So you can switch between these different plans and see, okay you know, oh, okay, I'm not getting a post possibility on this endpoint, but I do get that in the large plan. Right? \[00:16:58\] And so making that really \[00:17:00\] clear to the developer what, what they're working on with the whole, you know, triad functionality is a triad console there as well. Once you've subscribed to an API, the API credentials are automatically assigned to the documentation. So you can just start pressing buttons and it will do, you know, API calls for you and so on. \[00:17:19\] So, you know, trying to help the developer. And I, I'd say, you know, they're one of the key things as well is that we noticed, we noticed working at, at Swisscom and elsewhere as well, is that the decision maker to use an API is not always the developer, right. Developer may be the one like signing up using it, but we really feel that if we can get the decision maker to sign up for the API and put their credit card in and sign any contracts and go for the approval process, and then do invite the developer in a at the correct time. \[00:17:55\] That's much more useful for the developer because, you know, no developer likes \[00:18:00\] waiting for approval. Right. It, you know, you, you go, you go to a site, you sign in, you know, I wanna use this API, the worst thing a developer can see is, okay this is pending approval. Yeah. We'll get back to you when John comes off his vacation in two weeks time. \[00:18:15\] So, so we, we really try and, you know, work from our experience of seeing Okay. You know, who's making the decision to use the api? Sometimes it's the developer directly, right? For, you know, more enterprisey, contractee, APIs we, we let you know more of a business person take care of all of that stuff, and then just say, okay, within the team, invite in the developer to work on the API and then they just, you know, get straight in. \[00:18:42\] The subscription is there, they get the API key and they start working. Okay. \[00:18:46\] \*\*Track 1:\*\* Yeah. I am, I'm victim or I have been victim of that in the past where hey, here's the next great thing we're gonna use, or maybe I have the idea to, Hey, like, our team should probably adopt this product. And you get there and you're hit a door stop because it's not security \[00:19:00\] approved or I. Accounting hasn't approved the expense or whatever the case may be. It's a really interesting approach. I, I think typically when we're talking about API products it's not really a facet that comes up and it's definitely like an important angle, especially I'd imagine if you're dealing mostly with enterprise level customers, there's a lot more of that like red tape in place to keep the business safe and secure. \[00:19:18\] That makes a lot of sense. \[00:19:20\] \[00:19:20\] \*\*Track 1:\*\* You, you touched on something a minute ago that I, I think is interesting too, that you were talking about differences in plans for APIs and making different things available. I noticed on your site that there's some monetization features available as well. Can you talk a little bit about that? \[00:19:33\] \*\*Allan Knabe:\*\* Yeah, that's, that's really exciting. Right now we're working on some of the more advanced monetization cases as well. At the moment we've got, you know, more of a like standard subscription base that if you subscribe to use these API endpoints, you can use them for, I don't know, 10,000 times a month, 10,000 API calls for 50 bucks or something like this. \[00:19:54\] So that's kind of where we started on. A lot of our early adopters have used this, but \[00:20:00\] we are getting into some cases now where we're doing more usage based pricing. Which is what people expect with APIs, right? So per API call, you're gonna, we're gonna charge you $1, or whatever it is. The caveat there is that we, we are not a proxy to the API, right? \[00:20:17\] So we do not get involved in like the networking layer of your API. We leave that to the API gateway. So unlike some other solutions which proxy everything and add latency, we don't do that. The downside of that, of course, is that we then don't going get to count the number of API calls. So we have to do that afterwards via the the log file, which is what we're working on right now. \[00:20:38\] We hope I mean may, maybe even by the time this podcast comes out we'll have completed this for these advanced cases where. You know, you can charge for a particular endpoint for a particular method, you can have a different price to anything else, right? So especially with AI companies, they if they really are interested in these cases of \[00:21:00\] saying, okay, certain calls are more valuable than than other ones. \[00:21:04\] And we get 'em requests now for credits as well to say, okay, someone could buy a package of credits. And then depending on the kind of course they're making it uses more or less credits. So, so these, you know, it's really interesting time to be working in this and yeah, luckily the team is doing a really good job, so. \[00:21:24\] \*\*Track 1:\*\* Yeah, that's a big change in paradigm that I think we faced probably somewhere in the last two years where LLMs in particular have made it so that like the same API call costs differently every time you call it, based on how much input and output that it's, it's using which. I don't know if, if you use it back to, call it February of 2020 that was a pretty foreign concept not too long ago and is very, very like standard these days for, for a lot of the creative AI based tools. So really cool to see being able to support that. There, there's a lot here. I'm, I'm honestly really impressed by the amount of things that, that API will. Provides, and \[00:22:00\] even like the granularity you're getting into for, for monetization between seeing freemium and usage based and different subscription tiers and all those things, there's, there's quite a bit that goes into that. Can you tell me a little bit about what it would be like for a team that's interested in using api, able to adopt API able, what does the integration process look like? \[00:22:18\] \*\*Allan Knabe:\*\* For the integration process, we really tried to make it. As simple as possible. I know everyone one says that, but we, we saw like when we were at Swisscom and elsewhere, that there are products you can get and you can do a lot of your own development to, to integrate, et cetera, and you are ready some weeks later. \[00:22:35\] But we, we had, I mean if we take just that one case of pirate weather very small company in Canada, I think they were up and running in an hour or something. They just connected their. Amazon API Gateway, which you can do with a role right in the Amazon. Like 10 minutes to connect that. And once you've got your APIs well then you just need to, you know, put a pretty picture on it and say how much you want to \[00:23:00\] charge basically. \[00:23:00\] And it's kind of working, right? So yeah, we really tried to do it like, you know, product led in, in that case, so that it was, you know, off the shelf. Very quickly. Obviously for enterprise customers, they might want a little bit more, more help with the onboarding process and so on. And for that we have, you know, pilot programs where we sit with them and, and discuss their needs and so on, which is fascinating because then you get to uncover like these, like really urgent needs they have, like the, that's how we got to the API catalog part as well, right. \[00:23:31\] Just by talking to, you know, people who wanted to use it and then figuring out that actually they want a catalog to seal their APIs, right? And being able to add that. That quite quickly. But it, it does drive my development team insane because it, it's just constant feature requests, you know, from me. \[00:23:50\] And I apologize to the team if they're listening to this, but I, I do keep like a constant flow of new features coming in. So at some point I have to stop and we \[00:24:00\] just need to tidy everything up, but yeah. \[00:24:03\] \*\*Track 1:\*\* Well, that's the joy of building a product that people want, is that there's lots more ways to use it that come up along the way. \[00:24:09\] So one of the interesting things about the product you're building then, and, and from what I'm hearing is that a pable is probably from the. Implementation standpoint of the dev teams that are integrating with API bullet, you're sort of language ag agnostic, right? \[00:24:22\] There's really just needs to be able to point to and understand the sort of the shape of a gateway. Is there do you find yourself working with particular I don't know, flavors of team more often than others? Like, do you, do you find that Ruby developers come to you or rust developers or something like that? \[00:24:37\] Or is it coming from all over the map? \[00:24:39\] \*\*Allan Knabe:\*\* It is all over the shop, to be honest. We, we, we get everything come through and at the moment we're getting all shapes and sizes of company come across. It's, it's obviously really interesting working with startups. And, and you know, them getting like Python requests and stuff like this, it's really interesting when they come along because \[00:25:00\] they, they wanna get going immediately and have it finished in two weeks and they don't mind giving you a hundred requests at the same time. \[00:25:07\] So very useful speaking to those guys. Very useful. The enterprise is, you know, first of all, you have a conversation for six months with someone who's not actually ever gonna use the portal. And then, then you get to the real team who are, who are kind of using it and connecting gateways and stuff. \[00:25:24\] So it's two, two different worlds. But yeah, in terms of like, you know, languages, it's, it's a huge chasm. \[00:25:31\] \*\*Track 1:\*\* I could imagine. What about open source work? Are you doing anything with open source projects? \[00:25:36\] \*\*Allan Knabe:\*\* Well, well, I hinted at rapid Doc. Is, is one that we use within our products. It's honestly very, very well put together. You, you can find it at a rapid doc web.com. I guess you'll put a link as well in. \[00:25:51\] \*\*Track 1:\*\* Yeah, absolutely. \[00:25:52\] \*\*Allan Knabe:\*\* Yeah, that's great. So it, it, it's it's a really good project. I think, you know, probably more people could get value from that if you just \[00:26:00\] want to, you know, if you've got a simple API you wanna document it and put it up somewhere it's a really nice project for that. \[00:26:06\] And we're, we're hoping to, to support that community more and more as we go forward. So, so that's exciting. Other stuff we're kind of like interested when we get five minutes because like I said, it's a constant like feature frenzy at, at api able, but one of the things we're looking at is more on the oof two side as well. \[00:26:27\] Somehow helping. And this was, this was like an idea we had a few years ago working at Connor. And that's when you turn around to a product manager when you're building, for example, an app and development team starts to ask some questions like, what, what flavor of OO two should we use for securing this application? \[00:26:46\] Right. And getting blank stares back. Right. So the, the, there's some kind of a translation needed there. That I think is really doable. Some kind of like a tool that you could build to \[00:27:00\] say, okay, in, you know, more, more human terms. How would you like to secure something? What are the cases, right? You know, is it on a per user who's logging in basis or is it more of a machine to machine case, wherever. \[00:27:14\] So some kind of like, maybe just utility that. You know, you go through that flow and out of it comes like the correct o or flow or you know, JWT, what you should be using as a technology to secure it for that given case. Right? So that's kind of one open source tool that we would like to, to put out at some point. \[00:27:36\] 'cause OF two is, you know, well, it's developer, right? And it doesn't translate very well to, you know, you can't send, you know, we've done this, right? We've sent like business people the o to a specification and said, yeah, just choose the right flow, right? And they've looked at it and go, oh my God, this is crazy. \[00:27:55\] So, so there's that kind of thing there. The other one that we would \[00:28:00\] like to look at maybe get a chance this year a little bit more is to say, okay, how green are your APIs? Right. So I'm, I'm sure Phil would be interested in this one as well. But we're in a, a very good place since we're connected into all of the API gateways across the organization. \[00:28:19\] And we can, you know, see from the log files what developers are all doing. And start to drop in some hints for the organizational owners, the customers to say, okay what could you be doing to reduce the footprint here? Because I mean, APIs are, is it 83% of all internet traffic or is it more than that? \[00:28:39\] I can't remember. It's around that kind of like area, a huge amount of work happening with APIs and you know, are there some quick wins, for example. If we see a developer is getting the status of something every two seconds, right? It could be you know, a potential then to turn around \[00:29:00\] and say well, maybe this should be behind some kind of a web hook, right? \[00:29:03\] Say, okay, instead of some polling all the time, you know, and, and that generating a lot of backend work, you could say, okay, we'll update you in like four hours when this thing has changed status, right? So. That's like an immediate idea, but I'm sure we could uncover like, a lot more to say. Okay. Just, you know, let's, let's reduce the number of API calls if possible. \[00:29:25\] And also not forgetting the, the load on the back ends as well. \[00:29:30\] \*\*Track 1:\*\* Yeah. Oh, that's, that's a really interesting product angle and you're certainly well positioned to be able to advise on that. Given what you must know about how, you know, gateways are deployed and consumed and all that really fascinating. Yeah. I think that is, one of those things that teams rarely think about, right? \[00:29:46\] Like, not everyone has a Phil Sturgeon on their team who is very, very cognizant of the existential crisis facing all of us with, with environmental friendliness. \[00:29:54\] And it's hard. It's, it's something that's subtle, right? \[00:29:56\] Like, it seems really simple to make a polling call and like, yeah, \[00:30:00\] every two seconds we're sending a couple of packets across the web. \[00:30:02\] But like, \[00:30:03\] you stack that up for a day and a week and a month, and you're, you're, putting a lot of energy through the wire that doesn't necessarily need to be done there. That's super cool. \[00:30:10\] I like that. \[00:30:11\] Um, oh, sorry. Go ahead. \[00:30:12\] \*\*Allan Knabe:\*\* I was gonna say, it's, it's actually a, it's probably an entirely new company and product of its own Right. But that, that, that's, you know, if someone wants to run with this idea, go for it. Yeah, again, my team hate me because I always have these crazy ideas. I. \[00:30:27\] \*\*Track 1:\*\* I like it. I'm into it. Well, I, I will make absolutely sure to drop your GitHub organization in the show notes as well, if people are interested in following the open source and exposed code and projects that you're working on, on the web. And certainly if folks are interested in chasing you down for working on some of these projects there, there'll be ways to do that there too. Let's let's talk about what's to come for you. What is API bull looking at building next? What are the things on the horizon that are exciting for you? \[00:30:53\] \*\*Allan Knabe:\*\* Yeah, I think I already, I already touched on kind of like these, these features coming in advanced monetization cases.\[00:31:00\] Also along with that, the analytics that we can get to say, okay, how are you using the APIs a little bit. This like green thing is in there as well. Of course, you know, when we get to see, okay, what you, what you're doing with the APIs and, you know, our developers you know, signing up to use your api I, but then not making the first API call. \[00:31:20\] Or did they, or they, were they a regular user of your API and certainly dropped off and things like this? There's lots of very interesting things we can do there as well, but I, I think we're pretty much like feature complete. Other than that we'll probably add more API Gateways, we support Amazon, API Gateway and Kong and Appg and Azure. \[00:31:41\] And so we'll probably add some more API gateways this year. Lots of people on like MuleSoft and stuff like this, so, we'll, we'll, we'll add those in. But that's yeah, we're, we're, we, we need to, you know, not do more features and so I'm \[00:31:55\] \*\*Track 1:\*\* Yeah, features versus focus is a perennial startup founder challenge. I get \[00:32:00\] that. Okay, so what about your team? Are you expanding? Are you looking at hiring? \[00:32:04\] \*\*Allan Knabe:\*\* Okay. So we're, we're currently going through our seed round. So, so we're doing another raise at the moment, and as soon as that's complete, then yes, we will be looking for developers to join the team. You know, especially we have need for DevOps type people, site reliability engineers to, to take care of the infrastructure because there's a lot of moving parts. \[00:32:25\] We opted for a single tenancy application which is, which is great. But at the same time takes a lot of, you know, maintenance and looking after. So that's, that's definitely one role we'll be looking for in the future. \[00:32:39\] \*\*Track 1:\*\* Got it. Yeah, that's another link I have in the show notes is the, the correct sort of URL for careers or job listings for api. I able too. That's great. So before we wrap up, Alan where, where's the best place for listeners of the show to go to find and chat with you if they're interested? \[00:32:55\] \*\*Allan Knabe:\*\* Okay. I spend most of my time on, on LinkedIn. I'm not cool enough \[00:33:00\] to be on X, so I'm, I'm mainly on, on LinkedIn. Other than that, you know, it just come to the website and then just, you know, throw in a contact request or something and you know, get a, get ahold of me. That way as well, \[00:33:12\] \*\*Track 1:\*\* That's great. And just so that we say it out loud, what is the website. \[00:33:16\] \*\*Allan Knabe:\*\* a API able io so you can think of it's, we make you API able or you could be asking a question. As a developer, when you look at some code, you can say. Is that a pable? \[00:33:28\] \*\*Track 1:\*\* I love it. Again, another one of those things we'll drop in the show notes. And Alan, thanks so much for, for joining me today. It's been a real pleasure. I'm super excited about what y'all are building, and happy to have you come back on the show anytime if you're you know, rolling through new feature announcements. \[00:33:41\] So there's some you know, environmentally friendly things we want to get into for a deeper conversation. I'm happy to do that too. Super, super cool to have you here. Thanks for joining today. I appreciate it, Alan. \[00:33:49\] \*\*Allan Knabe:\*\* Yeah. Thanks a lot, Mike, for having me on. Really enjoyed it. \[00:33:52\] \*\*Track 1:\*\* Of course. Yeah. Take care. Talk to you soon. \[00:33:54\] \*\*Allan Knabe:\*\* Thanks, bye. \[00:33:56\] ​\[00:34:00\] Mike Bifulco ### Climate Friendly APIs URL: https://apisyouwonthate.com/newsletter/climate-friendly-apis/ Last updated: 2024-02-01T14:44:48.000Z Reducing the environmental impact of the software we write is crucial in helping our industry limit its environmental impact. Due to a lack of inaction and the continuous push of the fossil fuel industry, we all must do our part. Sometimes, things can feel overwhelming, or life circumstances limit your ability to live in a van and plant trees daily. Thankfully, you can adopt green practices into your code, meaning you can have a real impact with every release you push. While this newsletter focuses on software, if you are interested in what you can do on a more personal level, Phil has written a fantastic blog series. Starting with [explaining the problem](https://philsturgeon.com/unf-king-the-climate/?ref=apisyouwonthate.com), he then digs into [food](https://philsturgeon.com/unf-king-the-climate-food/?ref=apisyouwonthate.com), [waste](https://philsturgeon.com/unf-king-the-climate-waste/?ref=apisyouwonthate.com), [electric cars](https://philsturgeon.com/unf-king-the-climate-electric-cars/?ref=apisyouwonthate.com) and [transport](https://philsturgeon.com/unf-king-the-climate-freedom-from-car-dependency/?ref=apisyouwonthate.com). \-- Alexander ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2024/01/aspen-ad.png) ## Meet Aspen: Revolutionize Your API Testing Workflow! Explore Aspen by Treblle, the advanced HTTP client revolutionizing REST API interactions with AI-powered integrations and instant code generation. Enjoy secure, local data storage and exceptional performance on MacOS, ensuring minimal memory and CPU usage. [ Download Now (No account required)](https://getaspen.io/?utm%5Fsource=newsletter&utm%5Fmedium=apis-hate-sponsorship&utm%5Fcampaign=aspen) ### The API Round-Up Our monthly round-up of articles, news and updates from APIs You Won't Hate and around the web. ### [What is Green Software](https://philsturgeon.com/what-is-green-software/?ref=apisyouwonthate.com) Unsure where to start on the green path? Phil covers what green software is and why you should care. He gives a high-level overview of carbon grid intensity and temporal/spatial shifting. A great place to start if you are unsure. ### [Green Software Foundation](https://greensoftware.foundation/?ref=apisyouwonthate.com) The Green Software Foundation is on a mission to build a trusted ecosystem of standards, tools and best practices. They have some fantastic resources to get you started covering green software practices and a whole host of patterns you can use to reduce your software footprint. Their Carbon Aware SDK provides a web API and command line tool to help you pick the greenest times and locations to run your applications. ### [Cloud Carbon Footprint](https://www.cloudcarbonfootprint.org/?ref=apisyouwonthate.com) Working across multiple cloud providers, or maybe you're using AWS, and you can't get the granularity you need on the emissions of your software? Cloud Carbon Footprint has got you covered. You can run their full-fledged API and web portal or set up the API. The emission reporting is entirely transparent, and the accuracy beats many of the default reporting offered by cloud providers. ### [Electricity Maps API](https://www.electricitymaps.com/?ref=apisyouwonthate.com) This API allows you to measure the origin and footprint of consumed electricity. It can also be used with the Cloud Carbon Footprint tool above for more accurate emissions. The API has real-time data and forecasts—a great tool to help build carbon-aware platforms. --- ### APIs You Won't Hate The latest from the team at API's You Won't Hate. ### [🎙️ Redocly, OpenAPI, and Learning to Moonwalk with Lorna Jane Mitchell](https://apisyouwonthate.com/podcast/redocly-openapi-moonwalk-lorna-jane-mitchell/) Phil sits down with Lorna Jane Mitchell to get the latest on what's going on at Redocly, the state of OpenAPI 3.1, and what's to come with OpenAPI's proposed 4.0 spec, Moonwalk. ### [🎙️ Microsoft built Kiota to keep an API with 20,000 endpoints humming](https://apisyouwonthate.com/podcast/microsoft-kiota-graph-api-darrel-miller-vincent-biret/) In this episode of the podcast, Phil has a chat with Darrel Miller and Vincent Biret from Microsoft's Graph API team. They discuss Kiota, Microsoft's API-wrangling toolset which was born from a need to manage 20,000 endpoints on their Graph API's v1 endpoint. ### From Our Community Articles written and shared in our free [Slack community](https://slack.apisyouwonthate.com/?ref=apisyouwonthate.com). ### [How Rigorous API Testing Can Prevent Outages During Disasters](https://danaepp.com/rigorous-api-rate-limiting-testing?ref=apisyouwonthate.com) Dana has been sharing some fantastic articles in our Slack community. In this issue, he focuses on testing and how covering all your bases with tests can prevent severe outages. The article offers a great example of a disaster app that hits a rate limit during a disaster. A great read; thank you, Dana. ## Support APIs You Won't Hate When you become an member, you'll get access to members-only content while directly supporting our work. Your support helps us to keep making resources for the API community. [Become a member today](https://apisyouwonthate.com/#/portal) Thanks so much for reading, and for supporting APIs You Won't Hate. Until next time! Alexander, Phil, and Mike ### Redocly, OpenAPI, and learning to Moonwalk with Lorna Jane Mitchell URL: https://apisyouwonthate.com/podcast/redocly-openapi-moonwalk-lorna-jane-mitchell/ Last updated: 2024-01-22T07:00:47.000Z ## Show Notes - Lorna Jane Mitchell - [lornajane.net](https://lornajane.net/about?ref=apisyouwonthate.com) - Redocly - [https://redocly.com/](https://redocly.com/?ref=apisyouwonthate.com) - [Meet Redocly CLI: The Modern OpenAPI Sidekick ](https://apisyouwonthate.com/blog/redocly-cli/) - [OpenAPI 3.1 Spec](https://spec.openapis.org/oas/latest.html?ref=apisyouwonthate.com) - [OpenAPI Moonwalk (4.0 proposed spec)](https://www.openapis.org/blog/2023/12/06/openapi-moonwalk-2024?ref=apisyouwonthate.com) - Find great tools for working with OpenAPI at [https://openapi.tools](https://openapi.tools/?ref=apisyouwonthate.com) [Meet Redocly CLI: The Modern OpenAPI SidekickRedocly CLI is a brilliant new tool from the folks who made ReDoc, the first beautiful API reference documentation tool powered by OpenAPI. This CLI tool goes a lot further than documentation, and helps with “linting” (automated API Style Guides), and solves the biggest problem that I had previously been![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/size/w256h256/2023/03/apis-logo-square.png)APIs You Won't HatePhil Sturgeon![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2023/11/meet-redocly-cli.jpg)](https://apisyouwonthate.com/blog/redocly-cli/) --- #### Transcript \[00:00:00\] \*\*Phil Sturgeon:\*\* Hello everybody and welcome to another episode of APIs You Won't Hate With me, Phil Sturgeon and no Mike, the co-host. So I'm still muddling through trying to figure out what buttons I meant to press, but slowly get in the hang of it. And thankfully, I'm joined today by Lorna Mitchell. Say hello, \[00:00:17\] \*\*Lorna Mitchell:\*\* Hi. Hello. Happy to be here. \[00:00:20\] \*\*Track 1:\*\* Nice. \[00:00:20\] How's it going? I think you've been on the podcast before. I meant to actually look that up back in the day talking about something or other, \[00:00:26\] but if not. \[00:00:27\] \*\*Lorna Mitchell:\*\* I actually don't think I have, I, I have written for APIs you won't hate, and \[00:00:31\] I like to write. I should do that again. I'm not sure we've done the podcast thing before. \[00:00:36\] \*\*Track 1:\*\* alright, well either way welcome. And for folks who dunno who you are, could you describe a little bit about yourself? \[00:00:44\] \*\*Lorna Mitchell:\*\* Sure. I am Lorna. I am currently VP of Developer experience at Redley. But my background is I am been a software engineer for a long time. I'm a conference speaker. I'm a published author and I am mad passionate about APIs and \[00:01:00\] open source. \[00:01:01\] \*\*Track 1:\*\* Ha, sorry, I was just doing some research live on a podcast like a real professional . Fantastic. Yeah, no, your, your name pops up everywhere in open API. And yeah, you were pretty helpful in the open API 3.1 stuff when we were trying to get all of that across the line. And what was it? What was the main thing you were working on? \[00:01:20\] I completely forgot because I, my whole thing was like, Jason Schema needs to match open API or what are we even doing? But you were working on other bits? \[00:01:27\] \*\*Lorna Mitchell:\*\* Yeah, so for 3.3 0.1, which is when I really came into working on open API itself, I was working on the webhooks thing. \[00:01:34\] That's when I was at Vonage and with their two-way messaging APIs, you need webhooks callbacks are not the same. So that was I in my not humble at all opinion. One of the best things about the 3.1 release of open API. \[00:01:49\] \*\*Track 1:\*\* Absolutely. I, I forgot that. Was that you that helped push that through and yeah. We're using it, using it right now for some webhook stuff that protect Earth's doing. So yeah, it, it's really nice that that's there. \[00:02:00\] 'cause it was always a bit weird trying to hack that into callbacks. You'd have to like. \[00:02:03\] Pretend you'd have to make some sort of fake URL called like web hooks or something, and then just kind of pretend what it was for. But no, this is, it's useful to be able to say, sometimes there are messages that are gonna come from this server that aren't simply in response to a request you just made. \[00:02:18\] You can register for a web hook through the website or whatever. I mean, Shopify's super weird. We're we're registering for webhooks. With them through like a CLA command, which is awkward. But then it will just forever spit web hooks at us on, on topics that aren't related to a single thing. So it's really handy to have that in there. \[00:02:35\] \*\*Lorna Mitchell:\*\* Yeah, I think there's loads of use cases. You know, callbacks gave us half of it. You know, it's in response to an incoming request, but webhooks lets you have that payload going across in response to something else. \[00:02:48\] \*\*Track 1:\*\* Yeah. Brilliant. And so you've been involved in loads of stuff with Open api. I, and obviously you're working for Redoc re Dock Lee now. But yeah, we, we, we've both been kind of in the same spaces working on trying to make like \[00:03:00\] a sidekick for open API. Right. I did \[00:03:02\] that recent blog post about how Redoc Lee has kind of got there now, but you were around in, in Specky trying to make that useful. \[00:03:08\] And, and spectral. And can you just talk a little bit about those tools and where you feel like they fell short and you know, what they're good at, what they're bad at? And then, and then what's going on with Redoc Lee? And its kind of open API psychic now. \[00:03:22\] \*\*Lorna Mitchell:\*\* Yeah, no, I think, I think there are so many great API tools now, but it's not as if they just came out of thin air now, like they're ready to use now. Like you say specky I think was one of the first ones that at least I knew of, which really helped me to validate and check things. I'm not sure I was customizing it a lot. \[00:03:42\] And so that can be quite frustrating if you're using out of the box rules. I think too many people do that and then it can be a hard experience. \[00:03:49\] \*\*Track 1:\*\* Yeah, \[00:03:50\] I think specifically I think you were the person who, well, a lot of people really hated all of the default rules about like tags and stuff. \[00:03:56\] But I seem to remember you giving a bunch of feedback about that. Like why is \[00:04:00\] it constantly complaining that I need to use tags, I don't actually need to use tags, and that was just in \[00:04:04\] there for some reason. \[00:04:05\] \*\*Lorna Mitchell:\*\* I, I, think it was an API with like two end points. I, I really don't need to tag them, but thanks. \[00:04:09\] And then we used Spectral a lot. I realized the other you'll love this. I realized the other day that my personal blog, so that's Lorna jane.net will put it in the show notes. My personal blog has instructions of how to link APIs with spectral. But it does not have how to link them with Reduc, CLI, which is, that's what I do \[00:04:29\] now. \[00:04:29\] \*\*Track 1:\*\* time for an updated blog post. I think \[00:04:31\] \*\*Lorna Mitchell:\*\* definitely. Well, and we'll just write all the blog posts for \[00:04:34\] all the tools because I think they're all great. But yeah, I used Spectral and that's when I started customizing rules. I was working for Vonage and what's interesting about this is I think I did Specky when I was working for Doing like integration consultancy. I, I worked on spectral when I was at Vonage. They're an API provider, so that, but they have a lot of APIs and they're quite different from each other. So working \[00:05:00\] on spectral there and making sure that we're enforcing that consistency. And, you know, I'm not sure developer experience was a buzzword then, but that's how I would call it now. In between. I worked at a cool, another cool Cloud SaaS and they had an API and that's when I started using Redoc Lee tools because I really enjoyed their linting and of course Redoc API reference documentation. So now I'm out redly working on Redoc Lee, CLI, which as you say is the API sidekick. \[00:05:34\] And I think I'm still seeing some of those same problems like. People use the recommended rule set and then tell us it's wrong. it's, you need to cus \[00:05:42\] \*\*Track 1:\*\* Yeah. Yeah. \[00:05:43\] \*\*Lorna Mitchell:\*\* 1 0 1 run it. We'll give you some good feedback, you know, and I'm seeing some more of those upload your API and we'll tell you what you've done wrong. Like Yeah, but every API is different, so you need to customize your rule set and use it yourself. So Redly CLI is open source. \[00:06:00\] It does do the linting, but it does a couple of things that I would describe as Either side of that in the process. One is it has functionality for splitting your open API up into multiple files so you don't have to deal with a hundred thousand lines of diff. You can, and also you can look at like, which segments changed and that can be easier for review. It's also easier on your editor. And then you \[00:06:27\] can \[00:06:27\] \*\*Track 1:\*\* yeah, whenever you are trying to like send a pull request on a file that's 10,000 lines long, firstly it's just not gonna show you the preview half the time. And secondly, it's more, more likely to conflict. 'cause two people worked on something completely different, but it didn't realize it was completely different and therefore, but, \[00:06:42\] \*\*Lorna Mitchell:\*\* Yeah, and it's harder \[00:06:44\] to reason about which sections should have changed \[00:06:46\] when it's just one long thing. \[00:06:48\] \*\*Track 1:\*\* I think before, before we go past it too quickly. I think one of the most impressive things, so when I was doing the review of re doley I was like, okay, yeah, I know what this is about. They got like a liny thing and they got like a Bundy thing and they got like \[00:07:00\] a preview docs. These are all good, these aren't new things. \[00:07:02\] They've been kind of brought together in a, in, in one new tool that does a whole bunch of things, which is great. But what I was really, really excited about discovering was the split thing. 'cause the number of times, like I'm looking at some massive open API file going, ugh, like it's been generated from HCTP or whatever. \[00:07:18\] You're kinda like just, you're just staring at that massive far going. This is gonna take some time. And I think like stoplight studio at some point had vaped some functionality in like, move this to a model. But then the button broke and it was disabled, and I begged and begged and begged to get that button put back in and they were, oh, we're too busy. \[00:07:34\] I was like, no, because taking a, taking a giant opiate open, API file. Like, I feel like no one's doing enough refactoring, \[00:07:40\] automating, refactoring, like move this to move this to models. These two, these two in place things look the same. Change that to one ref. And I think, you know, split, split does a lot of that. \[00:07:51\] They, they noticed that you've got refs and they were like bung 'em off into files and it did also notice that two things were the same thing \[00:07:57\] and put them into the same file. And I was like, God damn, \[00:08:00\] that's useful. \[00:08:00\] So I was really glad to see that even though it's like a, a undersung kind of feature. \[00:08:05\] Yeah. \[00:08:06\] \*\*Lorna Mitchell:\*\* it is. But I think it does really help, and particularly As we mature in our API practice, it's not just about, I wrote an open API file or I generated one. There you go. I've done that tick You know, those are living, changing documents and so the maintenance aspect gets quite big. So yeah, the, I think the, the splitting bundling and we also have join, so where people are maintaining different, every \[00:08:31\] team maintains an API each, but you publish it as a single surface that's actually really messy. \[00:08:38\] I've seen some horrible merge. Attempts and \[00:08:41\] join attempts to handle that, which I think is really nice. So from an API management point of view, the other thing that I love about Reduc, CLI, which I am now seeing in other places, but Reduc CLI has had it for a long time, is decorators the ability to \[00:08:59\] repeatably \[00:09:00\] transform something. an open API document so you get something it's not perfect or you want to add something for the next stage in your pipeline. There's a decorator's set. There are some built-in decorators, but you can also extend it by saying whenever you see a, I dunno, an an operation parameter, do this. \[00:09:23\] \*\*Track 1:\*\* Interesting. Like, like what? Make it more tangible for my struggling brain, \[00:09:28\] \*\*Lorna Mitchell:\*\* So we do things like you can add, you can replace all the descriptions. This, this operation Id should have this description. This one should have this description. So if \[00:09:35\] you're getting poor quality generated content or you wanna add, you know, the descriptions in open API support markdown, so you can add really beautiful, rich links and whatever. \[00:09:49\] Do you want to maintain that in your source code? When you are generating from that, you know, \[00:09:54\] \*\*Track 1:\*\* Oh, I see. \[00:09:56\] Yeah. \[00:09:57\] \*\*Lorna Mitchell:\*\* you are maintaining the YAML for the markdown \[00:10:00\] in your source code. I just, I do understand why people don't enjoy that. \[00:10:05\] \*\*Track 1:\*\* Absolutely. Yeah. I think it's the, the people that advocate most for the kind of code first B it all into annotations in our code approach are the people who aren't ever gonna put very much effort into their documentation. And therefore you know, you're not, you're not gonna see these like . Hundred lines of markdown showing up in the source code purely because they wouldn't bother to write it and the tech writers can't be faf to go and like get in there. \[00:10:31\] Um, \[00:10:31\] \*\*Lorna Mitchell:\*\* Not even have access. \[00:10:33\] That's really common. You get frustrated tech writers who could add really great, you know, descriptive content. Example values. \[00:10:41\] If it's generated from source code and not, and they're not giving access to the source code, it's very difficult to include that. And we're starting to see tools as well. \[00:10:49\] Obviously the documentation tools, Redoc and friends are the very good API Documentation tools are available. \[00:10:57\] \*\*Track 1:\*\* Standard BBC response. Yeah. \[00:10:58\] \*\*Lorna Mitchell:\*\* Well, I just feel like, I \[00:11:00\] don't mind which one you use, but please give me some documentation, you know, \[00:11:03\] \*\*Track 1:\*\* Yeah. Yeah. \[00:11:03\] \*\*Lorna Mitchell:\*\* And the SDK generators as well. You know, we've just seen speakeasy release the overlays tool, Because their tool does better with some added metadata and you don't wanna \[00:11:14\] maintain that in wherever upstream your open API. Things are coming from. \[00:11:19\] \*\*Track 1:\*\* Great. Well that brings us on nicely. So I've been meaning to pick your brain about overlays a bit 'cause it's, it's something that there's increasing interest in. I mean, I, I work with the, the folks at Bump as well, and they were asking about, about a bit more about overlays and there's a lot of people in the docs space who were interested in that idea for exactly what you're saying. \[00:11:36\] Like, if you know the, the code first, people dump out some open API. And it's whatever it is. And if you edit it, then it's just, you know, gone the next time they dump it \[00:11:46\] out. Or you've got the people that are design first where the, the YAML is the source of truth and they do all their edits and, and that's that, and that's probably a little bit easier to add things into. \[00:11:55\] But then maybe you are trying to expand somebody else's published design first. Stuff \[00:12:00\] that you don't have any control over. So there's this kind of constant idea of taking . Some open API and then like programmatically adding stuff to it for whatever reason. Tech writing, you could do translation, you could add a bunch of stuff in. \[00:12:13\] I've, I've seen your name come up for quite a few blog posts and overlay overlays. And while I was doing some research, I noticed the folks over at Redoc aren't quite so impressed with it. So I was wondering if you could explain where it all sits. \[00:12:24\] \*\*Lorna Mitchell:\*\* controversial? \[00:12:25\] Yeah. \[00:12:25\] I think so. There's an older blog post on the Redoc website where Adam, you know, It didn't love it when he saw it. It's, it's not a brand fresh site, but it is top hit for overlay. So it's \[00:12:39\] not a secret And I think the point there is a good one, that it's yet another complex structure. \[00:12:47\] We've got a specification with no tooling at all, and it \[00:12:50\] requires use of JSO path. Now, if you've been using Spectral, that is very JSON path heavy. \[00:12:59\] \*\*Track 1:\*\* \[00:13:00\] Yeah. \[00:13:00\] \*\*Lorna Mitchell:\*\* But v Doley, CLI isn't, we do that by considering the open API as like a tree structure and the different data \[00:13:10\] types. So everything's very type aware. And you can filter on fields and it knows, oh, this is a parameter you know, it'll have an in then We don't, with JSON path, it's just blind data structure. So we don't have JSON path currently in the Marine Dole CLI ecosystem I don't think at all. And so that extra learning curve, you sort of think, well, I don't know who could use this. Like it's a good solution, but there's no validator for the format. And the learning curve puts it beyond a lot of the users that \[00:13:44\] at re doley we're trying to enable. \[00:13:47\] \*\*Track 1:\*\* Right. And so I think when we are talking about the use cases for decorators, like overlays kind of intended to be a more generic implementation of that, that \[00:13:55\] anyone could theoretically use, whereas decorators is more of a like redly feature. \[00:13:59\] And \[00:14:00\] could theoretically be, you know, ji to, to be the same thing. \[00:14:02\] But what, what you're saying is yeah, the. right now. Overlays are designed to use JSON Path and for anyone who's not familiar if you've been using Spectral yet, then yeah, that's, that's the absolutely batshit bunch of things. You're typing into those rules to try and make anything, make work. Make work. \[00:14:19\] You can tell I'm tired. I was saying before the show, I've just come back from API days and, and normally API days gets you a hangover, but this time I've just been like sick for a couple of days and now I've just not slept. I decided to go and get myself into a hotel room early and then like got into bed at eight o'clock and there were just like homeless people screaming outside my window until 4:00 AM and then the fire alarm went off at 8:00 AM and I'm like, what is reality? \[00:14:44\] So, yeah, JSON Path is even more confusing than my brain right now. And basically you have to kind of tell it what you would like to filter through, kind like a CSS selector but for js. And then, so JSON Path doesn't have a standard. They're kind of working on one, but it's not there yet.\[00:15:00\] And so much like what markdown had, like markdown plus and a million different markdown flavors. \[00:15:05\] JSON path now has JSON path plus, which stoplight was using for quite a long time and, and used as the main thing for. For spectral rules, but then that wasn't quite enough. And then like Nier appeared, which is Jason Path Plus with some other bits and like it's even Jason Path Plus not Nier is a mixture of a lot of just squiggles and doodles and back ticks. \[00:15:28\] But also you can bung red rejects in there. So you can kind of say, I would like to dot gets you into the property of something and then square brackets can get you into like the array of, of something. And then there's like stars to say all of them. And there's tilts to say, parent, and you can kind of fetch all of the headers where the parent is on this and the, the name begins with an X and you can do really powerful things. \[00:15:54\] But it's . Almost impossible in any sort of . \[00:16:00\] I've made lots of spectral rule sets because I built an NPM rule set and, and I built an NPM test suite using jest so that I can type in these crazy things and go, am I close yet? Am I close yet? Am I close yet? \[00:16:12\] And the result will show me what I've done. \[00:16:14\] But most people are just writing a YAML file. And, and that doesn't work. So you have to like, literally switch over to your terminal, run it, and it just blows up at you in strange ways. And it, it can be very hard to do. So it's one of the things I really appreciated about about Redoc Redoc Lee, CLI was the, was the switch to using just named selectors. \[00:16:34\] So you can say like. Header instead of going like, you know and I think spectral iss trying it, it made early attempt to fix that by adding aliases. But I'm just going, going off on a bit of a tirade now. I've gotta get \[00:16:47\] some stuff off my chest. \[00:16:49\] \*\*Lorna Mitchell:\*\* Carry on. \[00:16:50\] \*\*Track 1:\*\* The roadmap for aliases was powerful and impressive, and I've smoked to Smart Bear who have now inherited this to try and get \[00:16:56\] this back on the table. \[00:16:57\] But basically aliases, the idea was \[00:17:00\] let's come up with a list of aliases. Common open, API, aliases common A and KPI, aliases, and then people can use those instead. So instead of dollar. Which is root and then paths, brackets slash post you, you would just say like you would just say paths and then you could do the square bracket slash post to pull out all the post ones. \[00:17:19\] So you would cut out a lot of the nonsense by using aliases to replace a certain chunk of of the path. Whereas I think Redley would've said, you know, paths and then you would have. Header if, if, if post or something, right? So yours is more of a DSL approach, but aliases seemed like a good step forward. \[00:17:39\] And yeah, the, the aliases ended up getting defined in the core rule set of like the core open API rule set. And they weren't extendable. So if you extend the core open API rule set, you can. Get those aliases out, they're only available inside the place. They're defined. \[00:17:59\] So that meant \[00:18:00\] you couldn't actually use those to help. \[00:18:01\] Like no one using, no one writing their own rule sets, extending open API or no one writing their own rule sets at all, could use those aliases \[00:18:08\] unless you copy and paste them. And so I'm, I've been suggesting that like not only should they be extendable, but they should be defined in like the format. So you should, you know, anyone using the open API format. \[00:18:19\] Whether it's the core rule set or not, you should get those aliases. But that's, that's a bit more of a, like a, a roadmap. One day they might fix it kind of thing, but \[00:18:28\] I, \[00:18:28\] completely recognize the need to not Jason path and therefore avoid that entirely in overlays. Right. \[00:18:33\] \*\*Lorna Mitchell:\*\* yeah. Well, I don't think it is avoidable in overlays. And I feel a little bit like I've done it, an injustice there. Like leading with that, leading with that negative, the controversial blog post. You know, not everyone's a fan. I really think that overlays. Solves a real problem. Like I see this all the time we just talked about when it's generated from code and it hasn't got enough anything. I'm also seeing much more \[00:19:00\] complex pipelines. Now. People have multiple teams maintaining different bits of the a p surface. They all ship an open API. We join it into one, and then the, the downstream outputs, whether that's documentation, SDK, generation API, gateways, like whatever. All of them need their own enrichment and overlays solves that problem where you don't have to, you can just, yeah, add a translation, add an example, add the hint for the correct data type for the target language in the SDK and the team that deals with the SDKs can do that. \[00:19:37\] We don't have to go all the way back in a, in a, in a big organization that's hard \[00:19:42\] to find. The team that maintains the thing. That lives in the house. The Jack bill, right? So ,we, we can do that here. Overlays solves that problem. I don't think it's a coincidence that one of the implementations, there are not many \[00:20:00\] overlays, implementations. One of them is mine. We can link to it. It's very hobby. \[00:20:05\] I just, I, this is how I get things out of my brain. I had to write code that did it so that I could look at \[00:20:10\] it. I need to \[00:20:12\] do \[00:20:12\] \*\*Track 1:\*\* that's really helpful. I, I remember, I remember reading the, the official page of the overlays working group. And oh, just a quick thing on working groups. \[00:20:20\] So open API used to do everything by everyone getting on a weekly call and then getting homework assigned, and then like, maybe it was done by the next weekly call, but that \[00:20:28\] was how everything ever happened. \[00:20:30\] And if there wasn't enough interest in certain things, then they just wouldn't happen. But now things have been split up into working groups and I think one of them was like workflows that I was . Briefly involved with, it's quite exciting. \[00:20:41\] And there's a bunch of other ones and SLAs and, and yeah, so overlays was, was one of them. \[00:20:46\] And it's great to see an early kind of finished I don't know what phases and stages there are, but I read the repo and it said like, we're basically stable. Now but no one's really implemented this, so can, you know, we need some big \[00:21:00\] implementations to go and do it. And, and it's cool to hear there's some hobbyist ones or some implementations out there, \[00:21:04\] but that, that's the hard part. \[00:21:05\] We had this problem at stoplight. \[00:21:07\] Everyone has this problem where you want to support a feature, but you get a few user requests come in saying like, could you use this feature? Could we. Could we have this feature? And then you're like, oh, I'm not sure we should implement it because it's not stable yet. \[00:21:20\] And the people who have like come up with it have said, well, we'll only call it version one when it's had some implementation and some feedback. So you get in this chicken and egg problem of everyone hoping somebody else \[00:21:29\] will implement it, so that then it stabilizes. But then if they're the first to market, then you lose out, but you're not prepared to commit the resources to it. \[00:21:37\] So \[00:21:38\] So where are we at \[00:21:39\] \*\*Lorna Mitchell:\*\* Good news. I have a hobbyist thing and I think Mike Sson has something as well. He donated me a \[00:21:45\] Tests. Mine's \[00:21:46\] open \[00:21:46\] \*\*Track 1:\*\* course Mike does brilliant \[00:21:48\] \*\*Lorna Mitchell:\*\* does. It's like he's got his finger on the pulse. But there's another implementation, which is from speakeasy. Now they do call modern SDKs. \[00:21:58\] It is not a coincidence \[00:22:00\] that it is them that have gone to implement with overlays because. Open API does not have all of the metadata that you need to generate a great SDK in every tech stack. Some of them need more data type instructions or serialization information, or, you know, there's a lot that you need to add that wouldn't be in a standard open API description. So they have these extra extensions, the x dash fields that they add. And rather than everyone, like you say, you can't add them because then when you update your open API, they are lost. So not a coincidence that they implemented overlays. Their tool is available, it's open source, it includes a validator. Kind of embrace and extended the, the implementation a little bit, but I think that's an interesting use case too. So yeah. More than one tool to look at. And I'm \[00:22:53\] excited because I think it, it really solves a real problem. \[00:22:57\] \*\*Track 1:\*\* That is really cool. And once again, I'm googling mid podcast \[00:23:00\] 'cause I am extremely professional. But I'm looking at the speakeasy dev. I'm just seeing AI powered end-to-end API maintenance. \[00:23:06\] And again, there's like, I've just come from API days. Every other talk was about AI and I just, you know, my, my eyes are in pain from rolling, but this looks like a, a really useful implementation. They're like . They've noticed that you've type that they've noticed that you've typed in a duplicate schema and \[00:23:20\] then just like changed it to a ref , which is amazing. \[00:23:24\] And there's a lot of other pretty cool looking stuff on this marketing page. Let's, let's be clear, but I like their intentions. \[00:23:32\] \*\*Lorna Mitchell:\*\* Yeah. And I think \[00:23:33\] \*\*Track 1:\*\* dig in more. \[00:23:34\] \*\*Lorna Mitchell:\*\* I'm with you in in team Rolling your eyes. When it comes to Yeah. AI is gonna do everything but \[00:23:40\] actually . The open API, that standard machine readable specification is gonna enable a bunch of AI applications. So, and also chat GT four knows quite a lot of open API and can \[00:23:54\] like quickly give you, quickly give you back some hilarious and sensible example fields. \[00:23:59\] For \[00:24:00\] example, I wouldn't let it write \[00:24:01\] my API, but when I'm \[00:24:02\] working on examples, it's like, ah, this could be better. Like, \[00:24:05\] make me, make me something fun with circus animals. It can do that. \[00:24:09\] \*\*Track 1:\*\* Brilliant. Okay. Yeah. 'cause writing, you know, arbitrary examples was always the hardest part. I would always really struggle to do tutorials or demos 'cause I was like, just coming up with contrived examples is so frustrating. And like, everyone's got a bloody to-do app and no one cares. And, and just, I, I fundamentally refuse to use the pet store for anything ever \[00:24:28\] So, \[00:24:29\] yeah, I, I have \[00:24:29\] \*\*Lorna Mitchell:\*\* we can share a link to Redley just published a museum, API for more or less these reasons. \[00:24:34\] Also, the \[00:24:34\] pet store's really outdated. \[00:24:36\] SmartBear only just gone to 3.1, so we've been using a variation of this museum thing internally and we were like, we should ship this. \[00:24:44\] \*\*Track 1:\*\* right? Well, So I mean, , the, what really annoys me about the pet store is that it's a. Bad. API like it's describing a bad API. It's got like a bunch of really bad conventions in the actual API that it's describing. \[00:24:58\] it it uses a weird amount of, \[00:25:00\] it uses like a, an odd amount of open API to do it. I think it's being copied from like swagger two and, and \[00:25:05\] not being fleshed out. \[00:25:07\] And then, so now the pet store lives in open API three when like 3.1 came out, 20 20, 20 21. \[00:25:15\] \*\*Lorna Mitchell:\*\* 2021\. By the time we press the button, yeah. \[00:25:17\] \*\*Track 1:\*\* Yeah, so that's, you know, two, it's nearly, yeah, it's a while. \[00:25:22\] So don't use the pet store. There's other better examples and I like that AI can help with that for sure. I mean, I have been using copilot built into VS code and it's been helping me out for the protector. \[00:25:34\] API quite a lot. Like I was just adding an array of species to sites so that we can . Ahead of time, we can say, look, these are the only species we're gonna plant at this site. So when I'm going through the field and it's raining and I've gotta take a photograph of 4,000 bloody trees, we just planted for proof for our funding partners. \[00:25:50\] I don't wanna be scrolling past like sea buckthorn, which we've never planted, \[00:25:53\] Just to get to like the, the, the subset of the eight species that are actually in that bloody field. Right. \[00:25:58\] 'cause we've got like 50 different species \[00:26:00\] we may plant there. So yeah, like I was just putting that in and I just typed on the, on the sites. \[00:26:04\] List I typed in species and it was like, oh, would you like to add all of these obvious properties and this whole schema and then reference the docs slash schema no sorry. Schema slash species that I see over there. I was like, yes, yes, I bloody wood. And it just built like loads of stuff for me. \[00:26:20\] And even building, building that that one schema was great, but there have been lots of times where it's just completely hallucinated and done something absolutely bonkers, and I've just like pushed it. So \[00:26:30\] I, I still don't trust it to do anything much apart from like, kind of assist me, but I, I, I'll never do dictated but not read with AI \[00:26:38\] \*\*Lorna Mitchell:\*\* No, \[00:26:38\] \*\*Track 1:\*\* it's not there yet. \[00:26:39\] \*\*Lorna Mitchell:\*\* think as well, like it's a, it's, it's a time enhancer for me. One of the things that I find time consuming is I'm no longer particularly technically specialist in any given. Programming language. I write about \[00:26:52\] four with equal levels of danger. PP is my original community, but it's moved on. \[00:26:57\] I wouldn't recognize it if I walked past it in the street. \[00:27:00\] And so and so actually I can write all of those languages and I can debug all of those languages, but I just like throw the thing I want. In and I get something back and then that's enough. So otherwise you're like, how do I iterate? How do I concatenate? What have I done wrong with \[00:27:16\] \*\*Track 1:\*\* Yeah. Yeah. \[00:27:17\] \*\*Lorna Mitchell:\*\* on this? You know, like I know that it JavaScript array is a reference, but I need it to not be. And so that kind of speed up is quicker for the things that I don't know. It just lies and I can't catch it. So yeah, I'm with you. That \[00:27:31\] \*\*Track 1:\*\* That that is absolutely the problem. And it's totally fine if this decay like this podcast decay into complaining about ai. 'cause we've been meaning to talk more about AI on it for a while. We've got someone coming on soon to talk about things that it is useful for and so we can get some of the bullshit outta the way. \[00:27:44\] But yeah, even with PHP, like people have said, oh, it can help you write your tests or whatever. And I was writing a test for a model and it just kept recommending that I like write these really complicated tests for relationships, which you don't really need to do. 'cause if you have an OMM that has relationship logic and you \[00:28:00\] just say, I. \[00:28:01\] You know orders should have organization. You don't really need to write a test. That's, that says orders do have organization, \[00:28:06\] but the test it wrote was invalid and didn't fundamentally work at all. And kept doing loads of other stuff. It was trying to test methods that didn't exist \[00:28:13\] like it was trying to find out if that I. Order was allocated to something, but orders aren't allocated, like units are allocated to orders. So it, it sore enough to have a real college try, but it didn't actually do the right thing. And it just confused me more than if it didn't. 'cause if, if I hadn't spent like 10 minutes trying to faff around with the, the bad test that it made, I would've just Googled how do I test this thing \[00:28:37\] and then done it \[00:28:38\] So. \[00:28:39\] \*\*Lorna Mitchell:\*\* And I do worry about how much it helps us to learn new things, and especially if you are earlier in your career, like actually, is this helping you to be the best that you can be? It, I'm, I'm just not sure, and time will tell, I think. \[00:28:54\] \*\*Track 1:\*\* Yeah, one, one devil's advocate there is that it's, it's just sped up stack \[00:29:00\] overflow, copy pasting, which is, you know, and it has a bit of knowledge of your system \[00:29:03\] where a stack overflow has none. So there's plenty of times that I've like Googled for a problem and like copied and pasted the close enough thing into my code base and, and hope for the best. \[00:29:12\] And so if it's, if it's doing that, but slightly better than, okay. \[00:29:17\] \*\*Lorna Mitchell:\*\* It \[00:29:18\] isn't \[00:29:18\] \*\*Track 1:\*\* still not really adding that knowledge and there's, you're missing the, like, the ability to see down votes and, you know I mean, I think you were around in the code United Days, right? And, \[00:29:28\] and if not early PHP days where a lot of people were doing really dumb stuff and really dumb stuff was popular \[00:29:34\] and, and the popularity of something, the frequency of which you see something doesn't make it better. \[00:29:39\] And so you kind of have to hope that at least with Stack Overflow and some of those forums, you'd have someone posting a really terrible idea that may even be accepted and then a bunch of people down voting and going, whatcha doing underneath it? \[00:29:52\] Which is good. And with this it just goes, now take that. \[00:29:54\] May it'd be fine. \[00:29:55\] \*\*Lorna Mitchell:\*\* Accepted answer. Go for it. Yeah. And what you want is you want the most voted answer. \[00:30:00\] And you also want, did this change recently? You know, stack Overflow is not a \[00:30:03\] \*\*Track 1:\*\* yeah. Yeah. It was the correct answer for years Yeah. \[00:30:06\] \*\*Lorna Mitchell:\*\* was correct but after 2015, you should do it this way. Like that's a, that's a real answer, and I feel that. Machine necessarily on our side with that \[00:30:17\] because they can't take in that context. \[00:30:21\] \*\*Track 1:\*\* can I, can I do a, a complaint about oh, it's my podcast to do what I want? \[00:30:25\] \*\*Lorna Mitchell:\*\* I love, please, please continue. \[00:30:28\] \*\*Track 1:\*\* one of the talks I saw by a lovely bloke, well-intentioned talk, but the whole thing fundamentally seemed to be. Integrating with other people's APIs is really hard, so here we can get AI to do it for us. \[00:30:43\] So here is an, an AI doing all of this really complicated, amazing stuff and it was truly impressive what the toy demos were doing. But towards the end of the question and answer, it was very much, how much does the AI \[00:31:00\] need the API to be built in a specific way? And and he was like, oh yeah, basically there are certain ways that AI will expect the API to be built. \[00:31:10\] And if it's not built following those very specific assumptions, then this basically won't work at all. So what some people do is build a new API specifically for the ai. \[00:31:20\] \*\*Lorna Mitchell:\*\* Mm-Hmm. \[00:31:21\] \*\*Track 1:\*\* Where it does follow all of those conventions and then it can save you loads of time. It's like, oh, okay. We're all just completely rebuilding all of our APIs and, and doing BFFs for every client just so that AI can then save some time instead of looking at the docs we spent ages making. \[00:31:36\] I, that's, that's sort of thing is where my, my, my bullshit flag goes up. \[00:31:41\] 'cause it's like, is this just early days and the AI will eventually be able to talk to any API or are we just kind of saying . Like the wizard will fix it and pass in more work onto API teams who probably would've had a better time sitting down with their stakeholders and asking what they really need instead of building some shit that was so confusing in the first place, that \[00:32:00\] everyone's scared of looking at the documentation and then ask for an AI to be built so they can figure out what the hell you did. \[00:32:05\] Like \[00:32:05\] maybe we could just make better APIs. Yeah. \[00:32:08\] \*\*Lorna Mitchell:\*\* Well, and I, I'm an optimist, so I, I wanna flip that argument and say you're right, but it, what you're actually saying is that the people who have, well-designed, well-documented APIs described with open API. Can integrate easily and well both with other platforms and with ai if that's what people want to use it with. \[00:32:27\] It's good for all integrations. People who have badly designed APIs incomplete API documentation, they are gonna have a hard time. I don't think the wrapper APIs add anything. You, \[00:32:39\] you'd be better just to start again, talk to the actual users talk to the AI if you like. They're very chatty. And make, and make things better. \[00:32:49\] \*\*Track 1:\*\* That's cool. Yeah, I think it's a good point is that some of those, some of those people were coming from a place of, you don't have any open API or anything similar. So if it's just \[00:32:57\] like you have an implementation and then \[00:33:00\] like some Jason happens, I guess then how would anyone know anything about this? \[00:33:04\] And I think open API is an interesting way that it can add in more. You can, you can train an API on that a bit. And even if they don't understand what all of the objects are moving through. Part of something that a, the API community needs to do anyway. Part of my like 10 year plan, 20 year plan for like what I'm gonna be blathering about in API world has always been step one, get people defining what the hell this data is, right? \[00:33:28\] Like open API isn't the end goal that we all, we do this and then everything's brilliant. Even though I talk about it plenty, it was was you have random Jason flying about the internet and no one knows what the fuck is going on. This is not great. So getting people to like . Write that down, make a contract that, that, that was step one and then, okay, great. \[00:33:47\] We know what your implement, you know, your version of a tree and your version of a person and your version of a whatever account. But then like, can we try and have shared collections? And at \[00:33:56\] stoplight we were doing design libraries where you can have like a \[00:34:00\] repository of open API models you can reuse within your organization. \[00:34:04\] 'cause there was one company that had . 500 different versions of a flight, which is \[00:34:08\] literally plane goes from there to there. Like that's not very complicated. With some seats, but there are a lot of different versions. And so they've, they've kind of squished that down to one version of a flight now. \[00:34:18\] So it's literally describe the mess and then like reduce the, the, the mess and then like reuse those outside of the organization \[00:34:27\] with, you know, shared data models and, and concepts like schema.org. But maybe not that 'cause it's like. A lot of that is biological medical data, which doesn't seem very helpful, , but that kind of concept. \[00:34:38\] And then after that, like everyone is more trained, Haos becomes more useful in that world because \[00:34:43\] you don't need to train it to know exactly what that URL is. You train it to know what these generic models are and how you work with them. But it also makes AI more useful 'cause it then has that same shared knowledge of what's coming back. \[00:34:53\] So I, I like that that approach seems to be. The same as my existing approach. , I'd have to \[00:35:00\] change my thoughts too much. Great. But yeah, the other, there is a lot of like magical Wi Wizard stuff in the world of AI and I do wanna kind of get people to be a bit cautious about what they go all in on. \[00:35:10\] 'cause it, \[00:35:10\] some of it doesn't seem very helpful. \[00:35:12\] \*\*Lorna Mitchell:\*\* agreed. \[00:35:13\] \*\*Track 1:\*\* Last bit we were gonna talk about was open API four. I have been very out of the loop on this. It has a fun name, moonwalk, what's going on, help. \[00:35:22\] \*\*Lorna Mitchell:\*\* Yeah. I'm not sure why it needed a project code name. But I think we've been kicking this around for a while. I mean, I went off and spent two years working in the database space and came back. We were still talking about it, so it's not going anywhere. \[00:35:36\] \*\*Track 1:\*\* I was gonna say you vanished for a while and then came back to open \[00:35:38\] \*\*Lorna Mitchell:\*\* I did, I had \[00:35:39\] \*\*Track 1:\*\* said yay. \[00:35:39\] \*\*Lorna Mitchell:\*\* absolutely lovely time doing loads of derell with loads of open source databases at Ivan, which was \[00:35:43\] great. Now I'm back in the API space. That's great as well. Did I mention I'm an optimist? So news, news from the open API community. Then I mean, I think I. There's news in both the V three and the V four worlds. The V three news is just, we are seeing \[00:36:00\] a lot of adoption. We are seeing people using 3.1\. I think some of those educational resources are starting to get out there. Companies like Redley, and, but it takes the whole ecosystem \[00:36:10\] and, and we, I think even though we are commercial competitors, we know we share a community and we collaborate really well. I'm sure that's true in other industries as well. But, so yeah, three, going from strength to strength, new tools all the time. General goodness, four. We've just put out a blog post from the Open API initiative committing to launching a version of four in 2024. \[00:36:37\] \*\*Track 1:\*\* Okay. \[00:36:37\] Getting people ready for it instead \[00:36:39\] \*\*Lorna Mitchell:\*\* ticking. Yep. And there is, if anyone wants to see, we'll put it in the show notes. There's a repository full of discussions on GitHub about what we want to solve. At a high level um, it's, we are \[00:36:52\] looking at simplifying the structure, making it a bit more resource oriented \[00:36:59\] and a little \[00:37:00\] bit more approachable as well. Jason \[00:37:02\] \*\*Track 1:\*\* Just to explain on that a second, the resource orientated, meaning it's a bit less about like paths. What are your paths? 'cause that's been a, a concern about . Not really being very resty. \[00:37:13\] 'cause like paths, some people will say that in a rest API like paths are the least important thing. \[00:37:18\] And they could literally be like a random MD five Check sum is the most extreme example I've heard someone say because it really is just, it's about interacting with resources as like a state machine. \[00:37:29\] And the homepage is a resource that then like lets you see other resources you could interact with in various ways and describes how. \[00:37:36\] And so the fact that open API has always been like, tell me your paths, resources are optional, has always felt a little bit RPC ish to many people. So \[00:37:44\] is that something that's \[00:37:45\] \*\*Lorna Mitchell:\*\* so \[00:37:46\] well, and the goal I think, with moonwalk is to get away from paths so that we can include more types of API, including the ones \[00:37:54\] that are less restful. So \[00:37:57\] with RPC, where you have one end point \[00:38:00\] and maybe different query parameters or. \[00:38:03\] Body data makes different actions happen. You can't represent that right now in \[00:38:09\] open API. If you can just have, it's like the path and verb combination. Is it get, or is it post? What's the exact \[00:38:17\] path? Not including query parameters, that's your unique identifier. \[00:38:22\] \*\*Track 1:\*\* Yeah, someone had to make open api sorry, open RPC, which is like a copy paste of open API, but tweaked it so that you could do RPC stuff. And \[00:38:30\] so yeah, that would be nice if. \[00:38:32\] \*\*Lorna Mitchell:\*\* Yeah. \[00:38:32\] And looking more at, there are more, the more things than that, that go into what we are now calling a signature. Like what endpoint is this? How would you root it in your code? You know, does your API do different things on header or on some other thing that we could take into account. Otherwise, you end up with this very lots of polymorphism because you can't. Correctly represent that these two things look completely \[00:39:00\] different. If it's, you know, completed orders or incomplete orders, \[00:39:03\] it might be a filter on an end point, but you're gonna get different types of object back. Moonwalk is gonna cater for that as well. But there's \[00:39:11\] a thriving community, lots of chats. Th three isn't going anywhere. That's a lively community too. And as with the open API initiative, we're looking more at how we can Better respond to questions in that community, stuff like that. I have triaged on those repos and just trying to get everyone to be a bit more present. \[00:39:30\] \*\*Track 1:\*\* Nice. I will have to dive in a little bit more. I mean, bloody Yeah, we had Darryl Miller on on the episode on the podcast, and I didn't ask him a single question about it, but we're talking about other stuff. I think the reason they came up with moonwalk was that they wanted a bit of a experimental playground where they could just try and hash out some ideas \[00:39:48\] without everyone going. \[00:39:49\] That's definitely what's happening. Panic, panic, don't do four. Um, Which would definitely happen. We've had that before. So I can understand that. And it sounds like it's kind of got enough agreement amongst the \[00:40:00\] people that were paying attention that it. Is moving towards possibly actually being four. \[00:40:05\] So that's, that's what you want. So we'll just have a normal name. It will just be called open API four. \[00:40:10\] We're not doing like cool branding now \[00:40:12\] \*\*Lorna Mitchell:\*\* Well, \[00:40:12\] a \[00:40:13\] \*\*Track 1:\*\* that's cool. ' \[00:40:13\] \*\*Lorna Mitchell:\*\* cause you're actually walking backwards. I don't know. \[00:40:15\] \*\*Track 1:\*\* Yeah. , this is, this is several very casual steps backwards. Brilliant. Well, yeah. Great. I'm glad to hear those changes. \[00:40:22\] I mean, one, one, . Small thing in my head is that. There are certain things that open API won't let you do that. I've always been quite happy about, in a \[00:40:31\] way, cause like open API will let you perfectly describe a bad API. But there are some of the things you shouldn't be doing that you can't currently describe. \[00:40:40\] And so by being able to describe more of those things you shouldn't be doing, I'm a little bit like flexibility is important so that you can get all the old APIs on and then you can edge them in the right direction with, you know automated linters. And, and get them doing the right conventions that way. \[00:40:54\] But part of it was also you. Man, why did you do that? Please don't do that. Change that quickly.\[00:41:00\] \[00:41:00\] \*\*Lorna Mitchell:\*\* Yeah, I think that's a, I think \[00:41:02\] that's a real concern. And, you know, I want, I wanna represent and respect everyone else's work, but looking at the adoption for \[00:41:08\] 3.1, which was a very obvious improvement and upgrade for 3.0 users. some tools were available quite quickly. I know some took some time. \[00:41:19\] There's a whole raft of new tools have come out for 3.1\. We're really seeing the adoption now. 4.0 doesn't offer anything to existing users. \[00:41:27\] \*\*Track 1:\*\* Mm. \[00:41:28\] \*\*Lorna Mitchell:\*\* brings more people into the fold, and if you're an existing tools vendor. Which obviously that's my job now. I'm not sure where we're going with that. It's going to be significantly more difficult to build for \[00:41:40\] as things look at this very early stage. So I am supporting everything that's happening in the Open a BI initiative. Very excited to see just better. Education, better appreciation in the industry for what we do with Open API three and 3.1 and how it can \[00:41:57\] help. And watching what's happening in \[00:42:00\] four. It takes a long time to write these specifications and for tools to follow. So nobody needs to panic. \[00:42:08\] \*\*Track 1:\*\* Yeah. Yeah. It's not appearing anytime now. I mean, even if, even if, didn't you say that mo version four is committed to appearing sometime in 2024? \[00:42:17\] Well, that's, that's 12 months long, isn't it? So \[00:42:20\] \*\*Lorna Mitchell:\*\* And I don't know, you know, open API initiative does not build tools, and I'm not, it's not clear to me right now who will. \[00:42:27\] \*\*Track 1:\*\* Right. Well, that's the thing. So yeah, like in a year's time we could have a spec out and then still no one will use it. 'cause it takes a long time for, for tool and vendors to catch up. Which is why it's important to like, you know, plow on whilst, but some tools have only just upgraded to 3.1\. Yeah, this all takes a long time. \[00:42:43\] So like, someone needs to be thinking about the future instead of just going, don't change anything. It's scary. \[00:42:47\] \*\*Lorna Mitchell:\*\* Me it is \[00:42:48\] about doing good rep, good support \[00:42:51\] for 3.1\. Lots of people are coming into APIs or raising their API \[00:42:56\] game now with what's available today. \[00:43:00\] And I \[00:43:00\] think, you know, I work in developer experience. I really care about stuff that you can do today, and so that's my focus. \[00:43:06\] \*\*Track 1:\*\* absolutely. Yeah. Gotta do both. But as long as, as long as the new one thinks about migration path, like as long as, as long as it's made in such a way where there is a migration path and, and that can be automated, I think that's pretty important. \[00:43:18\] But also. \[00:43:19\] \*\*Lorna Mitchell:\*\* is one of the requirements. Yes, \[00:43:21\] \*\*Track 1:\*\* Okay, good , because various tooling vendors. \[00:43:24\] \*\*Lorna Mitchell:\*\* They got it. \[00:43:25\] \*\*Track 1:\*\* Yeah. Well, Mike Ralph's there saying like, he, you know, he, he made swagger to open API and he is gonna make open API three to moonwalk or whatever. Like he, he's on it. But yeah, that, that is important. And then the other thing, like Eric Wild made this really good point at a API days Paris about how \[00:43:43\] Async API released a async passer, and it is like the defacto passer. You can pass a document any way you like. You can also use theirs, and I'm sure it's just written in, in TypeScript and nothing \[00:43:54\] else, but this thing, this thing exists and, and the majority of tools can use it. And an open \[00:44:00\] API majority of the tools are TypeScript or no something. \[00:44:03\] So I'm, I'm really thinking that like that is the next most important step is for somebody probably Mike, let's be honest, he loves to write some code. Somebody to uh, just volunteered him either way. Somebody to write like an open API passer, that is the default. 'cause there's been lots of these other little ones here and there. \[00:44:18\] Like I've just deprecated one of them that I awkwardly inherited from some guy I tried to help and then, and then I got stuck doing all the code and I was like, I don't have time for this. There's a lot of these like awkward ones floating about, but if there can be one sample represent you know implementation. \[00:44:31\] Initially written by a person, but as more people use it, more people will help maintain it. And then that can be okay, you've got, you've got your passer that works for 3.1, but there's this other passer over here that works for four and that makes it a lot easier to switch to. 'cause you don't have to rebuild your passer to support four, you just use the thing. \[00:44:49\] And then you don't have to worry about all the intricacy, awkward bits of yaml. It's just you have some models to, to work with. I really hope they can do that. And I'll be nagging people to give that a try. Without volunteering myself 'cause I'm crap \[00:45:00\] at code these days. \[00:45:01\] \*\*Lorna Mitchell:\*\* I think it's a tricky one, you know, 'cause I work for a tools vendor now, so I was like, oh, you are gonna build some of the tools. Do you wanna tell me what else you're gonna build so that we don't invest our commercial expertise in \[00:45:12\] \*\*Track 1:\*\* Yeah. \[00:45:12\] \*\*Lorna Mitchell:\*\* you know, and I'm .I'm not really sure if the proliferation of many tools in open API is a feature or a bug. \[00:45:19\] But I think feature, I think the competition has helped because it's very open source, isn't it? It's, it's a, \[00:45:25\] it's an open standard. There's loads of different tools. You know, open API tools tells that story loud and clear. You can choose, it's in your tech stack. There's always something. Whereas \[00:45:37\] I think if we go to one official version, we're tied down to the speed that the Central initiative works at, which isn't always the same speed the industry does \[00:45:46\] \*\*Track 1:\*\* Yeah, I mean, hey, maybe that could be a working group right in, in, instead of it just being , very overworked, Mike and trying to do the thing to make all the vendors happy, who are all getting paid to work on their tools and he's not. Maybe that could be a working group where we're like, let's hash out like what we \[00:46:00\] need from a, a shared passer. \[00:46:01\] And then if anyone in the group ends up not really getting something that. They're happy with and they can just go and make their own however they want. But yeah, definitely it is a difficult line in general with like, should the open API foundation be making all this stuff? 'cause they're like, well we've just made a documentation tool. \[00:46:17\] You know, like that would be pretty messed up. \[00:46:19\] \*\*Lorna Mitchell:\*\* well, and the members are documentation tool vendors, so that's awkward Async. KPII have the opposite one, and you might have seen this as well from having Async KPI support spectral. Reduc, CLI has Async API support as well, just for linting. We don't have it in docs yet, and honestly there's the Async API community doesn't really talk about anyone else's tools. \[00:46:42\] It doesn't have the same ecosystem. They just build their own tools. They're all quite closely coupled. It's the same people. \[00:46:48\] \*\*Track 1:\*\* Right. \[00:46:49\] \*\*Lorna Mitchell:\*\* Postman pays them. That's what you get. And so it's very, very different despite the fact that those two projects have quite a lot of shared philosophy. They also don't, \[00:47:00\] and the tools is \[00:47:00\] a really big differentiator. \[00:47:02\] \*\*Track 1:\*\* That is interesting. Well, still, I'm always excited about the potential for collaboration and at some point I'll be picking your brains about I chatted to all the other API linkers and we're all gonna have a conversation about like coming up with a shared rule set format. And even if it's not one that anyone particularly wants to make their their default, it's just like, can we make some tools that swap them in all directions? \[00:47:21\] 'cause that'll be cool. Although I gotta talk to the optic team about, they've just come up with um, \[00:47:26\] lint, GPT, where you just write your rules as a, an array of strings and hope that the wizard is consistent. But \[00:47:32\] I'm gonna give that to all a try \[00:47:34\] \*\*Lorna Mitchell:\*\* I think Optic currently have the best converters, so they understand this problem space, but because. \[00:47:40\] \*\*Track 1:\*\* Yeah. Yeah. \[00:47:41\] \*\*Lorna Mitchell:\*\* Redoc, CLI, all our linting is very like a ST data structure driven. It's not \[00:47:49\] JSON path driven like the others. My, I had a quick look and I was like, oh, this is gonna be really difficult. \[00:47:55\] Interrupt because we are not describing, you \[00:47:59\] know, \[00:48:00\] we're like it, oh, you have a response example here. This is what we do with this response examples. It's not like it's this path, this path, this path. \[00:48:08\] \*\*Track 1:\*\* Right. \[00:48:09\] \*\*Lorna Mitchell:\*\* Turn, turn left at this array. Then take the third success response element. You know, like \[00:48:14\] \*\*Track 1:\*\* Yeah. I gotcha. cause it's based on, it's based on OAS kit. Right. And I remember how that works. Like OAS kit just walks through and then goes, oh, I found one of these. Anyone care about this? Yeah, you do. Cool. Run some rules on it. Whereas we're like, go through the rules and then like . Delve into the depths following this arcane direction. \[00:48:29\] I shouldn't have opened up this can of worms so late in the podcast. \[00:48:32\] We like \[00:48:32\] \*\*Lorna Mitchell:\*\* we're gonna still \[00:48:33\] \*\*Track 1:\*\* 35 minutes normally, and I yeah, I could, I could talk to you for hours, honestly. Yeah, it it's really great to have you on the show and we'll have to get you back on to talk about other stuff in the future. \[00:48:41\] For sure. \[00:48:41\] \*\*Lorna Mitchell:\*\* That \[00:48:42\] would be awesome. \[00:48:43\] \*\*Track 1:\*\* for coming, \[00:48:43\] \*\*Lorna Mitchell:\*\* for having me. \[00:48:44\] \*\*Track 1:\*\* Cheers. Bye. \[00:48:46\] ​ ### Low Code, No Code APIs URL: https://apisyouwonthate.com/newsletter/low-code-no-code-apis/ Last updated: 2024-01-15T13:03:35.000Z In this issue of our newsletter, we will focus on low-code and no-code APIs. Phil has written some fantastic articles over the last month, and I highly recommend you check out his review of different [no-code and low-code API tools](https://apisyouwonthate.com/blog/no-code-low-code-api-solutions/). Developers can give low-code and no-code tools a harsh dismissal. Some will say tools like this are for start-ups without tech founders or testing out an MVP. I find many of these tools amazing, and they grow with your needs while also automating away grunt work. Sure, they are not suitable for everyone or every situation. A counter to using these tools tends to be scale issues. The reality is that not many places need the scale and customization of FANG-size companies. --- Thanks so much to our sponsor ## Unlock the Future of API Technology: Discover the 2023 Treblle Report Dive into the cutting-edge world of APIs with Treblle's comprehensive 2023 report, **Anatomy of an API: Exploring the State of APIs in 2023*. This insightful report is your gateway to understanding the latest trends, innovations, and best practices in API development and integration. [Download the Report now (no email required!)](https://report.treblle.com/?utm%5Fsource=newsletter&utm%5Fmedium=apis-hate-sponsorship&utm%5Fcampaign=api%5Freport) --- ### The API Round-Up Our monthly round-up of articles, news and updates from APIs You Won't Hate and around the web. ### [Supabase](https://supabase.com/?ref=apisyouwonthate.com) Phil touches on Firebase in his article, a tremendous low-code tool, but have you checked out Supabase, the open-source alternative? I would never pick Supabase based on its no vendor lock-in marketing line; we always lock ourselves into something. However, it has some fantastic features: auto-generated APIs, Auth, Edge Functions, etc. Honestly, I love Supabase and would recommend it again and again. ### [Retool](https://retool.com/products/workflows?ref=apisyouwonthate.com) You may know Retool as a low-code solution for building admin portals; however, they now have a workflow feature. You can visually create workflows based on any data source, connecting API's and other third-party apps. Retool now offers many low-code solutions, not just an admin portal builder anymore. ### [n8n](https://n8n.io/?ref=apisyouwonthate.com) An open-source node-based workflow automation tool. Users can automate tasks and integrate various applications and services using a visual workflow editor. n8n has a tone of features, can connect to almost anything and has excellent community support. ### [Appwrite](https://appwrite.io/?ref=apisyouwonthate.com) Another fantastic low-code tool like Supabase is Appwrite, an open-source tool that allows you to build backends quickly. It gives you databases, authentication, storage and more. It helps you create REST, Graph and real-time APIs for connecting to your server. Honestly, why build anything when you can just use Appwrite? --- ### APIs You Won't Hate The latest from the team at API's You Won't Hate. ### [Low Code & No Code API Solutions](https://apisyouwonthate.com/blog/no-code-low-code-api-solutions/) Strap in for a great read. Phil gives a breakdown of some great low-code and no-code tools. Covering Headless CMSs and no-code Application Platforms and then moving on to low-code tools like AWS Amplify, Strapi and Firebase. It is a fantastic dive into some of the best tools in the space available. ### [Utilize AI for API Governance with LintGPT](https://apisyouwonthate.com/blog/ai-powered-api-linting-with-optic/) Phil walks us through the latest tool from Optic, LintGPT. Linting with style guides is a great way to keep your API spec in shape. With LintGPT, you can define powerful rules by giving it human readable prompts instead of writing long, messy rules yourself. ### [🎙️ Kiota: Microsoft's solution for managing SDK codegen for an API with 20,000 endpoints](https://apisyouwonthate.com/podcast/microsoft-kiota-graph-api-darrel-miller-vincent-biret/) On this episode of the podcast, Phil has a chat with Darrel Miller and Vincent Biret from Microsoft's Graph API team. On this episode of the podcast, we discuss Kiota, Microsoft's API-wrangling toolset which was born from a need to manage 20,000 endpoints on their Graph API's v1 endpoint. --- ### From Our Community Articles written and shared in our free [Slack community](https://slack.apisyouwonthate.com/?ref=apisyouwonthate.com). ### [Webhooks & Callbacks in OpenAPI/Swagger](https://www.speakeasyapi.dev/post/openapi-tips-webhooks-callbacks?ref=apisyouwonthate.com) Nolan Sullivan has written a great article on best practices for callbacks and webhooks in OpenAPI. See how they can enhance a REST API with these tools and how to describe them in OpenAPI. Thanks for sharing in our Slack community, Sagar. ## Support APIs You Won't Hate When you become an member, you'll get access to members-only content while directly supporting our work. Your support helps us to keep making resources for the API community. [Become a member today](https://apisyouwonthate.com/#/portal) That's all for now - thanks so much for reading and supporting APIs You Won't Hate. Until next time! \- Alex, Phil, & Mike ### Microsoft built Kiota to keep an API with 20,000 endpoints humming URL: https://apisyouwonthate.com/podcast/microsoft-kiota-graph-api-darrel-miller-vincent-biret/ Last updated: 2024-01-15T12:44:24.000Z ## Show Notes - [Microsoft Kiota](https://github.com/microsoft/kiota?ref=apisyouwonthate.com) on GitHub - [Darrel Miller](https://hachyderm.io/@darrel%5Fmiller@mastodon.social?ref=apisyouwonthate.com) - [Vincent Biret](https://about.me/vincentbiret?ref=apisyouwonthate.com) [GitHub - microsoft/kiota: OpenAPI based HTTP Client code generatorOpenAPI based HTTP Client code generator. Contribute to microsoft/kiota development by creating an account on GitHub.![](https://github.githubassets.com/assets/pinned-octocat-093da3e6fa40.svg)GitHubmicrosoft![](https://opengraph.githubassets.com/197a20a17b61f54491e167477d0c22e828f60e9681601385816855ca646a01ec/microsoft/kiota)](https://github.com/microsoft/kiota?ref=apisyouwonthate.com) #### Transcript \[00:00:00\] \*\*Phil:\*\* Hello everybody, and welcome to another episode of APIs. You won't hate this time. It's just me, Phil, with no mic. I, I don't know what I'm doing. He, there's a lot of buttons on my screen and I'm a little bit scared and confused, but I I think it's recording. I think this is a podcast and thankfully to keep me company with this world of confusing options. \[00:00:19\] I have two wonderful guests Darryl Miller, who has been on the show before, and a new guest Vincent Bere. Did I get that right? \[00:00:27\] \*\*Vincent Biret-1:\*\* Yes. Perfect. \[00:00:29\] \*\*Phil:\*\* Fantastic. Welcome both start with you Vincent. Would you like to just tell everyone who you are? \[00:00:34\] \*\*Vincent Biret-1:\*\* Yeah, sure. Hi everyone. Nice to meet you. Nice to meet you in person. Making our quotes at the same time, Phil. And I am based out of Montreal. I'm a principal developer on the Microsoft Graph client experience team. And I work a lot with Darl on the day-to-Day basis. \[00:00:51\] \*\*Phil:\*\* Nice. And Darryl, who, who are you? \[00:00:53\] \*\*Darrel Miller:\*\* I work at Microsoft as an API architect on the Microsoft graph and in my. Spare \[00:01:00\] time. I spent some time working on the open API specification and a few IETF HTP related specifications. \[00:01:06\] \*\*Phil:\*\* Nice. And I'm really glad that you could both come today to talk all about open AI and the latest happenings with Sam Altman. \[00:01:14\] \*\*Darrel Miller:\*\* Our lips are sealed. We can say nothing. \[00:01:16\] \*\*Phil:\*\* Okay. Yeah, no, I do often confuse open AI and open API. So yeah, let's, let's talk about that one. So you guys have made with a lot, a lot of other people involved. I'm sure you've made another SDK tool that's a little bit different from some of the others. \[00:01:32\] But first we have to ask ourselves, what is an SDK? What is this tool all about? \[00:01:38\] \*\*Darrel Miller:\*\* Well, I, I, first of all, I wanna say we, we are trying not to call it an SDK generator because. SDK is a term comes with a lot of baggage and is confusing and people have different understandings of what it means, and so we like to call it an API Client Code generator, which is a bit more of a mouthful, but it is a bit \[00:02:00\] more explicit. \[00:02:01\] Vincent, tell 'em what it does. \[00:02:02\] \*\*Vincent Biret-1:\*\* Well, it will for any rest, API that you have, that has an open API description, it will take that and generate client code to call your. Rest API with fluent API in the code and models, and it'll handle ization, ization and a number of different aspects for you so you can get going calling your API and you focus on what matters, writing the code for your application and not just, you know, handling ization digitalization and, and nitty gritty details like that. \[00:02:33\] \*\*Phil:\*\* Brilliant. And so we've had, we've had a couple of cool, a couple of podcasts about kind of SDK generators, client stuff in the past. I think depends on what happens between recording and actually publishing stuff. But like one of the last ones was about that. We've talked about Fern in the past and we've talked about amatic and so. Now we're talking about Kyo, not \[00:02:57\] \*\*Vincent Biret-1:\*\* Yes. \[00:02:58\] \*\*Phil:\*\* KTA from \[00:03:00\] Microsoft and. It's a little bit different. I noticed that you are trying to take the approach. I mean, some of these SDK generation tools kind of come across like you are the API development team and you don't have time to be right in a Go library and our library and other languages that you might not know. \[00:03:17\] So just run this command. And then tadda, it's been published and all different, A PA developers use all different. SDK generators and some they write by hand and some they use different tools. And so everyone's A-P-I-S-D-K looks a little bit different. It seems like you folks were going for a slightly different approach of kind of you're an API client and you could, you could grab a bunch of different open APIs from different people and generate a whole bunch of SDK well API client libraries yourself so that you can work with other people's APIs. \[00:03:49\] Is that about right or have I just butchered it? \[00:03:52\] \*\*Vincent Biret-1:\*\* No, it's perfect. Actually. You could, you could do the podcast on, on your own by yourself. You already know everything here, but \[00:03:58\] \*\*Phil:\*\* later. \[00:03:59\] \*\*Vincent Biret-1:\*\* yeah, \[00:04:00\] exactly. I'm, I'm gonna grab a coffee instead, but yeah. The idea is to. As you can still use Kda to as an API producer to put SDKs out there to reuse your own term and, and offer those to your customers. \[00:04:13\] You can perfectly do that with Kda of course, but we set out on a different approach, as you said, to. Make sure that you have a consistent experience across multiple APIs. More and more nowadays projects or applications have to integrate between multiple APIs, multiple vendors that we use, and having to, you know, find out whether or not they have a package for you in your language first, and then learn the semantics and the different aspects of this specific package for each and every API you consume. \[00:04:44\] Potentially get into, de dependencies, conflicts and having to resolve that, which is always awesome to do. We, we really want to get rid of all that, that experience for, for people and instead say, all right, you have your API consuming tool and API discovery tool for that matter. \[00:05:00\] Because KRA also offers search and discovery commands and, and, and features as well to allow you to discover public APIs out there. \[00:05:08\] And then you pick and choose not only the different APIs you're interested in from the different vendors or partners or whatnot, but you can even get down to choosing a specific. Operation under specific endpoint. So now you get a client that is very specific to you, what you actually need to do, and you get this consistent experience code writing experience across multiple API providers. \[00:05:30\] \*\*Darrel Miller:\*\* Yes. So Phil, the problem is developers have opinions. We all have opinions about HTTP and when people go and say, Hey, I need to build an SDK for my API, so that everybody will use it. So PHP developers will use it and Ruby developers will use it. We all go and build these libraries that say, well, this is the way that an H-T-P-A-P-I should be projected into into the native programming language. \[00:05:59\] \[00:06:00\] And the problem with having every API provider. Have their opinions on how their API should be projected. It means that poor API consumer now has to deal with everybody's opinions of every API that they go and talk to. So the Kyoto approach is, well if you can put it with our opinion, the Kyoto opinion on how you project H HB APIs, you can then use K for every API that you want to talk to and you only have to learn how to do it once. \[00:06:29\] And we are. We are as un opinionated other than . Having HTTP opinions, like we don't take the HP methods and say, well, usually when it's a post, it means create. Therefore we're gonna use the create verb on a method. We just tell you, there's your post, here's how you put HP headers in. Here's how you put query parameters in. \[00:06:50\] Here's how you format a request body. You understand the uniform interface, we're just translating it into your programming language, and we'll deal with the really annoying stuff, \[00:07:00\] like figuring out how to present in code parameters in a query string, and how to use a UI template \[00:07:05\] to construct. Yeah, yeah. All those ugly things. That those edge cases of HP we take care of. But you take our projected code, which people have said, yeah, this is not how, this is not an optimum experience for this API. And I'm like, yeah, I know it's not an optimum experience for the API put an adapter in front of it and create a perfect experience for your. \[00:07:27\] A particular application that you're building and use it as a mocking interface and then hide our projected code behind it. You're never gonna get beautiful projected code. And we keep trying to do that based on a whole bunch of people's opinions, and you'll never get everybody to agree on what the perfect projection of an HT P API is. \[00:07:48\] \*\*Phil:\*\* Yeah, absolutely. I mean in, in most kind of worlds of frameworks, there's a whole bunch of different. Coding patterns that people prefer. And then, you know happens in Laravel all the time. Everyone goes from repository, \[00:08:00\] pattern, event bust, all these different things and like different different naming conventions and everything else. \[00:08:05\] So yeah, certain communities can't be consistent with themselves and even if they do reach consistency at any point, it changes. So it can be really hard to try and make something that, that, that makes everyone happy. And I saw Remember, I remember seeing you give a talk at one of God knows how many conferences a couple of years ago about \[00:08:27\] \*\*Darrel Miller:\*\* The \[00:08:27\] kit back in SDK was what it was called. \[00:08:31\] \*\*Phil:\*\* yeah, that's the one. \[00:08:32\] It was like, yeah, how to make good SDKs or whatever. And a lot of it was talking about Middlewares. Do you still have a bunch of middleware logic in Kyo Did you manage to get that in \[00:08:40\] \*\*Darrel Miller:\*\* Yes, indeed. It's all built on that patent. Vincent, you wanna go into a bit of the details of the underlying bits? \[00:08:47\] \*\*Vincent Biret-1:\*\* Sure. So the. code we generate only relies on a set of abstractions that we publish, of course which means that as soon as \[00:09:00\] we, you generate a client for an API and that you pull the abstractions package, you'll be able to build your project solution application whatever. It won't do anything, of course, because you'll be missing a bunch of implementations, but at least it will build. \[00:09:13\] And then for those attractions that are for executing the HTP request serializing, decentralizing, and the authentications aspects as well. We provide default implementations that you're more than happy more than welcome to use and import in your project or your application. But if you're not happy with those because you need a different ization format or you prefer a different library, for example you can re-implement those abstractions and, and swap those away. \[00:09:41\] And if we focus on HTP and executing the request one of the abstractions we have is the request adapter and we have a default implementation that also provides a bunch of middlewares for retry handling, compression handling, redirection handling, and a different of of things set of our \[00:10:00\] things as well. \[00:10:00\] And, and now if you want to add custom behavior to that, you can of course just write. Additional middleware. If you're happy with a client of choice, we've selected, or if you're not happy at all with a client, which typical client we've selected you can of course reimplement the whole interface and, and, and, and do whatever you prefer here, but. \[00:10:19\] \*\*Phil:\*\* That's brilliant. Yeah. 'cause middlewares are just increasingly becoming the thing I'm the most interested in when it comes to . Pretty much any part of working with APIs, right? 'cause the, the actual business logic of, of most APIs is quite small. It's usually like, oh, we're gonna call a model and add some stuff and do a bit of validation. \[00:10:37\] But that's like three lines of code and it punts it off to a whatever service some sort of service class. But the most important, like API logic, Both, both client side and server side. Most stuff can be implemented as middlewares and everything from you know, rate limiting and caching. And I was just doing a whole series of posts about like item potency, keys and, and things like \[00:11:00\] that. \[00:11:00\] And both the client and the server can just kind of install one package and just register it and, and, and use that. So is there a bunch of like pre-built stuff? For Kyoto, you like, do you have support for item parent C keys? 'cause that'd be fun. \[00:11:15\] \*\*Vincent Biret-1:\*\* Now, so this one specifically we don't have but it'll be very easy for you for any language we provide and support today to say I wanna add support for potency, keys, and write up a middleware and add it to your chain and use the same exact client you just generated with qal. You don't need to change it for generated. \[00:11:32\] Code at all or touch any of that. You just need to add your crosscutting concern, implementation and, and, and you're good to go. And, and better than that we work a lot in the open as an open source project, so if you feel like what you've built will be valuable to others, we'll be more than, well, happy to see pull requests coming our way and, and, and, you know, make others benefit from your hard work here. \[00:11:54\] So, Yeah. \[00:11:55\] \*\*Darrel Miller:\*\* Did you notice? \[00:11:56\] \*\*Phil:\*\* sounds like it does. \[00:11:57\] \*\*Darrel Miller:\*\* Did, did you notice Phil in the latest \[00:12:00\] update to the item potency, item potency, spec in the ITF? That hasn't quite . Been gone through last call yet there was some wording that was changed that now allows a client to send an item potency key, even if they don't know whether the server supports it or not. \[00:12:18\] And \[00:12:18\] this, this came from browser vendors. cause the browser vendors were like, Hey, it would be really handy when we're. To send an item potency key in the hope that maybe the server supports it. And then if a server does, then, then, then, cool. So you don't need this pre-negotiated contract and Oh, I know that the server implements it, therefore I'm gonna send it. \[00:12:41\] You can just always send it on a post if, if you want that post to be ideally, \[00:12:47\] \*\*Phil:\*\* that makes a lot of sense. I mean, they already kind of take a huge amount of guesses at various cash in logic and, and do a bunch of things for you. And like most browsers will already say like, are you sure you want to refresh this page? 'cause that means you're gonna. \[00:13:00\] You know, resend this data. \[00:13:01\] Now they can change that to be like, are you sure you wanna refresh this? Because it might resend the thing. Or, but if they've had like good responses and they know that it does support it, then they can be like, yeah, you can refresh this and not have to show you the message. So that's pretty handy. I cool Bit of a tangent on item potency keys there, but there's a blog post. I'll tell you all about it. I wrote one for the APIs. You won't hate blog. That was mostly just me complaining about getting charged three times for a hotel. But then I wrote another one for HT TP toolkit on their blog, which is actually useful. \[00:13:31\] So go and check that one out. Now that I'm done plugging my stuff on my stuff, I am curious what Microsoft's interest in all this is. Why did you start building \[00:13:41\] kta? Did you just wanna save API developers from writing boring SDKs, or is there a bigger business reason \[00:13:48\] \*\*Vincent Biret-1:\*\* Both, like both actually. I think so, so for context both Darl and I work for Microsoft Graph, which is one of the largest rest APIs in the \[00:14:00\] world, both in terms of traffic and in terms of number of o of operations we support. I. To give you an idea, we have about 20,000 different unique operations in, on the V one endpoint. \[00:14:12\] So that makes for 70 plus megabytes open API description for the whole thing. \[00:14:17\] \*\*Phil:\*\* Good Lord \[00:14:18\] \*\*Vincent Biret-1:\*\* yeah. And, and so we, we found ourselves in a situations where the traditional, the existing generators out there. Would not scale in terms of how fast we can generate things they would not scale in terms of naming conventions and other things that, that Darl mentioned earlier. \[00:14:36\] Because they would try to generate clever methods name and that would not work at that scale of course. And they would also not allow. For more advanced scenarios like selecting the different endpoints you care about or not selecting the different operations you care about or not, and generating code just for those things. \[00:14:55\] And, and so this is how we set out to, or this is why we set out to \[00:15:00\] build yet another claim code generator for REST APIs a couple of years ago with Darl to not only solve our business needs, but we built it in such a way that is not. Just for Microsoft Graph and and the Devex developer experience for Microsoft Graph, but also that it's really useful for the broader community and the industry at large. \[00:15:19\] And hopefully people find it useful. Yeah. \[00:15:22\] \*\*Phil:\*\* Oh, I see. So you just released the, you, you really, you've released the software so that other people can fix your bugs for you, and then you can say, well, open source, \[00:15:30\] \*\*Vincent Biret-1:\*\* yeah, of course. Our personas is great. We get people work for us for free. No, but we, we, we are already collaborating with Red Hat on, on, on, on qda and, and we made a a key decision to ship a bunch of clients based off Qda for different, set of projects they have. We are also working with GitHub for their client experience as well there are a number of different players out there that not only, you know, get benefits from the open source, and the fact they can use it right off the bat, but they also contribute back to the project. So it's not \[00:16:00\] just, you know, us trying to offload book fixes to the community at this point. Right. \[00:16:03\] So. \[00:16:04\] \*\*Darrel Miller:\*\* So the other benefit of the approach that we've took, well there's, there's two benefits. There's, one is, as Vincent said, it's really big, our api, so nobody wants to use the entire API. So they want this ability to just project Code just for the parts of the API I they care about. But the other thing is a lot of our customers who are doing work with Microsoft Graph which stores like all your, your data for if you have an an M 365 license, so your email, your calendar, your contacts, all of that kind of stuff. \[00:16:33\] A lot of it is integration scenarios. Either integration with internal systems or integration with other third party systems. And we found a lot of our customers were doing work where they are making two APIs talk to each other, which is where this consistency story starts to come into play. And we found there were a lot of our customers. \[00:16:55\] Who just, who weren't using our SDKs. And our understanding \[00:17:00\] was yes, it's just another SDK that they have to learn. And this was, this was a big piece of feedback that we got from customers that people are just tired of going and learning yet another SDK. And they're like, I know how to make HTTB calls, and they'll just. \[00:17:14\] Go do it themselves because they don't want to have to deal with multiple different SDKs. And so Kyoto was, was a reaction to that kind of scenario that our customers are running into and we need to go build them anyway. And Vincent and I had lots of conversations about how, how open source this would be. \[00:17:34\] And it's taken us a little while to really convince our management that we should be building things that for customers who are not directly our customers, there's a lot of indirect benefits to shipping something that conforms very well to a standard like open API. \[00:17:49\] \*\*Phil:\*\* that's, that's always a really hard. A hard one to push, isn't it? It's like, Hey, can we do loads of work that won't make us money directly , \[00:18:00\] because they're like, yeah, open source, the SDKs. That's great. You don't need to open source. The thing that makes the SDKs, I mean, how, how much extra work has gone in that's like how long has a piece of string, but like how, how much extra work has gone into making it something you could, you know, confidently release to the public versus something that was just fine for your needs. \[00:18:18\] \*\*Darrel Miller:\*\* Yeah, that's \[00:18:19\] \*\*Vincent Biret-1:\*\* Ooh, that's . a, that's a very good \[00:18:21\] \*\*Darrel Miller:\*\* We don't wanna admit that publicly. \[00:18:24\] \*\*Vincent Biret-1:\*\* yeah. \[00:18:27\] \*\*Darrel Miller:\*\* So the, and and this is the challenge is it's usually when you run into, there's two scenarios. One is when you run into a problem that we have with our API. And there's an obvious easy shortcut that you can take that will solve it for us, but won't solve it for anybody. And you've gotta sell that to management as like, no, we can't take the shortcut because that won't deliver the bigger picture. \[00:18:50\] And then there's the other scenario, the classic one that we keep running into is we'll call an API and they'll return a list of things as an array, adjacent array \[00:19:00\] and \[00:19:01\] We don't ever do that in Microsoft Graph because we follow a set of O data conventions that says everything has to be an object at the root. \[00:19:08\] And even collections are an object with a values property that contains an array. And so we'll keep going and we'll try it against an, oh damn, there's a scenario that doesn't exist in graph. And then we've gotta justify going, spending the time to go fix that. And our, our management are. Great. They have, they have come to terms with the fact that integration scenarios are an absolutely worthwhile thing, and we're starting to gain a lot more traction with other teams, either across Microsoft and across the industry, which again, helps to justify doing it. \[00:19:40\] But yeah, it, it's a non-trivial amount of work and it requires continuous convincing of management that it is worthwhile doing. \[00:19:48\] \*\*Phil:\*\* Hmm. \[00:19:48\] \*\*Vincent Biret-1:\*\* Yeah. And, and I think also another part that we spend time on, which we wouldn't, if we had not open sourced the, the generator. Is the experience of a tool itself, like how \[00:20:00\] to well structure the commands and provide actionable feedback to users using the the generator. We also built a vs code extension, an an integration of the tool in VS code. \[00:20:11\] All of that, of course, is not directly supporting our core business. But it brings broader adoption, it bring of the tool itself, and it brings more people to contribute to the tool. And, and that has largely paid off in case any of our managers are listening to it post ca, post cast. It has paid off already. \[00:20:30\] Like for example, on the Java front red Hat. Has done tremendous work to get to a much better Java story in terms of generation. And, and if we had not open source for generator, if we had not, you know started building a community around, around Kda, we would not have gotten those benefits. \[00:20:48\] Right. So. \[00:20:48\] \*\*Phil:\*\* Ah, that is brilliant and I'm glad you threw that in there just for your managers in case they are listening. Slightly different topic. I mean, I don't think we've talked about Microsoft graph on it before and that that sounds \[00:21:00\] like a massive API that sounds blooming complicated. So is it a, a series of smaller APIs or is it just one massive API with loads of operations? \[00:21:09\] And is there \[00:21:11\] any difference between those two things really, apart from implementation \[00:21:14\] \*\*Darrel Miller:\*\* hundreds of APIs. Produced by completely different organizations at Microsoft that in different languages, on different tech stacks, that all gets sewn together into kind of like a federa, like a GraphQL Federation data, a gateway, but using O data conventions instead. And this is one of our other interesting challenges. \[00:21:40\] We actually Don't natively use open API to describe our APIs. It's ironic. It's like in some of the backends do use open API as they're building it, but then when they come to Microsoft Graph, they have to provide us with what's called A-C-S-D-L description, which is XML based format. And it should see some of the, the \[00:22:00\] more junior engineers as they join Microsoft and we tell 'em, yeah, you've gotta go and create this XML description of your API. \[00:22:07\] \*\*Phil:\*\* Use this language that's like, it sucks and there isn't types and everything's like a Boolean or a string, and just \[00:22:12\] \*\*Darrel Miller:\*\* there's lots of, there are lots of types, but, but, but it's, it, it has its challenges, but we're using that language. We can basically sew the the ap, all of these different APIs together into one coherent, description and \[00:22:26\] provide a single surface area. And the technology problem is not the biggest thing that we solve. \[00:22:33\] It's getting different parts of Microsoft's organization to talk to each other. I. Like how many different companies have you dealt with where you ask them to change your address and then you change your address and then you get a letter from them three weeks later that's going to the wrong address because they have your address stored 74 times. \[00:22:52\] Well, we provide that role as a centralized place that says, no, there should only be one representation of a user's \[00:23:00\] profile photo. There should be only one representation of a user's contact information. And that's, that's kind of the, the business value of Microsoft Graph is it provides that one unified surface area to our customers for doing integrations with that data. \[00:23:16\] \*\*Phil:\*\* That's awesome. So yeah, that's, that's the more interesting part I feel like of API governance that a lot of people don't get round to talking about. 'cause they haven't quite got that far yet. Like when, when people start trying to figure out. What API governance is and how to do it. Then it's mostly just like looking at pull requests and saying, don't do that. \[00:23:34\] But it's, it's usually kind of focused on that's not the right naming and convention, that's not the right security thing. Like I, you know, I've, I've been pushing this for a long time 'cause I build tools that help you do that. But it's kind of getting people started with the idea of it. But the, the more interesting thing has always been trying to automate some of that dumber stuff, some of that more simple stuff. \[00:23:54\] Patterns, standards, conventions, automate as much as you can so that you can then start to focus on \[00:24:00\] more domain modeling where you're like, yeah, don't, there's, there's already a dress over there. How are you gonna keep it in sync? Oh, you're gonna do a two-way sync, not on my watch son. You know, and, and just kind of just bounce in I bad ideas out of the entire ecosystem is, is far more interesting. \[00:24:16\] So \[00:24:16\] \*\*Darrel Miller:\*\* one of the most interesting conversations that we have with teams is when they show a user's name, and the first question we ask is, are you getting that from our central identity location? No. We, we, we keep a copy of it over here and we're like, oh, no, you don't, because people change their names. And it can do serious damage to people if they change their name. And that doesn't immediately get propagated throughout our entire system. \[00:24:47\] And we get some funny looks from teams when we're like, oh, no, no, no. You've gotta go back and change your system. You can't store that. You can cache it for a short period of time, but no, you have to guarantee that you are going to sync that name before, \[00:25:00\] we'll, actually you ship the API. \[00:25:02\] So it, it, it's interesting the areas we get into conversations about in what is, you know, API governance. \[00:25:09\] \*\*Phil:\*\* Mm. Yeah. Awesome. I mean, we had the same, same problem that WeWork where users could live in multiple different systems and there was a, a three-way sync between, you know, two different monoliths that had roughly half the data that everything needed. And like Salesforce, I. That we also had a copy of Bloom and everything, and they all had different validation rules. \[00:25:29\] So when you updated it in one place, it might make it to one or two of the other systems, but it might not . So you just had all this like, crazy different information and they all used like email as a unique key across all the systems, which would change and break everything. So yeah, trying to keep data in sync is. \[00:25:47\] The dumbest part of the, kind of, was the hardest, most annoying and often dumbest part of microservice architecture. 'cause you're like, what if we split up all this code into multiple separate, distinct \[00:26:00\] things that then don't have access to anything and we'll just copy and paste everything everywhere and everything's worse. \[00:26:06\] So you really need those very experienced people that can just sit there and say, yeah, not under my roof. Whenever someone triess to copy stuff, \[00:26:12\] \*\*Darrel Miller:\*\* the response is, but, but, but we have to ship this next week. \[00:26:15\] \*\*Phil:\*\* Yeah, it'll be okay this time. I promise it won't be like every other time. Cool. I mean, what other, what other troubles do you have corralling that many different APIs into one architecture? \[00:26:28\] \*\*Darrel Miller:\*\* I mean, it, it's the reinventing of the wheel, which is no, please don't implement another RAC type security system that's specific to your particular . Product in the corner of the system. I think that's one of the other common things that we get into. I mean, \[00:26:45\] \*\*Phil:\*\* Role-based authentication, \[00:26:48\] \*\*Darrel Miller:\*\* role-based authentication, access control role-based access control. \[00:26:51\] That's what our backs from. \[00:26:53\] \*\*Phil:\*\* There we go, \[00:26:55\] \*\*Darrel Miller:\*\* yeah, I think that's probably the other biggest area. The other \[00:27:00\] area is, is it is continues to surprise me. How a lot of people don't think of resource modeling and they just look at a URL as just a string of characters that has no necessary organizing. It's just a bunch of nice words that you string together. \[00:27:17\] And they don't think of, you know, how are people going to go and use this API beyond the very specific scenario that . We are designing for today and we are shipping for a particular product. It's a case of no, you have to think about that customer who needs to get access to the data and they might want to do something different with the API than you would considered. \[00:27:43\] How can we make that possible without generating a massive amount of work and building an infinitely capable machine that nobody is gonna use all a large part of the stuff that you built. So, I, think it's that just generalized design for serendipitous reuse \[00:28:00\] that's, that's hard for people to grok. \[00:28:01\] \*\*Phil:\*\* Mm. Yeah. 'cause you, you either go the, this is just raw data, do with it what you will, maximum flexibility, no control. Or, or you kind of go like. These are the workflows we have really optimized for. And they're gonna be sick if you wanna do this thing with it, but if you wanna do anything else, it's awful. \[00:28:17\] Yeah, that is the constant slating scale of terribleness. That is \[00:28:20\] \*\*Darrel Miller:\*\* And you need to find some happy medium, somewhere in between. And make sure you also design it in a way that's gonna work well for the client code that we're going to project for you. Because sometimes you can design APIs that are like, oh, that's just not gonna work well in native programming languages. \[00:28:38\] You know, like people who do lots of APIs that return very dynamic responses, or APIs that return raw. Oh yeah. Well, we don't wanna schematize this part of the API. \[00:28:48\] \*\*Phil:\*\* Yeah, yeah. We've got an API that just like one endpoint, just like spits eye calendar format at you . And we've managed to get rid of that 'cause it was a bit gross and weird. But yeah, it's just like none of the tools are okay \[00:29:00\] because they're expecting Jason, maybe X ml. And then all of a sudden it's like, what the fuck is that \[00:29:06\] So yeah, understand that. I mean, bringing it back to Kyoto for a second \[00:29:09\] You just by way of wrapping up as well, could probably get into that time. But obviously you guys are doing such a wonderful job at making Microsoft Graph be very consistent and therefore any You would expect if, if the open API is fairly consistent and follows patterns and the APIs are fairly consistent and follow patterns, then those SDKs are gonna be pretty, you know, standard with each other. \[00:29:35\] But for other. People like the example of a client who is talking to 10 different APIs made by 10 different people, and you've cotified a lot of them. \[00:29:45\] How much does different open API affect the kind of consistency of the SDK? Well, the API client code that is then generated from it. \[00:29:55\] \*\*Vincent Biret-1:\*\* I would say not that much because a, again, a as, as Darryl mentioned at the beginning \[00:30:00\] one one of the key designs aspect is that the only opinionated choices we've made were around HGTP and the conventions and whatnot, and we stuck to as close as H GTT P conventions as. As we could and, and, and for example, if you take the fluent API surface that we project for you to call the different operations and, and points and whatnot. \[00:30:19\] One the way it is structured is that it. follows the path segmentation of your API. 'cause not only it's easier to map mentally to know, oh yeah, this method or this code path on in my code maps to that operation on the API, but also it allows us to. The structure, the code we generate and potentially avoid any conflicts if you have things with different names, but at with the same names, but at different levels and, and, and whatnot. \[00:30:49\] So now if you look at different descriptions and different APIs, even though. Their domains are different, even though the terms they're using are different. \[00:31:00\] The organization of affluent API Surface and the models and the other conventions are gonna be the same. They are going to be mostly based on the HT P conventions. \[00:31:10\] And there is gonna be another layer of conventions added by, by Qda. But it, we try to keep it as thin as possible. So it's, it's super easy for you to mentally map your way through the different clients we generate here. \[00:31:22\] \*\*Phil:\*\* Brilliant. Alright, well I'm gonna recommend that all you listeners go and have a look at kta because you might think you're being clever by just doing a, like a file, get contents and bunging, A URL. And then just like hope that thing works. But every time you, you'd kind of directly just write your own little SDK, you've just built a bad SDK and it's brittle and there's a lot of things that can go wrong and it's gonna be different every time. \[00:31:52\] And you don't have the ability to wrap that in beautiful, fantastic middlewares that can, can chuck in timeout logic consistently, easily, that \[00:32:00\] can chuck in caching logic consistently, easily. So yeah, if you have a bunch of like direct dependencies that you've written yourself. Delete those and maybe Kyoto can be the way to replace them. \[00:32:11\] Thank you very much Vincent and Darryl for coming on and I think, I think this has actually recorded the whole way through, so I think we're okay. I think we did a podcast \[00:32:19\] \*\*Vincent Biret-1:\*\* Hopefully, yeah. Thank you so much for having us. \[00:32:23\] \*\*Phil:\*\* Cheers guys. \[00:32:24\] \*\*Vincent Biret-1:\*\* Bye. ### Battle of the API Ratings: API Insights vs Rate my API URL: https://apisyouwonthate.com/blog/api-insights-vs-rate-my-api/ Last updated: 2024-01-10T10:32:37.000Z There's plenty of tools out there which can warn you about problematic things it notices, but that's rather different from letting you know if an API is good quality. Not triggering any warnings doesn't mean it's great, it just means it's not bad. Thankfully two tools have popped up to take care of that. [API Insights](https://apiinsights.io/?ref=apisyouwonthate.com) by Trebble, and [Rate my API](https://www.ratemyopenapi.com/?ref=apisyouwonthate.com) by Zuplo. When I was helping to build [Spectral](https://github.com/stoplightio/spectral/?ref=apisyouwonthate.com) (and [Speccy](https://github.com/wework/speccy?ref=apisyouwonthate.com) before that) the goal was to build a tool which could help you stick to certain conventions, use certain standards, and avoid pitfalls. This was coming from an API Governance angle, helping people doing API Design Reviews not have to keep checking for the same problems, making sure the right HTTP authentication method was being used, using JSON:API not rando-JSON, etc. It always felt very "don't do that" centric, and what I really wanted to see was something more like the [SSL Test](https://www.ssllabs.com/ssltest/analyze.html?ref=apisyouwonthate.com) by [SSL Labs](https://www.ssllabs.com/?ref=apisyouwonthate.com), which gave you a A-F rating and let you know what you were doing well too. Both API Insights and Rate my OpenAPI aim to do this, and whilst they are very new tools they are both showing a lot of promise. Let's have a play with them and see how they handle the Tree Tracker API a rag-tag crew of volunteers have built to power [Protect Earth](https://www.protect.earth/?ref=apisyouwonthate.com)'s reforestation work. ## API Insights Getting started with API Insights is easy, you can either go to [apiinsights.io](https://apiinsights.io/?ref=apisyouwonthate.com) or grab the [API Insights](https://apps.apple.com/hr/app/api-insights-by-treblle/id6468354004?mt=12&ref=apisyouwonthate.com) app from the Mac Store. The idea is you load an OpenAPI document into the tool, and it will inspect not just the OpenAPI itself, but the actual hosted API listed in the servers array, which helps give feedback on server settings controlled by the implementation that might not live in OpenAPI. At first I had a bit of a fumble trying to get my OpenAPI document into the tool, as I've split it up using loads of [$ref](https://docs.bump.sh/guides/openapi/advanced-ref-usage/?ref=apisyouwonthate.com)'s and it wanted me to provide *one file*. Thankfully [Redocly CLI helps bundle OpenAPI documents](https://apisyouwonthate.com/blog/openapi-bundling-tools-compared-blog-post/) into a single YAML/JSON file and then it's easy to shove that into either the web app or the Mac app. ``` $ redocly bundle openapi.yaml -o openapi.bundled.yaml ``` ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2023/12/Screenshot-2023-11-26-at-8.17.57-PM.png) With document loaded in, I just needed to hit "Get Insights" and a few moments later I had my API score. ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2023/12/api-insights-not-great.png) Oh dear, I built an API that this tool hates! ### What does API Insight score your API on? There are three sections: Design, Performance, Security. I'll start with security, because its the least controversial. ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2023/12/api-insights-security.png) These are all the things that I always forget to do with an API, that should probably be enabled by more Web Application Frameworks by default, especially the ones specifically aiming to be API-centric, or have an API mode. How much do they matter? I'm not sure, the tool doesn't educate my on why I need to do any of this, which could be as simple as linking to MDN to explain things like why I need the [Content-Security-Policy](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Security-Policy?ref=apisyouwonthate.com) header. There are definitely a lot of people who will want to know why they need to do things, especially if it involves a breaking change to their API, but just as equally there are plenty of people who will dive in and start doing this stuff anyway. I found myself in the "rushing to do it regardless" camp because I want the high score, and this is a large part of why API Scoring can help in an API Governance program, because however much you'd like to deny it, people are motivated by shiny stickers. Some of the checks will need tweaking, because they are pushing you towards slightly outdated practices. For example it recommends the `x-frame-options` header, which has two issues. 1. `x-` headers should [never be used for anything](https://www.mnot.net/blog/2009/02/18/x-?ref=apisyouwonthate.com) and are being [deprecated with a vengeance across](https://datatracker.ietf.org/doc/html/rfc6648?ref=apisyouwonthate.com) the world of HTTP. 2. The `Content-Security-Policy` HTTP header has a frame-ancestors directive which replaces `x-frame-options`. I mentioned this to the team and they're going to see how they implement support for both. It could be a case of giving you an A for using the modern approach, a B for using the older approach, and a F for using nothing. Either way it's good to get people thinking about these headers. Another check was Strict Transport Security (HSTS), and it's not entirely clear where API Insights is expecting to see those headers. From the description of the check I think they are looking to see it in my actual OpenAPI document, like this: ``` responses: '200': description: successful operation headers: Strict-Transport-Security: schema: type: string example: max-age=31536000; includeSubDomains ``` I gotta be honest, I'm never going to do that. Not even if I make it a shared reference and $ref it into every response, I just... would never ever bother for any reason. not only does this feel far to much like a infrastructure implementation detail to be in OpenAPI, but if I pop this header into the OpenAPI it's going to confuse my [OpenAPI-based contract testing](https://apisyouwonthate.com/blog/contract-testing-apis-laravel-php-openapi/) which is now expecting a header that hasn't appeared. I'd need to move it into my source code instead of letting the server (nginx, Cloudflare, etc) add it... which... maybe I should? Thoughts in the comments please. Performance has a great list of checks. - Cache Support - CDN Usage - Compression Support - Load Time - Response size - HTTP/2 Usage Thanks to using Cloudflare it looks like I got a great score without having to do anything. ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2023/12/api-insights-performance.png) I always forget to add `content-encoding` to APIs I build, and stupidly just fire full-fat JSON around worsening the 4% of global CO2eq emissions caused by the internet. I should be using gzip or br to squash my response sizes! Thank you for reminding me API Insights. Learn more about [content encoding in APIs from HTTP Toolkit](https://httptoolkit.com/blog/http-encodings/?ref=apisyouwonthate.com), and google around until you find some sort of middleware or copy/paste code on Stack Overflow. The Design section is a bit more controversial to me than these other checks. ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2023/12/api-insight-design.png) - Consistent Noun Usage - Consistent Pluralization - JSON Support - Multiple HTTP Methods - Rate Limiting - Robust Responses - Versioning It's really hard for a tool like this to try and make a single set of checks for everyone making any type of API, and API design has a lot of different opinions floating around. API Insights works by defining one set of quality metrics and doesn't have "rulesets" like other API linters, so tool needs sensible defaults and I'm not sure they've got them all right just yet. "Consistent Noun Usage" wants more than just GET methods to be used, but if you have a read-only API you will only have GET methods. I do not think a read-only API is a bad one, but this will ding your score. A more advanced check could look for GET or POST methods with words like "delete" in the URL, summary, or operationId, and ding your score for dangerously misusing HTTP. "Consistent Pluralization" means pluralizing your `/widget` to `/widgets`, which is not really a requirement for REST or even necessarily good practice. You might have a `/me` endpoint which definitely does not need to be `/mes`. Not sure what to suggest here. The Rate limit is a good rule which looks at your OpenAPI to see if Rate Limiting is defined. Again it currently focuses on an outdated non-standard approach, requiring `x-ratelimit-*` which is a convention from Twitter and Vimeo. There's a draft web standard that attempts to standardize this with [IETF Draft HTTP RateLimit Headers](https://datatracker.ietf.org/doc/draft-ietf-httpapi-ratelimit-headers?ref=apisyouwonthate.com). Adding equal or higher score for using those headers would be a good improvement. Finally, API versioning... this one is always going to be hell to try and write any one rule for, because [API Versioning Has No "Right Way"](https://apisyouwonthate.com/blog/api-versioning-has-no-right-way/). The API I am using switched from API versioning to API Evolution (we literally removed the /v1) and so far all updates have happened in a non-breaking way thanks to changing /trees to /units with a type. API Evolution is a solid alternative to API versioning, but admittedly it's a big change for a lot of people, and API Versioning is still the predominant strategy. Whichever you prefer, should this tool tell me I am wrong for using API Evolution and prompt me to start API versioning again? Thoughts in the comments please. ### API Insights Summary API Insghts is a little different from the current field of API linters (Spectral, Redocly CLI, Optic) which focus on what they can see in the OpenAPI document, as it will go over the wire via HTTP to see what your API is up to too. Requiring an API implementation to exist means this will likely not be a tool you use during the API Design phase, but something you use to see how your current API is doing, and give you direction for improvements to some things to consider for the next version. Whether these checks are looking at OpenAPI or going over the wire, I think an "Explain" section would be super handy in the interface, which shows you what the tool expected, what it found (or didn't find), what it would prefer I do about it, and *why*. Together this Explain section would be a powerful tool for helping developers make improvements, like Google Lighthouse for APIs. Finally, the API versioning check raises the question: should API rating tools bases their scores off of what is currently popular, or should it be more prescriptive trying to push people towards API design deemed "better". If so, who gets to decide "better"? It's not an easy line to walk but the Trebble team are only just getting started and I am excited to see how this tool evolves. ## Rate my OpenAPI Zuplo's [Rate my OpenAPI](https://ratemyopenapi.com/?ref=apisyouwonthate.com) tool aims to do the same sort of thiing as API Insights, and wins a point for being the first released. Just like API Insights it supports a single OpenAPI document as a file upload which will need to be bundled if you split your document into multiple components. Unlike API Insights there is no support for pointing to OpenAPI hosted on a URL, and there's no Mac app if that's something you find important. Once you've uploaded your OpenAPI document it will ask for an email address, then it will email you the report. This only takes a minute to come through. You will receive a score out of 100 like API Insights, but there is no A-F rating. In API Insights my API got a F (55/100), so how did this score in Rate my OpenAPI? ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2023/12/rate-my-api-score.png) Straight away I'm getting advice on specifically what I need to do. > **Summary** > > Well, well, well, looks like we've got some issues with your OpenAPI file. Brace yourself, because here's the lowdown in just 2 lines: You've got security and server problems, along with some missing error responses and rate limit issues. Oh, and let's not forget the unused components and missing descriptions. Time to roll up those sleeves and get to work! The summary is interesting because it seems a bit more conversational, and it's pointing me to exactly which line number my first issue is on. > Alright, let's dive into the summary of issues in your OpenAPI file. Brace yourself, because we're about to embark on a snarky adventure! > > Issue 1: apimatic-security-defined (1 occurrence) Oh dear, it seems like your API is missing some security definitions. You wouldn't want just anyone waltzing in and wreaking havoc, would you? Time to tighten up your security game! > > Issue 2: oas3-server-trailing-slash (1 occurrence) Ah, the notorious trailing slash issue! It appears that your server URL has a sneaky little slash at the end. While it may seem harmless, it can cause confusion and lead to unnecessary redirects. Let's trim that slash and keep things tidy! > > Issue 3: owasp:api3:2019-define-error-responses-500 (22 occurrences) Well, well, well, looks like your API is throwing some unhandled 500 errors. It's like a party for internal server errors, and nobody wants to attend that party! Time to define some error responses and give those 500 errors a proper send-off. > > Alright, those are the top three highest severity issues from your OpenAPI file. Fixing these will give your API a solid foundation to build upon. Remember, snarky tone aside, we're here to help you create a kick-ass API! This made me chuckle, because it reminded me a lot of Speccy, the original OpenAPI linter which I described as an automated #WellActually tool, nagging you about doing things wrong. The names of the "issues" (e.g.: `oas3-server-trailing-slash` and `owasp:api3:2019-define-error-responses-500`) look pretty familiar, they match the names of two "rules" in the Spectral default [openapi ruleset](https://docs.stoplight.io/docs/spectral/4dec24461f3af-open-api-rules?ref=apisyouwonthate.com), and the [spectral-owasp ruleset](https://github.com/stoplightio/spectral-owasp-ruleset?ref=apisyouwonthate.com). Is Rate my OpenAPI a wrapper around Spectral? I had to know, and thankfully finding out was easy. Sure I could have asked the team at Zuplo, but I didn't need to as their implementation is [open-source on Github](https://github.com/zuplo/rate-my-openapi/?ref=apisyouwonthate.com) and after a quick bimble around their source code I spotted some chunks of Spectral are being pulled into the app. ``` "@stoplight/spectral-core": "^1.18.3", "@stoplight/spectral-parsers": "^1.0.3", "@stoplight/spectral-ruleset-bundler": "^1.5.2", ``` There's also a few references to Vaccum, a Go-based OpenAPI tool which is compatible with Spectral rulesets. Seeing these tools in there helps me understand the intentions of the tool a little more. Clearly from the name "Rate my OpenAPI" we could already tell the tool is more focused on reporting issues it can see in OpenAPI documents than it is about hitting the API over HTTP to score the implementation too, but seeing Spectral & Vacuum in there confirms that. Of course, layers could be added on top to hit the actual API, but that might be going against the intention of the name. ### What does Rate my OpenAPI score your API on? The overall rating is broken into four categories which each have a score. - Documentation - Completeness - SDK Generation - Security First things first, documentation. ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2023/12/rate-my-openapi-documentation.png) At first I thought this was the dreaded "you should use tags" rule which *nobody* ever liked in Spectral, but thankfully not. This is correctly pointing out that I have referenced a tag in an operation which I did not define in the global tags array. Oops! How did I know that was the case? Well, Rate my OpenAPI told me. I can click on the 📄 icon next to the issue, and it will show me where it found it in my OpenAPI document. ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2023/12/rate-my-openapi-context.png) Well, ok it didn't tell me exactly, it's more like it pointed to where the problem occurred, and then they have an Additional Details link which sends you off to the corresponding rule documentation in the Vacuum documentation, and *that* told me. ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2023/12/vacuum-rule-docs.png) This is really helpful, and most of what I would have liked API Insights to do with an "Explain" section. Rate my OpenAPI could do a little more within the app and say something like "We expected to see Home defined as a tag in the global `tags:` keyword." to save you a click, but this does get you there. No complaints about these rules, and Completeness also didn't seem to have any problematic issues either... possibly because I use Spectral locally and had these rulesets enabled. 😅 ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2023/12/rate-my-openapi-completeness.png) Next! SDK Generation caught my eye. ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2023/12/rate-my-openapi-sdk-generation.png) I'm not entirely sure what these checks have to do with SDK generation. The more information screen showed this: > Apimatic: It is recommended to define at least one authentication scheme globally using the securitySchemes property in the components section and use it globally to authenticate all the endpoints or use it for specific endpoints. Oh hey Apimatic! We talked about their [SDK generation tooling](https://apisyouwonthate.com/blog/apimatic-review-dx-as-a-service/) recently, but I wasn't expecting to see them pop up. It turns out Rate my OpenAPI utilized [recommendations from Apimatic](https://www.apimatic.io/blog/2022/11/14-best-practices-to-write-openapi-for-better-api-consumption/?ref=apisyouwonthate.com), and they have a rule that says: > **At Least One Security Scheme** > > In order to protect itself from outside cyberattacks, an API should define an authentication scheme and should authenticate endpoint requests made by end users. For OpenAPI specification, it is recommended to define at least one authentication scheme globally using the securitySchemes property in the components section and use it globally to authenticate all the endpoints or use it for specific endpoints. This seems a bit out of place. Perhaps the Security category should be giving me feedback about security, but SDK generation doesn't care. If I was feeding this OpenAPI into Apimatic to generate SDKs it would care, as from memory I think they assume you'll have some sort of security defined globally. This feels a bit disconnected and end-users of Rate my OpenAPI would only ever get this context by digging through the source code, and finding the link to that blog post in the documentation, which is probably only ever going to be me doing that. Whether this rule belongs here or not, this is another vote for an Explain section which really explains **why**, instead of just what. > Potentially unused component has been detected This rule makes a lot of sense for SDK generation. Why build code that's not actually going to be used? That's going to confuse and pollute the code documentation and have people trying to call classes and methods that don't do anything. Trim the fat, remove that OpenAPI, and keep the SDKs easy to work with. ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2023/12/rate-my-openapi-security.png) Here we go, this feels familiar. It's all based on the OWASP ruleset Stoplight had me put together before I vanished off into the woods forever. > WARN: 'security' must be defined and not empty Heh, yeah you want to define some authorization and authentication for your API or it's going to be wide open to everyone. Sometimes you want this, and it might be a bit sad to eat the lower score for an API that is legitimately a completely public read-only API, but these tools can't do everything. > WARN: Operation is missing responses\[401\]. warn WARN: Operation is missing responses\[401\].content. Absolutely to the first, I should be defining 401 errors in my OpenAPI and to be quite frank I have been lazy in not bothering. This is a bit redundant though, because it's telling me I have not defined how the content looks in the 401 response that I have not defined, so... we only need the first error to show up in this instance. > INFO: All 2XX and 4XX responses should define rate limiting headers. This rule showed up in API Insights too and dinged my score saying I must have rate limiting headers set up. Rate my OpenAPI classifies it as an info and to be honest I'm not sure if that's effecting my rating or not. It feels a bit less like a demand, which you might prefer depending on how much you care about rate limiting, and whether its defined by your API or your API Gateway. ## Which Tool is "The Best"? Well I don't think I can pick one, and not because both of these tools sponsor APIs You Won't Hate. 🤣 They are fundamentally different. API Insights sets out to rate your OpenAPI and your API implementation; in fact that abstract the difference between the two things, but they have far fewer rules and some of them may be a bit too opinionated for you. Perhaps I am too opinionated, but it definitely gave me some things to think about and the focus on pointing me towards adding security headers I had entirely forgotten to add in production was massively helpful. Rate my OpenAPI focuses on rating your OpenAPI, and does that using a lot of very standard rules like the OWASP ruleset for Spectral. Of course I like that, I wrote it, but those rules are all based on the OWASP API Security project, and they are all things you should do. It does not mention security headers in the API implementation because it isn't looking there, but that isn't the goal. Give them both a spin and see what you think. Please post your thoughts in the comments, because both of these tools are being actively developed and can only get better with feedback from people like you. ### Low Code & No Code API Solutions - Are they Right for You? URL: https://apisyouwonthate.com/blog/no-code-low-code-api-solutions/ Last updated: 2024-01-16T11:44:12.000Z Recently I introduced the idea of using [low/no code solutions to avoid designing/building APIs](https://apisyouwonthate.com/blog/low-code-no-code-and-automation-for-apis-gadget-or-godsend/), highlighting the practicality and trade-offs of these approaches. This article is going to dig into that concept a little more, and explore the different categories and showing off some of the tooling available. API developers are a funny category. Are you a standard software developer who sometimes build monolithic backend applications which might spit out HTML or JSON? Are you a Full Stack developer who builds an API to communicate data from your database to your frontend? Are you a DevOps person who dabbles with a bit of code. Your API needs to live somewhere, and it has to be built somehow, but those different groups all have different needs and motivations. The full-stack devs might not want to spend ages building that backend, and the software developers might not want to get stuck into infrastructure setup. If you're anything like me you'll get a migraine just thinking about spinning Amazon Web Services up, and if I don't have a team with me for a project I'm probably looking for alternatives. What options are there apart from "do everything yourself"? No code and low code are two brilliant emerging categories which are helping API teams, taking over various amounts of the infra/config/functionality, and letting you either click your way to a functioning application, or write a little bit of code to help out without needing to write *everything*. This is a long article, so feel free to hop about to the bit you're most interested in. - [No Code APIs](#no-code-apis) - [No Code Headless CMSs](#no-code-headless-cmss) - [Do You Need Database Access to a Headless CMS?](#do-you-need-database-access-to-a-headless-cms) - [No Code Application Platforms](#no-code-application-platforms) - [Low Code APIs](#low-code) - [AWS Amplify](#aws-amplify) - [DreamFactory](#dreamfactory) - [Google Firebase](#google-firebase) - [Honorable mentions](#honorable-mentions) - [Automations](#automations) - [Summary](#summary) ## No Code APIs There's a few different types of No Code API, from the simple headless content management system, to "Mega Spreadsheets" like Airtable which do absolutely everything you could ever think of. ### No Code Headless CMSs Content Management Systems used to be the king of the Internet. They still power 64% of the internet, but it felt like everything outside of major corporations started by installing a CMS. In many ways that was a much simpler time. You would install a CMS on your server, and it would give you Blogs and Pages. Eventually everything because a Page or Entity with custom data making them even more powerful. It was only a matter of time until these CMSs started adding HTTP/REST APIs for that custom data so you could do whatever you wanted with the data. Users would disable the default server-side generated frontends, and build entirely custom interactive frontends with JavaScript. This happened enough that some CMSs ditched their server-side frontend altogether, and exposed data purely over the API. Content management systems that focus on providing an API are known as "Headless CMSs" and they come in all shapes and sizes, with some supporting REST, some supporting GraphQL, and some support both. Let's take a look at some no code headless CMSs that you can use to create an API and backend with minimal setup. #### Strapi [Strapi](https://strapi.io/?ref=apisyouwonthate.com) is one of the most popular headless CMS around. This beautiful CMS comes in [open-source](https://strapi.io/features?ref=apisyouwonthate.com) or [enterprise](https://strapi.io/enterprise?ref=apisyouwonthate.com) flavours. You can host it yourself, or host it on [Strapi Cloud](https://strapi.io/cloud?ref=apisyouwonthate.com) if you really don't want to handle the hassle of hosting/configuring the code, database, email provider, and CDN. Everything is a Content Type. Build collections and single types. Pick all the fields, and it will build everything up for you. Unlike some similar systems, instead of everything being purely configured by the database, it will build a bunch of code for you in the filesystem, giving you the opportunity to edit and customize that if you like. This could put Strapi in the low-code camp, but I've left it here as it's optional. ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2024/01/strapi.png) - **API:** REST & GraphQL - **Webhooks:** ✅ - **Self hosted:** ✅ - **Cloud hosted:** ✅ - **Database Support:** PostgreSQL, MongoDB, SQLite, MySQL & MariaDB #### Contentful The focus for [Contentful](https://www.contentful.com/?ref=apisyouwonthate.com) is clearly completely on developing your content, and not having to worry about where it's going or what it's doing. They have developed infinite integrations and extensions, so you can link it up to popular web frameworks to build out a frontend with minimal effort. If that's no good you can use their [Content Delivery API](https://www.contentful.com/developers/docs/references/content-delivery-api/?ref=apisyouwonthate.com) or [GraphQL Content API](https://www.contentful.com/developers/docs/references/graphql/?ref=apisyouwonthate.com) to build your own integration, which is especially helpful if that content needs to go somewhere more complicated than onto something like a Gatsby site. ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2024/01/contentful.png) Define content types, create entries, assign tags and relationships, and build up everything you need without having to write any code. - **API:** REST & GraphQL - **Webhooks:** ✅ - **Self hosted:** ❌ - **Cloud hosted:** ✅ - **Database Support:** ❌ *(It's cloud hosted and they do not permit connections to their database)* #### Sanity [Sanity](https://www.sanity.io/?ref=apisyouwonthate.com) is apparently not a headless CMS, it's a "Composable Content Cloud", which walks and quacks a lot lot a headless CMS. Whatever, people come up with fancy marketing terms all the time. When I was building a headless CMS there was a whole phase of calling them Content Management Frameworks, then the even fancier people called it a Content Management Platform... it's all the same thing. ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2024/01/sanity.png) Use the open-source [Sanity Studio](https://www.sanity.io/studio?ref=apisyouwonthate.com) tool to structure your content, then all the data lives in a [Content Lake](https://www.sanity.io/content-lake?ref=apisyouwonthate.com) which is accessible via [various types of API](https://www.sanity.io/docs/reference?ref=apisyouwonthate.com). It's got loads of CLI helpers and image management logic, asset libraries, etc. and can take care of a *lot* of boring nonsense for you. - **API:** REST, GraphQL, and [GROQ](https://www.sanity.io/docs/groq?ref=apisyouwonthate.com) - **Webhooks:** ✅ - **Self hosted:** ❌ *(you can host the studio but not the API/data)* - **Cloud hosted:** ✅ - **Database Support:** ❌ ### Do You Need Database Access to a Headless CMS? Most cloud hosted headless CMSs will not let you access the database directly, and will have you read from the content API, and manage content through the interface or some other content management API. Self hosted headless CMSs will need you to host the database yourself, so you do get access, but the list of database engines they support might not include your favourite. Even if your database platform of choice is included, the CMS will tightly control how that data is structured. This might be absolutely fine, for two reasons: 1. Maybe you don't care how your data is stored. You aren't a database admin, and don't want to have to think about the pros and cons of normalizing to the nth degree. You can assume this CMS has probably set it up to work in a sensible way, and deferring control to them is one less headache. 2. Maybe you can just connect to the DB directly to do particularly complicated things like sweeping up loads of data for crunching stats, data warehousing, syncing to another system, or building a partner API later. So long as the API the CMS produces is usable (or you can configure it to be useable), a headless CMS can get you a long way with minimum effort, especially if it is just making some CRUD content for use elsewhere. ### No Code Application Platforms Another approach to building no code APIs is to use a hosted solution like [Airtable](https://airtable.com/?ref=apisyouwonthate.com). In the [previous article in the no/low code series](https://apisyouwonthate.com/blog/low-code-no-code-and-automation-for-apis-gadget-or-godsend/) I wrote more about how [Protect Earth](https://www.protect.earth/?ref=apisyouwonthate.com) used Airtable to build our "Tree Tracker", saving us a huge amount of time, and money we did not have. ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2024/01/airtable.jpg) Airtable can handle creation of tables, which have all the common fields you would expect: string, dates, numbers, ranges, emails, images, relationships, etc. Then with no code you can configure views, like grid view for images, table views for data, and user facing views allowing you to generate data entry forms to gather more data. This is really helpful, but fundamentally Airtable positions itself as an "application builder", that just so happens to have an API. The [Airtable API](https://airtable.com/developers/web/api/introduction?ref=apisyouwonthate.com) is not particularly beautiful, but it let our iOS developer upload trees planted from the field, and upload images easily. We needed a little bit more flexibility on the JSON output and there was no ability for us to customize it, but as with most of these APIs you can simply code a thin proxy API which pulls in data from the main API, then smushes the JSON around a bit, and puts it out in whatever format you like. It's entirely cloud hosted and there's no access to the data other than through the API, so complex queries are not possible. You'll need to sync the data down, which is cumbersome and frustrating, then query that data, making it no code for some, but code heavy for more advanced use cases, and may only be a matter of time until you switch away. Maybe that's ok, but you'll need to be thinking of an exit strategy when you start using it or you might get stuck. - **API:** REST only - **Webhooks:** ✅ - **Self hosted:** ❌ - **Cloud hosted:** ✅ - **Database Support:** ❌ There are other cloud platforms like Notion (which I use extensively for project management, knowledge bases, etc but it's not quite so good at this sort of use case). If you'd prefer a self-hosted open-source Airtable clones, there are loads: - [Baserow](https://baserow.io/?ref=apisyouwonthate.com) - [Grist](https://support.getgrist.com/?ref=apisyouwonthate.com) - [NocoDB](https://www.nocodb.com/?ref=apisyouwonthate.com) Some folks will love these sorts of tools, but the more techie you are the more likely you are to want some code involved, which takes us onto the next category: Low Code. ## Low Code APIs What about low-code? There are a few low code headless CMSs around which provide all the functionality of the no code headless CMSs but with a bit more configuration required. We've talked enough about this type of tool so I'll just list a few off you should check out: - [CraftCMS](https://craftcms.com/?ref=apisyouwonthate.com) is a popular PHP-based headless CMS that you can self-host and extend, that lets you get a whole load of work done without writing any code. It offers GraphQL out of the box, but does not have REST out of the box. - [Payload CMS](https://cms.payloadcms.com/?ref=apisyouwonthate.com) \- A free and open-source dev-first TypeScript headless CMS. Where the low code API world really gets interesting is a whole category of tools that are an interesting balance between application builder and configurable API-centric infrastructure. ### AWS Amplify Most software engineers will have come across Amazon Web Services at some point, and it can be a huge headache to work with. People will shout "it's simple!" but they're people who have spent enough time grinding through it all to become an infrastructure person, and everyone else will shuffle around making expensive mistakes and misconfiguring security holes in everything. AWS Amplify aims to rectify that situation, helping beginners build full-stack apps, and mobile apps, but for this audience the most interesting part is the ability to create scalable and secure APIs. Using a GUI you can set up databases, file storage, authentication, authorization, and continuous deployments. This eliminates the need for manual configuration and reduces the risk of security vulnerabilities, ensuring that the APIs are robust and reliable. Amplify also offers a wide range of prebuilt components and integrations, making it easier for developers to add functionality to their APIs. These components include features like real-time data synchronization, user management, file storage, and more, most of it being powered by extensible services like Amazon S3, AuraDB, etc. under the hood. By leveraging these prebuilt components, developers can save time and effort, as they don't have to build all this out from scratch, or glue a bunch of stuff together. The API logic in Amplify is powered by another AWS service called AppSync, which helps you spin up APIs with a single command. ``` amplify init amplify add api ``` From there you just [define some GraphQL Types](https://docs.amplify.aws/react/build-a-backend/graphqlapi/data-modeling/?ref=apisyouwonthate.com) and you've got a [GraphQL API](https://docs.amplify.aws/react/build-a-backend/graphqlapi/?ref=apisyouwonthate.com). If you want REST then you're out of luck, but if this is only being used by your frontend or some internal clients that might not be an issue. ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2024/01/aws-appsync.png) Unlike tools like Airtable, you've got a much better escape hatch here, because you could blow away all of the frontend logic and API logic, and all of your data is still sat in a perfectly useful hosted database that you could connect to from any other server where your new API runs. ### DreamFactory I really like the idea of [DreamFactory](https://dreamfactory.com/?ref=apisyouwonthate.com), which takes the opposite approach from most of the other tools we've been talking about. Instead of helping you define a bunch of content types, making you a database and helping you hook it all up together with content types and modeling done just to create the database structure, DreamFactory does the opposite. DreamFactory is a hosted API platform that makes the assumption you've already got a database full of data. When you sign up you point it are your database, then you can expose that via a REST API to another team as an internal/private API, without needing to grow an entire API/infrastructure team to be responsible for all the complexity of designing and building an API, handling authentication, authorization, deployments, setting up an API gateway, email, push notifications, and even generates API documentation. You just get all of that out of the box. ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2024/01/dreamfactory.png) DreamFactory sits in the low code camp, because whilst it does all this for you, it's not just a dumb data-in/data-out system, it lets you run attach Javascript, PHP, and Python, to handle whatever business logic or other transportation you need. This sets it apart from other "database as an API" tools and platforms, because vomiting out an entire database as an API is rarely useful. You want to control what data is output, and how, and evolve that over time, which DreamFactory will let you do with code. DreamFactory also easily has the [largest database support](https://www.dreamfactory.com/connectors?ref=apisyouwonthate.com) out of any of the tools we've talked about so far. You can hook it up to MySQL, SQL Server, Oracle, PostgreSQL, MongoDB, and IBM DB2, SQLite, memSQL, Snowflake, CosmosDB, CouchDB, DynamoDB, Cassandra, RedShift, Firebird, SAP SQL Anywhere, Alloy DB, Maria DB, Azure DocumentDB, and some other databases I've never even heard of. Setting it up was incredibly easy, and with a few clicks I had connected to a test database, had an API key, and was making calls to my data via curl. ``` $ curl -X GET "https://greenturtle.apps.dreamfactory.com/api/v2/phils-test-db/_table/departments/" -H "accept: application/json" -H "X-DreamFactory-Api-Key: " | jq . { "resource": [ { "dept_no": "d009", "dept_name": "Customer Service" }, { "dept_no": "d005", "dept_name": "Development" }, { "dept_no": "d002", "dept_name": "Finance" }, { "dept_no": "d003", "dept_name": "Human Resources" }, ``` I'm pretty sure with a few more clicks I could have pointed this up to the Protect Earth database hosted on AWS with a SSH Jumpbox, but I'm over 2,000 words in and there's one more tool to talk about. ### Google Firebase Google Firebase is another powerful platform that lets developers focus on building the core functionality of their APIs, while leaving the infrastructure and backend management to the platform. One of the key features is Firebase Cloud Functions, which allows developers to write serverless functions that can be triggered by events or HTTP requests, eliminating the need to set up and manage traditional server infrastructure. Those functions can then be secured with [Firebase Authentication](https://firebase.google.com/docs/auth/?ref=apisyouwonthate.com), with a variety of authentication methods, including email/password, social media logins, OAuth 2.0, and OpenID Connect. Seeing as this is another "Toolbox that can do a million things" it can initially be a bit confusing as to how and what you're meant to do, but [Geekflare](https://geekflare.com/?ref=apisyouwonthate.com) have written an excellent tutorial that takes you through [building an API with Google Firebase](https://geekflare.com/build-api-with-firebase/?ref=apisyouwonthate.com). The API you build can be [REST](https://firebase.google.com/docs/reference/rest/database/?ref=apisyouwonthate.com), but if you want GraphQL you'll have to [build a wrapper](https://developer.ibm.com/articles/awb-use-graphql-firebase-realtime-database/?ref=apisyouwonthate.com). For database storage you've got [Firebase Realtime Database](https://firebase.google.com/docs/database/?ref=apisyouwonthate.com) and [Cloud Firestore](https://firebase.google.com/docs/firestore?ref=apisyouwonthate.com), two NoSQL databases that provide real-time synchronization and offline capabilities. This makes you a bit more tied to Firebase for your data storage, which is less flexible than DreamFactory, and is slightly less helpful than AWS which has hosted MySQL databases, but it's a good enough escape hatch if you compare it to the cloud hosted headless CMSs or platforms like Airtable. ## Honorable mentions Tools which would have been mentioned in here if I wasn't running out of time: - [App Write](https://appwrite.io/?ref=apisyouwonthate.com) - [API Platform](https://api-platform.com/?ref=apisyouwonthate.com) - [Hasura](https://hasura.io/?ref=apisyouwonthate.com) - [Supabase](https://supabase.com/?ref=apisyouwonthate.com) - [WunderGraph](https://wundergraph.com/?ref=apisyouwonthate.com) ## Automations We've not had a huge amount of space to talk about Automations, but all of these tools come with a whole load of extension points. You can either write webhooks to trigger events on other systems via HTTP calls, or link them up with [Zapier](https://zapier.com/?ref=apisyouwonthate.com), [Pipedream](https://pipedream.com/?ref=apisyouwonthate.com), [Make](https://www.make.com/?ref=apisyouwonthate.com), or other similar tools, so you don't need to worry about not being able to control the code entirely, just glue stuff together. ## Summary There will always be the developers who want to build literally everything completely from scratch, but thankfully those developers aren't usually the ones making business decisions, because nothing would ever get done. For public APIs being presented to end users, you might want to have full control over everything, and that makes perfect sense. Headless CMSs have made breaking changes to their API layers in new versions of the API, and you don't want that sort of implementation detail being leaked out to important client integrations. For internal and private APIs there will likely be very different requirements. Sometimes your team want to rapidly iterate on an idea without a huge amount of funding and an infinite amount of time to work on it. Maybe you just need to get data from one place to another, another department, a partner who needs rapid integration, or want to validate an idea before massively investing in it. API Design First is still absolutely amazing, and seeing as some of these tools support OpenAPI you can actually combine the approaches. Low code and no code are not the opposite of design first, they are the opposite or writing all the code yourself. You can use the design first mindset to plan an amazing API, then build that API with these tools instead of writing all the code. Ideally you can be familiar enough with these various approaches that you can recommend the most appropriate one, especially when it seems like your team is about to rush into something that is going to cause problems. If you pizza when you wanna french fry, you're gonna have a bad time. ### Utilize AI for API Governance with LintGPT URL: https://apisyouwonthate.com/blog/ai-powered-api-linting-with-optic/ Last updated: 2024-01-09T10:31:29.000Z API Linting has exploded in popularity, powering an increasing chunk of API Governance programs at organizations large and small. Instead of wasting infinite human time checking API descriptions like OpenAPI, AsyncAPI, gRPC, etc. for mistakes, robots have been deployed to take care of the tedious part themselves, leaving humans to reason about larger more important things. [Automated Style Guides for REST, GraphQL and gRPCAsk 100 developers where a semicolon should go, and you’ll either get 100 answers, or a all-on-all fist fight. To save this from happening at work, most folks implement a style guide, which beyond helping with consistent style to avoid new developers getting shouted at for “doing it wrong”. Linters![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/size/w256h256/2023/03/apis-logo-square.png)APIs You Won't HatePhil Sturgeon![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/size/w1200/posts/rules-regulations.jpeg)](https://apisyouwonthate.com/blog/automated-style-guides-for-rest-graphql-grpc/) Spectral was a huge chunk of getting us to this point in the OpenAPI community, but the way you're required to build rules is really complicated, with every single rule needing arcane scribbles of [JSONPath-Plus](https://jsonpath-plus.github.io/?ref=apisyouwonthate.com)\-with-changes and possibly a bunch of regex to get the job done. ``` "owasp:api4:2019-rate-limit": { message: "All 2XX and 4XX responses should define rate limiting headers.", given: "$.paths[*]..responses[?(@property.match(/^(2|4)/))]", then: { field: "headers", function: schema, functionOptions: { schema: { type: "object", oneOf: [ { required: ["RateLimit-Limit", "RateLimit-Reset"], }, ``` [Redocly CLI](https://redocly.com/redocly-cli/?ref=apisyouwonthate.com) attempts to make rule creation easier by using "identifiers" and an improved Domain Specific Language (DSL) which is a lot nicer to work with, but fundamentally is a very similar approach. [Meet Redocly CLI: The Modern OpenAPI SidekickRedocly CLI is a brilliant new tool from the folks who made ReDoc, the first beautiful API reference documentation tool powered by OpenAPI. This CLI tool goes a lot further than documentation, and helps with “linting” (automated API Style Guides), and solves the biggest problem that I had previously been![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/size/w256h256/2023/03/apis-logo-square.png)APIs You Won't HatePhil Sturgeon![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2023/11/meet-redocly-cli.jpg)](https://apisyouwonthate.com/blog/redocly-cli/) An interesting third tool has popped up with an entirely new approach: AI-powered Style Guides. I know, I know, there's a lot of nonsense floating around in the AI world right now, and I can barely stop my eyes rolling when somebody starts another "AI is going to change your life" tech talk, but this one is different. Optic has rolled out a new tool currently in beta: [LintGPT](https://www.useoptic.com/docs/style-guides/lintgpt?ref=apisyouwonthate.com). I've reviewed tools made by Optic before, and they've definitely produced the best modern [OpenAPI Learning tool](https://apisyouwonthate.com/blog/turn-http-traffic-into-openapi-with-optic/), so I figure they're probably not just phoning it in and playing buzzword bingo. Let's find out. ## How does LintGPT work? Instead of writing rulesets in a complicated DSL, you define an array of ChatGPT prompts, which it will interpret and effectively create the rules for you. ``` ruleset: - lintgpt: v3_standards: rules: - "GET requests that have an ID at the end of their path (ie /teams/{teamID}) should define a 404." - 'POST requests should return a 201 or 202 status on success instead of 200.' - "All operation URL path components MUST be nouns, not verbs." - "Properties should have an example" - "Properties that sounds like they are dates should use format: datetime or format: timestamp" ``` At first glance you might be thinking "what's the point", and I'll admit I missed the real power here at first. This is not just avoiding writing a DSL, this helps us write far more powerful rules that would be pushing the limits of what Spectral could handle (or entirely beyond them). For example, asking the AI if we think a particular property is a date field: ``` - "Properties that sounds like they are dates should use format: datetime or format: timestamp" ``` Doing that in Spectral would involve creating a rule that checks for `created_at` or `created_on`, which would miss `created_date` or `date_created`. So then you feel like you've got it all and somebody starts using `createdAt`, `Created-At` or `DateCreated`, and don't forget this would all need to be regex not just a list because there's `updated_at` and `planted_at` to catch too. Trying to preempt every possible name of a date field is boring, and a rule able to do it would be ludicrously slow. Whilst working with AI is not quicker, the way this chunks down the documents and walks through them, checking for any property related rules along the way, means it can be a lot quicker than writing a bunch of really vague Spectral rules. ``` ⚠ [date-like-properties-should-use-datetime-or-timestamp] The property 'planted_at' sounds like a date but does not specify 'format: datetime' or 'format: timestamp'. It is important to specify the format for date-related properties to ensure they are interpreted correctly by the API consumers. at api/openapi.yaml:44:1039 ``` It caught 'PlantedOn' too. All just from this one prompt: ``` ruleset: - lintgpt: v3_standards: rules: - "Properties that sounds like they are dates should use format: datetime or format: timestamp" ``` ### Quality not just Presence Another interesting ability of this approach is looking beyond simply "if a keyword is there", but attempting to analyze it for quality. Think about the sorts of descriptions people add to parameters in documentation. They're usually not very good. Spectral has rules saying that parameter descriptions should be present, and you can even set a rule saying it should be a certain length, but that sometimes just gets people adding descriptions that are long and useless. What if we can ask for them to be useful, define what we mean by that, and specifically request a writing style? ``` - "Request property descriptions should be written from the client's point of view. ie "set the earliest date you wish to revive records from" ``` At first I found this to be pretty gung-ho as it wanted me to define descriptions on *all* properties, but amongst the "Add a description here and here and here" prompts were some classic examples or useless descriptions that it wasn't going to accept. > ⚠ \[descriptive-property-descriptions\] The description 'The category of the error.' is not sufficiently descriptive and appears to just repeat the property name 'errorType'. A more useful description would provide additional context or explain what constitutes an error category. Yeah, fair. `errorType` is pretty clearly a type of error! Come on folks we need to do better than this. Where can I find out what types might be defined. What do they mean. > ⚠ \[descriptive-property-descriptions\] The description 'The PSP reference of the payment.' is not sufficiently descriptive and appears to repeat the property name without providing additional useful information. Again, useless description is useless. Of course a property called `pspReference` on `/payments` is a PSP Reference of the payment. What is a PSP? Where does this come from. How can I find out more. Spectral would not have flagged these as it just wanted the description to be there (that's probably why these specific descriptions were added like this in the first place) but that would just leave me being the person saying "Come on, can we do a bit better". With Optic you can call out poor quality descriptions automatically, and reduce the friction of API Design Reviews by getting that all out of the way earlier on. ### No Error Responses over 2XX One rules I've always wanted to write but struggled to with Spectral was a "No Errors on 200 OK" for [**apisyouwonthate/style-guide**](https://github.com/apisyouwonthate/style-guide?ref=apisyouwonthate.com). The idea would be to look for any signs of an error message being defined over 200 (or 2XX), but not just seeking specific properties because there are infinite. - error: - errors: - warning: - warnings: - errorMessage - errorMsg - message I never figured out how to do this well in Spectral. Checking for a million various properties in the root and hoping it wasn't wrapped in anything just seemed daft, but LintGPT handled this with a single line. ``` - '200 response bodies should not be used to return errors. Those must use 4xx or 5xx' ``` I made a naughty OpenAPI document with all different types of errors/warnings/errMsg properties and it caught the lot. ``` x GET /error-single: added x [200-response-no-errors] The 200 response body contains an error field, which is not appropriate for successful responses. Errors should be indicated with 4xx or 5xx status codes. at api/openapi-errors.yaml:10:179 x GET /errors-array: added x [200-response-no-errors] The 200 response body contains an 'errors' field, which suggests it is being used to return error information. According to best practices, successful responses (2xx) should not include error details. Errors should be indicated with appropriate 4xx or 5xx status codes. at api/openapi-errors.yaml:28:681 x GET /warnings-array: added x [200-response-no-errors] The 200 response body contains a 'warnings' property which suggests that it is being used to return issues that should be communicated with 4xx or 5xx status codes. Using 200 OK for errors can be misleading and does not follow HTTP status code semantics. at api/openapi-errors.yaml:48:1248 x GET /single-property: added x [200-response-no-errors] The 200 response body contains an 'errorMsg' property, which suggests that errors are being returned with a 200 OK status. Error messages should be returned with appropriate 4xx or 5xx status codes to indicate client or server errors respectively. at api/openapi-errors.yaml:67:1809 ``` It even spotted a `{ some-wrapper: { errMsg: "" } }` which is how you know it's good. ## The Future of API Governance Linting is an integral part of API Governance, reducing how much time needs to be spent on API Design Reviews, but even the evangelists have always tried to explain that "API Linting", "automatic Style Guides", etc could not handle everything. LintGPT still doesn't handle everything, but it takes a large step forward, moving the boundary on what can and cannot be handled with linting. Maybe Spectral could handle 70% of your review, and LintGPT is up to 80%. Should you rush out and replace all of your API linting with LintGPT right this minute? I'd advise against it! For one it's still in beta with some functionality still being worked on, like linting on operation parameters and headers. Can't do that yet. Having different ruleset formats is a bit of a pain in the backside, and conversations need to happen between the teams working on [Redocly CLI](https://redocly.com/redocly-cli/?ref=apisyouwonthate.com), [LintGPT](https://www.useoptic.com/lintgpt?ref=apisyouwonthate.com), [Spectral](https://stoplight.io/open-source/spectral?ref=apisyouwonthate.com), and [Vacuum](https://quobix.com/vacuum/?ref=apisyouwonthate.com). We need to see if there's anything we can do to standardize rulesets amongst us. Thankfully we're all already friends and everyone has told me they're up for a chat, but it's not going to be an easy job from a technical perspective. Seeing as LintGPT lives in [Optic CLI](https://www.useoptic.com/cli?ref=apisyouwonthate.com), which [already supports Spectral rulesets](https://www.useoptic.com/docs/style-guides/spectral?ref=apisyouwonthate.com), anyone already happily using Spectral can give Optic *and* Spectral a whirl together. That way you can start experimenting with a few extra LintGPT rules on top of your existing ruleset, and not have to worry about rebuilding what already works great. You may find some rules are easier to build in LintGPT than they are in Spectral, but you may find some are easier to build in Spectral. There are some things LintGPT just cannot do, like suggesting particular endpoints that should exist. It's early days for [LintGPT](https://www.useoptic.com/lintgpt?ref=apisyouwonthate.com), that much is clear, but the Optic team have a reputation for making great stuff, releasing early, releasing often, and keeping the momentum going, so I recommend giving it a whirl and seeing how far it takes you on your API Governance adventure. *I hope the robot overlords thank me for doing their bidding when they take over.* ### Low Code, No Code, and Automation for APIs - Gimmick or Godsend? URL: https://apisyouwonthate.com/blog/low-code-no-code-and-automation-for-apis-gadget-or-godsend/ Last updated: 2023-12-20T20:51:44.000Z Designing an API using OpenAPI and going through modeling, mocking, etc. can save you wasting a bunch of time building the wrong thing, and having that OpenAPI can speed you up along the whole API lifecycle as you build the API and write tests, but is it always necessary? What if you don't want to design an API? What if you don't want to even build one? Is this a good idea, or are you a lazy bad developer? I am proud of my part in helping the API Design-first workflow go from relative obscurity five years ago, to a place where it's being adopted by teams all over the world. I've worked on infinite tools, helped rewrite chunks of specifications, spent years working on education in the space, so you'd probably expect me to use API Design-first for every API right? Wrong! Sometimes I completely ignore the whole concept. 🤯 There's an old Spanish proverb "the cobbler’s children have no shoes", and it basically means sometimes the expert is so busy helping others that they're too tired to sort out their own problems. It's why my chef friends knock out microwave meals when they get home from a long shift. It's why some hairdressers have shockingly bad hair, It's also why I've been using Airtable to make APIs instead of designing and building top quality APIs. ## Use-case Drive APIs vs Data APIs API Design intentionally takes a bit more time earlier on in the process. Think about the domain, split up various models into resources on distinct endpoints, start adding in some methods, headers, payloads, all the whilst looping in stakeholders to make sure the APIs do what you want. This is exactly what you need to do for Use-Case Driven APIs that may have complex workflows with predefined paths and "next available steps" available through HATEOAS links, but if the API is a "Data API"... do you really need to plan out that you're going to Create, Read, Update, and Delete each of the Things in your API? Why? Sitting there and writing a `create()`, `read()`, `update()`, and `delete()` for every controller is mind-numbing, and no amount of "generators" or "scaffolding" can save you from this pain. Then there's the whole faff of building a CRUD admin interface to go with those CRUD endpoints. In 2006 I remember Rails telling me I'd never have to do this again, and it was never true. ## Skipping all that CRUD In 2020 my reforestation startup [Protect Earth](https://www.protect.earth/?ref=apisyouwonthate.com) needed a simple iOS application to photograph all the trees we plant, and save a bit of metadata about species, supervisors, GPS coordinates, and a photo. This would help us keep track of all the trees we planted, and hook them up with Organizations paying for them through Orders to support our work. If I was going to design and build this API I'd start with some CRUD API endpoints: - `POST /trees` - `GET /species` - `GET /supervisors` - `GET /orders` - `GET /organizations` I'd then bash together some MVC application with a database that has tables for all of those entities, and build the 12,096th admin panel I've had to build. Then it starts getting a bit trickier: I've got to handle image uploads, which is a bit of a faff now that I have to set up a Amazon S3 bucket, set up some access tokens, make sure my API is configured to support it in both dev and production. Soon I'll likely need to make custom views for those organizations to see various reports, and set up Slack alerts for new orders. This is all a whole lot of code I need to write which is taking me away from the woods/bike/pub. You can see where I'm going with this. It's all a bit of a faff, and with almost zero budget it made no sense at all for me to spend months doing it all the "most proper way" possible when there was nobody there to judge me. I didn't need to build a shining castle on the hill, I needed to build a simple cabin that would get me through winter as snow was falling all around me. ## No/Low-Code APIs With more no/low-code solutions popping up it was incredibly simple to pop this all together. After trying out a few headless CMS at the easy end, and Firebase at the more complicated end, I settled with Airtable, which had a super simple API, let me muck about with it programatically, and had enough integrations baked in I could spin off a [Zapier](https://zapier.com/?ref=apisyouwonthate.com), [Pipedream](https://pipedream.com/?ref=apisyouwonthate.com), or [Make](https://make.com/?ref=apisyouwonthate.com) for anything I needed to tack on in the future. The API might not have been super lovely, but it was fine enough for our iOS developer to work with, and our iOS app was build in record time uploading images and data to Airtable. We got out there photographing all our trees, building up that dataset ready for the next stage. Our funding partners had particular ideas about what the API should look like, and throwing them directly at Airtable apparently was not going to work. Alright, easy, we just made a simple proxy API that would mush the Airtable API data into the format they wanted. No complicated logic, just turning some JSON into other JSON with an AWS Lambda. ## The Arc of the Universe Bends Towards Design & Code Airtable served my charity well for two years until it started to creak at the seams. Too many records in some bases, photos too big, too many automations being triggered, not enough granular control over permissions for views, etc. Fair enough, two years of service were massively appreciated. Making a wrapper around Airtable had given us a huge amount of flexibility for a migration to *anything else*, so long as the new system was mushable from some datasource into the same JSON our partners were expecting. We ended up going back to my roots, and building the API using Laravel (PHP), but deferring all the admin panel work to their powerful low-code solution [Laravel Nova](https://nova.laravel.com/?ref=apisyouwonthate.com). This was more configuration than coding, and helped with CRUD forms and image uploads nicely. As it is basically a standard PHP application that supports whatever database system and cloud filesystems you want, it meant we could migrate our data from Airtable into MySQL (Amazon RDS), and move our images from Airtable into S3/CloudFront. ## Consider the Exit Strategy The migration from Airtable was admittedly a bit of a faff, and its made me think a bit differently about how I might want to do this in the future. Tools like [DreamFactory](https://www.dreamfactory.com/?ref=apisyouwonthate.com), [Google Firebase](https://firebase.google.com/?ref=apisyouwonthate.com), [AWS Amplify](https://aws.amazon.com/amplify/?ref=apisyouwonthate.com), all seem like I could have got a lot of the power of Airtable right out of the box, all whilst having my data somewhere more useful. AWS Amplify for example would let me just have a RDS database right there, and I know how to handle that. Google Firebase has a fancy sounding [Google Firestore](https://firebase.google.com/docs/firestore?ref=apisyouwonthate.com), which I am sure is absolutely brilliant once I figure it out. DreamFactory would let me hook up *whatever* data source, including things I've never heard of, so I could build whatever as a database then just have it do the API magic for me. All of that lookedl ike it would need a bit more learning than I had the mental capacity for whilst also trying to learn about reforestation grants and different saplings and infinite other stuff, so I punted it, and went with the easiest thing I could find. That might not be the right approach every time. There is an art to finding the tool/platform which can solve your needs right now, and finding a tool/platform that you arent just going to outgrow before you have capacity to replace it. If you can find something that's likely to cover your needs as you scale, you can have your cake and eat it too, and whilst I didn't have time to evaluate that all before, I'd like to spend some time delving into various tools to help you make those decisions for upcoming projects. ## Data-first is ok for Data APIs Design-first is mostly helpful at avoiding change later down the line at more expensive parts in the lifecycle, like getting a team to rewrite all their controllers and models to some new name you decided, but with most of these no-code solutions you just rename a Model and it's all done for you, so long as you've not got a plethora of clients all suddenly broken by the change then that's not really an issue. These data-driven approaches can save a whole lot of effort in scenarios where you don't need all the bells and whistles of a shiny hand built artisinally crafted API. Many of us API experts have said infinite times "Your data model is not your API domain model" and that is usually true, unless you are literally building a Data API. In which case... it might be. Sure you don't want to literally expose every single relationship and pivot table exactly as it lives in your database, but you don't need to get massively creative when you just need to get that data over the wire in a sensible way. If you're in a startup or working on an experimental idea, low-code or no-code could really help you focus on delivering functionality for your customers so you can extend the lifeline of your company instead of wasting time you don't have rebuilding various wheels for the umpteenth time. It can also help spin up prototypes in larger businesses, proving an idea before you're able to get buyin for larger efforts. Perhaps you can use these tools well enough that you can outlast "rough experiment" and get into "pretty useful for ages" territory, before commiting huge resources to going the entirely custom and extremely well planned approach, and when you do, you'll have all this experience to build from. ### Introducing the Webhook Standard URL: https://apisyouwonthate.com/newsletter/introducing-webhooks-standard/ Last updated: 2023-12-17T18:18:49.000Z Svix offers [Webhooks as a Service,](https://www.svix.com/?ref=apisyouwonthate.com) but they have been cooking up something extraordinary for the last year. Working alongside developers from other companies like Zapier, Twilio, Supabase and more, they have been working on a webhook standard. A standard protocol we follow for webhooks will make our lives much easier, especially if it already has support from multiple large providers in the industry. It was only released a few days ago and is already seeing adoption. You can find more info on [their blog](https://www.svix.com/blog/standard-webhooks/?ref=apisyouwonthate.com) or [check out the standard](https://www.standardwebhooks.com/?ref=apisyouwonthate.com). I am super excited about this one. Great work, Svix! **One note:** With the holiday upon us, the API's You Won't Hate newsletter will skip the issue on the 1st of January as we all enjoy some time off. Happy Holidays from all of us here, Phil, Mike and Alexander. Hopefully, you all have some time off and get a break from your screens and keys. \-Alexander --- Thanks so much to our sponsor ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2023/12/newsletter-2.jpg) ## Your Ultimate API Governance Tool Elevate your API game with API Insights. Visit [apiinsights.io](https://apiinsights.io/?utm%5Fsource=newsletter&utm%5Fmedium=apis-hate-sponsorship&utm%5Fcampaign=api%5Finsights) and experience instant analysis and detailed breakdowns, all designed to enhance your API's performance, security, and design. Join the API governance revolution [Get Started Now](https://apiinsights.io/?utm%5Fsource=newsletter&utm%5Fmedium=apis-hate-sponsorship&utm%5Fcampaign=api%5Finsights) --- ### The API Round-Up Our monthly round-up of articles, news and updates from APIs You Won't Hate and around the web. ### [Huma: A New API Framework for Go](https://github.com/danielgtaylor/huma?ref=apisyouwonthate.com) A simple, fast, and modern HTTP framework for building REST and RPC APIs in Go. Inspired by Python's FastAPI, Huma has first-class OpenAPI and JSON Schema support. Some interesting choices have been made around accept headers and JSON error types. It's worth checking out. ### [Idempotency Keys](https://httptoolkit.com/blog/idempotency-keys/?ref=apisyouwonthate.com) Unlock the potential of idempotency keys for fail-proof retries! Join our very own Phil as he delves into practical examples, tackles security and scalability, and introduces the 'Idempotency-Key' header in the new RFC! ### [Bruno is Growing](https://www.usebruno.com/blog/bootstrapping?ref=apisyouwonthate.com) I mentioned Bruno, a git-based open-source API client, a few issues back. Well, since then, it's grown to 25,000 monthly active users. With more contributors, issues, and PRs than the owner can handle, Anoop is moving to work on Bruno full-time starting in January. If you haven't checked out Bruno yet, what are you waiting for? ### [Tidy First](https://www.oreilly.com/library/view/tidy-first/9781098151232/?%5Fgl=1%2A1hhb2w5%2A%5Fga%2AMTg0MzIyMTE4Mi4xNzAyNTM4NTQx%2A%5Fga%5F092EL089CH%2AMTcwMjUzODU0MC4xLjEuMTcwMjUzODYzMC40NS4wLjA.&ref=apisyouwonthate.com) I love a good book, and Tidy First is one you should pick up. Chapters are short and offer insights on how to make your code more manageable and readable. The first 15 chapters are an easy ready building on top of each other. Once you get past these first 15, you move onto big-picture refactors and eventually more profound questions like what software design is—a great read. ### [FrankenPHP](https://laravel-news.com/frankenphp?ref=apisyouwonthate.com) PHP in Go? FrankenPHP gives you a production PHP server with one command. It's a PHP server written in Go. FrankenPHP just released v1.o and supports Symphony, Laravel and WordPress. Honestly, the Frankenstein elephant sells it more than I ever could. ### [Level Up Your Skills](https://codecrafters.io/?ref=apisyouwonthate.com) I had some learning budget to burn before the end of the year and was looking for something more than your basic book or course. I found most courses do not go as deep as I want. Enter Code Crafters. Rebuild Redis, Docker, Database and more in a variety of languages. You also get feedback on your work; I highly recommend it. --- ### APIs You Won't Hate The latest from the team at API's You Won't Hate. ### [2023 Most Read Article](https://apisyouwonthate.com/blog/understanding-rpc-rest-and-graphql/) I thought I would share our most-read article from 2023\. Top of the list comes **Understanding RPC, REST and GraphQL**. I have to admit I was surprised by this one. Phil breaks down the technical differences between RPC, REST and GraphQL without touching the honest nest of which one is best. We all know it's SOAP API's anyway. --- ### From Our Community Articles written and shared in our free [Slack community](https://slack.apisyouwonthate.com/?ref=apisyouwonthate.com). ### [HATEOAS corpus](https://changelog.com/friends/24?ref=apisyouwonthate.com) Jamie shared an amazing podcast episode from Changelog & Friends. In this episode, they dive into all things API Design. Covering GraphQL, why do we do REST wrong, and what is HATEOAS? ### [Finding Dark Data](https://danaepp.com/finding-dark-data?ref=apisyouwonthate.com) Dana is back with another blog post, working us through Dark Data. Dana covers what it is and the risks it brings. He walks you through common approaches for checking for Dark Data in your API. A great read, as always, Dana. ## Support APIs You Won't Hate When you become an member, you'll get access to members-only content while directly supporting our work. Your support helps us to keep making resources for the API community. [Become a member today](https://apisyouwonthate.com/#/portal) Thanks so much for reading. Have a great new year - until next time! ### Navigating Network Failures URL: https://apisyouwonthate.com/newsletter/navigating-network-failures/ Last updated: 2023-12-01T15:14:53.000Z I read a lot, and I mean *a lot*, but there's one blogger from whom I eagerly await a new post. Sam Rose creates some of the best visual explainers I have ever seen. [His latest blog](https://encore.dev/blog/retries?ref=apisyouwonthate.com) explores the challenges of handling failed network requests in software development and different retry methods. It is a fantastic read. You'll learn how to give your system the best chance to recover when failure happens. \-Alexander K --- Thanks so much to our sponsor ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2023/12/newsletter-2.jpg) ## Your Ultimate API Governance Tool Elevate your API game with API Insights. Visit [apiinsights.io](https://apiinsights.io/?utm%5Fsource=newsletter&utm%5Fmedium=apis-hate-sponsorship&utm%5Fcampaign=api%5Finsights) and experience instant analysis and detailed breakdowns, all designed to enhance your API's performance, security, and design. Join the API governance revolution [Get Started Now](https://apiinsights.io/?utm%5Fsource=newsletter&utm%5Fmedium=apis-hate-sponsorship&utm%5Fcampaign=api%5Finsights) ### The API Round-Up Our monthly round-up of articles, news and updates from APIs You Won't Hate and around the web. ### [Laravel Pulse](https://pulse.laravel.com/?ref=apisyouwonthate.com) Let's be honest: the Laravel community and ecosystem are amazing. Laravel Pulse is a new open-source tool coming to the ecosystem which offers insights into your application's performance and usage. It looks incredible, and I can't believe it's open-source. Go Laravel! ### [AWS Lambda Get's Node 20](https://aws.amazon.com/blogs/compute/node-js-20-x-runtime-now-available-in-aws-lambda/?ref=apisyouwonthate.com) AWS Lambda now supports the Node.js 20 runtime, offering enhanced performance and new language features while streamlining certificate loading for faster cold starts and improved efficiency in serverless applications. I am a big fan of Serverless, so I am super excited about this one. ### [API Keys](https://unkey.dev/?ref=apisyouwonthate.com) I came across this new tool while looking at API key options for an API built on Cloudflare. A tool for providing open-source user-facing API keys, focusing on low latency by storing the keys globally. Keys on the edge. Check it out. ### [Ktor](https://ktor.io/?ref=apisyouwonthate.com) At Atlassian, I use a lot of Kotlin and Spring Boot. Even though Spring Boot is rock solid, so much of it still leaves a bad taste in my mouth. Say hello to Ktor, a simple and fun framework for building server applications. Finally something that brings a more straightforward approach to web frameworks that I have come to expect from other languages. ### [State of JavaScript 2023](https://stateofjs.com/en-US?ref=apisyouwonthate.com) It is that time of year again. The state of JavaScript 2023 survey is here. If you use JS in your work, head over and ensure API developers are represented in the results. --- ### APIs You Won't Hate The latest from the team at API's You Won't Hate. ### [OpenAPI & AsyncAPI: Advanced $ref Guide](https://apisyouwonthate.com/blog/openapi-asyncapi-ref-advanced-guide/) Phil walks us through using `$ref` in OpenAPI and AsyncAPI, helping you manage complex YAML/JSON documents and streamline your API descriptions. It is time to clean things up by splitting description documents with `$ref`. ### [Craftwork](https://www.forbes.com/sites/brucerogers/2023/11/17/craftwork-out-to-modernize-home-services-market-scores-6-million-seed-round/?sh=49ea943d2be5&ref=apisyouwonthate.com) Mike has been busy. His team just raised $6 million in a seed round for his start-up [Craftwork](https://craftwork.com/?ref=apisyouwonthate.com). Congratulations Mike. Craftwork streamlines planning and ordering home painting services, focusing on transparent pricing and quality assurance. --- ### From Our Community Articles written and shared in our free [Slack community](https://slack.apisyouwonthate.com/?ref=apisyouwonthate.com). ### [Bypassing API Rate Limiting](https://danaepp.com/bypassing-api-rate-limiting-using-ip-rotation-in-burp-suite?ref=apisyouwonthate.com) Another cracking article from Dana about using IP rotation in Burp to bypass API rate limitations. It's a great read; as always, thanks, Dana. --- ## Support APIs You Won't Hate When you become an member, you'll get access to members-only content while directly supporting our work. Your support helps us to keep making resources for the API community. [Become a member today](https://apisyouwonthate.com/#/portal) Thanks so much for reading - until next time! ### OpenAPI & AsyncAPI $ref: Advanced Guide URL: https://apisyouwonthate.com/blog/openapi-asyncapi-ref-advanced-guide/ Last updated: 2023-11-28T09:20:07.000Z After using OpenAPI or AsyncAPI for a while, you might notice your description documents have become a rather unwieldy mess of YAML and JSON. You end up with a whole lot of repetition, and this huge mess just loves to trigger merge conflicts as multiple developers change different things but Git seems none the wiser. You can avoid this pain by splitting description documents up with `$ref`, using various reusable components, but how exactly you go about doing that can be a tricky one to work out. ## OpenAPI Reusable Components The [OpenAPI Documentation](https://learn.openapis.org/?ref=apisyouwonthate.com) includes a brilliant example of an API for playing the classic board game Tic Tac Toe. This has several parts that are used several times, so instead of copy-pasting everything they’ve defined reusable `components` for both `schemas` and `parameters`. ``` # Whole board operations /board: get: summary: Get the whole board description: Retrieves the current state of the board and the winner. tags: - Gameplay operationId: get-board responses: "200": description: "OK" content: application/json: schema: $ref: "#/components/schemas/status" # Single square operations /board/{row}/{column}: parameters: - $ref: "#/components/parameters/rowParam" - $ref: "#/components/parameters/columnParam" get: # ... Hidden for readability... put: # ... Hidden for readability... components: parameters: rowParam: description: Board row (vertical coordinate) name: row in: path required: true schema: $ref: "#/components/schemas/coordinate" columnParam: description: Board column (horizontal coordinate) name: column in: path required: true schema: $ref: "#/components/schemas/coordinate" schemas: errorMessage: type: string maxLength: 256 description: A text message describing an error coordinate: type: integer minimum: 1 maximum: 3 example: 1 mark: type: string enum: [".", "X", "O"] description: Possible values for a board square. `.` means empty square. example: "." board: type: array maxItems: 3 minItems: 3 items: type: array maxItems: 3 minItems: 3 items: $ref: "#/components/schemas/mark" winner: type: string enum: [".", "X", "O"] description: Winner of the game. `.` means nobody has won yet. example: "." status: type: object properties: winner: $ref: "#/components/schemas/winner" board: $ref: "#/components/schemas/board" ``` This is not particularly unmanageable, but let’s pretend there is 50 or more endpoints. You could imagine how this one file would be getting a bit much to handle. How people split up there files has been completely unique to the developer for a long time, but certain conventions are starting to emerge with tooling leading the way. ``` ├── paths │   ├── board.yaml │   └── board_{row}_{column}.yaml ├── components │   ├── schemas │   │   ├── errorMessage.yaml │   │   ├── board.yaml │   │   ├── coordinate.yaml │   │   ├── status.yaml │   │   ├── winner.yaml │   │   └── mark.yaml │   └── parameters │   ├── columnParam.yaml │   └── rowParam.yaml └── openapi.yaml ``` This convention splits each type of `components` into their own subdirectory, and then puts them into their own unique file. Now the `openapi.yaml` is a whole lot lighter. ``` openapi: 3.1.0 info: title: Tic Tac Toe description: | This API allows writing down marks on a Tic Tac Toe board and requesting the state of the board or of individual squares. version: 1.0.0 tags: - name: Gameplay paths: /board: $ref: paths/board.yaml /board/{row}/{column}: $ref: paths/board_{row}_{column}.yaml ``` The `paths/board.yaml` looks like this: ``` get: summary: Get the whole board description: Retrieves the current state of the board and the winner. tags: - Gameplay operationId: get-board responses: '200': description: OK content: application/json: schema: $ref: ../components/schemas/status.yaml ``` Finally, `components/schemas/status.yaml` looks like this: ``` type: object properties: winner: $ref: ./winner.yaml board: $ref: ./board.yaml ``` All the filepaths are relative to their current location, and can traverse up and down the filesystem, with the standard `../` to go up a directory. The chance of getting a Git conflict when two different developers add two different paths or expanding properties in a schema is now a fair bit smaller. If a conflict does occur, the diff will be a lot less confusing to work out. One downside of splitting up components into different documents like this is that it becomes harder to follow API changes, either directly or by looking at files in GitHub. Changing a schema in one document can effect how multiple different endpoints work, and that can caused a bit of confusion. API change management tools like [Bump.sh](https://bump.sh/api-change-management?ref=apisyouwonthate.com) or [Optic](https://www.useoptic.com/docs/diff-openapi?ref=apisyouwonthate.com) can help by spotting breaking changes and reporting them on PRs, so that you can easily see problems that could otherwise slip through. ## AsyncAPI Reusable Components AsyncAPI is thankfully the same when it comes to `$ref` and `components`, so if your event-driven API is struggling as much as your HTTP API then it’s time to split things up. ``` v0/rust/servers/{server_id}/players/{steam_id}/events/banned: description: Channel for notifying a server banned a player parameters: server_id: "$ref": "./components/parameters.json#/server_id" steam_id: "$ref": "./components/parameters.json#/steam_id" subscribe: operationId: ServerPlayerBanned message: "$ref": "./components/messages/ServerPlayerBanned.json" ``` This example is taken from the [Gaming API](https://github.com/GamingAPI/definitions/?ref=apisyouwonthate.com) example projects, and highlights a slightly different approach of using a single `parameters.json` document and referencing a parameter within that file, instead of using a `parameters/` subdirectory with a file for each parameter. You could do either with either OpenAPI or AsyncAPI, it’s a matter of personal preference. To learn more about the `components` keyword in AsyncAPI, head on over to [their documentation](https://www.asyncapi.com/docs/reference/specification/v2.6.0?ref=apisyouwonthate.com#componentsObject). ## Using $ref with URLs Filepaths are not the only way to work with `$ref`, you can also use URLs. This is particularly helpful when you have a “data model” that is shared across multiple APIs or microservices. Perhaps you don’t want each API to define a User, Company, or Payment separately, and get stuck with infinite different variant models. Simply publish those shared components as JSON, YAML, or both, on a static site or S3 bucket somewhere and let people $ref them into their API. ``` responses: '200': description: OK content: application/json: schema: $ref: "https://schema.example.org/status.json" ``` ### Benefits of URL `$ref` Doing this has several benefits. Not only can other API teams all work together to make a single repository of all the most command/shared components, but API consumers can use them too. Perhaps clients want to implement some [client-side validation](https://apisyouwonthate.com/blog/json-schema-client-side-validation/) to make sure form submissions are valid before they waste time and carbon emissions going over the wire talking to the API with an invalid request. Specifically splitting the “schemas” out is brilliant because its not just helpful for [OpenAPI tooling](https://openapi.tools/?ref=apisyouwonthate.com), but for [JSON Schema tooling](https://json-schema.org/implementations?ref=apisyouwonthate.com) too. There’s even more JSON Schema tooling than OpenAPI tooling so its handy to be able to use both. ### Downsides of URL `$ref` One downside you’re probably already thinking of is that doing all of this requires a bit of work. This is jokingly called SpecOps (API descriptions are also known as “specifications”). Setting up deployment pipelines and hosting to make those reusable components available is a faff. Other complications can appear depending on which tools you’re using. Some tools do not support URLs in `$ref`, either for security concerns or because the tool maintainers never got around to it. You need to programmatically replace all the $ref’s with URLs to be local refs, and whilst there are tools which can “bundle” your API descriptions up for you, it’s another bit of work, and adds another copy of the API description document to keep track of and keep updated. Finally there’s authentication. Some people have their API descriptions in a private Git repositories and cannot access it with `https://raw.githubusercontent.com/org/repo/main/content/schemas/foo.json` because it would need some sort of access token and how’s that going to work? Making a GitHub Action / Continuous Integration step that deploys the API descriptions or schemas to a public S3 bucket or other public static site is probably the best thing to do there. Others hide their OpenAPI and AsyncAPI by choice for security reasons, but that’s never made much sense because Stripe, PayPal, Box, GitHub, and plenty of other massive API companies have their API descriptions out in public and nobody has hacked them. APIs should be protected with firewalls and API keys, but OpenAPI and AsyncAPI information can be plastered all over the place. Another vote for the public static site. There is an argument for making public APIs public and keeping internal API’s private, and some hosted API documentation tools can help with that, or you can host internal API docs on a different static site that’s only available on the company network. Either way you’ll need to keep your public APIs public, and keep your shared components public, then hide the internal APIs that reference those. That gives you the best of both worlds. ## Propagating Changes Using tools like Bump.sh you get all the benefits of a tool that understands $ref, but without any of the hassle of needing to bundle documents up. Like any tool which uses a build step, this has the pro and the con of meaning that documentation is built at a certain point in time. Changes that happen to the $ref’ed resources - whether they’re in another repository, or being pulled in via URL - will take some time to appear in your API. For example, if the Widget API is using a shared Company schema via `$ref: https://widgets.com/schema/company.json`, and company decides to add VAT number as a property, your Widget API documentation is not going to mention that property until your next build. Is that a good thing or a bad thing? It can be both depending on the scenario, but having changes appear in your API without your knowledge is probably not ideal. ## Tools for Bundling & Splitting Bundling is usually only needed if you are working with older or strange tools which do not support $ref properly (or at all). If you are working with Bump.sh CLI you won’t need to bundle, but if a tool wants you to import a single `openapi.yaml` document you might need to bundle. ``` $ redocly bundle openapi.yaml -o openapi-bundled.yaml bundling openapi.yaml... 📦 Created a bundle for openapi.yaml at openapi-bundled.yaml 105ms. ``` This will grab all of the `$ref`’s that use “external files” or URLs and move the contents into the relevant subsection of `components` in the `openapi-bundled.yaml` document. Splitting does the opposite. If you have a massive painful document (maybe generated from HTTP or converted from Postman) you can split it down into multiple documents with a sensible folder structure, ditch the original, commit all that to Git, and push it up to Bump.sh with all the `$ref`’s intact. ``` redocly split generated-openapi.json --outDir api/ bump deploy api/openapi.json ``` To give Redocly CLI a try, in combination with the Bump CLI, install them both: ``` npm install -g @redocly/cli bump-cli ``` ## Further Reading If you’d like to learn more than you could ever possibly want to know about AsyncAPI $ref then head on over to [The Reference Rabbit Hole](https://www.asyncapi.com/blog/the-reference-rabbit-hole?ref=apisyouwonthate.com) by [Jonas Lagoni](https://github.com/jonaslagoni?ref=apisyouwonthate.com). ### Rising Above the Clouds: Cloudflare Leveled Up URL: https://apisyouwonthate.com/newsletter/clouds-cloudflare-leveled-up/ Last updated: 2023-11-15T23:22:27.000Z I'll be up front: this suggestion won't be for everyone, nor will it be the right choice for most. However, while we are all deploying solutions across AWS, Azure and other providers, Cloudflare has come out swinging. Recently, I have been building full-stack web applications and APIs using workers and pages. Cloudflare's suite of tools and services makes getting an idea off the ground easy, as you no longer think about infrastructure. You get to focus on your API and what it does; that's it. Local development easily replicates the deployed environment thanks to their wrangler/mini flare tools. If you are curious about workers, the Cloudflare team were recently on the [Syntax podcast](https://syntax.fm/show/691/cloudflare-workers-are-next-level-with-rita-kozlov-and-brendan-irvine-broque?ref=apisyouwonthate.com), and it offers a good intro. --- ### The API Round-Up Our monthly round-up of articles, news and updates from APIs You Won't Hate and around the web. ### [Zuplo's $9M Raise](https://zuplo.com/blog/2023/11/08/announcing-9M-raise?ref=apisyouwonthate.com) Zuplo announced a $9M seed raise led by Trilogy Equity Partners, aiming to revolutionize API management. Zuplo makes API management a breeze and offers high availability across global data centres. I love that they have taken edge computing into account here and made deploying to Earth easy. Honestly, the tagline `Get Stripe quality documentation` sold me. Go check them out. ### [Build Better APIs with AI-powered Testing Tools](https://blog.stoplight.io/build-better-apis-ai-testing-tools?ref=apisyouwonthate.com) Stoplight gives us a gentle walkthrough of using AI-powered tools to level up our testing. Even though the constant hype around AI can be grating, there are some incredible areas where it can be applied and improving tests is one of them. ### [Milo: A New Node HTTP Parser](https://github.com/ShogunPanda/milo?ref=apisyouwonthate.com) Many of our readers use PHP, but Node powers many APIs, as JS is the best language ever conceived 😉. I mean, who doesn't love `['a'] + ['b'] = "ab"`. *I'm getting off-topic.* There's a plan to replace the current HTTP parser in Node with a new one written in Rust. Check out Paolo's talk at [Vancouver's Node Collab Summit](https://youtube.com/watch?v=L-VONzXQ944&ref=apisyouwonthate.com) for some rationale behind the move. ### [Handling a Regional Outage](https://newsletter.pragmaticengineer.com/p/handling-a-regional-outage-comparing?utm%5Fsource=post-email-title&publication%5Fid=458709&post%5Fid=138459678&utm%5Fcampaign=email-post-title&isFreemail=true&r=1v83fz&utm%5Fmedium=email) Gergely Orosz covers how a regional outage is handled by the big three cloud providers: AWS, Azure and GCP. Covering how they dealt with the communication during the incident, details shared and their postmortem. This is a great read that might help you choose a cloud provider for your next API. --- ### APIs You Won't Hate The latest from the team at API's You Won't Hate. ### [Meet Redocly CLI: The Modern OpenAPI Sidekick](https://apisyouwonthate.com/blog/redocly-cli/) Phils seems to be bouncing with excitement as he walks us through the capabilities of the Redocly CLI. Redocly CLI is an open-source tool for working with OpenAPI, handling linting, bundling, documentation and more. For more info, give Phil's post a read. ### [Popular OpenAPI Bundling Tools Compared](https://apisyouwonthate.com/blog/openapi-bundling-tools-compared-blog-post/) Comparing Swagger CLI, JSON Schema Ref Parser, and Redocly CLI, Phil shows us how they stack up against each other. If you read the above article, it's no surprise that Phil picked Redocly and is now deprecating Swagger CLI. ### [Podcast: Catching up with Danny Sheridan from Fern](https://apisyouwonthate.com/podcast/catching-up-with-danny-sheridan-from-fern/) In this episode of APIs You Won't Hate (the podcast), cohost Mike Bifulco has a chat with Fern's CEO and cofounder Danny. Since their last chat early in 2023, Fern has continued to release updates across their product stack: their ambitious and slick client library SDK codegen product, and their new docs site generator. --- ### From Our Community Articles written and shared in our free [Slack community](https://slack.apisyouwonthate.com/?ref=apisyouwonthate.com). ### [Proving API exploitability with Burp Collaborator](https://danaepp.com/proving-api-exploitability-with-burp-collaborator?ref=apisyouwonthate.com) Dana walks us through using Burp Collaborator to find vulnerabilities in API's. Dana always hits us with an exciting blog that leaves you with some new skills. Thank you for sharing. ## Thanks to our members! Thanks so much to Sahand, Danny, JUXT, Kin, and Oleksii, for showing your support to APIs You Won't Hate with a paid membership. Your support means the world to us! ## Support APIs You Won't Hate When you become an member, you'll get access to members-only content while directly supporting our work. Your support helps us to keep making resources for the API community. [Become a member today](https://apisyouwonthate.com/#/portal) ### Catching up with Danny Sheridan from Fern URL: https://apisyouwonthate.com/podcast/catching-up-with-danny-sheridan-from-fern/ Last updated: 2023-11-15T16:01:24.000Z Mike and Danny Sheridan from [Fern](https://buildwithfern.com/?ref=apisyouwonthate.com) chat about updates to Fern: client library SDK codegen, and their great new docs site generator tool. - Fern - [https://buildwithfern.com/](https://buildwithfern.com/?ref=apisyouwonthate.com) - Danny Sheridan - [https://www.linkedin.com/in/sheridandanny/](https://www.linkedin.com/in/sheridandanny/?ref=apisyouwonthate.com) - [Amazon Smithy](https://smithy.io/2.0/index.html?ref=apisyouwonthate.com) - [Palantir Conjure](https://github.com/palantir/conjure?ref=apisyouwonthate.com) - [Fern OSS on GitHub](https://github.com/fern-api/?ref=apisyouwonthate.com) - [Cohere](https://www.bing.com/aclk?ld=e8u8V2zO6ymPmozbD6%5FkG53zVUCUwadzADZ6VOShR7v2ekcSghKBw5h10geO8lA80x4Tk7QkveVHMcoqGLdvOsP-RO8XdHIO1A5KtWGG-EP-E6drfrsjN3v%5FLDbn3L-4pnIfkUZOefW8iM1qp9OJou%5FCInuVoHQuiJ0WfMM59ovYk3pgMq6564PcsVL-PUI5WiKKIfvQ&u=aHR0cHMlM2ElMmYlMmZjb2hlcmUuY29tJTJmJTNmbXNjbGtpZCUzZDQ0MTg1MTRiMDZkNzEwMDBmM2MxMzQxZmVmMTg1NjJjJTI2dXRtX3NvdXJjZSUzZGJpbmclMjZ1dG1fbWVkaXVtJTNkY3BjJTI2dXRtX2NhbXBhaWduJTNkQ29oZXJlJTI1MjBCcmFuZCUyNnV0bV90ZXJtJTNkY29oZXJlJTI2dXRtX2NvbnRlbnQlM2RCcmFuZA&rlid=4418514b06d71000f3c1341fef18562c&ref=apisyouwonthate.com) - [APIs You Won't Hate: Make your API Idempotent](https://apisyouwonthate.com/blog/idemptoency-keys/) - [Fern Careers](https://buildwithfern.com/careers?ref=apisyouwonthate.com) #### Transcript \[00:00:00\] ****Mike Bifulco:** Hello, and welcome back to APIs you won't hate. . My name is Mike Biko one of your APIs you won't hate. Co-hosts, co-founders, whatever you wanna call \[00:00:08\] ****Mike Bifulco:** me. Today I'm sitting down to chat with my good friend Danny Sheridan from Fern. Danny's been on the show before about six months ago. We talked in March of 2023, shortly after I believe Danny, you had just finished going through Y Combinator's Winter Batch for this year. \[00:00:23\] And it's great to have you back. I'm really excited to, to hear what's happened since then. And what's going on with you? How are you doing today? \[00:00:28\] ****Danny Sheridan:** I am ecstatic today. I think that as we talked last, like you said, six months ago. My team was just coming out of the Y Combinator program. We were able to use that acceleration that they provide to go raise the seed round of fundraising. And at this point, we've been growing the business by increasing the number of customers that we work with, which we can talk about today, as well as expanding the functionality that Fern provides for companies that have public rest APIs. \[00:00:59\] ****Mike Bifulco:** Love that. Yeah, \[00:01:00\] I'm, I'm super interested to get into it. As you know you know, we're, we're big API fans around here, so nerding out on this stuff is definitely in the wheelhouse. Before we do that, let's take a step back and why don't you tell me the audience listeners about Fern from a 1 0 1 0 1 level, what is Fern? \[00:01:15\] What is it for, and who would be interested in it? \[00:01:17\] ****Danny Sheridan:** Yeah, I think the best way to start there is to talk about the inspiration from Fern, which is that I was at AWS as a product manager before Co-founding Fern, and my co-founder, whose name is Deep, was at Palantir Technologies as an engineer. And one of the things that both of us got to experience was that Palantir and AWS. Both design APIs, starting with the API specification or the API contract, they don't start by writing backend server code. They don't start by writing API documentation. Instead, they start in A-A-J-S-O-N or a YAML file describing the \[00:02:00\] endpoints, the data models. And the errors that can be thrown by those endpoints. \[00:02:06\] And so that was a big inspiration. I wanna give, we stand on the shoulders of giants here at Fern and specifically we do that with Amazon's Smithy, which is an open source project they created. And Palantir's Conjure and Mike, we'll include those in the show notes for folks that wanna check 'em out. Both, both of them are, are in-house built tools. AWS and Palantir that really showed us the power of going API first, they allow a front end and backend engineer, maybe even a product manager to collaborate on designing the API. And then once the API contract is established, you run it through code generation technology and those code generators produce a couple of different artifacts. They can produce server boilerplate code. that looks like maybe you're using the Java springing \[00:03:00\] framework or the TypeScript Express, or even Python's fast, API. And you wanna take my API contract and make sure that I create all of those routes and I'm implementing them. \[00:03:13\] The second thing that those tools created were client libraries. \[00:03:16\] So these are the SDKs. It's nice to have a typed client that you get auto complete in your ID of whichever you're using. Typically, we see VS code as the most popular. But when I like hit, when I'm using the SDK, it'll show me all of the resources that are available and the operations that I can perform on those resources. \[00:03:35\] And so that might be like payment create. And that's a really nice developer experience because they don't have to leave. Their IDE while they're coding to go check what the documentation says. It's part of the SDK and they, they've kind of got docs on hover where they can see details about each, each of those resources or operations, uh, how they behave. And the third \[00:04:00\] thing, Mike, is in addition to some server code and SDKs, is that these tools, these conjure and SMITHY tools from the companies that I mentioned, produced a version of API documentation. In AWS's case, it was literally the A-W-S-A-P-I docs. I don't think it'll be a surprise to folks and Mike, tell me if I'm wrong, that AWS does not hand write all of their API documentation and their API reference. \[00:04:25\] ****Mike Bifulco:** Yeah. I think if you've browsed it once or twice, it's probably apparent that either someone from a different universe made some of them, or they're, they're not necessarily human generated. Yeah. \[00:04:34\] ****Danny Sheridan:** It was really inspiring to me, though, to see an example. a human is not responsible for updating the docs anytime that the API changes because at Amazon we had that concept of good intentions, which \[00:04:50\] is that good intentions. Mike, when someone says, anytime we change the API, we will remember to update the docs. Good \[00:04:57\] intentions fail because people \[00:05:00\] are busy \[00:05:00\] ****Mike Bifulco:** That's \[00:05:00\] ****Danny Sheridan:** and instead Exactly. And instead, what we should be doing is having tooling. That supports us in accomplishing our goals. And so Fern is a way where companies with public rest APIs use Fern to offer SDKs in popular programming languages and API documentation that documents those rest APIs. \[00:05:23\] ****Mike Bifulco:** I think our listeners will identify with this quite a bit. I think we're in an interesting, like. Point in history, this sort of inflection point where maybe even between now and the last time you and I chatted I think something has happened sort of in like the, the general public mindset that people suddenly trust machines to do more intelligent things. \[00:05:42\] And by that I mean like the, the zeitgeist of AI tools, AI wrappers for everything has made people a lot more lenient to let the the, the . Software, write things for them, create things for them, do creative work, do work. That was prior quite manual tasks. And I think Fern is happening at a really interesting \[00:06:00\] time in that inflection where suddenly it's like, oh, actually maybe I can run my, my open API spec through something and it can generate client libraries and also maybe generate meaningful documentation where I think if you asked me that two years ago, I would've been like, not a chance. \[00:06:14\] I'm gonna hire an army of technical writers and they're gonna do a great job for me on this. I'm curious to hear your sort of feelings on that and, and also your feelings on like the, the world of working with people who are generating code as opposed to writing code manually too. \[00:06:28\] ****Danny Sheridan:** let's talk about the biggest objections that I hear \[00:06:31\] when I talk to customers or prospects specifically about Fern. One of the objections that is common is I don't trust a machine to generate my SDK. My developer experience is important to my business. It increases our revenue if we can acquire new customers, and I don't wanna trust that to something that. eVery time I run the code generator could have a different output. Right. And what I tell people \[00:07:00\] is, Fern does not use LLMs to generate code. \[00:07:04\] ****Mike Bifulco:** Hmm. \[00:07:05\] ****Danny Sheridan:** We have bitten, we have built a rule-based system and if we're getting more specific, we built a compiler that takes in an open API spec and generates TypeScript or Python or Java files. \[00:07:18\] ****Mike Bifulco:** Yeah. \[00:07:19\] ****Danny Sheridan:** And so one of the important things to folks right now is determinism. I run the fern generator more than one time, will I get the same SDK output? If, if the SDK would change every time because of determinism, that, that would be a, a major blocker to adoption. And so right now, the most popular way is to generate SDKs are all, there is no L-L-M-S-D-K generation tooling that has become popular. \[00:07:48\] Not to say it can't happen one day. I think folks want determinism and then they also wanna see why, like why did the code generator output a specific line of code? \[00:08:00\] And they really like being able to poke into the open source source code. And so Fern is open source because our customers are developers and they like to see how the magic happens. \[00:08:09\] ****Mike Bifulco:** Yeah, that's an important point too. And I'll make sure we have notes on a link to fern's GitHub in our show notes so our listeners can go and kind of browse that and, you know, also contribute if if they're so inclined. Yeah, I think, I think that's. A valuable point to make that like deterministic LLMs are not a thing in, in the current state of the art. \[00:08:28\] They may one day be, I don't know if that's even possible based on, you know, what an LLM technically is, but you want your generated libraries to be predictable and the same and by that nature also testable. And I feel like that's something that probably comes out of people using Fern is you can generate a library and if you can generate it the same every time, you can test it and be certain that what you're getting is something that at least passes the tests that, that you've provided. \[00:08:50\] ****Danny Sheridan:** aNd Mike, I'm thinking about one of our customers. The company is called Cohere. They're an AI platform for enterprises that is growing in popularity. \[00:09:00\] They are LLM experts and they have chosen fern and do not wanna use LLMs to produce their SDKs because their main source of revenue is their API \[00:09:10\] ****Mike Bifulco:** Yeah, \[00:09:11\] ****Danny Sheridan:** and a problem in their SDK would literally impact their ability to generate income. \[00:09:17\] And so they, they, they were almost willing to put an engineer against it when they saw that they could, instead of having to build. SDKs by hand. They could buy an SDK generation tool. They \[00:09:30\] opted to buy and not build. \[00:09:31\] And I think you're right. You mentioned things are changing around the API developer experience market, should we call it that \[00:09:39\] the developer experience market um, things \[00:09:41\] around SDKs, API documentation. We are seeing a shift where in the past companies would've said, and we've been working on FERN for 18 months now. 18 months ago, my folks would've said. We, the only option to provide great SDKs is to put an engineer against it. \[00:09:58\] ****Mike Bifulco:** Yeah, definitely. \[00:10:00\] Definitely. \[00:10:00\] ****Danny Sheridan:** do this by hand. This is a, a ticket in our engineering queue. \[00:10:04\] And I think that there's been a shift in the last 18 months where more engineering leaders, this typically is happening at the CTO or an engineering manager, platform engineer who says there are now tools like Fern where I can pass in an open API specification. I can get out high quality idiomatic SDKs that I'm comfortable putting in front of my customers. \[00:10:29\] ****Mike Bifulco:** Sure. Yeah. I think also worth pointing out that even the discussion you and I had six months ago has changed pretty dramatically in as much as back then at least, I don't think we talked too much about documentation either. But I feel like that's maybe part of like the, the holistic picture of providing great developer experience to people consuming your client libraries, whether they're internal or external to your company. \[00:10:49\] GEtting both a functioning client library and then the documentation to build it is the story. That's, that's the story. You need to have both of those things for your thing to be valuable. And, and I think there's probably like, I \[00:11:00\] don't know. I would imagine you've probably seen patterns in your customers that they're probably all looking for like a platonic ideal that they want to have, you know, great developer experience and great documentation to go with it. \[00:11:09\] What, what does that like lifecycle look like for maybe finding, acquiring and, and delivering solutions to your customers? \[00:11:17\] ****Danny Sheridan:** Mike, you said kind of who is the gold standard, who is the ideal \[00:11:21\] right now for developer experience, and I think that over the last at least three years, the answer has not changed. The company with the best developer experience in the world is Stripe. \[00:11:33\] ****Mike Bifulco:** yeah, sure. \[00:11:34\] sure. \[00:11:35\] ****Danny Sheridan:** think \[00:11:35\] that folks would find that agreeable, but Mike disagree on the why. Mike, you have to \[00:11:41\] reason through it together. \[00:11:43\] ****Mike Bifulco:** Let's do it. And before we get into that, this is probably where I should mention. I'm a former Stripe employee. No bad blood between me and Stripe, or me and Danny as far as I know, at least but you know, a reasonable call out here too. Yeah. Let's reason through it. Walk me through it. \[00:11:56\] ****Danny Sheridan:** Let's talk about this as if we're making \[00:12:00\] a, a recipe and the outcome is the Stripe developer experience. I think the question, Mike, is what are the inputs. \[00:12:06\] ****Mike Bifulco:** Cool. Yeah. \[00:12:08\] ****Danny Sheridan:** And I usually like to speak in threes, but in this case, I've got five things on my mind that make the Stripe developer experience exceptional. \[00:12:16\] ****Mike Bifulco:** Yeah, hit me. \[00:12:18\] ****Danny Sheridan:** The first is consistent design, whichever endpoints you're using within Stripe's, vast API, you can expect similar behavior and patterns, like you'll learn it once and you learned it across the whole API and any endpoint you use. \[00:12:35\] ****Mike Bifulco:** Yeah. \[00:12:36\] ****Danny Sheridan:** To do, especially as a company grows over. I mean, Mike, they're well over a decade old, right? \[00:12:42\] ****Mike Bifulco:** Sure. Yeah, absolutely. Yeah, pat patterns are huge. And you know, much like you'd imagine in some places there are teams of developers maintaining client libraries. And at AWS there are teams of people starting with that API spec. The discussion on consistency is massive at Stripe. Making sure that things make sense from one to the other so that \[00:13:00\] end users can learn one pattern and apply it everywhere is super, super valuable. \[00:13:06\] ****Danny Sheridan:** The second is documentation. Developers expect that they, it's easy to find what they're looking for and that the information is up to date. It's very frustrating when you go and check out what the JSON looks like for an API response, and it's wrong \[00:13:23\] because some developer changed it and forgot to update the docs. And so Stripe has done a phenomenal job of. Automating their documentation and they have a whole team responsible for it. \[00:13:35\] ****Mike Bifulco:** Yeah. Yeah, definitely. If you are reasonably good at Googling, you can find a, a talk I did on Stripe's internal documentation tools on YouTube somewhere. \[00:13:45\] ****Danny Sheridan:** The third thing are SDKs in popular languages, whether you are a Java, a Python, or a Ruby developer, you get a client library that's got auto complete. \[00:14:00\] Immediately, and you get that when you're starting your project. SDKs tend to make integrations happen in 50% less time, and Stripe understands that by my last count, they supported seven different languages or frameworks that they provide SDKs for. \[00:14:17\] ****Mike Bifulco:** Yeah, that sounds about right. Yeah. And then to, to that point too, idiomatic SDKs are important, right? Like, it feels like python, it feels like ruby, when you're writing in that language. And that's, that's a hard thing to to accomplish just as well. \[00:14:31\] ****Danny Sheridan:** All right. The fourth thing of five errors. When you run into an error in the stripe, API, the message that's output to the developer, it's actually helpful. It typically links you to a place in their documentation. So it's like documentation aware. It'll give you AURL where you can go to read more and it might even recommend a fix as part of the error message. And that's just, um, \[00:15:00\] Mike, it might be craft, that's the word on my mind. \[00:15:04\] ****Mike Bifulco:** Yeah, sure. sure. I think so. And, and that was a revelation too, right? Like this was not a common thing. In a world before stripes developer SDKs existed, like oftentimes as a API consumer. Your job was to set debuggers run console logs and pull your hair out until you figured out what, what comma was missing or what type you had incorrect, or you know, what you had done wrong. \[00:15:27\] And at best you maybe got a 500 back. Something like that. Yeah. \[00:15:32\] ****Danny Sheridan:** And the fifth thing, Mike, that's on my mind is around versioning. When you start, when you open your Stripe account, your pin to a version of the API, and from that day on you are, they expect that you'll be using that API version, and you have to choose to take an upgrade. And so Stripe does a phenomenal job of having backwards compatibility. They've actually built tooling that they call gating that we can talk more about in another episode if you'd like to in.\[00:16:00\] \[00:16:00\] ****Mike Bifulco:** Sure. \[00:16:01\] ****Danny Sheridan:** They've, to date, they've never broken their API. And I think that's a really expensive decision because it, you learn a lot after 10 plus years of doing business. And I'm sure there are moments that they would've loved to break the API, but to date they have not. \[00:16:16\] ****Mike Bifulco:** Yeah, yeah. The, those, so let me let me see if I can kind of sum that up then. So, versioning error messaging, client, libraries, docs and the fifth one is escaping me. A design right. Consistency are are definitely adding up for that the magic. That's, that's the experience I want my engineering team to have. \[00:16:35\] That's also the experience I want from every client library I consume from a third party service as well. \[00:16:40\] ****Danny Sheridan:** Yeah, but Mike, how many employees does Stripe have? \[00:16:43\] ****Mike Bifulco:** Engineers. We have four right now. \[00:16:45\] ****Danny Sheridan:** Sorry. At Stripe. \[00:16:46\] ****Mike Bifulco:** Oh, it's Stripe. Oh, sorry. Yeah, yeah, yeah. Stripe. Gosh, rounding out a few thousand easily, I think probably close to five or 6,000. \[00:16:52\] ****Danny Sheridan:** Yeah. So \[00:16:53\] let's compare craft work to stripe's \[00:16:55\] engineering ability, maybe I'll call it four to a thousand, something like that ratio.\[00:17:00\] \[00:17:00\] It's really, \[00:17:01\] it's not feasible for companies to hire engineers to go build this developer experience infrastructure. \[00:17:10\] For their company. And so that is why the market is turning to buy a solution that's available for you. And Fern is filling that place in the market. And so the, the idea of Fern is that we will bring a stripe like developer experience to your API. We will do that over. That's, that's the vision that we are working toward. And today we will deliver two things for our customers SDKs in multiple languages that are idiomatic. We take care of publishing them to NPM and Maven Central and So we take care of getting 'em to the package managers and publishing the source code to GitHub. And the second product that we offer is API documentation. Everyone wants the Stripe like Docs website that fits your, it matches your brand. \[00:17:52\] You've got the infinite scrolling page. You can see the SDKs documented in the API reference. That's a \[00:18:00\] really nice feature, Mike. And Stripe does in a way where it's auto-populated. \[00:18:04\] So human does not have to go and update the code snippets in all seven SDK languages for every endpoint. Whenever the API changes, \[00:18:12\] they have automation for that. And that's something that Fern has now rolled out for our customers is you can get what we call code snippets for every endpoint showing how to use your SDK in any of the languages that you generated for an SDK for. \[00:18:26\] ****Mike Bifulco:** Yeah. That's, that's incredible. That's, that's quite an offering and, and a bold statement to make. And I think what I wanna know from you is like, what, what are your consumers need to provide? How do they, how do they get that? What's \[00:18:38\] ****Danny Sheridan:** They provide, they provide an open API spec. \[00:18:41\] ****Mike Bifulco:** Hmm. I. \[00:18:42\] ****Danny Sheridan:** We generate SDKs and we generate the API docs. Additionally, if they want to, they can provide markdown files, which will be the pages in their documentation if they'd like to do things like how to get started. How to authenticate with our API. Maybe they have item Marker \[00:18:57\] ****Danny Sheridan:** potency. \[00:18:58\] Those are some of the examples that we tend to see \[00:19:00\] how pagination works. So they can add custom pages in addition to their API reference. And in my mind, Mike SDKs and API docs are like peanut butter and jelly. They just go better when they're together. You can have 'em separately, but \[00:19:20\] it's really nice when your SDKs, yeah, when the SDKs are documented next to the API reference, it feels most natural as a developer that wants to go figure out, how do I integrate with this company's API. \[00:19:32\] ****Mike Bifulco:** Sure. Yeah. So are you able to share any stories about customers that you've had who are using Fern? \[00:19:39\] ****Danny Sheridan:** Absolutely. How about we talk about, I mentioned cohere already, so I'll pick another one. hOw about merge Dev to \[00:19:46\] give a little? Are you familiar with that, \[00:19:48\] Mike? \[00:19:49\] ****Mike Bifulco:** Past guests of APIs you and hate. We have, we recorded an episode with them maybe a year and a half ago at this point. \[00:19:54\] ****Danny Sheridan:** Oh, was that my hunch was probably either Shei or Gale, the CTO. \[00:19:59\] ****Mike Bifulco:** It was Gil. Yeah. We \[00:20:00\] chatted with Gil on, on APIs, you and hate. Yeah. \[00:20:02\] ****Danny Sheridan:** Fantastic. \[00:20:03\] ****Mike Bifulco:** In case someone hasn't listened though, what's the 1, 1 0 1 on merge? What do they do? I. \[00:20:07\] ****Danny Sheridan:** Yeah, merge is a fast growing company that developed a unified API that simplifies integration development. So there are a lot of popular SaaS products out there that you might want to integrate with. And I'll just give an example. If I wanna add ACRM integration to my app, I might want to connect with Salesforce data, or HubSpot data, or Pipedrive data. And so for me to go build integrations with each of those three is pretty burdensome on my engineering team. And instead I can go use Merge. And so Merge is the, is a unified API platform and is the difference between building one integration with Just Merge or building 10 with multiple different SaaS tools that have changing APIs. \[00:20:50\] ****Mike Bifulco:** Yeah, from, from what I remember, talking to GI Merge, they have a astonishing number of third parties that they unify into one place to call, at, at the time when we \[00:21:00\] chatted, it was like dozens upon dozens, and I'm sure it's grown since then. It's, it's definitely a big product. \[00:21:05\] ****Danny Sheridan:** WE can talk about the, the problem that merge came to us with, a merge customer uses the merge SDK and not a Salesforce or a HubSpot, SDK, \[00:21:17\] ****Mike Bifulco:** Right. \[00:21:17\] ****Danny Sheridan:** instead of having to deal with those \[00:21:19\] ****Mike Bifulco:** place. \[00:21:20\] ****Danny Sheridan:** Exactly. You call 'em all from one place. Well, that means that merge needs, I'm gonna call it a kick ass SDK. They need to have an extremely developer friendly, well document consistent across all languages experience. But they are API experts. They're not necessarily SDK experts, and so they teamed up with the firm team and they move really quickly. Merge is a fast moving company from first initial call with us to learn about what we do to actually going live with. Was a four week time period, so really impressive at the speed that they've been able to move. \[00:21:54\] ****Mike Bifulco:** Yeah, \[00:21:54\] my God. \[00:21:56\] ****Danny Sheridan:** at this point they've now released a node, Python, \[00:22:00\] Java, go, SDK, using Fern, and we're together collaborating on C Sharp and Ruby as the next languages that'll come out. It's pretty nice when you have a customer who pushes you to increase selection. \[00:22:12\] ****Mike Bifulco:** Yeah, definitely. Right. So, so with that then, it sounds like you've probably expanded the languages that Fern has delivered on since March as well, right? \[00:22:20\] ****Danny Sheridan:** We have \[00:22:20\] since we last talked, we've added Python and go and we will continue adding languages. It feels like after the top eight to 10 programming languages, there's a pretty big drop off. \[00:22:31\] ****Mike Bifulco:** That's probably fair. Yeah. \[00:22:32\] ****Danny Sheridan:** but I could see us using a good chunk of 2024 to get there to that top eight or 10. \[00:22:37\] ****Mike Bifulco:** Cool. . Okay, so, so Merge is a a fern user and they've, they've then delivered their SDKs in, in four weeks. Is is honestly a staggering turnaround for something like that. Like if you're listening to the show, you should hop over to merge.dev and like browse, scroll to the bottom of the page. \[00:22:54\] The footer of their page shows how many APIs they integrate with. And it is . Like conservatively \[00:23:00\] a hundred different, different APIs they pull into merge, which must mean that Fern is generating like mountains of API client library and documentation for them as well. \[00:23:08\] ****Danny Sheridan:** Yeah, we generate hundreds of endpoints for them every time that they generate code with us. \[00:23:12\] And one of the things that I want to call out is at, I was at AWS before this, as I mentioned, at AWS Andy Jassy used to talk to the team that the most successful implementations of AWS happened when it was top down. It's just too hard for cloud infrastructure to go bottom up. \[00:23:33\] You need a C-level goal, such as the CTO or CIO setting a goal that gets everyone on the same North Star of like getting to the cloud. And in this case, we see the same trend happening with SDKs. It's really hard to go bottoms up. It's really easy when a leader like Gil from Merge says, Hey, everyone. We are using Fern, we're going all in on offering every SDK language. They, \[00:24:00\] they provide and puts an engineer. Actually in our case, there was an engineer and an engineering manager that we got to work with to go live with each of their SDKs. And I think that's just the right approach as a leader to identify like a single threaded leader who's gonna be responsible for this SDK project going from first meeting to completion. \[00:24:20\] ****Mike Bifulco:** Yeah, that's the right rights kind of investment and clearly shows you know, a pattern for success too, and that it worked and, and that merges now user of Fern. That's, that's really exciting. So you, you now have a client library product and a Docs product. I. For people listening to the show. \[00:24:36\] We also mentioned before that you to, to get started with Fern, you need an open API spec. What does it look like to get started? How long does that take? \[00:24:44\] ****Danny Sheridan:** Yeah it, you install an NPM package, which is the Fern CLI command line interface. then you run a command called Fern Generate and that takes your open API spec and starts generating a list of generators that you specify. So it takes \[00:25:00\] about five minutes to get started. Pretty often what our customers figure out is that their open API spec is of pretty poor quality. I \[00:25:11\] mean that this is like a trend that you've talked about on this show a lot \[00:25:15\] and that you've written about of their, having an open API spec is not the same as having a high quality open API spec for your organization. \[00:25:23\] ****Mike Bifulco:** For sure. Yeah, it's aspirational. Definitely intentional, you know, a lot of, a lot of companies want to do it, and in practice it's tricky. It's a bit like starting with test driven development, like it's, it sounds like a great idea, but it's hard to get people to commit to it. \[00:25:36\] ****Danny Sheridan:** I'll give two examples of where I think open API specs that I've seen fall short just to, to make it tangible \[00:25:40\] for folks. The first is that they mention a bunch of different properties of an object and they don't add a description of what that property is. \[00:25:52\] And I don't know your API as well as, you know, your API, and so I need some help. \[00:25:56\] I need a human readable description. Age \[00:26:00\] means like is age in days and months and years. It's really helpful. You know, an integer for age isn't very clear to me as a developer what I should be put using there. \[00:26:07\] And so one of the things that we see as a room for improvement is adding description fields across the entire API spec. The second area of improvement are adding examples. really helpful as a developer to see an example JS om response, \[00:26:23\] ****Mike Bifulco:** Yeah. \[00:26:24\] ****Danny Sheridan:** like very high value. But in order for. Fe to know what a response looks like. We need companies in their open, API specs to include an example for each endpoint and the great, the great customers that we work with that really want the exceptional developer experience make that part of their, like their, they can't ship update API without improving, without updating their open API spec. \[00:26:52\] And then improving things like description fields and examples. \[00:26:56\] ****Mike Bifulco:** Yeah, I'd imagine that's a bit of a cathartic experience too, because it's effectively like \[00:27:00\] forcing people to invest in their own reduction of tech debt. You know, we, we all have this notion that we want to do a great job of these things, but frankly, like human readable descriptions that make sense and examples and things like that fall by the wayside pretty quickly when you're under the gun to get things done. \[00:27:14\] But that'll turn around if the tooling takes. Those things end. Multiplies the value of it for you. Right. And, and I think that's what Fern at least, is purporting to do here, and I think that's a really exciting offering. \[00:27:25\] ****Danny Sheridan:** Mike, do you see a world where LLMs could be helpful here, where it's like, help me add some descriptions and help me add some examples. \[00:27:31\] ****Mike Bifulco:** Sure. Yeah, yeah, yeah. That, that's definitely a place where I think people would be excited to you know, to to, to add that sort of ai magic on top of things. And maybe an, an interesting application of that too. I could see maybe even something where you're, you're chatting with your code base, like, Hey, how would I. \[00:27:47\] Do whatever it is, like take an ACH payment from someone in Japan. Right? Like, how do, how do I do that? What does that look like? And, and if your documentation is in order and your open API spec is well written, I think that's something that, that is not too far off in the future \[00:28:00\] for us. \[00:28:00\] ****Danny Sheridan:** You just touched on something that I think is just probably a billion dollar idea, so let's explore \[00:28:06\] it for a minute together. \[00:28:07\] ****Mike Bifulco:** yeah. Far away. \[00:28:09\] ****Danny Sheridan:** Today API documentation is like an encyclopedia. \[00:28:13\] ****Mike Bifulco:** Yeah. \[00:28:14\] ****Danny Sheridan:** It's got a bunch of informations and, and a bunch of pages, and it's your responsibility to go either look at the table of contents or look at the glossary, kinda like the search bar, and you have to go find what you're looking for, get to the right section, read it, and digest it. \[00:28:30\] ****Mike Bifulco:** Yeah. \[00:28:31\] ****Danny Sheridan:** I speculate that instead of an encyclopedia experience where you have to like go find the right page. You're gonna be able to ask a question to the encyclopedia. I mean, this is what SIR and Alexa wanted to be back then, right? I can just ask a question and it goes and finds the information and presents it to me. And I could see a future where you get to someone's API docs and there's a text input. And that text input allows you to ask a question like, I'm using Jango to call the \[00:29:00\] Stripe API as an example. Well, let's pick the merge dev API I'm using to call your API. I wanna migrate, I wanna use the HubSpot and Salesforce integration to go create new leads and it starts writing code for you, basically a sample app of how you would, and you get to share that, right? \[00:29:22\] It's gonna use the Python, SDK 'cause it's, it knows you're in a Python framework \[00:29:26\] and it's going to live, generate you code that you can start using that you would never find in their docs. It's actually combining some of the word description guides with the API reference with the SDKs that now like an LLM Power Docs understands all of those things to me. \[00:29:43\] The day that we can go and, and like you said, have a conversation about what are you trying to accomplish here at Docs? Congrats. You've landed. What are you trying to do here? That is not an experience that I've ever seen. \[00:29:55\] Have you? \[00:29:56\] ****Mike Bifulco:** No. And I think that the, those tend to be, for me at least, the most exciting \[00:30:00\] applications of this LLM technology is where, where it's a second brain that actually is a second brain, right? Like in the past few years people have talked a lot about note taking and self-organization and things like notion and. \[00:30:12\] Rome research and obsidian and, and being really good about documenting what you're seeing. But when the L LMS start putting that together for you and take the need to document yourself out of the picture and can become an educator for you, I think that's really interesting and really empowering. \[00:30:27\] And, and I can see lots of different people getting into development as a result of that too, and I'm really excited by that. \[00:30:33\] ****Danny Sheridan:** I, I'll make a speculation here. 'cause it's fun to guess about the future, isn't it? \[00:30:38\] ****Mike Bifulco:** No, always. Yeah. \[00:30:40\] ****Danny Sheridan:** I think that the number one consumer of API documentation in the future, we'll call it three years from now, is going to be LLMs. \[00:30:49\] ****Mike Bifulco:** Hmm. That's really interesting. Wow. Well, I guess we're gonna have to put it on the calendar to have you back on the podcast in three years, but I'd, I'd love to have you back sooner if, if that's something you're interested in. \[00:30:59\] ****Danny Sheridan:** I,\[00:31:00\] \[00:31:00\] ****Mike Bifulco:** Danny, let, let's do this. So tell me about if people are interested in jumping in and getting started with Fern, where's the best place to go? \[00:31:06\] ****Danny Sheridan:** the best place to go is our website, which is build with fern.com. \[00:31:11\] ****Mike Bifulco:** We will of course have that in the show notes along with all the other things we've mentioned here today. One other thing I, I like to ask guests on the show is is Fern growing? Are you hiring at the moment? \[00:31:21\] ****Danny Sheridan:** Fern is growing. We are looking for a founding backend engineer. This person would work with our team in New York City. It's an onsite role, and they would be responsible for helping us expand selection of our SDK generators. So we're looking for someone who's excited to be a polyglot that might jump within a week between Java, Python, and Ruby, and is excited about languages and has a high attention to detail. \[00:31:48\] ****Mike Bifulco:** Yeah. Very cool that that sounds like a great opportunity. \[00:31:52\] ****Danny Sheridan:** And, And, \[00:31:52\] lastly, Mike, I'll, \[00:31:53\] I'll share that \[00:31:54\] the salary range that we're paying for that role we publish, which is 130 to \[00:32:00\] $160,000 a year, in addition to a half percent to 1% equity in a growing startup. \[00:32:06\] ****Mike Bifulco:** Brilliant. How's that for Open? I'm, I'm, I deeply appreciate that. I think especially in a world where lots of people are squirming to find the right place for them getting to that discussion around like, what does compensation look like? Where do you want me to be and what am I gonna be doing? \[00:32:18\] It's really hard to get to and, and I think a lot of companies. Can and should aspire to what you just did there. We'll drop a, a link in the show notes as well to the job opening here too. And Danny, where, where's the best place to get at you? If people want to shout at you about SDKs or documentation, where, where can you be found online? \[00:32:35\] ****Danny Sheridan:** The best place to talk to me is actually going to the build with fern.com website \[00:32:40\] and scheduling a call. Would love to ch if folks listen to this and they're like, Hey, I'm ready. I'm ready to make my API developer experience more like stripes. That's, if that's the intention, that's the place. Best place to find me is just book a call. \[00:32:54\] Let's hop on. \[00:32:55\] ****Mike Bifulco:** Yeah. That is an opportunity not to be passed up friends, I think . Danny a, \[00:33:00\] apart from being a, a very exciting person to chat with, I think you're also super knowledgeable and it's been great to have you hang out. I'm looking forward to six months from now seeing how you've shaped the, the planet of API developer experience in that marketplace too. \[00:33:11\] Thanks for joining so much. I appreciate having you here. And looking forward to talking again, \[00:33:15\] ****Danny Sheridan:** Let's build APIs that people will love. \[00:33:19\] ****Mike Bifulco:** add away into it. All right, Denny, take care. \[00:33:21\] ****Danny Sheridan:** Cheers. ## Support APIs You Won't Hate When you become an member, you'll get access to members-only content while directly supporting our work. Your support helps us to keep making resources for the API community. [Become a member today](https://apisyouwonthate.com/#/portal) ### Popular OpenAPI Bundling Tools Compared URL: https://apisyouwonthate.com/blog/openapi-bundling-tools-compared-blog-post/ Last updated: 2023-11-13T11:00:59.000Z When OpenAPI or JSON Schema documents get massive or repetitive, the contents can be split across multiple documents (on the filesystem, URLs, in memory somewhere) and joined together `$ref`. These split up API descriptions can then be joined back together as one document, with `$ref` pointing to an internal location instead of an external location. This is called "bundling". ## Swagger CLI, JSON Schema Ref Parser, and Redocly CLI Bundling has been struggling lately, with the most popular tools mostly being abandoned or taken in-house. I was [begging for help](https://philsturgeon.com/bundling-openapi-with-javascript/?ref=apisyouwonthate.com) with the popular JavaScript [json-schema-ref-resolver](https://github.com/APIDevTools/json-schema-ref-parser?ref=apisyouwonthate.com) for years, and was entirely ignored - despite it getting 2,395,495 downloads a week. Thankfully [JonLuca De Caro](https://github.com/jonluca?ref=apisyouwonthate.com) has stepped in to take over, and that project is now under some amount of maintenance again. The json-schema-ref-parser package is a low-level JavaScript/TypeScript library for bundling `$ref` and you nee to be writing JavaScript to use it as there is no CLI. Another popular package for that is [swagger-cli](https://www.npmjs.com/package/@apidevtools/swagger-cli?ref=apisyouwonthate.com), which wraps the bundle logic with a CLI layer so you don't need to write any code. I was also responsible for maintaining this tool, but thankfully I was able to deprecate it after [reviewing Redocly CLI](https://apisyouwonthate.com/redocly-cli/) and discovering it did a far better job of everything it set out to do. The approach it takes to bundling lined up far more closely with what most people seem to actually want. ## Bundling Output Compared Bundling can mean anything. Grab all the stuff and squish it into a file somehow. This behaviour being undefined leads to everyone being upset when their unique expectations are not matched, so let's have a look at what these tools output and see which you prefer. ``` openapi: 3.0.0 info: title: My API version: 1.0.0 paths: /things: get: responses: '200': description: 'OK' content: application/json: schema: properties: data: type: array items: $ref: './schemas/thing.yaml' /things/{id}: get: parameters: - name: id in: path required: true schema: type: string format: uuid responses: '200': description: 'OK' content: application/json: schema: $ref: './schemas/thing.yaml' ``` The json-schema-ref-parser library would bundle by pulling the content in from external references, replacing one of the $ref's with a subschema containing that, then point other matching references to the same thing to that location, wherever it happened to be. ### Swagger CLI - Bundling ``` $ swagger-cli bundle -t yaml openapi.yaml ``` ``` openapi: 3.0.0 info: title: My API version: 1.0.0 paths: /things: get: responses: '200': description: OK content: application/json: schema: properties: data: type: array items: $ref: '#/paths/~1things~1%7Bid%7D/get/responses/200/content/application~1json/schema' '/things/{id}': get: parameters: - name: id in: path required: true schema: type: string format: uuid responses: '200': description: OK content: application/json: schema: type: object properties: id: type: string format: uuid name: type: string type: type: string enum: - type1 - type2 ``` This functionality was constantly reported as a bug despite working as intended. Having a $ref of `'#/paths/~1things~1%7Bid%7D/get/responses/200/content/application~1json/schema'` is generally fine for computers, it's perfectly valid, but if this is something humans are meant to look at it can be rather confusing. If tools are using that `$ref` to build a name, or you want your API Reference Documentation to have a "Components" or "Models" section, then it's not going to work. ### Redocly CLI - Bundling Let's compare how Redocly CLI handles the same API description document. ``` $ redocly bundle openapi.yaml ``` ``` openapi: 3.0.0 info: title: My API version: 1.0.0 paths: /things: get: responses: '200': description: OK content: application/json: schema: properties: data: type: array items: $ref: '#/components/schemas/thing' /things/{id}: get: parameters: - name: id in: path required: true schema: type: string format: uuid responses: '200': description: OK content: application/json: schema: $ref: '#/components/schemas/thing' components: schemas: thing: type: object properties: id: type: string format: uuid name: type: string type: type: string enum: - type1 - type2 ``` Lovely! It's created a components section, added a shared schema, given it a name (inferred from the file but without the `.yaml`), and pointed all instances of that same subschema to the shared schema. *chef kiss*. ## Ye Oldé Dereferencing Both tools also support "dereferencing", which is the rough alternative to bundling, where all $ref's are replaced with what they point to, regardless of whether they were pointing to external or internal locations. This makes a much much larger output, because its super repetative, and should only be used if you absolutely need to use some old busted tool which doesn't understand what a $ref is at all. ### Swagger CLI - Dereferencing ``` openapi: 3.0.0 info: title: My API version: 1.0.0 paths: /things: get: responses: '200': description: OK content: application/json: schema: properties: data: type: array items: type: object properties: id: type: string format: uuid name: type: string type: type: string enum: - type1 - type2 '/things/{id}': get: parameters: - name: id in: path required: true schema: type: string format: uuid responses: '200': description: OK content: application/json: schema: type: object properties: id: type: string format: uuid name: type: string type: type: string enum: - type1 - type2 ``` Ooof yeah you can see why that can be problematic for large documents. It's literally going to repeat stuff every time. That doesn't just lead to large file sizes, but its rife for tripping over circular references that definitively cannot be represented this way. ### Redocly CLI - Dereferencing I was expecting more of the same for Redocly CLI but was pleasantly surprised to see they've got a trick to make this better. ``` $ redocly bundle -d openapi.yaml ``` ``` openapi: 3.0.0 info: title: My API version: 1.0.0 paths: /things: get: responses: '200': description: OK content: application/json: schema: properties: data: type: array items: type: object properties: &ref_0 id: type: string format: uuid name: type: string type: type: string enum: - type1 - type2 /things/{id}: get: parameters: - name: id in: path required: true schema: type: string format: uuid responses: '200': description: OK content: application/json: schema: type: object properties: *ref_0 components: schemas: thing: type: object properties: *ref_0 ``` HA! Using YAML anchors and aliases they've made this smaller, but again this is not nice for humans and circular references will probably just fail at a later point instead of breaking the dereference. Regardless, this is better for the majority of cases where you'd want to use $ref, so long as you're using YAML and the tool you're punting it into understands anchors/aliases. ### Other Tools Redocly CLI is, as the name suggests, CLI only, but the same logic is available in the [openapi-core NPM package](https://www.npmjs.com/package/@redocly/openapi-core?ref=apisyouwonthate.com). It's still considered somewhat internal, but the safe bits are documented, and I hear they'll be putting some more effort to stabalise the API over time. The Redocly CLI bundling strategy is very similar to the way the Export feature works in Stoplight Studio, meaning if you're using Stoplight Platform or still have a copy of the rug-pulled Studio Desktop then you can get a pretty similar experience by clicking buttons. Unfortunately the way they've implemented that was an undocumented in-house [fork of json-schema-ref-parser](https://www.npmjs.com/package/@stoplight/json-schema-ref-parser?ref=apisyouwonthate.com), so once again it's a "hunt through code and figure it out" sort of situation. I tried to find some other tools, but the sample code for [udamir/api-ref-bundler](https://github.com/udamir/api-ref-bundler?ref=apisyouwonthate.com) was not working and I couldn't fix it. Python has a [OpenAPI CLI Tool](https://pypi.org/project/openapi-cli-tool/?ref=apisyouwonthate.com) but that was throwing exceptions running the hello world command. *If you know of any other bundling tools that you think people should know about, try them out on that sample OpenAPI and see what the output looks like. See if you can find any awkward edge cases and post a gist. Links in the comments, or post it on [our Slack community](https://slack.apisyouwonthate.com/?ref=apisyouwonthate.com).* ## Summary As I said in the recent review, Redocly CLI is really impressive, and it's approach to bundling is a god send for users, and for me alike as I can deprecate Swagger CLI with a "hey check this out" instead of an abrupt rug-pull. Generally it's lovely to see tooling vendors stepping up and making awesome, open, free, reusable tools that are free from walled gardens. It makes sense for the people being paid to make tools to be the ones maintaining tools, and so long as they can keep the right balance between open-source and paid stuff then they should be able to keep these tools stable and growing for years to come, instead of relying on open-source-only maintainers running themselves into the ground, suffocating under a deluge of obscure edge case demands. Let me know how you get on with Redocly CLI in the comments, and if you're going to battle on with Swagger CLI do let me know why. 😅 ### Meet Redocly CLI: The Modern OpenAPI Sidekick URL: https://apisyouwonthate.com/blog/redocly-cli/ Last updated: 2023-11-10T13:33:44.000Z Redocly CLI is a brilliant new tool from the folks who made ReDoc, the first beautiful API reference documentation tool powered by OpenAPI. This CLI tool goes a lot further than documentation, and helps with “linting” (automated API Style Guides), and solves the biggest problem that I had previously been lumbered with trying to maintain: bundling! ## History Back in 2016 I built [Speccy](https://github.com/wework/speccy?ref=apisyouwonthate.com), with the help of the excellent Mike Ralphson who built OAS Kit which handled some of the internal logic. Speccy was the first CLI linter that would take programmable rulesets, arguably allowing you to create automated[ API Style Guides](https://apisyouwonthate.com/blog/automated-style-guides-for-rest-graphql-grpc/), a powerful component of any API Governance program. It also wrapped ReDoc, which are the time didn’t have a CLI. Then to make supporting multi-file documents easier it would bundle them up, and you could export that document with a bundle command. Speccy was abandoned by WeWork shortly after I left, and the void that created was partially filled by Spectral, partly swagger-parser, and various other piecemeal bits and pieces made by various teams with various support for OpenAPI v3.0 or v3.1. Redoc CLI is here to restore the one-tool to help you do all the most common things, without needing to login to some web interface or muck around with JavaScript code just to preview some OpenAPI. Let’s talk about linting, previewing, bundling, and something new: splitting. ## Linting When I joined Stoplight it was brilliant to see they’d secretly been working on a souped up rewrite of Speccy which could do infinitely more than Speccy’s linter could. They dropped the CLI docs and bundling as other tools were handling that, leaving Spectral hyper-focused on linting. As powerful as Spectral became, it can be pretty complicated to built rulesets. I battled through making rulesets covering [APIs You Won’t Hate](https://github.com/apisyouwonthate/style-guide?ref=apisyouwonthate.com), [URL Versioning](https://github.com/stoplightio/spectral-url-versioning?ref=apisyouwonthate.com), [OWASP Security](https://apisyouwonthate.com/blog/securing-apis-with-spectral-owasp-ruleset/), but it was complicated even when I had access to the core developers. I favoured the [NPM-module approach](https://apisyouwonthate.com/blog/distribute-spectral-style-guides-with-npm/) because it allowed me to [shim together a test suite](https://apisyouwonthate.com/blog/testing-spectral-style-guides-with-jest/) that became a rudimentary development environment. Part of the complexity comes from the idea that Spectral could be a rules engine for any structured JSON/YAML data, meaning not just OpenAPI, but AsyncAPI, Kubernetes manifests, GitHub Actions, or anything else. Flexibility comes at a cost, and the cost here was the inability to use named identifiers pointing to specific bits of the OpenAPI document like Speccy did. Spectral uses a barely known standard called JSONPath which is like a combination of CSS and regular expressions. ``` # The headers of any 429 response $..responses[429].headers # Each security scheme $.components.securitySchemes[*] # A list of paths $.paths[*]~ # A list of names for all the header parameters $..parameters[?(@.in === 'header')].name # Mime Types for error responses $.paths[*]..responses[?(@property.match(/^(4|5)/))].content.*~ ``` That’s not ideal, but it gets bit worse… even if you are familiar with JSONPath, this used JSONPath-Plus. Weeeeell no it used a modified version of JSONPath-Plus that sometimes was not compatible… It all reminded me far too much of the [State of Markdown](https://philsturgeon.com/state-of-markdown/?ref=apisyouwonthate.com) (tl;dr its an absolute state). ReDocly CLI aims to simplify this with named identifiers for all the OpenAPI objects like `Operation` and `Response`, and as I was writing this they released AsyncAPI support objects too. Combining these named identifiers with filters that can help you get more specific is a simple yet powerful combination. These three filters are exceptionally useful: ``` filterInParentKeys: [get, put] filterOutParentKeys: [delete] matchParentKeys: /^p/ ``` With Spectral you need to arcane spells like this to target responses with a status code 2XX or 4XX: ``` given: "$.paths[*]..responses[?(@property.match(/^(2|4)/))]", ``` Using the Redocly linting logic you’d swap that out for something like this: ``` - subject: type: Response matchParentKeys: - /^2/ - /^4/ ``` The named `Response` object removes the need for rule builders to worry about the structure of the rest of the document (`$.paths[*]..resposnes`), and the match parent keys allows you to ignore bits you’re not fussed about. There’s still regex in there, but you don’t have to smush it all into a one-liner. I am excited with this new approach, and whilst I’ve not had a chance to try and test it to this limits, I can see a huge amount of potential here. Give it a go and see how far you get. ## Previewing My favourite way to spin up API documentation is pointing a CLI tool at an `openapi.yaml` and having it launch a server that’ll change as I work on it. If it can watch and hot load then even better. This is exactly how Redocly CLI’s `preview-docs` command works. ``` redocly preview-docs openapi.yaml ``` Then when everything is looking ship-shape, I want to be able to run a single command to have it build static HTML which I can dump on Amazon S3, GitHub Pages, Netlify, or anywhere else static HTML is happy. This is exactly how Redocly CLI’s `build-docs` command works. ``` redocly build-docs openapi.yaml --output=docs/index.html ``` Incredibly simple, and handles gargantuan API description documents like [GitHub Enterprise Server API](https://raw.githubusercontent.com/github/rest-api-description/main/descriptions/ghes-3.0/ghes-3.0.json?ref=apisyouwonthate.com) with ease. ## Bundling & Splitting Having one massive OpenAPI document with thousands (or tens of thousands!) of lines is annoying, hard to work with, and prone to conflicts if two people dare do anything completely unrelated to each other. To avoid this developers split their API descriptions into multiple documents, then link it all back together with `$ref`. Unfortunately there is incomplete tooling out there which does not support $ref properly or at all, and you need a workaround. That workaround is some form of bundling ([also known as resolving or dereferencing](https://apisyouwonthate.com/blog/resolving-overloaded-terms-for-api-specifications-descriptions-contracts/)). ``` $ redocly bundle api/openapi.yaml -o tmp/output-bundled.yaml bundling api/openapi.yaml... 📦 Created a bundle for api/openapi.yaml at tmp/output-bundled.yaml 105ms. ``` That one command seems to handle huge documents quickly, and the quality of the bundled YAML is lovely. It will put everything into named `components`, unlike some other bundling tools which… do less beautiful things. Bundling differences need a whole other post, but… suffice to say its good enough a human could look at it without scratching their head, which also makes it look a lot better in any OpenAPI documentation tool you throw it at. The opposite of bundling is splitting, and whilst you should not need to bundle then split in the same workflow, you may at different times find yourself needing both. Perhaps somebody has just written their first OpenAPI description and the document is massive, or perhaps they’ve [generated the OpenAPI from HTTP](https://apisyouwonthate.com/blog/turn-http-traffic-into-openapi-with-optic/) and it’s *huge*. Split will take that massive file, slice it up into a sensible folder structure, and link the pieces together with `$ref` just like I have done entirely manually a bunch of times before. I thought I’d try and trip Redocly CLI up by using the [GitHub Enterprise Server OpenAPI description](https://raw.githubusercontent.com/github/rest-api-description/main/descriptions/ghes-3.0/ghes-3.0.json?ref=apisyouwonthate.com) but it happily sliced it up. ``` $ redocly split ghes-3.0.json --outDir split/ghes/ code . 🪓 Document: ghes-3.0.json is successfully split and all related files are saved to the directory: split/ghes/ ghes-3.0.json: split processed in 31970ms ``` Genuinely impressive stuff there, I’ll be using that in the future instead of manually mushing YAML around and hoping I got it right. ## Summary Redocly CLI is a brilliant modern OpenAPI tool that does a lot, and does it well. There’s good intentions behind modular tools that do one thing and one thing well, but much of this logic is required and multiple tools end up being awkwardly chained. Redocly CLI is therefore a great replacement for [Swagger CLI](https://www.npmjs.com/package/swagger-cli?ref=apisyouwonthate.com) (which I have now deprecated), as it handles linting (validation but better) and does bundling in a far more useful way, plus the beautiful documentation with a live reload, and much more. So, head on over to [Redocly CLI](https://redocly.com/docs/cli/?ref=apisyouwonthate.com) and give it a whirl! ### Who Needs DELETE? URL: https://apisyouwonthate.com/newsletter/who-needs-delete/ Last updated: 2023-11-01T11:40:52.000Z An interesting [tweet](https://x.com/matthewcp/status/1716549522015310116?s=20&ref=apisyouwonthate.com) made the rounds recently about only using the GET and POST verbs. The main reason is `
` only supports these two. Taking this further, another [tweet](https://x.com/jaffathecake/status/1716761776044335121?s=20&ref=apisyouwonthate.com) states that GET and POST have meaningful behavior differences in the browser. The rest don't. An example was given of how to cancel a reservation. A DELETE request to `/reservations/id` makes no sense, and you should use a POST to `/reservations/id/cancel`. Going to be honest, I don't care. If it works for you and it's well-documented, go for it. However, semantics matter in programming, they matter in HTML, and they should matter for HTTP verbs. Browsers are not the only clients of API's. Personally, I would make a `PATCH` request to `/reservations/id` with the new status in the body. Although I am sure not everyone will agree with that. I wish we could all agree on some solid standards across our industry, but hey, nothing to lose sleep over. \-Alexander --- ### The API Round-Up Our monthly round-up of articles, news and updates from APIs You Won't Hate and around the web. ### [Bruno](https://www.usebruno.com/?ref=apisyouwonthate.com) An excellent little tool called Bruno was recommended to me this week. An open-source API client where collections become first-class citizens, co-located with related information and easily version controlled. It's a rather interesting tool and worth a look. ### [Automatic OpenAPI Spec Maintenance](https://www.speakeasyapi.dev/post/release-speakeasy-suggest?ref=apisyouwonthate.com) Check out Speakeasy Suggest, an AI-powered tool that automatically enhances OpenAPI specifications. Give it an OpenAPI document, and it automatically suggests fixes, applies them, and outputs the modified spec. You can use it via a CLI or GitHub action. ### [Baselime](https://baselime.io/?ref=apisyouwonthate.com) I have been building a lot of serverless and edge APIs recently and was looking for a good logging solution when I stumbled across Baselime. It lets you discover events from anywhere in your system and is geared towards serverless setups. I'm underselling it here. Please go check it out. ### [Hypermedia Factors](https://gtramontina.com/h-factors/?ref=apisyouwonthate.com) A collection of existing Hypermedia Formats. An open wiki of sorts to keep track of all of them. Contributions to keep the list up to date are welcome. --- ### APIs You Won't Hate The latest from the team at API's You Won't Hate. ### [Make Your API Idempotent, Avoid Ruining Clients Lives](https://apisyouwonthate.com/p/82e2f5e9-9306-49a2-afa4-9bc6a80263c7/) Phil Sturgeon highlights the value of idempotency in APIs based on a story of accidentally booking three hotel rooms. He advocates for Idempotency Keys in POST requests to prevent such costly mishaps, providing real-world API examples for clarity. ### [Video: Modern API Tooling for the Entire API Lifecycle](https://apisyouwonthate.com/blog/modern-openapi-tooling-for-the-entire-api-lifecycling/) Step into the future of API tooling with this video Phil Sturgeon recorded for Treblle's Hacktoberfest called "Modern OpenAPI Tooling," part of our recent series of OpenAPI tools for every part of the API lifecycle.[](https://apisyouwonthate.com/p/82e2f5e9-9306-49a2-afa4-9bc6a80263c7/) --- ### From Our Community Articles written and shared in our free [Slack community](https://slack.apisyouwonthate.com/?ref=apisyouwonthate.com). ### [How to Define Union Types in OpenAPI](https://www.speakeasyapi.dev/post/openapi-tips-oneof-allof-anyof?ref=apisyouwonthate.com) Nolan Sullivan explores the use of the `oneOf`, `allOf`, and `anyOf` keywords in OpenAPI to create complex, flexible schemas. See why experts recommend using `oneOf` for union-type object fields, `allOf` for intersection/composite objects, and advises against using `anyOf` unless necessary due to potential issues in code generation. --- Thanks so much for reading the newsletter. Our membership program helps support the community by covering production and hosting costs for our articles, podcast episodes, and newsletter. ## Support APIs You Won't Hate When you become an member, you'll get access to members-only content while directly supporting our work. Your support helps us to keep making resources for the API community. [Become a member today](https://apisyouwonthate.com/#/portal) Thanks so much for your consideration. Until next time! ### Make Your API Idempotent, Avoid Ruining Clients Lives URL: https://apisyouwonthate.com/blog/idemptoency-keys/ Last updated: 2023-11-01T08:12:40.000Z Idempotency is the idea that doing something multiple times should have no different affects as doing it once, especially important for actions that can get expensive, like booking a hotel room, or sending a payment. ## Idempotency in HTTP Different HTTP methods have idempotency baked in. GET, HEAD, PUT, DELETE, OPTIONS, and TRACE can all be executed multiple times without any unintended side effects occurring. That could mean being able to make multiple GET requests for a record without it suddenly vanishing, or doing something weird like sending emails to people. An API can still do simple things like logging a "viewed" event so you know that excuse making customer has seen their invoice and can't pretend their email isn't working. This is important because if a connection fails or a client's timeout kicks in then they should be able to retry without fear of the item being mysterious gone. You also would not expect a GET to delete something, PUT is idempotent by design, because the main difference from PATCH is that you are sending a complete document to be recorded entirely. The client says "here is the whole document, save that" so if you send the same request a second time it's still... that. Whether it updates the database all over again or notices that it's the same depends on the implementation, and if no triggers are being fired off then its fairly moot other than being wasteful of energy. DELETE is idempotent because if you delete something, then a second attempt to delete it means that thing has still been deleted. Some APIs will show a 404 error saying "that could not be found" which is a bit of a weird thing to show a client who's trying to delete something and experienced a brief connection wobble triggering a retry, because if the resource is deleted and the client is trying to delete it that should really be treated as a success. Those are all idempotent by design, but POST and PATCH are not idempotent by design; at least not by default. ## Using Idempotency Keys to avoid POST spam POST is often used to create things, but does not have to be exclusively for that. It can be used to submit form data, trigger events, run commands, things that if you executed them twice you'd expect to record the information twice. As somebody who has cycled through ridiculously remote parts of the world then tried booking hotels on [Booking.com](https://booking.com/?ref=apisyouwonthate.com), I've had more than a few problems, and one in particular had me buy three rooms in the same hotel. I kept booking a room, then it would say sorry that room is taken... so I'd think damn, ok, then try and buy another room. Same again... wtf, this hotel must be popular. Tries a third time... Nope! I gave up and checked into somewhere in person, and when I logged into the hotel wifi I received three emails letting me know I had successfully bought all three rooms... 🤣 I wasn't running Wireshark at the time, but I imagine something like this was going on: ``` POST /bookings { "userId": "phil123", "hotelId": "1234", "checkinDate": "2023-01-01", "checkoutDate": "2023-01-02", "roomCode": "single-room-ab1" } ``` That request would venture off out into the tubes, possibly one of many required to make the booking successfully, and at some point the [Booking.com](https://booking.com/?ref=apisyouwonthate.com) client would decide things were taking too long and give it a retry. That second response of "Nope its booked" would come in a lot quicker, because yeah to be fair somebody just booked it, because even if the client gives up on the request, that doesn't mean the server did! Here's how you fix that. ``` POST /bookings Idempotency-Key: random-madeup-thing-235325 { "userId": "phil123", "hotelId": "1234", "checkinDate": "2023-01-01", "checkoutDate": "2023-01-02", "roomCode": "single-room-ab1" } ``` If the client had made up a completely random string (usually a UUID v4) then a second request using this key should reply with the same response as the first. This would have avoided making another payment, because much like caching, the server would have skipped all the functionality included in the server, and simply shown me the response that was sent to the first request. The client could retry on my behalf several times until it worked, and if I didn't leave the page I could retry myself and it would be the same effect. If I popped off to try and book another room, it would come up with a new Idempotency Key, so it would know that I was trying to make a new booking. Stripe was one of the first APIs I noticed using [Idempotent requests](https://stripe.com/docs/api/idempotent%5Frequests?ref=apisyouwonthate.com). ``` curl https://api.stripe.com/v1/customers \ -u sk_test_4eC39HqLyjWDarjtT1zdp7dc: \ -H "Idempotency-Key: 5mUP59iveriphkIJ" \ -d description="My First Test Customer" ``` [Adyen use Idempotency Key](https://docs.adyen.com/development-resources/api-idempotency/?ref=apisyouwonthate.com) too. [Go Cardless](https://gocardless.com/blog/idempotency-keys/?ref=apisyouwonthate.com) use them. ``` POST https://api.gocardless.com/payments HTTP/1.1 Idempotency-Key: PROCESS-ME-ONCE { "payments": { "amount": 100, "currency": "GBP", "charge_date": "2015-06-20", "reference": "DOLLAR01", "links": { "mandate": "MD00001EKBQ412" } } } ``` They are a little different as they'll return a 409 Conflict on the second attempt, and let you know that the resource was already created. ``` HTTP/1.1 409 (Conflict) { "error": { "code": 409, "type": "invalid_state", "message": "A resource has already been created with this idempotency key", "documentation_url": "https://developer.gocardless.com/pro#idempotent_creation_conflict", "request_id": "5f917bf9-df56-460f-a165-15d9e77414cb", "errors": [ { "reason": "idempotent_creation_conflict", "message": "A resource has already been created with this idempotency key", "links": { "conflicting_resource_id": "PM00001KKVGTS0" } } ] } } ``` Ok.. seems fine. A client can see that and decide to go grab the resource id and show that as a success. Not as seamless but works all the same. ### Making PATCH Idempotent PATCH is a bit of a weird one, and honestly I had to ask the [APIs You Won't Hate Slack channel](http://slack.apisyouwonthate.com/?ref=apisyouwonthate.com) why people would want to use Idempotency Keys for PATCH. As always, they came through. Thanks [Evert Pot](https://evertpot.com/?ref=apisyouwonthate.com)! Generally with PATCH you're just sending a bit of JSON, and what happens is not entirely defined by default. Maybe your PATCH looks like this. ``` PATCH /something/some-id { "foo" : 1, "baz" : 3 } ``` Should the omitted "bar" property be left as whatever it is on the server, or set to null? Or zero? Who knows. You can use [JSON Merge PATCH](https://www.rfc-editor.org/rfc/rfc7396.html?ref=apisyouwonthate.com) to clarify that, which says "yep, leave bar alone". Either way, when your API resources are just a bunch of JSON properties and you're updating a few of them at a time, it's essentially idempotent anyway, because if you're setting "foo" to `1` and that happens five times for some reason, foo is still going to be `1`. Where idempotency gets important for PATCH is if you're doing atomic requests, with modifications like `"incrementBy": 1`, where the end result of that being run accidentally run multiple times is a value far higher than intended. If you're making atomic requests, each attempt to `incrementBy: 1` (or whatever) should have it's own idempotency key, so that if a retry happens it's going to short circuit future efforts and show the previous result for any subsequent requests. ## Implementing Idempotency Key If you're using any sort of web application framework that supports middleware, there's a good chance there's a pre-built package you can install, or at least some tutorial code to copy. - [Laravel PHP](https://medium.com/@toby.okeke/how-to-setup-idempotent-apis-in-laravel-f3924e872797?ref=apisyouwonthate.com) - [Rails](https://github.com/qonto/idempotent-request?ref=apisyouwonthate.com) - [Spring](https://blog.stackademic.com/understanding-and-implementing-idempotency-in-spring-microservices-d732c8bcdb78?ref=apisyouwonthate.com) - [ExpressJS](https://www.npmjs.com/package/express-idempotency?ref=apisyouwonthate.com) ## Optional Keys Seeing as idempotency keys are optional you can roll this out on your APIs tomorrow without effecting any existing API clients. You can simply document the functionality, send out emails letting people know the option is there, and rejoice as people start using it. Each time a client implements this functionality, there will be a decline in support tickets coming into your helpdesk, which is time and money saved for everyone, and it means I won't spend my entire night in a hotel trying to explain to [Booking.com](https://booking.com/?ref=apisyouwonthate.com) how timeouts and retries work in a bid to get three hotel rooms refunded. ### Modern OpenAPI Tooling for the entire API Lifecycle URL: https://apisyouwonthate.com/blog/modern-openapi-tooling-for-the-entire-api-lifecycling/ Last updated: 2024-04-12T15:46:11.000Z Step into the future of API tooling with this video Phil Sturgeon recorded for [Treblle's](https://www.treblle.com/?utm%5Fsource=apisyouwonthate) Hacktoberfest called "Modern OpenAPI Tooling." We're covering OpenAPI tooling that can help you out with every part of the API lifecycle. If you're not sure what any of that means, dive into the video and we'll get it all covered! Let us know what you think in the comments. If you'd like a deep-dive on any of these particular tools let us know. We have also covered a few Open API Topics recently: [Generate a TypeScript REST API client with FernGenerate TypeScript client code easily with Fern, an automated tool that builds SDKs for major languages, Postman Collections, and OpenAPI definitions. Avoid the slow and brittle manual process, and publish directly to package repositories like NPM.![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/size/w256h256/2023/03/apis-logo-square.png)APIs You Won't HatePhil Sturgeon![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2023/08/fern-typescript.webp)](https://apisyouwonthate.com/blog/generate-typescript-rest-api-client-fern/) [Turn HTTP Traffic into OpenAPI with OpticCapture real HTTP traffic from production or anywhere else, and create OpenAPI from it, for documentation, mocks, SDKs, or contract testing.![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/size/w256h256/2023/03/apis-logo-square.png)APIs You Won't HatePhil Sturgeon![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2023/03/optic--1-.webp)](https://apisyouwonthate.com/blog/turn-http-traffic-into-openapi-with-optic/) [Contract Testing a Laravel API with OpenAPIYour API does a bunch of great stuff, and your OpenAPI document tells everyone about all the great stuff that your API can do, but making sure those two sources of truth agree can be a bit of a struggle at first. Whether you followed the API design-first workflow and![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/size/w256h256/2023/03/apis-logo-square.png)APIs You Won't HatePhil Sturgeon![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/posts/spiderman-pointing.jpg)](https://apisyouwonthate.com/blog/contract-testing-apis-laravel-php-openapi/) [Securing APIs with the Spectral OWASP RulesetRecently I’ve been on a run of making new all powerful Spectral rulesets, but so far it’s been focused on the concept of automating style guides, with the concrete example of the APIs You Won’t Hate: API Style Guide showing how it can be done with NPM and tested with![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/size/w256h256/2023/03/apis-logo-square.png)APIs You Won't HatePhil Sturgeon![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/posts/securing-apis-with-spectral-owasp-ruleset/cover.jpg)](https://apisyouwonthate.com/blog/securing-apis-with-spectral-owasp-ruleset/) ### API Specs, Schemas and Standards URL: https://apisyouwonthate.com/newsletter/api-specs-schemas-and-standards/ Last updated: 2023-10-16T14:25:32.000Z Recently, I tangled with JSON:API and found myself in a love-hate relationship with this specification. While it offers superb rules concerning pagination, relationships, and includes, it also injects complexity into projects that sometimes feel unwarranted. I'm not advocating dismissing it outright, but my recent frustrations nudged me to ponder specifications beyond JSON Schema and OpenAPI. I questioned the RESTfulness of the API I was crafting—was it genuinely REST or merely RESTish? Why can't JSON be more intuitive, and does the Stripe API serve as a good model to emulate? I turned to our resident expert, Phil, for insights, incorporating past discussions and some stellar community-contributed articles. So, the next time you find yourself overthinking like I did, these articles might steer you towards clarity. Well, either that or a refreshing stroll outdoors might do the trick 😉 \-Alexander --- ### The API Round-Up Our monthly round-up of articles, news and updates from APIs You Won't Hate and around the web. ### [JSON API, OpenAPI and JSON Schema Working in Harmony](https://apisyouwonthate.com/blog/json-api-openapi-and-json-schema/) Phil explores how JSON API, OpenAPI, and JSON Schema can harmoniously work together in API development. JSON API standardizes API design specifics, while OpenAPI and JSON Schema describe the service model and data instances. ### [Making the Most of JSON:API](https://apisyouwonthate.com/blog/making-the-most-of-json-api/) Digging further into our archives for this article on JSON:API. Phil covers managing compound documents, filtering relationships, and handling data pagination to sidestep performance hiccups. Through real-world examples, it sheds light on fine-tuning API design to adapt to evolving data needs and client expectations. ### [Let's Stop Building APIs Around a Network Hack](https://apisyouwonthate.com/blog/lets-stop-building-apis-around-a-network-hack/) Wrapping up a trilogy of reflections by Phil on JSON:API, this blog casts a critical eye on the age-old habit of tailoring APIs to network constraints. It questions JSON:API's stress on compound documents to minimize HTTP calls. The piece advocates for a shift towards more precise resources in API design, tapping into HTTP/2's offerings for a streamlined, contemporary approach. ### [**API links vs keys**](https://cloud.google.com/blog/products/application-development/api-design-why-you-should-use-links-not-keys-to-represent-relationships-in-apis?ref=apisyouwonthate.com) An article from Google on using links over keys to denote relationships in APIs. It underscores that links pave the way for self-descriptive APIs, facilitating more straightforward navigation and a superior user experience. ### [RESTful JSON](https://restfuljson.org/?ref=apisyouwonthate.com) A simple spec/media type for adding links to JSON APIs and drawing inspiration from how Stripe, Github and others handle links in their REST APIs. I love this spec as it is simple to follow and doesn't create any headaches while at the same time making your APIs more expressive. ### [Build APIs You Won't Hate](https://apisyouwonthate.com/books/build-apis-you-wont-hate/) A good fallback, as always, is the book that started my journey into building API's. Although the book is a bit old now, it's still a solid read, and the chapters on pagination and HATEOS provide a reliable guide for how to format your responses. --- ### From Our Community Articles written and shared in our free [Slack community](https://slack.apisyouwonthate.com/?ref=apisyouwonthate.com). ### [API Programs in 2023: From Functional to Transformational](https://netapinotes.substack.com/p/api-programs-in-2023-from-functional) Matthew Reinbold outlines an "API Program Progression Model" with five stages: Functional, Reliable, Intuitive, Strategic, and Transformative. It depicts the evolution from providing basic functionality to revolutionizing companies by creating new revenue streams and redefining value propositions through APIs. ### [What API hackers need to know about EPSS](https://danaepp.com/exploit-prediction-scoring-system?ref=apisyouwonthate.com) I am considering renaming our community section to "Dana's Domain", given Dana's consistency in pumping out content. In this blog, Dana explains how the Exploit Prediction Scoring System (EPSS) helps predict vulnerability exploitation. By integrating EPSS in API hacking, security teams can prioritize remediations effectively, adding a proactive layer to their cybersecurity strategy. --- ## Support APIs You Won't Hate Thanks so much for reading the newsletter. Our membership program helps support the community by covering production and hosting costs for our articles, podcast episodes, and newsletter. It would mean the world to us if you'd [consider supporting APIs You Won't Hate with a paid membership](https://apisyouwonthate.com/#/portal). Until next time! ### What Not to Do in API Versioning URL: https://apisyouwonthate.com/newsletter/api-versioning-what-not-to-do/ Last updated: 2023-10-02T12:51:58.000Z Dive into the realm of API versioning with this month's newsletter. As APIs evolve, the necessity for a sound versioning strategy emerges, though the journey isn't without its challenges. Our expert, Phil, generously shares his insights on this issue—when he's not [planting trees](https://twitter.com/ProtectEarthUK/status/1708174465232761314?ref=apisyouwonthate.com), he's nurturing our API knowledge. The articles below cover common missteps in versioning and offer guidance to navigate through them. While Phil sets the versioning stage, I included additional articles to build on and refresh some of the concepts introduced in his initial post. Hopefully, versioning won't be an issue for any of you after you finish reading the articles. \-Alexander --- ### Monthly API Round-Up Our monthly round-up of articles, news and updates from the web. ### [My Version of API Versioning is Better than Most ](https://blog.treblle.com/my-version-api-versioning-better-than-most-versiong-being-done/?ref=apisyouwonthate.com) Phil discusses the hurdles of API versioning and advocates for upfront API design to reduce early version rollouts. He suggests using OpenAPI to engage potential consumers in the design process, thus minimizing the need for multiple version updates and saving time and resources for both developers and consumers. ### [API-First, API Design-First, or Code-First](https://blog.stoplight.io/api-first-api-design-first-or-code-first-which-should-you-choose?ref=apisyouwonthate.com) Phil's versioning article raises approaches to developing APIs and how an improved design process can fix many of the issues we face when creating them. Stoplight talks us through the different approaches to building an API in case you need a refresher. ### [Deprecating API Endpoints](https://blog.stoplight.io/deprecating-api-endpoints?ref=apisyouwonthate.com) Continuing with themes raised in Phil's versioning blog post, let's look closer at deprecating endpoints in an API. Leaning on Stoplight's catalogue of excellent API resources, let's take a closer look at how we can use OpenAPI and HTTP headers, among other things, to achieve this. It's no surprise that Phil also wrote this blog. ### [API Evolution for REST/HTTP APIs](https://apisyouwonthate.com/blog/api-evolution-for-rest-http-apis/) We have struck hard in this issue on the need to avoid versioning and that there are better approaches to changing your API. If this has left you with more questions than answers, luckily, we have another blog from Phil in our archives. Phil talks us through the evolution of an API and how to add changes without causing pain for your end users. --- ### APIs You Won't Hate The latest from the team at API's You Won't Hate. ### [Should an API use Semantic Versioning?](https://apisyouwonthate.com/blog/versioning-apis-semantically/) Phil put together a great post on how versioning numbers should work in your API. While this week's content has focused on moving you away from versioning, that is not always possible. Learn why major versioning is the best approach. --- ### From Our Community Articles written and shared in our free [Slack community](https://slack.apisyouwonthate.com/?ref=apisyouwonthate.com). ### [Finding Hidden API Endpoints](https://danaepp.com/finding-hidden-api-endpoints-using-path-prediction?ref=apisyouwonthate.com) Dana Epp discusses enhancing API security by identifying hidden endpoints through path prediction. By understanding standard API design conventions and exploiting them, security researchers can predict obscured endpoints, aiding penetration testing and security auditing. Dana is constantly sharing incredible blogs in our Slack group. Thank you, Dana. --- ## Support APIs You Won't Hate Thanks so much for reading the newsletter. Our membership program helps support the community by covering production and hosting costs for our articles, podcast episodes, and newsletter. It would mean the world to us if you'd [consider supporting APIs You Won't Hate with a paid membership](https://apisyouwonthate.com/#/portal). Until next time! ### Should an API use Semantic Versioning? URL: https://apisyouwonthate.com/blog/versioning-apis-semantically/ Last updated: 2023-09-27T17:00:37.000Z An [article I wrote for Trebble](https://blog.treblle.com/my-version-api-versioning-better-than-most-versiong-being-done/?ref=apisyouwonthate.com), and the video, cover whether or not you should use API versioning for an API. Ignoring that larer topic, if you *are* using API versioning, how should those version numbers work? If you're an experienced developer who's new to API development, it's natural to try and bring over concepts you've used elsewhere in the world of software. It's common to see people trying to use Semantic Versioning for API releases, like you might for an NPM package or some other software library. For an API this would mean deploying and maintaining multiple versions of the API, all taking up server instances, which use up resources, costing money and creating carbon emissions: - [https://example.com/api/1.0.0/\*](https://example.com/api/1.0.0/?ref=apisyouwonthate.com) - `https://example.com/api/1.0.1/*` - `https://example.com/api/1.0.2/*` - `https://example.com/api/1.1.0/*` - `https://example.com/api/2.0.0/*` - `https://example.com/api/2.1.1/*` - `https://example.com/api/2.1.2/*` - `https://example.com/api/2.1.3/*` Beyond costing money, what would be the point? If you've fixed a bug in 1.0.0, then you probably want API consumer to get that fix, without needing to communicate with everyone telling them to update the URLs in their code/config to the new 1.0.1 URL. # What about minor versions? This gets folks thinking they should just use two version numbers, which in SemVer would be major and minor. Minor changes are meant to be "backwards compatible", meaning you can add new functionality without breaking anything already in use. If an API has a v1.1 it might add some new resources, or add new properties to a response. - `https://example.com/api/1.0/trees` - `https://example.com/api/1.1/trees` - `https://example.com/api/1.1/units` Sticking with my current domain of fighting the climate/biodiversity crises, this is a change our API has recently seen. We've gone from tracking planted trees, to tracking various types of units, which could be meters of hedgerow, square meters of wildflower meadow, or square meters of peat bog re-wetted. If we were using major/minor versioning we'd have added units to v1.1 because adding things is backwards compatible, but we'd have kept trees the same because removing them in *not* backwards compatible. Whilst at first it feels sensible to use this major/minor approach, it is equally as odd as the major/minor/patch approach discussed above. If an API has `/1.0` and `/1.1`, but any code interacting with `/1.1` will work the same as if it worked with `/1.0`, then... why bother maintaining those multiple instances? That `/units` resource could have been added to /1.0 and nothing would have broken for anyone. ## Major Global URL Versioning With minor and patch versions providing no value, taking them away leaves you with the most common approach in API versioning: major global versions. - `https://example.com/api/1/` - `https://example.com/api/2/` Whether this approach should be used compared to something like [API evolution](https://apisyouwonthate.com/blog/api-evolution-for-rest-http-apis/) is up-for-debate, but if you are going with URL versioning, please stick to a major version. Minor and patch versions are just noise. ## More reading Our friends over at Treblle recently posted a great [primer on API Versioning](https://blog.treblle.com/api-versioning-all-you-need-to-know/?ref=apisyouwonthate.com), which makes a fantastic companion for this article. Add it to your reading list if this is something you're thinking about. ### The Bun is out of the Oven URL: https://apisyouwonthate.com/newsletter/the-bun-is-out-of-the-oven/ Last updated: 2023-09-16T15:16:56.000Z Wow, what a week. I don't know about you, but I spent most of it updating all my Node APIs to use Bun 😉. [Bun](https://bun.sh/?ref=apisyouwonthate.com) is a new runtime for JavaScript, which just hit version 1.0\. It's much faster than Node, and it's been interesting seeing all the comments and drama about this newcomer unfold. The advancements and simplicities Bun has made make me excited for JS's future on the server. Many APIs are built using Node, so I have included some balanced thoughts on Bun vs. Node and some new Bun tools for API development. I'm glad we finally made development in JavaScript easier by having three runtimes: NodeJS, Deno and Bun. \-Alexander --- ### Monthly API Round-Up Our monthly round-up of articles, news and updates from the web. ### [NodeJS Performance](https://adventures.nodeland.dev/archive/my-thoughts-on-bun/?ref=apisyouwonthate.com) [Matteo Collina](https://twitter.com/matteocollina?ref=apisyouwonthate.com), author of Fastify and Pino, shares his thoughts on Bun and his reflections on the overall performance of NodeJS. Matteo's response is balanced and points to some of the issues in comparing speed between Bun and Node. ### [Fast and Friendly Bun Web Framework](https://elysiajs.com/?ref=apisyouwonthate.com) If you want to try out Bun for building your next API but are worried about compatibility with your favorite NodeJs framework, check out Elysia. Made for Bun, so there is no need to worry about compatibility issues and with a focus on developer productivity. ### [Nitro](https://nitro.unjs.io/?ref=apisyouwonthate.com) Forgive the JavaScript-heavy issue this week, but so much is happening in the space right now. If you are looking for an excellent tool for deploying and rapidly scaling a TypeScript server, then maybe it is time to give Nitro a go. ### [Zod OpenAPI Hono](https://github.com/honojs/middleware/tree/main/packages/zod-openapi?ref=apisyouwonthate.com) You may have seen my mention of Hono a few weeks ago; they now have an OpenAPI plugin via Zod. You can validate your values and types using Zod and then use this to generate OpenAPI Swagger documentation. --- ### From Our Community Articles written and shared in our free [Slack community](https://slack.apisyouwonthate.com/?ref=apisyouwonthate.com). ### [Automating API Delivery](https://www.manning.com/books/automating-api-delivery?utm%5Fsource=twitter&utm%5Fmedium=organic&utm%5Fcampaign=book%5Fnwaiwu%5Fautomating%5F6%5F13%5F23) I can't believe I missed this, but Ikenna Nwaiwu from our community shared his latest book, which is now in early access. "Automating API Delivery" outlines the APIOps approach to streamline API development and governance through DevOps automation, ensuring faster and consistent results. ### Smartbear Acquires Stoplight URL: https://apisyouwonthate.com/newsletter/smartbear-acquires-stoplight/ Last updated: 2023-09-05T11:56:12.000Z Huge news this month as we learn Smartbear, the company behind tools such as SwaggerHub, will be acquiring Stoplight. Stoplight has been my go-to tool for documenting APIs and is integral to my API design first approach. I am curious about what this acquisition will mean for Stoplight and Smartbear. There are many reasons for this deal to happen, from growing API tooling together to potentially killing a competitor. The acquisition proposes a few opportunities if you read the rumor mill thread in our Slack channel. Smartbear now has an integration into Postman through Stoplight. Smartbear has never had API rulesets or style guides. There is an option here to replace SwaggerUI with elements. Looking at other deals like this, I see the main reasons for the acquisition will be so the investors can get a return. I don't mean to sound pessimistic, but that is how VC-backed companies work. The return for VCs will come first over any better API tooling. Stoplight and its board have a fiduciary duty to their investors. Fingers crossed for an explosion of growth in API tooling. \-Alex --- ### Monthly API Round-Up Our monthly round-up of articles, news and updates from the web. ### [Smartbear Acquires Stoplight](https://blog.stoplight.io/stoplight-secures-additional-funding?ref=apisyouwonthate.com) If you want a more upbeat overview of the acquisition than my dark intro 😂 you can find a more upbeat overview on the Stoplight blog. ### [The Proposal to Enhance Go's HTTP Router](https://benhoyt.com/writings/go-servemux-enhancements/?ref=apisyouwonthate.com) Go has always had a solid HTTP server built into the standard library; however, the request router leaves much to be desired. Its minimal implementation means you often reach for a library. Jonathan Amsterdam of Google's Go team has raised an active proposal to improve it. ### [Fighting API Bots with Cloudflare's Invisible Turnstile](https://www.troyhunt.com/fighting-api-bots-with-cloudflares-invisible-turnstile/?ref=apisyouwonthate.com) Are you struggling with bot abuse on your API? Learn how Troy"Hunt of "Have I Been Pwned" tackled Cloudflare's invisible Turnstile. Implementing Turnstile led to a remarkable 91% hit rate of solved challenges, ensuring that bots are shown the door while real users get speedy, unhindered access. Say hello to superior UX and goodbye to pesky bots! ### [Chrome Dev Tools Get An Update](https://developer.chrome.com/blog/new-in-devtools-117/?ref=apisyouwonthate.com) Chrome developer tools recently shipped a new update with a feature to override fetch XHR requests. Want to test out how your API changes may affect a client? Now, thanks to these new updates, you can easily demo the changes. --- ### From Our Community Articles written and shared in our free [Slack community](https://slack.apisyouwonthate.com/?ref=apisyouwonthate.com). ### [Why API Hackers Should Embrace Failure](https://danaepp.com/why-api-hackers-should-embrace-failure?ref=apisyouwonthate.com) Dana Epp's latest article underscores the importance of embracing failure in API hacking. Epp offers actionable tips for overcoming setbacks and continues to add valuable insights to our API's You Won't Hate community. ### The Developer Experience Revolution is here URL: https://apisyouwonthate.com/newsletter/developer-experience-revolution/ Last updated: 2023-08-15T12:15:15.000Z There's never been a better time to be an API and web developer. There are some great tools, libraries and frameworks out there, so I thought I would share a few that have filled me with joy the last few weeks: [Fern](http://buildwithfern.com/?ref=apisyouwonthate.com) is a tool to generate SDKs for your API based on your OpenAPI spec. I can not get over how excellent this tool is. We can all have Stripe-level quality SDKs without hiring thousands of developers. [Fastify](https://fastify.dev/?ref=apisyouwonthate.com) has become my go-to Node web framework for building APIs. Its plugins and ecosystem are incredible and a joy to code and maintain. Plus, they have a killer OpenAPI plugin. Finally, [Hono](https://hono.dev/?ref=apisyouwonthate.com) is a lightweight and ultrafast web framework that can be deployed to any provider focusing on the edge. It caught my eye a few weeks ago, and so far, all my builds have deployed without a hitch. If you're looking for cross-platform deployments focusing on the edge, look no further. --- ### Monthly API Round-Up Our monthly round-up of articles, news and updates from the web. ### [Spring Boot in 20 Minutes](https://codaholic.com/spring-boot-guide/?ref=apisyouwonthate.com) Now I know we all like to knock on Java as much as the next person, no? Just me, then? Well, now I work at Atlassian; I have had to pick up Java, Kotlin and Spring Boot, and you know what? It's great. If you get past the fact everything needs to be a class, quickly building reliable and scalable APIs is easy. Check out this intro in Spring Boot; hopefully, it will make Spring Boot and Kotlin a new go-to for building APIs like it is for me. ### [Hypermedia Systems](https://hypermedia.systems/?ref=apisyouwonthate.com) Not strictly API or backend-focused, but Hypermedia is at the heart of a good REST API. A great deep dive into what a Hypermedia system is and how it can be used to build modern web applications. It includes a forward from [Mike Amundsen](https://training.amundsen.com/?ref=apisyouwonthate.com#h.p%5F6AntBa6tTn%5FJ), so you know it's good. ### [Postman API Test Automation](https://www.youtube.com/watch?v=zp5Jh2FIpF0&ref=apisyouwonthate.com) This course from Free Code Camp will talk you through the basics of writing API tests, automating them and integrating with Postman. Over fifty lessons are in this video, so you should walk away with a good idea of how to test your APIs. ### [What is an API? A Beginner's Guide](https://www.devchronicles.io/what-is-an-api-a-beginners-guide/?ref=apisyouwonthate.com) Sometimes it's worth going back over the basics. This beginner's guide provides an excellent overview of an API and its role in development. This article introduces people to the concept of an API or a quick refresher on the foundation; it's a great share for colleagues who might need a *superfast* API 101 or seasoned devs who could use a little brushing up. 😉 --- ### APIs You Won't Hate The latest from the team at API's You Won't Hate. ### [Generate a TypeScript REST API client with Fern](https://apisyouwonthate.com/blog/generate-typescript-rest-api-client-fern/) Phil talks you through how easy it is to set up a TypeScript client with Fern. Producing a TypeScript client SDK with Fern is surprisingly straightforward. The team at Fern have put some incredible work into their tooling! --- ### From Our Community Articles written and shared in our free [Slack community](https://slack.apisyouwonthate.com/?ref=apisyouwonthate.com). ### [Flow Diagrams from OpenAPI](https://diagrams.criteria.sh/?ref=apisyouwonthate.com) I love the APIs You Won't Hate community. You're constantly creating things, and James Moschou shared what he has been working on recently, and it's so cool. Load in your OpenAPI spec and get flow diagrams explaining the API. James is looking for feedback, so go check it out. --- ## Support APIs You Won't Hate Thanks so much for reading the newsletter. We recently rolled out a membership program, which helps support the community by covering production and hosting costs for our articles, podcast episodes, and newsletter. It would mean the world to us if you'd [consider supporting APIs You Won't Hate with a paid membership](https://apisyouwonthate.com/#/portal). Until next time! 🧶 ### Generate a TypeScript REST API client with Fern URL: https://apisyouwonthate.com/blog/generate-typescript-rest-api-client-fern/ Last updated: 2023-08-03T12:39:01.000Z In this tutorial, **we're going to generate TypeScript client code for an existing API** to help users of that API avoid writing their own HTTP-level interaction code. Why? It's slow and brittle, which often means they're not leveraging the full power of your API. The time a user spends this interaction code could be better used getting their product built and integrated, leading to more business for you. Building client code (also known as Software Development Kits) is not much fun, especially after you've made a few hundred. Automated tooling can be brought in to help avoid this monotonous task, especially with the fantastic code generator [Fern](http://buildwithfern.com/?ref=apisyouwonthate.com). Fern can help us build SDKs in major programming languages, not just TypeScript, so we can set up one pipeline and get a bunch of SDKs out at once. It will also generate a Postman Collection and OpenAPI definition, keeping them automatically in sync. Fern can even release your code up to package repositories like NPM or Maven, so you don't have to. ## 1.) Install Fern It's written in Node, like all the best API tools seem to be. ``` npm install -g fern-api ``` Pop into the directory of the API you want to build an SDK for, and generate the Fern config. ``` fern init ``` That sends you off to GitHub to create an account. If you don't have a GitHub account it's worth making one as lots of developer tools use it as a login. You don't have to host your code there. ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2023/08/Screenshot-2023-07-27-at-6.34.57-PM.png) Pop back to the terminal and when the command is done it will have created a directory of `fern/` with some config, and an `api/` directory (meaning you can have multiple APIs if you like). ## 2.) API definition formats Fern can run off its own DSL called the [Fern Definition](https://buildwithfern.com/docs/definition?ref=apisyouwonthate.com#an-example-of-a-fern-definition), which you can write and maintain by hand. Fern also supports reading from OpenAPI. ### Starting without OpenAPI If you don't have an OpenAPI definition and you want to start with one, you can: - [generate it from HTTP traffic](https://apisyouwonthate.com/blog/turn-http-traffic-into-openapi-with-optic/) - design it with a convenient [visual editor](https://openapi.tools/?ref=apisyouwonthate.com#gui-editors) - generate it from your server code like [FastAPI](https://fastapi.tiangolo.com/advanced/extending-openapi/?ref=apisyouwonthate.com#generate-the-openapi-schema) makes it easy to do ### Starting with OpenAPI (my preference) Since I always have OpenAPI defined for my APIs before I've even built them, it makes sense for me to point Fern at my `openapi.yaml`. Delete the default `fern/api` and make a new one from OpenAPI. ``` fern init --openapi api/openapi.bundled.yaml ``` I then "bundled" my OpenAPI via Redocly CLI so that Fern wouldn't have to worry about reading OpenAPI that's been [split over multiple documents](https://blog.stoplight.io/keeping-openapi-dry-and-portable?ref=apisyouwonthate.com). ## 3.) Generate the TypeScript SDK Fern will generate TypeScript by default so we can ignore config for now and just run the generate command. ``` fern generate ``` The TypeScript SDK is generated in the cloud and then downloaded to the local filesystem by default, which is handy for seeing how it looks before going any further. Have a click around, see if it looks about right, and if you're a fan, we can start using it in our client code. It's a fairly safe guess to assume having the TypeScript SDK sat in the source code of the API itself is not very useful, so let's publish it off somewhere that your client application can use it. Publishing it to the main NPM registry might feel premature at this stage, so Fern has their own NPM registry which you can publish to without everyone seeing it. To use this we can change `fern/api/generators.yml` to look like this: ``` default-group: sdk groups: sdk: generators: - name: fernapi/fern-typescript-node-sdk version: 0.7.2 output: location: npm url: npm.buildwithfern.com package-name: "@green-turtle-fern/tree-tracker" config: namespaceExport: TreeTracker ``` That `package-name` needs to be set to `"@-fern/"`, which is a little magic feeling at first, but when you've done that each time you generate the client it's going straight up to NPM. No messing around. ## 4.) Using the Node.js client code In your client application - the application that will be using the SDK - you'll need to let `npm` know you're trying to install code from a non-standard repository. Create a `.npmrc` with the following content. ``` # .npmrc @green-turtle-fern:registry=https://npm.buildwithfern.com ``` Change `@green-turtle-fern` to `@-fern` and save the file. Now we can install the SDK via NPM. ``` npm install @green-turtle-fern/tree-tracker ``` Sorted. Now to use our sweet, idiomatic TypeScript SDK. ``` # src/run.ts import { TreeTrackerClient } from "@green-turtle-fern/tree-tracker"; async function main() { const treeTracker = new TreeTrackerClient({ token: ``, }); const uuid = "a40d4174-f8c6-486a-adff-69364e0c1d18"; try { const response = await treeTracker.map.getUnitPins(uuid); console.log({ response }); } catch (error) { console.log("unknown error", error); } } (async () => { await main(); })(); ``` Now if I run this simple TypeScript file, Fern should show me either an HTTP error, or a whole bunch of "pins", which are all trees my charity has been planting around the U.K. ``` npx ts-node run.ts ``` ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2023/08/Screenshot-2023-07-30-at-6.28.08-PM.png) There they all are! Next let's dig into those objects a bit and see what data we have available. ## Type Safe JSON Bodies One of the biggest problems I've had in the past working with APIs is that people will just grab random JSON from one place, bung it off to another, occasionally mix it in with other random JSON from somewhere else, and smash that off to some other third place. This litters the whole codebase with weakly typed data which can change at literally any moment, and applications break in wildly unexpected ways as that happens. Having TypeScript power the SDK solves a lot of that, because it's looking at OpenAPI for what the JSON should be doing, and requiring developers to work with that data in a type-safe way. It's not going to solve all our problems, but it's already made me notice a few undefined properties in my own OpenAPI definitions. The TypeScript generator makes handy type definitions like this: ``` export interface MapUnits { features?: TreeTracker.MapUnitsFeaturesItem[]; type?: string; } export interface MapUnitsFeaturesItem { properties?: TreeTracker.MapUnitsFeaturesItemProperties; } export interface MapUnitsFeaturesItemProperties { id?: number; name?: string; species?: string; what3Words?: string; } ``` Fern's TypeScript generator will also case your properties accordingly to best practices in the language, which will often differ from the JSON. That's fine, because this is TypeScript not JSON. I did get a bit confused when I was told `what3words` did not exist, but then my code editor suggested `what3Words` and all was well again. Using it is as simple as writing normal TypeScript, it even helped work with `.features` and `.properties` which are part of the GeoJSON standard and have nothing to do with Fern. ``` response.features?.forEach((pin) => { const { id, species, what3Words } = pin.properties; console.log(`Tree ${id} is a ${species} planted at ${what3Words}.`); }); ``` Really simple to work with, and it will last a lot longer (and fail a lot more noticeably) than if I was randomly passing JSON around. ## Final thoughts If you are looking for a client code generator, for TypeScript, Go, Python, or Java, you would be hard-pressed to find a better option than Fern. Their combination of simple CLI interface and the speed of their generation in the cloud means you can build multiple excellent SDKs quickly. Plus, the ability to sync to Postman Collections means you can take care of a whole lot of your "SpecOps" all at once. [Fern | SDKs for your APIImport OpenAPI. Generate client libraries. Backed by Y Combinator.![](https://uploads-ssl.webflow.com/63be6666d7bf1725a4257f82/63be66f16e0953a0e152170e_android-chrome-512x512.png)SDKs for your API![](https://fern-image-hosting.s3.amazonaws.com/Open+graph+image.png)](https://buildwithfern.com/?ref=apisyouwonthate.com) Head on over to the Fern website to find out more. ### 2023's State of the API Report URL: https://apisyouwonthate.com/newsletter/2023-state-of-the-api-report/ Last updated: 2023-08-01T13:52:40.000Z Postman dropped their 2023 State of the API Report. **75% of respondents agree that API-first companies are more productive and create better software**. REST remains the top architecture but has lost some ground. Meanwhile, GraphQL finally took SOAP's spot. Some interesting stats in this report; however, while it was a survey, some of the data comes from Postman usage. Not everyone is on Postman, so worth bearing in mind. The full report is worth reading; you can find it on the [Postman site](https://www.postman.com/state-of-api/?ref=apisyouwonthate.com#key-findings). --- ### Monthly API Round-Up Our monthly round-up of articles, news and updates from the web. ### [Akita Joined Postman](https://www.akitasoftware.com/blog-posts/announcing-akita-has-joined-postman?ref=apisyouwonthate.com) Postman has acquired Akita, a tool for drop-in discovery and monitoring of your API endpoints. A real success story for an API-focused product. If you're curious about the founder Jean Yang's journey, raising funds and the tech stack, she sat down for an interview with [The Pragmatic Engineer](https://newsletter.pragmaticengineer.com/p/building-an-an-early-stage-startup?utm%5Fsource=substack&publication%5Fid=458709&post%5Fid=135308341&utm%5Fmedium=email&utm%5Fcontent=share&triggerShare=true&isFreemail=true). ### [Web Integrity API ](https://arstechnica.com/gadgets/2023/07/googles-web-integrity-api-sounds-like-drm-for-the-web/?ref=apisyouwonthate.com) A proposal from Google to learn about a browser's user, ensuring their not a robot or the browser has been tampered with. While you can argue there are positives to this proposal, it comes with many downsides, and that's not including that it's coming from Google. Another sign the death of the open web may be upon us. As we get forced into even more wall gardens, at least we can laugh about the closed [issues in Github](https://github.com/RupertBenWiser/Web-Environment-Integrity/issues?q=is%3Aissue+is%3Aclosed&ref=apisyouwonthate.com). ### [Web Framework for the Edges](https://hono.dev/?ref=apisyouwonthate.com) So many deployment options can now run across edge networks however finding a JS framework that supports all runtimes and platforms can be tricky. However, the JavaScript ecosystem is coming full circle and implementing the same standards. Hono takes advantage of this allowing you to build web APIs and deploy across multiple JS runtimes: Cloudflare Workers, Fastly Compute@Edge, Deno, Bun, Vercel, Lagon, AWS Lambda, and Node.js. ### [The Case for 'Developer Experience'](https://future.com/the-case-for-developer-experience/?ref=apisyouwonthate.com) Jean Yang makes it into two of my links in this issue. Jean dives into the developer experience, covering the history and why it is essential. As developers of APIs, developer experience is at the heart of our work. Development is so much more than writing code. Give this article a read. ### [Event-Driven Architecture](https://encore.dev/blog/event-driven-architecture?ref=apisyouwonthate.com) Not everything is a REST API; sometimes, you need to build something completely different. Matt Boyle walks us through event-driven architecture and why you might need to create one. --- ### From Our Community Articles written and shared in our free [Slack community](https://slack.apisyouwonthate.com/?ref=apisyouwonthate.com). ### [API Design Influences Everything](https://www.apimatic.io/art-of-developer-experience/podcast-richard-moot/?ref=apisyouwonthate.com) Sid Maestre shared a podcast episode where he talked to Richard Moot from Square. A great listen covering API governance at Block and how they approach things like versioning and deprecation. Thanks for sharing, Sid. --- ## Support APIs You Won't Hate Thanks so much for reading the newsletter. We recently rolled out a membership program, which helps support the community by covering production and hosting costs for our articles, podcast episodes, and newsletter. It would mean the world to us if you'd [consider supporting APIs You Won't Hate with a paid membership](https://apisyouwonthate.com/#/portal). Until next time! 🧶 ### Tech Twitter is Gone. URL: https://apisyouwonthate.com/newsletter/tech-twitter-is-gone/ Last updated: 2023-07-15T12:02:07.000Z I miss tech Twitter, not for the drama, clickbait or people dunking on each other's language of choice. I miss tech Twitter because it was a great place to find what others were working on and see the exciting things being shared. A quick scroll once a day left me with some great articles and videos to watch. Not to mention the ability to find incredible role models. Since the takeover, many people have left, and the feed is a mess. I still get hints of tech Twitter alongside a cesspool of hate and garbage. Mastodon, BlueSky and now Threads don't seem to have captured the same magic, and the incredible people I learnt from aren't all there. It looks like finding content for the newsletter is going to be harder from now on. --- ### Monthly API Round-Up Our monthly round-up of articles, news and updates from the web. ### [Building APIs in Laravel](https://juststeveking.substack.com/p/introduction) Steve McDougall drops an introductory post to his series on building REST APIs in Laravel. The series will be released in stages to subscribers on substack, and once finished, Steve plans to turn it into a book. To quote Steve directly, "We will start with a CRUD-style API and look at the steps we might need to take to refactor this API to be something even Phil Sturgeon would be proud of." ### [REST vs GraphQL vs gRPC vs SOAP](https://blog.postman.com/how-to-choose-between-rest-vs-graphql-vs-grpc-vs-soap/?ref=apisyouwonthate.com) Bruno Pedro talks us through the different API styles and their trade-offs in the Postman Blog. Bruno presents a simple framework he uses for picking the best architectural style for what your building. If you choose SOAP, though, I will come for you. I have many scars from integrating government services that used SOAP, possibly due to the government outsourcing all IT talent. ### [Swagger Now Supports Latest OpenAPI Spec](https://smartbear.com/news/news-releases/swagger-open-source-tooling-now-supports-latest-op/?ref=apisyouwonthate.com) SmartBear has finally updated the open-source tooling for Swagger to support the latest OpeAPI spec. It took them some time, but great to see they have caught up, considering how many companies use Swagger to document their APIs. ### [Running Github Actions Locally](https://github.com/nektos/act?ref=apisyouwonthate.com) GitHub Actions have become the place to run tests, deployments and builds for many projects. Using `act` you can run them locally, getting feedback and testing changes before committing. ### [Testing the dark scenarios of your Node.js application](https://practica.dev/blog/testing-the-dark-scenarios-of-your-nodejs-application/?ref=apisyouwonthate.com) I know most of our community are PHP developers driving around in Lamborghinis, but Yoni covers some necessary tests your backend application should not be without. What if you have a zombie process milling around or a service times out? Yoni delivers a fantastic read and helps you build better applications. --- ### APIs You Won't Hate The latest content from the team at API's You Won't Hate. ### [An API to abstract everything, with Roy Pereira of Unified.to](https://apisyouwonthate.com/podcast/an-api-to-abstract-everything) On this episode of the podcast, Mike and Phil chat with Roy Pereira from Unified.to, who are building a service that gives dev teams one API to use for integrating with external services. It's as ambitious as it sounds. --- ### From Our Community Articles written and shared in our free [Slack community](https://slack.apisyouwonthate.com/?ref=apisyouwonthate.com). ### [How to exploit an API](https://danaepp.com/how-to-exploit-an-api-using-prototype-pollution?ref=apisyouwonthate.com) Dana Epp shared with our community this incredible blog he wrote on exploring an API using prototype pollution. Dana walks us through what prototype pollution is, how to detect it and how to do it. Well worth a read; thanks, Dana! --- ## Support APIs You Won't Hate Thanks so much for reading the newsletter. We recently rolled out a membership program, which helps support the community by covering production and hosting costs for our articles, podcast episodes, and newsletter. It would mean the world to us if you'd [consider supporting APIs You Won't Hate with a paid membership](https://apisyouwonthate.com/#/portal). Thanks so much! ### An API to abstract everything, with Roy Pereira from Unified.to URL: https://apisyouwonthate.com/podcast/an-api-to-abstract-everything/ Last updated: 2023-07-13T12:12:18.000Z ## Show Notes - [Unified.to](https://unified.to/?ref=apisyouwonthate.com) \- one API to integrate them all - twitter: [@unified\_api](https://twitter.com/unified%5Fapi?ref=apisyouwonthate.com) - linkedin: [Unified-to](https://www.linkedin.com/company/unified-to/?ref=apisyouwonthate.com) - Roy Pereira online: - [roy@unified.to](mailto:roy@unified.to) - [@roymap on twitter](https://twitter.com/roymap?ref=apisyouwonthate.com) - [linkedin](https://www.linkedin.com/in/roypereira/?ref=apisyouwonthate.com) - Use Promo code `APISYOUWONTHATE2023` for 3 free months on Unified's Startup Plan #### Transcript \[00:00:00\] ****Mike:** Hello and welcome back to APIs You Won't Hate. My name is Mike Bifulco. I am one of your co-hosts of APIs You Won't Hate. And today I am hanging out with my good pal, Phil Sturgeon. Phil, how are you doing today? \[00:00:11\] ****Phill:** Hello. I am good. I'm in the Netherlands and it is beautiful and sunny, and there are sailboats going by the window. Fantastic. How you doing? \[00:00:19\] ****Mike:** There we go. I'm doing good. That means I can update the map I have on my wall of where is Phil today. With a finite location for once, which is always a, a positive thing for me. I'm, I'm home in North Carolina right now and have been en enjoying a fairly stable few weeks of, of life at home for once, which has been really nice. \[00:00:34\] Happily, we are joined today by a new friend and guest of ours, Roy Pereira from unified.to. Roy, it's really nice to have you here. How are you doing? \[00:00:42\] ****Roy:** Oh, I, I'm great. I'm in Toronto. It's a little rainy today, but quite nice spring. \[00:00:49\] ****Mike:** Gotcha. Yeah. I've been to Toronto a handful of times and I'm always struck by both how Unbelievably nice people are to me, but also like needlessly maybe needlessly from my American perspective, but wildly \[00:01:00\] helpful people are, I think I must always look lost when I end up in Toronto and someone has always pointed me in the right way in a way that's like disarming and, and shockingly kind to me. \[00:01:08\] ****Roy:** Yeah, that's typically a Canadian thing. But it's funny cuz if you're from the rest of Canada, they'll say that Torontonians are brew. So it's, it is a spec. \[00:01:16\] ****Mike:** Well, fair enough. Like all things I suppose. So Roy why don't you start a little bit by telling us about yourself. And of course we wanna talk about Unified, but I'd love to hear about your career how you got to where you're at today, and the story of how Unified came about. \[00:01:29\] ****Roy:** Yeah, for sure. So I'm I guess what you call a serial tech entrepreneur. I like starting companies. I've started about five of them. It, it really started right after university. I was taking computer science university. I got really bored. I wanted to go change the world. I dropped out and I started a startup and realized after a couple of years that I had no clue how to run a business. \[00:01:52\] So luckily I joined another startup, a real startup. This was during the.com era. We eventually got \[00:02:00\] acquired. That's when I also moved from coding into more product management as well. So that was really a great eyeopener. Yeah, I went to the, went to the valley, like a lot of tech people hanged out there. \[00:02:12\] Came back to Canada. And, and then actually I did a bunch of publicly traded companies, which we won't get to. I'll try to forget about that. But went back into startup land, which is where I really love. And of I've been here ever since. I'm a very technical founder and CEO. But I have a pretty big range, but I love tech and this is how I typically start, I typically start about thinking of what's what I need in the, in the market that's missing. \[00:02:40\] You know, that's the best way to actually start a company. Right? \[00:02:44\] ****Mike:** Sure. Yeah. So I guess that brings us then to the, the logical next step is what, what are you building now and, and what was the question you were asking that you're starting to answer? \[00:02:52\] ****Roy:** Yeah. It's interesting how I got here because when I look back at all of my startups, I really think I'm a one hit wonder.\[00:03:00\] I love abstracting data and abstracting APIs and connecting different APIs to do stuff. And each company does different stuff, right? I I, I, I like to do different industries and so forth. \[00:03:12\] And, you know, I, I've done ad tech, advertising tech where I've connected different ad servers together. The last company was scheduling and we needed a bunch of data to schedule meetings with people. And so we did a bunch of integrations with our customers accounts like CRMs and so forth. And so I, I keep doing the same sort of architecture in terms of building out these integrations to third party APIs. \[00:03:39\] And so I my last two companies were acquired and, and then the last one I was working for the acquiring company and I was thinking about what I wanted to do next, and I wanted to build out yet another SaaS application. This type was gonna be for salespeople to do something the relevant order was gonna do, because I started thinking about all the data that I would need to actually build this, this product \[00:04:00\] and data for my customers accounts. \[00:04:02\] And it, it really upset me cuz I just had done a bunch of these integrations, not just at the last company, but the previous company before that too. And I was like, I can't do another Salesforce integration. I just can't. And I, I just hit a wall. And, and Salesforce was the one that hit me hardest. It was like, no, I can't do that. \[00:04:21\] It's just like getting paid me every time I look at that API documentation. And so I actually put it on hold and I said, you know what? I'm just gonna enjoy my summer. I'm not gonna even think about that as too difficult. But then I started talking to colleagues, CEOs and CTOs and they were telling me how much of a pay that is actually. \[00:04:41\] And I was like, I know, I know it's a pay. But I talked to a CEO that basically said, it's not just a pay, it is a revenue limitation. I can't get my engineering team to build another integration to say HubSpot or some other crm. And our revenue has been impacted \[00:05:00\] greatly. And that's when the light bulb moment hit me. \[00:05:03\] And that's when I went back in. I said, this is the product that I need to build. I need to build a developer tool. That allows SaaS applications to easily build in these third party customer facing integrations. Because in today's world, it's all about data and the data's everywhere. And you don't own that data. \[00:05:24\] You are not the source of truth, right? And so there are existing sources of truth out there, the, the sales forces of the world and sales world workday for HR and so forth. And so the pain that I was feeling in terms of building a new company I turned that around and I said, I'm going to productize this concept and allow other apps among the other SaaS companies to easily integrate this unified API into their infrastructure, their product infrastructure, so that they can offer these customer \[00:06:00\] integrations without spending a ton of time and money. \[00:06:03\] ****Mike:** I think that is the sort of thing that if, if you've built things from scratch or been on teams that have had to develop new arms of a product, you find yourself answering the same questions over and over. And maybe a good litmus for that is are you adding the same, like, am I adding 12 a p i key environment variables to my project? \[00:06:19\] It's probably scratching an itch that's, that's somewhere similar to what you're doing at Unified. And in the end, I think all of the, I don't know, pick you, you pick your choice. CRM management, mailing list management error management, all of those things tend to feel the same. The APIs are similar, but they each have their own quirks. \[00:06:35\] It sounds to me like the goal of Unified is to give people the opportunity to do that without having to think about each API individually. Is that more or less right. \[00:06:44\] ****Roy:** Totally, and we actually have that written. It's like we should get tattoos on that because that is our ultimate goal, is to not have our customers, the developers read third party API documentation. So we want to unify as much as \[00:07:00\] possible for the greatest amount of use cases so that our customers only use us, only look at our API documentation, our SDK documentation, all of that, and never have to go and read a Salesforce API documentation or Workday API documentation and not really care what the difference between HubSpot and Salesforce is. \[00:07:21\] Because in, in a unified a p I environment, that should not matter. \[00:07:25\] ****Phill:** So I was just having a look at this and that, that, that felt weird in itself, that I actually, you know, prepared Sure. Podcast styling. But I went through and had a little play around and it was well helpful. I, I couldn't quite visualize how this is gonna work, right? Like, oh, it's an API that's distracted. \[00:07:40\] Everything else ever Is that, is that helpful? But looking, looking at the way you set it up, you kind of got integrations just like it's Zapier or something like that. \[00:07:48\] ****Roy:** It, it's called Zapier actually, cuz it rhymes with happier. I was told. \[00:07:52\] ****Phill:** Yeah, there you go. I've heard Americans say Zapier and I always what to complain about that, but Fantastic. But yeah, you kind of set up your integrations and the coolest \[00:08:00\] thing I noticed was that it was like, do you want to use your owno or tokens or do you want to use ours? And I was like, oh, I could like integrate with discord, for example. \[00:08:08\] Without, without actually having to go and register an application and set up all of my config and do all of this stuff. I could just use your keys. And then what? So when the users of my application want to connect with Discord, there's a little button that you've generated with a single line of Gerald trip to you in bed, you click on that and it opens up the whole flow of like, Hey, why don't you go and connect with Discord using our tokens? \[00:08:31\] That, that just cuts out so much time. Yeah. \[00:08:35\] ****Roy:** Yeah, we did that because we, with that time to the the aha moment is super important in any, any product, but as a developer, You have so many choices, you are Googling, how do I, you know, do x and you don't have a lot of time because you're on a sprint, right? And you have like a two week sprint cycle. \[00:08:54\] And so if you can get to that, ah, this does what I need to do. And so what we \[00:09:00\] wanted to, to allow our developers customers to, to do is to utilize our, our unified a p i to test it. And so we actually went out and we got all of our AWA credentials, it's called technically so that they can actually use ours. \[00:09:15\] Would they launch their product with ours? No. And in fact, you can't. Or you shouldn't. But it is a great way to test and to play around with the system. Most developers are great developers cuz they play with stuff right? Until they, they're comfortable with it. So that's what we wanted them to do. \[00:09:34\] ****Phill:** Yeah, exactly. I mean, if I'm, if I'm trying to sit around working out, I mean, you were just talking about when you've done 10 Salesforce integrations in a row, but when you are that person who's just start starting to try and work out what CRM to use and you've gotta go and try out 10 different CRMs you've gotta set up Oh, or a credential for 10 different apps to see how they go. \[00:09:50\] That is incredibly annoying. And its, \[00:09:52\] ****Roy:** It is, setting up o wall credentials is one of the worst things. That has not been fixed, by the way, \[00:10:00\] on. I, I think at some point we're gonna have to take a look at that. Or the industry's gonna have to come up with standards. It's, it's horrible and we get quite a lot of support calls from customers of developers who are sort of stuck, even though we have how two articles on it. \[00:10:14\] ****Phill:** yeah. Yeah. I mean, yeah, for sure. There's so, so much standardization involved in too, as, as to how the flows work and different processes work. But like, what's the, what's the user interface for it? Ah, I don't actually got out. It's Yeah, some, I mean, the way I've generally kind of gone about this problem in the past is using kind of installable packages when they exist. \[00:10:38\] So for example, there's things like Omni or that can help my specific application in Rubion rails. You can integrate with a bunch of different like Facebook and Twitter and all these different old login things and it'll kind of generate some views for you. So you get a bit of interface kind of made up. \[00:10:56\] Or we were talking on a recent episode about how there's lots of different, \[00:11:00\] like geo code APIs and you can find a, you can find a Geocoder php and it's got braver for 10 different, you know, APIs, things like that. There, there's been attempts to abstract certain, certain verticals all over the place. But yeah, there's pros and cons to having that as a SAS itself versus a bit of software that you are, you are controlling, but. \[00:11:20\] Those are very niche, you know, or, and geocoded, like what, what sort of API is a uron abstract? \[00:11:28\] ****Roy:** Yeah, it's a really good point. Phil. So there's been I think some unification of APIs in the past that we've seen authentication for sure. So you have Okta Off Zero, you have Firebase you have superb base. They're trying to unify them somewhat, and I think authentication's pretty well done. \[00:11:45\] I mean, it's a massive market. Okta is like 2 billion revenue in your year just for authentication. But if you go back a little further, you actually see now I'm blanking here. Plaid. There it is. Plaid unified API for \[00:12:00\] financial services, for transactions. How they were getting the data is a little different than what we're gonna talk about cuz we're all talking about APIs. \[00:12:08\] A lot of times Plat was screen scraping sites bank sites, but what they represented to their customers, I know but their, their representation to their customers was a singular unified a p i. And I thought that was really, really interesting and smart. Of course integrating into, say B2B applications like sales and, and HR applications has been done in the past. \[00:12:35\] I wouldn't call them a unified api I that much. There was a couple of companies like Cloud Elements and some others that have tried to do this in the last 10 years or so. But that unification was sort of lagging. And the issue there again is if a developer has to both implement your solution, but also read the p i documentation of the integration, And what sort of value are you really adding \[00:13:00\] there? \[00:13:00\] So I, I think pushing it to it's, it's its maximum in terms of unification is super important for us. So to answer your question though, Phil, so we're really targeting on two, what we call clusters categories. The HR space with r i s performance management directories and ATS applicant tracking systems for recruiting. \[00:13:23\] So that cluster is quite important for software companies that target employee benefits for example, or anything around an employee or anything around a candidate. And that seems to be a very large market around the world is about 15,000 or so software companies that, that do target that. \[00:13:41\] And it's also a really interesting industry because there are clear. Sources of truth, these platforms that store the employee data, especially employees. So Workday, s a p y d P, all of those big older software companies that have the lions show the market. \[00:14:00\] And when you talk about, say, older or more established companies, let's call them more established, you typically are talking about an a p i that maybe isn't as modern as it should be. \[00:14:12\] So it's very difficult to interface with, to, to integrate into your application and then to monitor and to maintain it. So that's one cluster that we target. We also target sales and, and, and marketing support as well. Those are also very important categories. There's quite a lot of sales software out there. \[00:14:31\] If you think about sales enablement or sales cadences reaching out to prospects and so forth. Mailing list, software, all of that. Trying to get that customer to go through your funnel, down to convert and to upgrade. So those are the two that we target today. And of course, our architecture allows us to expand into others. \[00:14:53\] We've had lots of conversations, you know truck shipping logistics for example, comes up \[00:15:00\] sometimes and it's like, oh, that may be a little bridge too far right now. Well, we are a small company. We wanna focus. \[00:15:06\] ****Mike:** Yeah. Roy, I'm really curious in hearing a little. About the story of how things got started. So it seems to me that one of the challenges with building a company like this is if you're going to unify APIs, you need to start with an MVP that tackles quite a bit of surface area From an implementation standpoint, what did the very first version of unified look like? \[00:15:24\] Where, where did you start in this mess of unifying APIs across all sorts of industries? \[00:15:30\] ****Roy:** Yeah. You know, it's funny, we started with the architecture. So we're actually just five months old. We started in January-ish of this year, 2023, and we started with the architecture. We, we did the hard work ahead of everything else. And it was just based off of experience, you know, what did we do in the past and past companies? \[00:15:52\] What worked, what didn't work? You know, we did some research into how much time we spent on integrations in some of the latter last companies. \[00:16:00\] Not quite a lot. And it wasn't just building, building actually was, I think the easy part in terms of time management. It was actually maintaining them. APIs change all the time. \[00:16:10\] Things break, sometimes the API documentation isn't actually what the API has. And so there's all sorts of issues there. Oh, a p i keys expire and off credentials expire and so forth. So so we basically started with what do we know worked, what, what didn't work? And we built the core architecture. \[00:16:32\] And what we found was it took a long time and we only really started building integrations in the last couple of months, to be honest. And those have been added very quickly now that we have the architecture. And so now we have 92 integrations that we've built out in the last. Two, three months. We're actually getting faster and faster at building them, but we also created, again, the, the core to also monitor and to manage them \[00:17:00\] after they're launched and after they're connected to a customer's account. \[00:17:04\] And so that's super, super important to us because we don't want it to break and not know but also to our customers because they rely on, on our stuff working, right? And, and if it doesn't work, then why are they using us instead of having an internal team, for example. \[00:17:19\] ****Phill:** Yeah, this is something we were talking a little bit before we hit record and it was a combintion and we should have just hit record. But I was talking about how I feel like there's kind of, whether it's two binary types of API developer or a bit of a sliding scale I was talking about how there's kind of the people that. \[00:17:35\] Just don't really care all that much about APIs and want to write the quickest, shortest, briefest happy path they possibly can, where they just get the thing and smash it through Jason's decode and hope for the best. And then there's the people who are often ferry places about APIs and want to build all sorts of wrappers around it so they've got a soft dependency and make that into a mapping service instead of talking to a map and get very kind of complex \[00:18:00\] about, about the architecture. \[00:18:01\] And I'm wondering which end, which type of customers you are, you are aiming to target there? Because I feel like neither them, them be interested even though all of them can benefit from it. Like how do you wedge in against those sort of people? \[00:18:16\] ****Roy:** Yeah. You know, we're all developers, right? And all developers feel like they can build it, right? Product manager comes in the room and says, Hey, I need X. And they're like, yeah, I can build that. No worries. And even as you get more senior, you still feel like you could build it. I think maybe you have maturity and you say, well, maybe I'm gonna look at alternatives, build versus buy kind of thing. \[00:18:37\] And so what we're finding our customers are the type that they've built it. So they know how much of a pain it is to build, but also the maintain. And really, again, that maintain part is really the, the thing that hits you, the tech debt and then the fact that you don't have time to build features, features in your product that are differentiate your product. \[00:18:59\] \[00:19:00\] That's really killer cuz then you're disappointing your product manager, your ceo, your CEO's complaining of L lack revenue growth and so forth. So we typically get those types of CTOs or developers coming to us, and what we're finding is that they want to expand. So they just got asked to build x, y, Z integration that they don't have. \[00:19:21\] They have like maybe one or two. And so they need to add in because the customer requests it. And so obviously the CEO wants it, salespeople want it. But they'll add that integration. But then they also will go back and replace their integration that they built natively as well, because once they get comfortable with the fact that this actually works better, the architecture is better done than their quick architecture that they did or lack of architecture. \[00:19:49\] And so they are, they're really happy. Like you can actually see like a sigh of relief, especially when you're having these video calls, you actually see their faces and, oh, like, oh my God, I don't have to think about that \[00:20:00\] anymore. And so to answer your question, yes, they are like the typical developer that loves to build stuff, but is sort of reluctant to really go deep and, and to own it because it's not core. \[00:20:12\] Like this is not what their job is. This is not what their product is. Their product does some other mousetrap thing that's cool and differentiates it. Talking to a customer facing API is like not sexy, and it's not really part of your core differentiator, but it's necessary. And so I think they understand that for sure. \[00:20:31\] And so they're, they're quite happy to, to give that off. And yeah, I sort of equate it to nobody hosts websites anymore on a server in, in your closet. You know, we gave that up a long time ago. We gave that control up. It's like, Hey, you know what? I trust AWS or Azure or Google. I don't have to think about this. \[00:20:51\] I'm just gonna let them do it. And now, you know, you saw. With plaid you saw with Okta with authentication, you know, Twilio with SMS and \[00:21:00\] voice. So there's a lot of examples out there. And so this is yet another infrastructure that we license, use a better word, but license I include in our products so that we can actually focus on our product better. \[00:21:14\] And the developers understand this. They don't want to build this stuff. \[00:21:18\] ****Mike:** Yeah, it strikes me that there's a pretty good parallel for this that, that most probably even non-developers can relate to in. Photography that, you know, for a very long time, people who were into photography were the ones who spent their lives like studying photography and figuring out how to use a camera and like what an aperture is and buying the right film, or whether they're gonna be outside or inside. \[00:21:37\] Then digital cameras came along and you know, we had a little more access to things like the, the equivalent to this might have been when services like Okta started appearing and maybe there were a bunch of choices and the APIs were a little clunky and whatnot. But what really made things go mainstream for all of us is suddenly we all have a camera on our phone and like the UI for the camera on your phone looks like the UI for the camera on my phone. \[00:21:58\] And it looks like, you know, an \[00:22:00\] Android one and a Palm OS one and a Windows OS one and all those kind of look the same. And suddenly the experts on you know, authentication and payment and H R I S and all those other things can do their thing. While the people who just need to use them in their day-to-day work can benefit from it. \[00:22:14\] And centralizing these sort of APIs and giving Maybe albeit a simplified a p i for some of these things really empowers a lot more functionality and broader adoption across the industry. What, what's the education side of things look like for you? Are you finding that the value proposition of like, you know, hey, maybe you came to, someone comes to unified for like a sales crm integration. \[00:22:36\] Are they finding their way into other bits of api? I facets that you're unifying, say call centers or recruiting things like that too. \[00:22:43\] ****Roy:** so we call those clusters because they are sort of intertwined someone, or can be depending on the, their use case, right? So if you're just doing sales enablement, you probably just want the crm and maybe the enrichment APIs that we built. But if you're doing some other sales or customer oriented \[00:23:00\] software, something around the customer, You may actually want to track the call center activity on that customer along with maybe the mailing list. \[00:23:08\] I don't know. So those subtexts can, can intertwine just like HR in in applicant tracking systems, candidates and employees. A candidate turns into an employee. So you want a good handoff. But those two clusters don't really merge or, or intertwine at all. We find but to, to answer your question a different way, we do have customers coming in, or prospects, let's say, who are looking for a Zapier clone. \[00:23:34\] And we're not a Zapier clone. We don't compete with Zapier or no code. In fact, we are code, you know, it's an api. You have to code with the api. We make it super simple, one line a go, but you still have to code. You're a developer, Zapier Enougher developers. And we don't offer the automations that a Zapier does. \[00:23:54\] And that's not our role. We're not an ETL either. We're not taking data from one database and putting \[00:24:00\] it into another database. We basically don't wanna add any business logic whatsoever. We want to give our customers the raw material making it super simple for them. But they are building a product and they are controlling their business logic the way they want to. \[00:24:16\] If they wanna store the data somewhere, it's up to them. Like they, it's not up to us if we're not gonna offer that. We are just making it super, super simple. We're, we're focusing very much on this one problem space. I think the other confusion that happens is we got a lot of co prospects coming in that wanna manage their own customer sorry, their own accounts. \[00:24:35\] So their own Salesforce account, they wanna automate their Salesforce account, moving data from it to something else. And that's really a Zapier use case again where you're managing your own accounts for your own business, not your customer's data. \[00:24:49\] ****Phill:** Yeah, I think if I try and make it a bit more concrete, cuz what I've, what I've said about it so far is like, you can skip over too and that's cool. But I have to think to, to talk about it a bit more. I, I was playing around with \[00:25:00\] it and, and I'm trying to think of like a, a similar kind of vibe and it, it reminded me of on ghost, I've be using Ghost a lot, the cms. \[00:25:06\] And it's super cool cause like you sign up to Ghost and then it's just got this massive pile of integration. So you can click integrate, integrate, integrate, integrate. And then it asks me for my, or tokens as the user of Ghost. And then kind of Ghost doesn't really care about those integrations, you know, cuz they could just be running through something like you and then in the background goes, can be using all that data to, to match things around. \[00:25:26\] So yeah, it, it, it's kinda like that, especially with the, with the handy little JavaScript. S D K or not, not sdk, the JavaScript one-liner that you've got. You can like have your embedded integrations. So I could be, oh, it all sounded very clever in my head, but if I'm building application like ghosts, right, and I need to add loads of integrations to it, I can just pick a bunch of your integrations, embed that one line of JavaScript, and then it's got that UI will built out with a list of all those integrations that, that my end user can then, can then click on and integrate their accounts with my \[00:26:00\] application. \[00:26:00\] Yeah. \[00:26:02\] ****Roy:** It's all about speed, right? The developer doesn't wanna waste three months of their time building out an integration on the back end and then building out the front end user interface for their customers. Like all of this is common stuff. Like we're, we're even gonna come out with some other common widgets actually. \[00:26:19\] I I, I won't go into detail, but how many other things can we standardize? That one line at coa? Just to back up, one of my favorite developer stories is I was running a, a startup and one of my junior developers came to me and said, I have a solution to the problem that you asked me to go and solve it. \[00:26:35\] That problem was we were having a lot of fraud with our payment system, right? And so I asked him to go and find some anti-fraud solution and he, he is like, I found, I found an even better solution. It's a new payment provider and they have integrated fraud and we should switch. And I'm like, but we're using like the big payment provider. \[00:26:55\] And I asked him some questions and, and he told me, well, it's a new startup out of Ireland. And \[00:27:00\] I'm like, oh, no way. Like what? Some new startup, we're not gonna put our payments through a new startup out of Ireland. And then he convinces me though, and this is the point, he's like, but there's one line of code that they gave me. \[00:27:13\] Their documentation is beautiful and I used it. I got to it working within 10 minutes and let me demo it and it solves all of our problems. And that one small Irish startup was strike. And so from that moment on, I've been using Stripe and it's because of that one line of code and that stuck in my head. \[00:27:35\] And so I wanted to have our developers have this one line of code that they can insert into their app for their customers and get it running and look like Rockstar. And so that, that's the, the background of that one line code. It's turned into not just JavaScript, but also react Component and a U J F component as well when building an angular component as well. \[00:27:57\] ****Phill:** I think especially for, you know, a lot of US \[00:28:00\] API developer pipes and sort of people listening to this podcast probably came from a mostly backend perspective, and I might hear a bit different, but a lot of us were kind of like doing backend and kind of vomiting something out to the front end and hoping someone would tidy it up later. \[00:28:13\] ****Roy:** I love that. I'm gonna, I'm gonna use that. I think exactly how I felt as well, but I think with age you sort of understand that you need some more balance. So yes, let's try and balance that. \[00:28:24\] ****Phill:** Yeah. Yeah. I mean hopefully that's other people too. Otherwise I'm just admitting to being a bad developer, but I definitely am not ballsack. And so things like Stripe, things like you know, even the Shopify buy button all these integrations that are just like this one line of code, it's not just the one line of code. \[00:28:37\] I could write a hundred lines of code. It's the fact that I don't have to go at a front end for this part of it. Right? Like, think about talking about the integrations. You know, you've got your, you are building an application, you've got a list of integrations of a bunch of third party SA systems that your users want to connect with for whatever reason. \[00:28:53\] Building the interfaith for that sucks hc, cuz you've gotta build this whole like, you know, create applications and, you \[00:29:00\] know, log in here and redirect back there. And then you've gotta like add some webhook URLs that all you're testing. And that's hard. Although we've done some podcasts about how to make that easier. \[00:29:08\] And then you've gotta have all of the various different forms and different things and all of just, just that building that is time I could spend doing literally anything else in the world. So for the fact that you've taken that interface off my hands is a, is a huge value in itself. \[00:29:23\] ****Roy:** But, but let's talk about all the stakeholders involved here, right? So we're just talking about developer, right? Which we typically only think of, right? And backend developers, to your point, Phil, not even front end. Yeah, forget about the front end guys, but there's more. There's a cto. And the CTO cares about different things than the developer, right? \[00:29:40\] They care about security, for example, massive, right? And if you're building an HR software company, you really care about security cuz you're handling data that is super sensitive. But look at support people cuz after you launch those poor support people are gonna have to ma to talk to your customers and like, Hey, I'm \[00:30:00\] trying to hook up to a workday. \[00:30:02\] Something's not working. So they need, they need help as well. They need something, they need documentation. But even marketers, we don't even talk about marketing. How do we launch this thing? How do we educate the marketer to launch? What do, what do they say on Twitter or what do they say on their release documentation for this new integration? \[00:30:21\] And it's not just one integration. Like when they add us, they get all of them. They get all 92\. Today, we're gonna be at a hundred in a couple weeks, but, so they can launch all of them at once. And so that's a massive amount of information that they actually have to do the work on. And, and I would actually pass it that the developer is actually the, the lucky one in this case cuz they can build this in an an afternoon. \[00:30:47\] The marketer is gonna have to build a ton of documentation and this support people are gonna have to build a ton of documentation and their FAQs to support all of these. \[00:30:56\] ****Mike:** I think that an important bit of additional context to provide \[00:31:00\] onto this story too is, so Phil, sort of cheekily mentioned before that I, I come to this world from a very different perspective. The reason Phil and I met is cuz he was the API developer looking for a, a, a front end full stack person. \[00:31:12\] Clean up his, his vomit mess as I think he put it before. And that's honestly, that's how we met, right? Like he, he had a bunch of really interesting ideas and I was used to taking the data from the interesting ideas and putting it into a presentable, human understandable experience. And you know, if I, if I could teleport back in time and talk to Mike from a bunch of years ago, I think the biggest thing I've learned from then is that like, much like you don't wanna keep building Salesforce integrations, I really don't want to keep building login screens. \[00:31:38\] I really don't want to keep building the same payment screen, you know, to, to like design the UX of how to put in a credit card number or something like that over and over. And I, I do think that coming from both angles of this, from a developer perspective, the value is there and it makes a lot of sense. \[00:31:51\] And now in the world I'm living in now where I'm building a company and you know, I'm, I'm Roy also in the startup world these days. And requirements change quickly and \[00:32:00\] needs of the team change quickly and the thing that on fire the most changes very quickly and. I think if you turned to me and said like, yeah. \[00:32:07\] And so the story to get your people plugged into the tools they need faster is a reasonably simple integration with documentation in one place with value proposition for help and security and all those things. Like that with the wisdom of time behind me sounds a whole lot like something that maybe I should be paying attention to and, you know, maybe perking up a little bit when something like that comes across my inbox too. \[00:32:29\] ****Roy:** Yeah. Yeah. And you, you're right. When you know you, you can build so much, and, and the, the requirements are never over your product. People will always give you new requirements, so you're always gonna be busy. Doesn't matter how much importance and time do you wanna spend on things and. I would rather spend time. \[00:32:48\] So back to my previous startups, when I look back and I try and do learn from my mistakes, one of my mistakes is that I think we spent too much time learning stuff that was core, and these customer integrations \[00:33:00\] weren't core. I should have spent more time on working on feature sets that were core, that differentiated my product from my competition. \[00:33:07\] So I wish I would've done more. I would've, I wish I, I had unified to to basically handle all of these customer integrations. So I didn't have to do that. And back to that one line of code. So it's not just the front end that's important. It's also the backend. We have, I think you brought it up, we have an s stk. \[00:33:25\] We have a couple of SDKs. One is a TypeScript node that you can easily use, but you can use one line of code basically with that as well, to, if you wanted to go and say, get contacts from A C R M or employees from an H R I S system. It's basically one line of code as well. So we're, we're trying to make it as simple as possible for both the backend and and front end, \[00:33:46\] ****Mike:** So I wanna talk a little bit about the implementation side of things too, if that's okay, Roy, from, from a consumer perspective what is the hello world experience like for Unified, if I'm starting to integrate with my product stack? \[00:33:57\] ****Roy:** right? So our end user developers, \[00:34:00\] right? So they come in, they, they go through a little onboarding. We try to get them to see the data from the API as quickly as possible. There's a couple, a couple of things they need to do. They need to say which integrations they're interested in and and then play around with that one line of code for that embedded directory, right? \[00:34:18\] So that shows like a matrix of available integration. So if they turn one on, they have one logo they can click on, that they can enter in an a p i key if it's that, or an oof. To screen right? Using our own credentials. They see, they visualize it through their customer's eyes. But once they authorize it, they also get a connection. \[00:34:42\] And that's like the core object in our, in our platform, everything runs off of a connection ID and it's abstracted. It's just an id, there's no authentication associated with it. There's no integration type. It doesn't matter, right? And so from a developer, then they get to use that connection id we give them a \[00:35:00\] little bit of curl code, which they can just copy and paste into their terminal and it calls that api. \[00:35:07\] It calls, get whatever employees, contacts, whatever that integration supports. And then they actually see the data in their terminal. And so I think that's really important. As a developer, you know, there's a lot of BS out there, companies that tell you that they can do X and it doesn't, or it's really hard. \[00:35:25\] And so we want to make sure that they understand that literally within five minutes, they're gonna see data, real data in, in a unified a p i. And that data object that comes back is unified. It is, it is one of the core benefits that we work our buds off. We expand those fields to the maximum that we can, and they're always the same format. \[00:35:47\] And so, as a developer, doesn't matter where I am, my customer's data's in, it's always gonna be the exact same format, unified data model. And that's actually how we start it though, abstracting \[00:36:00\] these data models away. That's one of our core, I think, benefits of this. And so that's what they're gonna see. \[00:36:07\] And then once they test that out, maybe they're testing it with their own accounts instead of their customer's accounts. Hopefully. You know, then they, they have this roadmap in terms of launching testing and then launching it in, in production. \[00:36:21\] ****Phill:** How about change management? Literally no one's favorite topic, but I'm doing it anyway. So something that seems like a benefit is that if there's, you know, your. You as an application develop building an application around an integration with some third party, and that's third party releases version two and three and four. \[00:36:39\] And they're not really important. They've renamed the methods for no obvious reason, and they've added a few parameters that you're not using. But as far as unified is concerned, like it's easy enough to update that integration so that I'm, I'm assuming that the, the users of unified don't really notice those version changes most of the time. \[00:36:56\] But what happens when a big change happens and like the data \[00:37:00\] is different? Can you h how much length do you go to to kind of help maintain that interface? And at what point do unified users need to do something differently if there's a big change? \[00:37:14\] ****Roy:** Yeah, so. As you can imagine with the unified data model, not all fields are available for all of the integrations. And we try super hard. Like there are some times when one API call into our system results in about 10 different API calls to the software that we're targeting just because we're trying to collect all the data so that we can build our own unified object model. \[00:37:36\] And so when those API calls change or they, their data changes, their format changes, whatever, our customers never see that. We do. And we actually have a lot of automated monitoring in place so that we are ahead of those changes. And that is of course one of our core benefits to our customers. They don't ever have to maintain that awareness.\[00:38:00\] \[00:38:00\] We do. So we're super hyper vigilant about that, about these changes. And we're also looking at the data coming across. Again, API documentation is usually out of date. Compared to what's actually happening on this a p I server, right? As you talked about, the front end guys typically get it last. And then the marketing people get it last. \[00:38:22\] And so there's this, this waterfall method where the developer is way ahead of everybody else, especially the API developer. So we do have quite a lot of automation to make sure that we're always ahead. But when let's say Workday changes their APIs or data models or anything like that, our customers will never see that change because we've already changed along with that. \[00:38:46\] And we have this unified system. We do expand that model. So there aren't, it's, it's not always static. So we will add into the field, like, you know, we'll add compensation or whatever. But that's in addition to, and it's not \[00:39:00\] backwards breaking. \[00:39:02\] ****Phill:** Okay, so whatever nonsense, they're absolutely you kind, ofra or unified, a unified. To face that is using evolution more than whatever breakthrough changes they've gone with. Cuz there's, there's easy examples slightly contrived and silly, but the, the like usage of names I've seen in API before where they're like, we used to have first name and last name and we realize that's not how names work. \[00:39:25\] So now we use name and they like literally somebody somewhere just writes a bit of code that says if first name and last name then mush them together to do name. And then you've got that covered all the other way around where they used to have name and they've gone, oh crap, we want to integrate with Stripe the demands first name and last name. \[00:39:41\] So somebody somewhere has written a explode space. And, and that again, it's not how names work but people do that sort of stuff. Like are you doing that sort of thing as versions change in the background or are you asking users to say, Hey, the data model changed, can you update somehow? \[00:39:59\] ****Roy:** No. So we are \[00:40:00\] monitor. Doing that ourselves. And again, we've, we've automated that. I think the example that you just gave is a really good reason why you actually don't wanna build these integrations in house because you're basically building an, an architecture that is inferior and it's not, you're not really thinking that long. \[00:40:15\] You're thinking, I only have this one integration to build. Product managers asked me to build Salesforce. I'm gonna build Salesforce and now I'm gonna cut corners. Yes. First name, last name. I'm just gonna like, add them, add the two strings. And then something changes and then your architecture can't handle it. \[00:40:29\] Or you're asked to add in a second integration and then your architecture really can't handle it. So we've done the homework there. We're actually gonna come up with an infographic in terms of all the differences. You know, first name, last name versus display name versus, you know, API key versus olf. \[00:40:47\] All of that's the differences. There are quite a lot of differences that. Are gonna make our head spin and are making our head spin right now. But we've automated an up a bit with our architecture that it, all of this works. Now \[00:41:00\] again, it's our responsibility to make sure that it is working. So if they do change first name last name to just display name, we'll fix, fix that before anyone notices. \[00:41:10\] I think what we've seen though more often is that they'll change the object. So they'll go from like, opportunity to deal or something like that. And then that, that, that's a much bigger change because the data model's completely different and maybe the associations between the different objects that they have are now different as well. \[00:41:31\] So that, that's a little bit more time for us to, to go in there and refactor that. \[00:41:36\] ****Phill:** Yeah, I suppose some of them, like I seem to remember Stripe doing something along the lines of changing payment into like charge and receivable or something. And that just like is a fundamental change to how it works. But I'm assuming that many of the integrations you've gone for have kind of settled down a bit. \[00:41:50\] There's that, there's that like early startup, everything's being renamed every five minutes kind of period. And then there's that like, It's strip, we're not changing, it's every had a weekend mate. \[00:42:00\] Like there's, you know, it kind of settles down at some point to a point. And also if they're mostly around the same verticals, like a team that a lot of them are, you know, it's, it's person and company and, and they're relatively stable domains in that sense as well. \[00:42:13\] ****Roy:** Yeah, so part of the unified API theory is, you know, we're not only unifying the API endpoint, right? S RAAs based API endpoint, we're also unifying the objects. So for ex in the c RM space, some c r M vendors call them opportunities. Some call them deals. We've chosen to use deals, but it means the same thing. \[00:42:35\] Same thing with company and account. And so we tried to unify those as well and we documented what they mean so that you can extrapolate if you know Salesforce, it's an opportunity, for example. And then of course the fields are also the same. So there's really like three levels that we try and, and unify, and in fact, there's even more unification. \[00:42:54\] So authorization, we talked about a little bit of that with our one line of code\[00:43:00\] to represent the embedded directory to your customers. That's for authorization. You're trying to get their authorization of their account. Most of the oof inventors out there have scopes, permission scopes. They're all different. \[00:43:14\] There's no standardization at all there. So we've actually gone ahead and unified all of them. We've standardized all of them, and so again, you as a developer, if you need access to contacts and deals, you don't have to go and read about what scopes you need and then put in custom scopes into our authorization directory. \[00:43:34\] You just can just say, I need to read and write contacts, and I just need to read deals, and then we'll figure that out. \[00:43:41\] ****Mike:** Sorry. I was gonna say one of the most striking things about the product altogether, which is really hard to convey through an audio medium like a podcast is the documentation that you've put together, which is sort of a meta documentation of, like you said, 93 APIs. And I think if you're listening to the show, and this sounds even remotely interesting, a really interesting \[00:44:00\] activity is to go just browse API documentation on unified two to, and check out what this looks like. \[00:44:06\] What in practice, what it means is there are you can look at sales crm, APIs, for example, and see a list of, I don't know, what looks like a dozen or more CRMs that are supported. And the really thoughtful way that Roy, your, your team has divine a A nomenclature that unifies all of those things in a way that's like human understandable. \[00:44:23\] I really like that you've published these I guess it'd be like an entity relationship diagram of the way that you, you think of the types and how they interrelate here for the things that, that are surfaced in your api. I, and so you can kind of see regardless of the c r m you pick, for example there's a notion of a team and a user and that user might have an address or they might become a lead and turn into a user and things like that, which fully zooms you out from having to think about, is this am I using front or HubSpot or Drift or Intercom or all these other things that's just getting into the meat of like, what do I need to get done here? \[00:44:54\] Rather than becoming an expert on the underlying principles that have been, you know, cast upon you by the people \[00:45:00\] delivering the software. Yeah. \[00:45:02\] ****Roy:** Oh, no, thank you Mike. You know, back to the Stripe example, their documentation is world class. It's always been the best API documentation. And you know, I'm not saying ours are, are that, but we really try and we have a new design coming out shortly and we'll always have one, but it just really super important to have it searchable detailed and go from the top where you see this UML diagram so you can understand how things work together, but also go right to the, the bottom. \[00:45:33\] Like what is each field and represent and so forth. So we've tried to encompass quite a lot of data. There maybe too much, but I think as a developer you wanna have that optionality. You know, you wanna get in there quick, five minutes, boom, boom, boom. Oh yes, it works. But then like, when something doesn't work, you want to have the detail there. \[00:45:51\] So we try quite, quite hard to, to to add that in. And the other thing I would say is after you build all of these integrations, like we have, I think \[00:46:00\] 24 HR, I guess, integrations right now, and we're building out quite a lot more of them right now. But after you really understand the commonality, right? \[00:46:08\] There are quite a diff a lot of differences, but the commonality in how they operate and how the relationships work and so we're giving that back to our customers. This is our understanding of how these, these elements, these objects work together. And so I, I think that's the API documentation that I want our customers to read and not Salesforce or Workday or anything like that. \[00:46:33\] ****Mike:** I suddenly have this overwhelming feeling that you can unplug yourself and see the matrix of how all these things are actually planned from behind the scenes and like that, that you, you've got a, a fourth dimension of vision for these things that I would love to develop. It's really fascinating. \[00:46:46\] ****Roy:** You know, it's, I'll give you a bad dad joke. Okay. Like, I, I, I always tell people that my, my second language is api. Because I really feel like I can sit and, and understand an \[00:47:00\] API within a matter of minutes. And it's just because I've always done this, right? It's, it's, you know, 10,000 hours, right? And so if you, if you've looked at APIs all of your career, you sort of are looking for specific things, you understand concepts. \[00:47:16\] So, yeah, I, I, I, I think this is the company that I think I've always should have built. \[00:47:21\] ****Phill:** I, I'm interested in the kind of, the way that you talk about it, you've unified these things, right? You kinda looked at lots of different companies and, and you've come up with unify nomen culture and I assume a team as well, not just you, but I'm wondering about things like standards. So schema.org is, is something I know that some of our listeners are super excited about. \[00:47:41\] It comes up on Slack all the time, not less, not just Jason Schema. It's a different usage of schema. Schema.org is like someone's, a large number of people have sat down and gone through everything. They've got, like they've got person which has got a address, additional name alumni or Birthplace Death place. \[00:47:57\] And then Place is is another \[00:48:00\] type which has various different events and coordinates and loads, properties, address, geo whatever, an organization which has address and employees and tax numbers. And I'm just reading random fields. But then they have really obscure stuff, which is like biochem entity, actually. \[00:48:18\] People like, okay. And Medical Entity and a bunch of other things like that. So there's some really useful stuff and some, you know, potentially useful stuff to a tiny place of population. But if you thought about working with something like Schema to say, instead of like, this is what Unify think that, you know, this is, this is what you think a person should be, maybe. \[00:48:40\] It could be your job to convert things to person and back again so that you are working with the standard interfaith that is standard. Standard, not your standard. \[00:48:48\] ****Roy:** Yeah. Yeah. So I, I have two comments on that. One is I love standards. In fact, in, in a previous life I was part of the I E T F, which is the internet standards body coming up with standards for \[00:49:00\] virtual private networking, for example. So I, I, I see how an industry explodes what you have standards and all of the companies within it can communicate. \[00:49:10\] So I'm all for that, and I'd love that there's not that many standards for say, crm. And so we had to build and abstract away that. I think that will happen over time. There are standards like Ski Skim for example which is for HR user provisioning, which has a user object. It's not bad. But I think that the comment that I would make to your schema example is that if you make something too broad, too big, then it sort of loses value, right? \[00:49:39\] And so if you, if you define a user object for everything, then it's too big. And so what we've done and, and you can take a look like, so there's there's actually a section on our API documentation called like user objects, which is like, Hey, we have a bunch of user objects. They're all called different things like contact or employee or candidate or customer.\[00:50:00\] \[00:50:00\] They're specific to the actual use cases. So A C R M contact is different than a HR employee object, which is different than maybe a, a ticketing customer object. And they're done for a purpose. They, they share quite a lot of information field, same fields, they email that kind of stuff. But because they're for different use cases, they're gonna have different fields that are relevant for that use case. \[00:50:26\] And so that's the, that's the angle that we picked. We don't wanna add noise by having this meta massive user object that contained everything other than the sun. So that, that's, you know, you can, we can argue whether or not that was the right way of doing it, but \[00:50:41\] ****Phill:** No, that's cool. I had to ask if I knew, someone would shout to me about why you didn't, you just use chemo.org if I didn't. So thank you for preemptively answering them. \[00:50:49\] ****Roy:** Yeah. No, I, I would love to use standards and I would actively promote and be active in defining those standards. That is something actually that we are looking forward to doing. At some point, \[00:50:59\] ****Phill:** \[00:51:00\] That's very cool. Yeah, I mean, it just, every time it's come up, like g stoplight brought it up a little while ago when I was working for them and you know, they've got customers that have 500 different models for a flight, and it's the same company, just has 500 different flight models, so things can get wildly varied. \[00:51:17\] So they've kind of gone with the approach of let's help companies standardize themselves so that they have like one thing to represent a flight. And then once there's lots of different companies that re flights, then you can start to standardize those versus if everyone's got snowflakes in their own home, then it's, it's not gonna be something you cane. \[00:51:34\] So that, that makes sense. \[00:51:37\] ****Roy:** you, you're right, you, you look like a company like Postman. Postman is, is growing very fast. Why aren't they trying to standardize some of these things, right? Readme Yo Spotlight, there's a bunch of these API management companies that are looking more at customer facing APIs instead of just internal APIs like the API management company like Kong. \[00:51:58\] So I, I think we will again there. \[00:52:00\] I'm not sure that there's a standard s spot out there right now, but maybe, you know, schema.org for example, could facilitate that. And like I said, already, there's a skim that has tried to do that, although it's definitely not unified. There's quite a lot of changes and differences within each one of those invitation. \[00:52:19\] ****Mike:** So Roy I, I realize we've been chatting about your product for a while here. A couple of things I wanna touch upon is if some of the folks listening to the show today are interested in jumping in and playing with unified. I noted on your site that at the moment at least it, it lists that you are sounds like either pre-launch or in sort of an early access mode. \[00:52:38\] What's the best way to get access? \[00:52:40\] ****Roy:** Yeah, we're I would say self launch. The entire platform is operational. It's very scalable, very, very secure. We're not storing any data. We have a ton of security functionality in there even to to store access tokens in your own database and so forth. But you can just self-register. It's we have a \[00:53:00\] free plan to test out the system. \[00:53:02\] It's free forever until you want to launch you, you know, you want your own identity. You don't want unified's identity in your app. And so anyone can go to unified.to and no, we are not based in Tonga, the South Pacific Island of Tonga. But it is it's a cool domain short, but we also do have a promo code if you guys want to test this out. \[00:53:25\] We're giving out three months free. And that that promo code is just APIs you won't hate 2023 all in capital, most basins. So that's a really easy way to to actually test this and put it into your product. And of course, you know, looking at our Twitter feed Unified underscore api, we always post in Twitter and on LinkedIn as well. \[00:53:49\] And we try and post quite a lot cuz as you can see from our change log, we are adding a ton of new integrations and a ton of Features within our \[00:54:00\] objects, like new fields and so forth. So we've, we've definitely figured out a way to automate the building of integrations and to put it into a very scalable architecture. \[00:54:10\] ****Mike:** Yeah, that's great. It's I think the kind of thing that, that is very encouraging too, to show developers like the, the price to try is free. And therefore, you know, theoretically the value proposition should be clear from there, right? Like, if, if the product speaks for itself, then it's easy to get started free and, and kind of dive in. \[00:54:25\] ****Roy:** And I think pricing model is really interesting that we didn't really talk about, you know, we talked about APIs and technical stuff, which I don't think we did enough of, but I think pricing is so important because you want this to work with your pricing model. If you're charging your customer 20 bucks, you don't want to pay 20 bucks a month for something like this. \[00:54:45\] So we, within startup operators for a very long time, that was something that we thought about. Really hard. We actually took a lot of time in coming up with a pricing model that would work, that we would pay for, and we're like cheap. \[00:55:00\] You know, typically you know, AWS costs shouldn't be, you know, a hundred percent of our profit in margin. \[00:55:06\] I, I think that is something that I, I'm quite proud of other than all the other a d I integrations that and all the API technology that we've put in there, you know, not only the rest a p I server, but the G R P C one, the GraphQL one in there also, that all of that tech that we built is, is really interesting. \[00:55:23\] But the, I think the pricing model is key. \[00:55:26\] ****Mike:** Yeah, there's a lot of rabbit holes that I think we could poke our way into and maybe good reason to have you come chat with us again sometime soon. Roy where's the best place to find you online? \[00:55:34\] ****Roy:** So Roy Unified to is my email address reach out to me. You know, I'm on LinkedIn I'm on Twitter. Of course, you can find my coordinates pretty easily. And yeah, I I am very out here in the Toronto tech scene. Love to talk to, to other tech entrepreneurs. Obviously very passionate about APIs, but I'm also \[00:56:00\] very passionate about the tech ecosystem anywhere and tech entrepreneurs as well. \[00:56:05\] ****Mike:** Fantastic. And the final question I usually ask before wrapping is are you hiring? And if so what sorts of folks are you looking for? \[00:56:12\] ****Roy:** Yeah, we are hiring. So yeah, we are, we're five months old. We are raising halfway through raising our pre-seed round right now, our first round of financing. And we're four people right now. We're hiring a fifth. We're hiring a we're calling a customer success manager. Basically post sales, a p I. \[00:56:32\] Technical but not coder list. Loves to work with customers. Get them to implement our E G I and technology. \[00:56:40\] ****Mike:** Got it. Oh, that's fantastic. Well, if that sounds like you and you're listening to this we will make sure that Roy's contact information is all over the show notes, just as well as show notes sorry, just as well as links to get to Unified as well as the promo code to get three months of free access with their startup plan. \[00:56:54\] So make sure you check the show notes to grab those. Roy, it's been fantastic having you chat \[00:57:00\] with us and hang out. Would love to have you back and, and talk about where things are in six months, three months, a year, whatever it may be. And kinda see how things are going for you. Thanks so much for your time. \[00:57:08\] I really appreciate having you here. \[00:57:09\] ****Roy:** Hey, thanks guys. \[00:57:10\] ****Mike:** All right. Take care. \[00:57:10\] ****Phill:** Yeah. Cheers everyone.\[00:58:00\] ### Death By Google URL: https://apisyouwonthate.com/newsletter/death-by-google/ Last updated: 2023-07-01T14:04:24.000Z Google Domains is dead. Utterly confused at Google's decision to kill something that is an on-ramp for Google Cloud. The domains currently being managed by Google will be sold to Squarespace. No shade at Squarespace here – it's a great tool if you want a website up and running in seconds. But seriously, Squarespace? I am left confused by this decision. As I write this, it's been two weeks since the announcement, and customers have still not been notified. At this point, we can safely say that you can no longer trust Google not to kill something you might consider a vital service. I guess it is hard to keep products alive when they don't produce the insane levels of profits that search/ads do. All I will say is if you're running something in Google Cloud, beware. If you want more details, [The Pragmatic Engineer](https://blog.pragmaticengineer.com/google-domains-to-shut-down/?ref=apisyouwonthate.com) covers the whole story. --- ### Monthly API Round-Up Our monthly round-up of articles, news and updates from the web. ### [Goodbye Kelsey Hightower](https://twitter.com/kelseyhightower/status/1673366087541600256?s=20&ref=apisyouwonthate.com) If you don't know who Kelsey is, imagine the kindest, most caring and most supportive person working in the cloud, and then somehow watch him blast past those expectations. Kelsey announced his retirement from Google, even getting a farewell tweet from the Google CEO himself. I mention Kelsey because he is an example we should all follow for being a supportive and innovative person in the tech community. ### [Smart Placements](https://blog.cloudflare.com/announcing-workers-smart-placement/?ref=apisyouwonthate.com) With edge computing options becoming more widely available in tech stacks and cloud providers, placements of compute vs data become a more significant issue. When moving API endpoints and other code to edge functions, people can forget the added delay when DB calls are made. Cloudflare is taking this issue off your plate. Using their new feature, Smart Placements lets them make network decisions for you so you can focus on shipping features. ### [Swift OpenAPI Generator](https://www.swift.org/blog/introducing-swift-openapi-generator/?ref=apisyouwonthate.com) A Swift plugin that generates client-side code to make HTTP calls when fed an OpenAPI document. The plugin doesn't stop there, though. You can also get it to create server-side code to handle the calls. Swift is not just for developing apps for Apple products; you can also build servers with Swift. The project is in active development but open-sourced now so the community can get involved and give feedback. ### [Database On The Edge With DenoKV](https://deno.com/blog/build-crud-api-oak-denokv?ref=apisyouwonthate.com) Recently Deno dropped a key-value database built right into the runtime called DenoKV. The database can store any JS value, is built on top of foundation DB and has different consistency read levels to handle running on the edge. See what all the fuss is about and how easy it is to use with this tutorial. You spin up a simple CRUD API, which can also be deployed with Deno Deploy in a few clicks. ### [When Should You Use Hypermedia?](https://htmx.org/essays/when-to-use-hypermedia/?ref=apisyouwonthate.com) I am unsure if I like htmx or the constant funny memes the Twitter account posts. Either way, htmx keeps dropping some solid gold advice. This blog post covers some background on Hypermedia, when it's a good fit and when it's not. Hypermedia can do much more than people think and is an underused tool. --- ### APIs You Won't Hate The latest content from the team at API's You Won't Hate. ### [Unlimited API Support From Phil Sturgeon](https://twitter.com/philsturgeon/status/1674824095320489989?s=46&ref=apisyouwonthate.com) Phil is now offering a special one-time deal where you can **get unlimited API support for six months for only €1000**. If you are building or maintaining an API, Phil is your go-to guy for help and advice. Act fast, as there are only six spots available. ### [How to Capture Quite a Lot of API Traffic](https://apisyouwonthate.com/p/68c14a44-0a09-4a1f-a30f-a413635faf26/) Need to re-write an API but don't want to break anything? In this article, Phil talks about using Optic to capture HTTP traffic to create an OpenAPI to work against. You can then use contract testing as the new API is built to pick up mismatches. ### [Creating world-class API devX with Sagar Batchu from Speakeasy](https://apisyouwonthate.com/podcast/sagar-batchu-from-speakeasy/) On this episode of APIs You Won't Hate (the podcast), Sagar Batchu from Speakeasyapi.dev shares his insights on helping devs live the dream of building APIs that have world-class developer experience. --- ### From Our Community Articles written and shared in our free [Slack community](https://slack.apisyouwonthate.com/?ref=apisyouwonthate.com). ### [Markdown Changelog from OpenAPI Git History](https://www.useoptic.com/docs?ref=apisyouwonthate.com) Aidan Cunniffe showed off a new feature from Optic, a tool to help you ship better APIs. You can now run a simple command and generate a changelog based on changes in Git from your OpenAPI doc. I have to say the output he shared does look pretty nice. ``` # TodoAPI ### Mon May 22 2023 - added `GET` `/todos/{todoId}/{userIdA}` ### Thu March 9 2023 - `GET` `/todos`: - changed `200` response `todos`.`items`.`name` to optional - changed `200` response `todos`.`items`.`message` to required - removed `exclude_completed` query parameter - removed `category` query parameter ``` --- ## Support APIs You Won't Hate Thanks so much for reading the newsletter. We recently rolled out a membership program, which helps support the community by covering production and hosting costs for our articles, podcast episodes, and newsletter. It would mean the world to us if you'd [consider supporting APIs You Won't Hate with a paid membership](https://apisyouwonthate.com/#/portal). Thanks so much! ### How to Capture Quite a Lot of API Traffic URL: https://apisyouwonthate.com/blog/capture-api-traffic-using-optic/ Last updated: 2024-04-05T12:45:15.000Z When I'm rewriting an API, I want to make sure I'm not unintentionally changing loads of stuff. After all, API specs are called *contracts* for a reason. To do this, my favorite new approach is [using Optic to capture HTTP traffic](https://apisyouwonthate.com/blog/turn-http-traffic-into-openapi-with-optic/), so we have an OpenAPI to work against. We can then ["contract test" against that OpenAPI](https://apisyouwonthate.com/blog/contract-testing-apis-laravel-php-openapi/) as the new API is built, and be alerted immediately if anything mismatches. [Turn HTTP Traffic into OpenAPI with OpticCapture real HTTP traffic from production or anywhere else, and create OpenAPI from it, for documentation, mocks, SDKs, or contract testing.![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/size/w256h256/2023/03/apis-logo-square.png)APIs You Won't HatePhil Sturgeon![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2023/03/optic--1-.webp)](https://apisyouwonthate.com/blog/turn-http-traffic-into-openapi-with-optic/) Some APIs a pretty "static" in what they return. Most responses look the same, so you can just capture a few responses and call it a day. however some APIs are more "variable" in what they return, so you want to get a good amount of traffic running through Optic to make sure you're not missing any of the possible variations. So here's what I am doing right now, writing this whilst a huge bash script runs through 45,000 odd HTTP calls. First, they've got an Airtable database they're trying to escape, so I've made a view on there which shows me aaaaaaaall of the UUIDs for the endpoint I'm most concerned about. ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2023/06/Screenshot-2023-06-26-at-6.46.46-PM.png) Airtable let me copy and paste 45,000 UUIDs out, and I've dumped them into VS Code. ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2023/06/Screenshot-2023-06-26-at-6.48.15-PM.png) Using the multi-cursor feature (CMD + Shift + I) I have changed this pile of UUIDs into a list of bash commands using [httpie](https://httpie.io/?ref=apisyouwonthate.com) (because nobody has time to remember how curl works), turning off SSL verification and sending it via the Optic Proxy which is running from `optic oas capture openapi.yaml https://api.example.com`. ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2023/06/Screenshot-2023-06-26-at-6.51.08-PM.png) VS Code got a bit upset about 45,000 lines, and decided to only let me work on 10,000 at a time, which is fine. I popped an `exit` in there and saved the Untitled file as a bash script called `import.sh`. Now I can crack open a new terminal window and run `./import.sh`. The bash scrip then runs through the first 10,000 requests quite happily, and I can delete those lines and pop an exit below the next 10,000. I also could probably have turned that untitled file full of UUIDs into a JavaScript array and run it through node doing `fetch()` calls, but this has worked out fine. You do you. When I have finished capturing requests, I use the verify command: ```bash optic oas verify openapi.yaml ``` That shows me what its found, and I can update the `openapi.ymal` with either all of the endpoints, or just some. I only hit one endpoint so this doesn't matter. ```bash optic update openapi.yaml --all ``` Job done! From there I can upload it to Optic for quick "this is what it looks like", shove `openapi.yaml` into git to use for development, or plop the file into the project management software for some other developers to pick up as they work on the new functionality. This might all seem like a bit of a faff, but it gives me an `openapi.yaml` that I can be confident accurately represents the old API, because I have put the *entire* API through it, contract testing in development, then [using it as documentation](https://apisyouwonthate.com/blog/turning-contracts-into-beautiful-documentation/) when I'm done to avoid having to do that far more boring extra work later. Give it a try. Did it make better or worse OpenAPI than you did by hand? Answer in the comments! ### Creating world-class API devX with Sagar Batchu from Speakeasy URL: https://apisyouwonthate.com/podcast/sagar-batchu-from-speakeasy/ Last updated: 2023-07-01T15:06:59.000Z ## Show notes - [Sagar Batchu](https://twitter.com/sagar%5Fbatchu?ref=apisyouwonthate.com) - - [join speakeasy slack](https://join.slack.com/t/speakeasy-dev/shared%5Finvite/zt-1cwb3flxz-lS5SyZxAsF%5F3NOq5xc8Cjw?ref=apisyouwonthate.com) - Speakeasy - SDKs for your API [https://speakeasyapi.dev](https://speakeasyapi.dev/?ref=apisyouwonthate.com) - - [join beta](https://app.speakeasyapi.dev/?ref=apisyouwonthate.com) - [sdks as runtimes - terraform](https://developer.hashicorp.com/terraform/docs/terraform-tools?ref=apisyouwonthate.com) - openapi for chatgpt - OSS - - [openapi validator](https://github.com/speakeasy-api/speakeasy?ref=apisyouwonthate.com) - [templating engine](https://swagger.io/docs/specification/api-host-and-base-path/?ref=apisyouwonthate.com) - [AI release - clippy for your OpenAPI Spec](https://speakeasyapi.dev/ai/?ref=apisyouwonthate.com) #### Transcript \[00:00:00\] ****Mike:** Hello and welcome back to APIs you won't Hate. My name is Mike Bifulco. I am your co-host of the APIs You Won't Hate podcast, flying Solo today. Phil and I, have been doing our usual thing where we dance around each other's schedules, and being an East Coast American, working with a middle of Europe and or middle of UK and or tree dwelling, co-founder is a little challenging. But, Today I am super, super happy to be able to sit down and chat with my new friend, Sagar Batchu from Speakeasy. Sagar, it's really nice to meet you. Thanks for joining today. How are you? \[00:00:33\] ****Sagar:** Doing well, thanks Mike. You know, I've been following your and Phil's work for a long time, and very much admire what you guys have been doing and with APIs You Won't Hate. \[00:00:42\] So, really excited to talk today. \[00:00:45\] ****Mike:** Yeah. Thanks so much. I'm, I'm honestly, deeply flattered by that, and it's always really encouraging to hear, we, we love the community we've built and it's super cool to see us having real impact on people's lives and careers and things like that. We are of course, here to talk about your work and what you're \[00:01:00\] doing at Speakeasy. \[00:01:01\] But before we get started doing that, I'm gonna surprise you with a question that I didn't tell you I was gonna ask you, but I went on your company page on speakeasyapi.dev, and I looked at your profile there. And the second thing it says under your name is you love chicory in your coffee. \[00:01:13\] So, we need to start by talking about coffee because I'm a, a massive, massive coffee nerd. Tell me, tell me a bit about your favorite way to have coffee. \[00:01:21\] ****Sagar:** Yeah, thanks for checking that out. My favorite way to have coffee is traditional pour over. So if ideally I have beans at home, grind them up, a nice ritual in the morning before you get started to work to actually grind coffee, smell it, have a nice drink. That statement comes from my, my family actually has a background in growing coffee, so past couple of generations in one side of my family has been growing coffee in India for a while. So that means I have to be a coffee snug, there's no other way. \[00:01:53\] ****Mike:** Yeah. Got it. I think you just pulled credentials on me. So I am certainly outranked here, but I drank an awful lot of coffee and I'm, I'm a \[00:02:00\] big fan of the morning ritual too. I'm an Espresso drinker myself. I tend to do a shot of espresso first thing in the morning. But yeah. Good, good to know. I'm in good company and more than one way here. \[00:02:10\] Alright, well let's start there. Why don't you tell me a little bit about yourself, your career, your story leading up to where you are today, and then let's talk a little bit about Speakeasy and, what you're building now. \[00:02:21\] ****Sagar:** Totally. So my journey probably starts, back in university and, In senior year of university, I had this existential crisis where I realized years of work sitting in a physics lab wasn't adding up to what I wanted to be doing. \[00:02:40\] Unlike most of my friends, wasn't gonna go get a PhD. Didn't really want to continue being in a lab underground for a number of years. And so had a mild freak out, decided to take as many computer science classes as possible one year, or try to be this pseudo computer engineer. And from there, \[00:03:00\] very much realized that this is what I wanted to be doing. \[00:03:02\] I wanted to be building infrastructure, building software that other companies get to build on top of. So that began, career in software engineering. And most recently before Speakeasy, I had the opportunity to work at a company called LiveRamp, which was a massive identity data platform powering a lot of advertising in the US as well as elsewhere in the world. \[00:03:26\] And with any massive scale data product like that, you have huge, you know, internal data infrastructure and developer tooling needs internally. And this is, you know, for all of us who've been doing engineering, who do open source, who do developer work, A lot of the best projects out there start as projects internally at companies. \[00:03:46\] So this was very much, you know, foundational experience for me. While I was at, LiveRamp, I moved out to the UK, and actually started to build out an engineering team for the company there. \[00:04:00\] And through that experience, really found that I loved this zero to one process of going from having no idea what you're doing to a couple drawings on a whiteboard to an early prototype to figuring out, you know, if people pay for a product. And that was an amazing experience because doing that with kind of a backing of a company and having a whole machine around you that supports decision making, is not something everyone gets to do. \[00:04:26\] So very, very fortunate to have done that. A lot of what I did at LiveRamp was focused around scaling our internal developer tooling our infrastructure data infrastructure. And so, towards the end of that experience, very much realized what I wanted to do next was actually, commoditize some of the things I had been exposed to internally, to developers everywhere. \[00:04:49\] And then also bring that to other enterprises where there's so much value locked up that, is waiting to be unlocked. \[00:04:56\] ****Mike:** Yeah, I think that's a fairly common feeling for folks who worked \[00:05:00\] at large companies with big engineering teams, that you have hundreds and thousands of man hours that can often get bottled behind decisions that were made long before, people who are still at the company. \[00:05:10\] And, some of the value of things being built, lacks the capacity to scale because things can't always move as quickly at big companies as they can at, zero to one shops as, you might put it. Yeah. Okay. So I guess that brings us to where you are now with speakeasy. So what's the elevator pitch? \[00:05:27\] What's the value proposition for speakeasy? \[00:05:30\] ****Sagar:** Totally. So, SPEAKEASY is a API developer experience platform. We make it really easy for API producers, so people who actually build APIs both internally, externally, to offer world-class develop experience for their users. And for any of us who, you know, worked at these API companies, we know that the mass amount of platforms during effort, that goes into actually building great API, scaling it, doing all the tool last mile tooling to ensure that you know your users are happy.\[00:06:00\] \[00:06:00\] And so speakeasy is that API platform that lets you serve your users, make sure you're able to, you know, unlock new developer communities and double your API usage quarter on quarter. Today we've kind of coming at the problem actually in reverse and have started with the day zero problem that you know, API consumers face when they integrate with APIs, and that's integrating with an API is trial and error. \[00:06:26\] You look at docs. You look at your id, you try out something doesn't really work. There's often no usage examples to go off of. There's very little guidance. And so you, you are either in Slack with, you know, an API team figuring out what to do, or you're hand rolling your own client and SDK to figure out, you know, to integrate something repeatable that maybe other people on your team or your friends can take advantage of. \[00:06:51\] So that problem is something we've seen repeated over and over. And so we realized one of the best ways to kind of enter this problem space is provide a \[00:07:00\] managed SDK offering. And that what that means is, when you are an API producer, you just wanna build your api, but you also want to serve your developers in different communities. \[00:07:12\] And to do that, you need great SDKs. These are kind of how, as API users, this is our primary way of interacting with APIs today. And so a great SDK makes our usage of an API loyal. It ensures that we have less hours. It's, it's honestly you know, creates this great experience around using API I and, and kind of this proof in the pudding here, in that the top 1% of API companies that drive traffic today all have great SDKs and, and kind of huge teams internally that maintain them. \[00:07:46\] At LiveRamp as well, we had kind of an internal DSL for API building that we created and eventually worked towards this goal. And we've seen this pattern repeated many, at many other companies before. Palantir, Stripe, \[00:08:00\] Twilio, all have this kind of internal, experience around the end-to-end type safety and making sure the end users get ergonomic experiences. \[00:08:08\] So that's where we're starting today as we move forward the managed SDKs we're starting to offer more components of API infrastructure that help you scale your API. So things like self-service authentication that works with any API gateway self-service, troubleshooting and visibility so your end users can actually understand how they're using your API. \[00:08:30\] And then eventually going down, or going upstream I should say, into, so the server side, helping you maintain your API, create API specs, this is, you take out of all of those tough problems that API producers face. \[00:08:43\] ****Mike:** Yeah, there's a lot there. I think that you're tackling a problem space that is obviously quite diverse in what you need to be able to address, but also is one where people can really relate to the series of problems that you've just described. \[00:08:58\] Especially I like that you \[00:09:00\] said something along the lines of, what you're doing is helping API engineering teams just build the API right? They're spending their time doing the thing that probably their company does best, which is have some super specific engineering know-how about building something, whatever. \[00:09:16\] It's, you know, it can be anything from Phil's company building things to track trees being planted in the middle of the UK to you know, whatever it is up and down. But when those groups of engineers spend their time figuring out how to do SDKs in various languages correctly, they're essentially wasting man hours on unsolved problems to some extent. \[00:09:36\] Is that roughly the gist of why having a managed SDK offering is something that you're interested in providing? \[00:09:45\] ****Sagar:** Exactly. It's, we want to take the burden of engineering teams looking to scale. So this is the bottom line for companies. At the same time, we, we think there's like a top line advantage as well. \[00:09:57\] So companies bring into, you know, being \[00:10:00\] able to break into new developer communities means potentially more revenue as well. So we, we like this kind of dual approach of yeah, reduce your cost, but also scaling usage. \[00:10:09\] ****Mike:** Yeah. Okay. So, let's talk about the nuts and bolts of it then. So you, you offer managed, SDKs I'm sure in a variety of languages. \[00:10:17\] What does that look like? \[00:10:19\] ****Sagar:** Yeah, Clear question. So for those of you who are familiar with like open API and open standards in the ecosystem, there are, you know, tons of open source generators today that will get you 50% of the way there. A lot of teams like hand roll SDKs as well, is a common pattern we've seen. \[00:10:37\] But to have really a great SDKs, something that's truly ergonomic, to your point, Mike, you know, built out in a variety of languages hosted in GitHub published to package managers taking, you know, that whole life cycle of concerns is actually really difficult. And so our product, can be thought of like a zero touch experience where we connect directly to your GitHub, wherever your API \[00:11:00\] specs are hosted, and actually create a GitHub workflow that automatically validates, you know, checks, enriches your spec, and then creates SDKs and we do something like seven languages now and then takes care of publishing them as well to package managers and keeping them validated and up to date. So that whole, you know, box diagram that you people probably have internally of API spec to SDKs is something that we're taking care of today. \[00:11:28\] Yeah. Okay. Okay. And so which spec formats do you work with? \[00:11:34\] Yeah, so we, we work with Open API primarily today, but have started to support Postman collections. Want to expand to the kind of breadth of the whole JSON Schema eventually? We know that's like an ongoing internal discussion of how wide to go and eventually in the future, even look at other formats potentially put out something of our own. \[00:11:54\] But today, open API has, you know, such amazing adoption, continue to be, you know, \[00:12:00\] evangelists of that ecosystem and, and see a really strong mutual existence there. \[00:12:05\] ****Mike:** Yeah, I think I can understand that, especially because Open API is so broadly adopted it's something that there's, there's often domain expertise for at least in the design process. \[00:12:15\] And, you know, things, things can go all kinds of haywire depending on the way the engineering team is set up once the spec is created. And you know, we, we've talked about on the show all sorts of challenges that come up with that. Even at that too, the Open API spec is an evolving thing, right? \[00:12:31\] Like it is, it has moved and continues to move in for good reason. As we learn more about building APIs and client libraries from scratch, we, we want more naturally out of what, we think the spec should be able to do. And for a long time, I think that's also where JSON Schema has been an interesting topic too. \[00:12:47\] Talking about validation and automatic generation of UIs and stuff like that is really, actually kind of an interesting, problem set to think about. So, tell me a little bit about the, the, I guess the customers, that \[00:13:00\] are looking at Speakeasy or the engineering teams that are using Speakeasy. Do they have like a commonality? \[00:13:04\] Is there a part of the engineering journey that they're typically at when they come to you? \[00:13:09\] ****Sagar:** Yeah, it's, it's a great question. We, have been working with a variety of companies across couple different verticals, e-commerce, FinTech, developer tooling, other infrastructure companies. And we see largely two modes of operation. \[00:13:25\] We have teams that are very early in the journey. They're essentially doing the first API launch, they're launching the product, launching the business. And so these are, these are what I would call lighthouse teams who realize that, you know, from day one, they want great SDKs, great documentation, good authentication. \[00:13:43\] They want everything that makes an API experience world class. And so we're able to help these kind of teams take work off the plate, ensure that great API launch happens for them. And then you have teams that, you know, are, most of them mature with the APIs usually, \[00:14:00\] you know, dozens of endpoints, hundreds of operations and their API spec, and have a sprawling API ecosystem internally. \[00:14:07\] And so for them, the value proposition of speakeasy is not just helping the external developer experience, but also the internal developer experience. Often you get to that scale where you have teams with, you know, dozens of API specs. Each team has their own open API and suddenly there's a question of how do you manage all these interfaces? \[00:14:27\] Are you going to have one massive monorepo with SDKs for all your APIs, or do you want separate ones? And there's all these questions that pop up, at that scale that, and that's where we come in. We actually help capture a lot of that decision making in our product. \[00:14:42\] ****Mike:** Yeah, that actually touches on a really interesting point that I feel like I've seen maybe rumblings of a lot lately. \[00:14:48\] I feel like for particularly large engineering teams, maybe not even large, but ones that have existed for a while, there's this notion of almost for lack of a better term, I'll call it internal developer relations, where part of the \[00:15:00\] challenge becomes how do we make it better to be a developer working on this thing? \[00:15:04\] And just knowing the breadth of tools available and like the, capacity to do better and make the engineering process better is actually kind of a big treasure trove of value for engineering companies. You know, engineers burn out because their product is hard to work on, or because the process of getting code from ideation to production becomes really challenging. \[00:15:24\] And I think companies are really starting to get wise to that, where, being able to lift some of the weight off the shoulders of engineers makes life easier and makes them, you know, as close to making it joyful to write code as it can possibly be. You know, I don't think I would say that code writing code is always joyful, but, it makes it better. \[00:15:40\] Right? And, and, having tool sets that do that is certainly something that will help with that kind of conversation. Really interesting to hear that you're, you're sort of larger and maybe more experienced customers are thinking about that too. \[00:15:53\] ****Sagar:** Absolutely, in a way, we almost treated as a way to onboard unto product where we recommend teams to use us for an internal \[00:16:00\] use case first before going external. And if you can prove to your own self and developer teams that there's a massive productivity as well as, you know, overall value proposition, then hopefully that is obvious to your end users eventually. \[00:16:14\] ****Mike:** Yeah, Ideally it's a toolset that speaks for itself, the value should become apparent. So speaking about adoption, then, what is the strategy for pricing? Is there an onboarding experience that allows people to test out Speakeasy without having to spin up a , you know, a new SaaS charge? \[00:16:33\] ****Sagar:** Yeah, so you can get onboard with us today. We have very generous free tier within the product. All it takes is an open API spec to get started. And you know, even if you don't have one, we actually will work with you to put one together. We've been doing some fun Hack week projects internally to use the latest, LLM and ChatGPT tech to actually trade specs for you if you don't have one. \[00:17:00\] Yeah. Fun stuff around that with, you know, fun stuff with that around the corner and hope to share more. But yeah, so that, that's all we need to get started. And then as you become a customer of Speakeasy, we basically charge in the number of operations you manage to us, so, Yeah, an operation is one rest word per endpoint. \[00:17:20\] We're basically incentivized to help you grow your API and grow usage. So we grow as you grow pretty friendly with, with companies. You know, we're, we're pretty early ourself and so always iterating, trying to understand where the maximum value for teams. \[00:17:35\] ****Mike:** Yeah, sure. And it goes a long way to be able to let people just get in and try the thing and, and give you feedback, see what's gonna work for them. \[00:17:43\] ****Sagar:** That's totally true. One, one of the ways we we'd like to work with people is if we see that you have an API out there publicly hosted, we'll actually just create an SDK, send it to you and say, Hey, this is yours. Feel free to use it. You know, trying to leave a little gift on your \[00:18:00\] dose step and see what happens. \[00:18:02\] ****Mike:** Sure. Yeah. Wow, that's, that's a very pleasant surprise, I'm sure to wake up to. Well what an interesting idea. So you talked a lot about focusing on idiomatic SDK generation, which I think for folks who have used client libraries that are idiomatic versus not idiomatic, it's always apparent when you bump into something that's very thoughtfully made. \[00:18:22\] I'm sure there's a lot of interesting decisions you have to make to generate client libraries in different languages that feel natural to, to both languages. What does your approach to that look like? \[00:18:33\] ****Sagar:** Yeah, it's, it's a great question. You, you're absolutely right. The devil's in the details when it comes to ergonomics. \[00:18:38\] I, I think, think of it in two ways. First and foremost, there's some top level philosophies that we try to maintain. Type safety is number one. We want to make sure that end users get the type safe experience. This is particularly a problem in, in rest where, you know, server to client type safety is not always guaranteed. \[00:18:56\] We wanna make sure they're human readable. So in the case \[00:19:00\] that the SDK fails or there's an, or there's some kind of error, you should be able to actually understand how the SDK is working. We want to give you batteries, include experience. So if you've come across a good SDK, you know it's batteries included when it has built in telemetry and retries and pagination, and it kind of takes care of the long tail of problems. \[00:19:20\] You come you, you know, you bump into as you do API integration. And then finally be fault tolerant. What's worse than this? You know, not having an SDK is an SDK that's broken. That is something no developer likes. And so that's another top level concern for us. So those are kind of a high level philosophies, but as we get into each language, we found that people are extremely opinionated about what a good Go SDK versus a good C Sharp SDK looks like. \[00:19:49\] You know, we try our best to maintain some stability and develop experience across them, but we also provide a set of extensions and hooks for people to actually customize the SDK \[00:20:00\] output. Because we know that, you know, we, we can be opinionated and prescribe what a good Go SDK looks like. \[00:20:05\] However, each company and team sees an SDK as actually a representation of their ensuring philosophy. And so we need to ensure that we're customizable enough that people can feel like it's their own and we're taking off, you know, getting rid of all the heavy lifting of that repetitive coding they need to do, but at the same time, they can customize and have a branded expense when they use us. So as we get into more languages, we're, we're exposing more extensions and eventually would love to get to a place where we actually expose the underlying generator as well for people to extend and, you know, provide their own system of customizing the SDKs. \[00:20:45\] ****Mike:** Sure, yeah. That's the tacit dream of an idiomatic, SDK, is that not only should it feel good in that language, but it should maybe feel good and also feel like something that's flavored like your team, right, flavored like your company, and maybe uses your style guide or \[00:21:00\] your you know, language features that, that certain teams have adopted and prefer to use. \[00:21:05\] Along those lines, I guess I'm interested in hearing a little bit about how documentation works then for the SDKs that you're generating. \[00:21:11\] ****Sagar:** Yeah. This is somewhere I would say we're still pretty early in. We do provide usage snippets and markdown for every SDK that's created. So we wanna make sure we're giving you the tools for you to be able to take that and embed it in your own docs site or your own app. \[00:21:28\] Longer term we would actually love to ship react embeds around this so that you could, you know, one click get a SDK documentation in Docusaurus or any kind of open source Docs provider. So slowly stepping towards that. And you're right, this is, this is so critical because when you work with something like Open api, it's, there's so much breadth, right? \[00:21:47\] You have, you know, types of serialization, you have ways of managing request and responses. You have all of these data types and things that really should just be captured in usage examples for people to just copy, paste \[00:22:00\] and get started. So today we have markdown documentation. Tomorrow we do wanna launch a SDK first docs product. \[00:22:09\] There's lots of great API docs providers out there, but we haven't seen an SDK docs provider. And from our point of view, it's a, it's a completely different kind of user journey. \[00:22:18\] ****Mike:** Yeah, I think that's fair. So you and I haven't spoken about this, but I'm, I'm a former Stripe employee and, and having a fair bit of insight into what it looked like for Stripe to generate documentation for the broad variety of, of client libraries that Stripe produces. \[00:22:35\] It's a pretty custom thing, like it, it takes a lot of work to make it. Not only look presentable, but also feel good and be informative for the people consuming it. So it's worth being thoughtful there. And I can understand the need to kind of take your time and get to a place where you're building something that not only is functional, but you're proud of and sort of ticks all the boxes for the people who want to consume your SDKs. \[00:22:57\] What are you thinking about next? What's Speakeasy working on \[00:23:00\] now? \[00:23:01\] ****Sagar:** Yeah, so for us, SDKs has been this awesome wedge into this rich API ecosystem. Moving forward from there, there's. Many more ancillary concerns around the API that we're trying to address. Things like self-service auth. So to give you an example, you know, like companies like Stripe or GitHub have amazing self-service API auth experience. \[00:23:24\] I can get a GitHub personal access token and I can provide scopes and get going completely by myself. But how many APIs out there have that kind of self-service experience? So powering all of those experiences is something that we are going after. And then as, as we progress from starting to grow up the value chain to the server side and support things like server side generation giving you middleware to embed various parts of the API lifecycle directly into your build. \[00:23:50\] So really exciting path forward for speakeasy on, on the SDKs itself as well. I'll throw in a plug that, a lot of our early customers have been \[00:24:00\] amazing and pushed us to think about SDKs. It's not just languages, but also run times and, and novel surfaces. So, We started to invest in generating Terraform as another SDK type. \[00:24:12\] A lot of infrastructure companies need Terraform providers and it's something that can be taken created directly from your open API spec. Eventually, you can think of things like command line tools, Zapier plugins, chat GPT plugins, these all s SDKs of an api. And should all you know, the moment you have an API spec should all become immediately available to you. \[00:24:33\] ****Mike:** Yeah, that makes a lot of sense. There's, a lot of ancillary value you can provide by just plugging, plugging into those little things. Having a, an API that works well with Zapier, for example, enables a lot of no-code and low-code experiences for people who may not be, happy in Postman or, you know, trying to, to send curl requests to figure out how something works but can certainly provide value you know, almost straight out of the, the gate there. \[00:24:59\] Yeah, \[00:25:00\] there's, it sounds like you have a broad array of things to tackle and definitely like, have, have come a long way to provide value for your customers as is right now. That's it's a very interesting journey and I think one of the things that we're seeing in, in the world of APIs right now too, is a lot of companies are thinking about this as a more holistic view of it's not just write the spec and then hope someone builds the spec. \[00:25:21\] It's write the spec and then generate value from that as much as possible. I'm personally enjoying being able to see all these things come into existence because it's making, the, the sort of middle developer experience really nice where you consume APIs and build things that are functional from that has suddenly become much easier and nicer across the board too. \[00:25:39\] ****Sagar:** Absolutely. I think especially to today as we see this new wave of AI companies pop up. At the end of the day, every AI company is an API company. It's really interesting to see, you know, ChatGPT adopt open API as the standard for plugins. So it's, this ecosystem is just seeing so many big tailwinds and \[00:26:00\] continues to grow and we want to be at the center of it and, and help this ecosystem continue to grow. \[00:26:06\] ****Mike:** Yeah, well that's, that's really encouraging to hear. I I'm one of those people that spends a lot of time thinking about ChatGPT and, and AI based products, and. Have dipped my toe in it a few times for sure. But I'm also a very happy end user of it and nice to see that, like the standards I'm already familiar with will be really helpful for that future. \[00:26:25\] Tell me a bit about how speakeasy is working with Open Source. Are there, are you working with like open api to, to develop future specs? How's how's that side of the house work for you? \[00:26:38\] ****Sagar:** Totally. So the core product we have is closed Source today. It is an ongoing discussion internally on how we make more parts of our product open source. \[00:26:49\] In the meantime, what we've done is we've adopted this philosophy of any tooling that we build that makes our lives easier. That is, you know, not, you know, our core code generation is \[00:27:00\] stuff that we open source. So I'm happy to provide some links to some interesting projects we work on, like open API validators. \[00:27:06\] Open sourced new templating engine that allows us to work more flexibly with open api. So there's lots of ways we see we can add value to the open source community while still keeping our core product close source. We support, you know, the newest versions of Open api. I think as we grow we'll and we have dozens of companies working with us, we'll be in a really good place to help inform and provide feedback to the Open API Foundation on, on future iterations of the protocol. \[00:27:35\] ****Mike:** Yeah, I would love to see some of those open source projects. And I think the open api foundation and the spec itself gets better with more voices in, in the room. In particular, when you're thinking about expanding the tooling that comes from open API and the things that we're building based on it especially people in position you're in, right? \[00:27:55\] Where your company is building tools that directly consume and build from open api, I think your feedback becomes \[00:28:00\] particularly valuable because you've got insight on both sides of that, consuming the spec, delivering things with the spec. And all the little holes that kind of develop as, as a result of that and the bumps along the way there. \[00:28:10\] So tell me about I guess the future from here. You, you have a, a series of releases coming out. I actually really liked, I was browsing your site earlier. It looks like you have a public roadmap, which I think is really interesting. Where consumers or consumers, people are interested in speakeasy can go see what you're bringing, bringing next, but also vote on things and leave feedback on what's coming with the roadmap. \[00:28:34\] Are there, are there things you're interested in getting around to building in the future? Like, are there releases coming that your team is kind of chomping at the bit to get out the door? \[00:28:42\] ****Sagar:** Yeah, totally. And we, we always welcome feedback and uploads downloads as well to our, to our roadmap. Always looking for people to, to be loud and opinionated. \[00:28:52\] Yeah, so some really excited things coming up on the immediate front with SDKs. You know, this is, it's amazing once you get into s STKs, the \[00:29:00\] number of things, number of value you can start to capture and, you know, make lives easier for developers. So, I'm excited for support for mobile, STKs supports for OAuth Lins STKs, you know, never refreshing token again. \[00:29:14\] Pagination, those kind of things that we see can all be embedded into the interface for developer. So all you have to do is integrate and you get all that stuff for free. As we think p beyond SDKs, we've started to get really excited and experiment with how we can give you embedded components for your infrastructure. \[00:29:36\] So one of the things I mentioned, we've been developing self-service art with to various gateways. You know, everyone uses an API gateway, but, but struggles to actually figure out how to provide self-service keys. So that's an area that we've continued to develop and, and push forward. Some of the areas I'll highlight, we very much believe that, you know, open API is an amazing standard, but for a lot of \[00:30:00\] teams they just want to get all this stuff in place and not even worry about open API sometimes. \[00:30:04\] Right? Sure. So we're trying to go really upstream here and we have a really exciting API release coming out, which, You can think of as a clippy for your open API spec that actually helps you walk through your spec, keep it up to date. Automat suggest changes, make automatically fix things for you. \[00:30:24\] And we've talked a lot of engineering products, technical rider teams that really spend a lot of time just on spec maintenance. And it takes time away from kinda core product development. So I think we're very uniquely positioned in this. World of SDKs and API infrastructure to actually start capturing all of that value and start making their lives a lot easier. \[00:30:45\] So really excited. When my API spec writes itself. \[00:30:50\] ****Mike:** I'll say, yeah, gosh, that's living the dream at that point. Exactly. So tell me a little bit about your team. How, how large \[00:31:00\] is the team building speakeasy and what sorts of engineers do you have on staff? \[00:31:04\] ****Sagar:** Yeah, so we're a team of eight right now split between the UK and the US. \[00:31:10\] Mostly on kind of West coast and, and GMT. Really amazing team, super fortunate to be working with them. Most of the developers on the team are platform engineers who very familiar with APIs, AI infrastructure. Some of them have built a p i platforms before at at pretty large companies. I think one thing that makes our team super unique is the spirit of experimentation is really strong. \[00:31:35\] We have, you know, every developer is a product manager. Everyone is driving on threads and driving on different angles themselves. Often will come to a standup and someone has worked on something completely new and completely fresh and. Based on customer feedback and, and ideas. So really excited to be working with them. \[00:31:55\] We are growing and hiring, so we're looking to hire several more founding \[00:32:00\] engineers, founding developer marketing role as well as thinking about roles like for deployed engineer account executives and sales as a little bit further down the road as we scale. \[00:32:13\] ****Mike:** Got it. Yeah. Well, what an exciting place to be in. It sounds like there's lots of growth to be had there. And it sounds like a really interesting team. So I, I will make sure to put some links in the show description to make sure that your open roles are something we link to. Are you hiring just in on the West coast and in the UK? That's usually something that our audience is keen to hear about. \[00:32:31\] ****Sagar:** We are remote first, so we are hiring everywhere. We do like to hire folks between the West Coast and GMT just to keep our time zone spread limited. But look, we're, we're always open to talking to amazing people everywhere and. I think first and foremost, passion and interest in the space is what stands up. \[00:32:52\] ****Mike:** Without a doubt. I think that's probably a good way to characterize early, early hires at companies like yours. Makes a lot of sense. \[00:33:00\] Okay, so Saar from here which, what's the best place for folks to get in touch with you if they're interested in \[00:33:06\] talking about APIs? Totally. So best place would be just joining us Slack community, having a chat with us. \[00:33:14\] ****Sagar:** If you go to our website at speakeasy api.dev, you can join us Slack community that way. You also drop us an email link through some of the information provided there. If you want to try out the product there's a button to join our beta. We'll immediately reach out to you and help you get started. \[00:33:33\] We also, most of our teams serve around APIs. You own hate Slack as well, so you'll find us there. You can always bug us there directly as well. I was telling Mike it is the second most used Slack community. For us after our own. So happy to talk to them as well. Yeah, I'm gonna, \[00:33:49\] ****Mike:** I'm gonna get you to write that down on a piece of paper so I can frame it. \[00:33:53\] That's, that's always very, very exciting for us to hear. Yeah, so Sagar and I were actually chatting on the APIs. You won't hate \[00:34:00\] Slack, so if you happen to be a member there, feel free to join. That is also free. We make lots of noise about lots of things in there and lots of people smarter than me to learn from in there, which is always really encouraging. \[00:34:11\] Sagar, thank, thank you so much for joining me today. I really appreciate it. We'd be happy to have you come on and chat anytime in the future if you've got new releases and new features, new functionality to talk about. It's been a real privilege. Thanks so much. \[00:34:24\] ****Sagar:** Nice to meet you, Mike. And yeah, likewise. \[00:34:26\] ****Mike:** All right, take care. ### We Build Better Together 🏳️‍🌈 URL: https://apisyouwonthate.com/newsletter/we-build-better-together/ Last updated: 2023-06-16T00:45:01.000Z Happy Pride Month. With all the recent attacks on the LGBTQ community, we think it's worth a reminder that we build better things together. Accepting people for who they are shouldn't be something that is debated. The LGBTQ community has contributed to programming from its inception. [Alan Turning](https://en.wikipedia.org/wiki/Alan%5FTuring?ref=apisyouwonthate.com), solving Enigma and helping shorten WWII. [Sophie Wilson](https://en.wikipedia.org/wiki/Sophie%5FWilson?ref=apisyouwonthate.com) helped design the architecture behind ARM processors and the BBC Micro. There are more people than we can fit in an email, and we recommend reading about some more [here](https://www.thecodingspace.com/blog/2022-05-27-10-queer-pioneers-of-computer-science/?ref=apisyouwonthate.com). The point is the world would not be as unique, bright, or interesting without people from the LGBTQ community. To our LGBTQ readers, keep being you, adding colour to the world. Please know the APIs You Won't Hate team loves having you here. 🫶 Phil, Mike, and Alexander --- ### Monthly API Round-Up Our monthly round-up of articles, news and updates from the web. ### [Build an API with Deno, Deno Deploy and DenoKV](https://deno.com/blog/build-crud-api-oak-denokv?ref=apisyouwonthate.com) Deno has come a long way since its first release. If you dismissed Deno at first, it might be worth another look with the release of DenoKV, a database built right into the runtime. This tutorial will show you how simple it is to use DenoKV by making a CRUD API. ### [REST APIs Must Be Hyper-Text Driven](https://htmx.org/essays/how-did-rest-come-to-mean-the-opposite-of-rest/?ref=apisyouwonthate.com) The author of htmx covers the history of RESTish APIs and what the term has come to mean. I had a good laugh after reading this, as it highlights two key points. REST is used to describe any JSON API, and as a whole, the industry has given up on hypermedia controls. ### [State of Edge Functions](https://deno.com/blog/state-of-edge-functions-2023?ref=apisyouwonthate.com) Edge functions are here to stay and have uses for both backend and frontend development. You may not think there for building APIs, but there is still a lot of thought to be given about edge optimizing specific routes with API Gateways. You could even move certain APIs entirely to the edge. Deno shares their results after surveying developers on the current and future of the edge. ### [The Next Generation of Serverless](https://www.fermyon.com/blog/next-generation-of-serverless-is-happening?ref=apisyouwonthate.com) Serverless is evolving. Tools like Cloudflare, Deno and Vercel offer serverless compute without cold starts. Matt Butcher talks us through what serverless is, the evolution of the cloud and the lineage from PHP. Worth a read. ### [Migrating from Supabase](https://blog.val.town/blog/migrating-from-supabase?ref=apisyouwonthate.com) Val Town, a service for running, deploying and sharing JavaScript snippets, shares their journey of migrating from Supabase. Let's steer clear of being dogmatic here. Supabase is an excellent product for building your backend; however, it did not scale for Val Town. Their post breaks down database issues, migration and ensuring uptime. --- ### APIs You Won't Hate The latest content from the team at API's You Won't Hate. ### [Rich Data For Everyone on Earth](https://apisyouwonthate.com/podcast/rich-data-for-every-location-on-earth-with-ed-freyfogle-from-opencage/) On this episode, Mike and Phil chat with Ed Freyfogle from OpenCage about their API Client libraries for Geocoding, reverse geocoding, and the infrastructure required to keep accurate data points for every location on Earth. --- ### From Our Community Articles written and shared in our free [Slack community](https://slack.apisyouwonthate.com/?ref=apisyouwonthate.com). ### [Tips For Better Documentation with OpenAPI](https://lornajane.net/posts/2023/tips-for-better-documentation-with-openapi?ref=apisyouwonthate.com) Lorna Mitchell shares her top tips for creating better documentation with OpenAPI. The best APIs have the best docs and should be a joy to read, making Lorna's tips worth reading. ### [Composing API Models with JSON Schema](https://apidesignmatters.substack.com/p/composing-api-models-with-json-schema) David Biesack has been writing a series on why API design matters. In this issue, he talks about further modelling for requests and responses with JSON Schema. Don't worry if you feel lost jumping into the middle of the series; there is a link to the start in the first paragraph. --- ### Rich data for every location on Earth, with Ed Freyfogle from OpenCage URL: https://apisyouwonthate.com/podcast/rich-data-for-every-location-on-earth-with-ed-freyfogle-from-opencage/ Last updated: 2023-06-12T12:54:18.000Z ## Show Notes - [OpenCage](https://opencagedata.com/?ref=apisyouwonthate.com): Convert coordinates to and from places - [OpenCage Client Libraries](https://opencagedata.com/sdks?ref=apisyouwonthate.com) - [Environmental Commitment](https://opencagedata.com/about?ref=apisyouwonthate.com#environmental-commitment) - **Ed Freyfogle**: Cofounder of OpenCage ([freyfogle.com](https://freyfogle.com/?ref=apisyouwonthate.com), [LinkedIn](http://www.linkedin.com/in/edfreyfogle?ref=apisyouwonthate.com), [Mastodon](https://mastodon.social/@freyfogle?ref=apisyouwonthate.com)) - [OpenStreetMap](https://www.openstreetmap.org/?ref=apisyouwonthate.com#map=4/38.01/-95.84) - [Localistico](https://localistico.com/?ref=apisyouwonthate.com) \- Taking Customers from Search to Store - [Paper Towns](https://en.wikipedia.org/wiki/Phantom%5Fsettlement?ref=apisyouwonthate.com) - [GeoMob](https://thegeomob.com/?ref=apisyouwonthate.com) - [meetups](https://thegeomob.com/events?ref=apisyouwonthate.com) - [podcast](https://thegeomob.com/podcast?ref=apisyouwonthate.com) - [OpenCage on Mastodon](https://en.osm.town/@opencage?ref=apisyouwonthate.com) ## Transcript #### Rich data for every location on Earth, with Ed Freyfogle from OpenCage \[00:00:00\] **Mike:** Hello, hello, and welcome back to APIs You Won't Hate. I am your co-host today, Mike Bifulco, one of the, uh, I guess founders of APIs you won't hate, and I'm hanging out with, uh, my good pal, Phil Sturgeon. Phil, good morning. How are you today? \[00:00:14\] **Phil S:** Hello. I should know where I am today. Usually you ask me where I am and I'm like, today I'm in Bristol and it's lovely and I'm having a great time. \[00:00:20\] How are you doing? There we go. I'm doing \[00:00:22\] **Mike:** great, man. It's, uh, it's good to see you. Definitely like one of these nice proper spring mornings over here while we're recording. Of course, you know, when it releases, it could be, uh, anything weather-wise and, and wherever you are. But yeah, having a nice morning over here. \[00:00:34\] Um, and I'm, I'm happy today. We're gonna get to, uh, sit and meet and chat with our new friend here, uh, ed Fry Fogel, uh, ed. Thanks so much for joining us. Um, appreciate you being here. You are working on a project called Open Cage, among other things. Um, tell us a little bit about yourself and, uh, how \[00:00:49\] **Ed Freyfogle:** you got to where you're now. \[00:00:50\] Hi guys. Thanks for having me on the show. I am a longtime listener, uh, longtime API practitioner. Um, yeah, so the main project I work on is a company called \[00:01:00\] Open Cage. We have been doing that now for about eight years. Uh, it's a geocoding api, so we offer forward and reverse geocoding. For those that don't know forward geocoding is you have an address or, or let's say, uh, some sort of text string that refers to a location and you wanna know the geographic coordinates of that. \[00:01:20\] And reverse is the opposite. You have, you have geographic coordinates and you wanna know the, the location description. Uh, and that's what we do. Uh, that's it. That, that sounds like a little bit, actually, it keeps us quite busy. Um, we. I guess the differentiator of our services that we do this only with open data, so we can talk a bit about exactly what open data is. \[00:01:42\] But, uh, listeners may have heard of projects like Open Street Map is probably one of the most famous, um, open data sources. We we're, we're heavily involved in the open street map, but also there are other open data projects as well. Um, yeah, so, so happy to go deep on all of that and talk about kind of the challenges of.\[00:02:00\] \[00:02:00\] Of running an API based business. Particularly what, I mean our biggest competitor is Google Maps. You might have heard of them. It's a Right, yeah, sure. Small company out of California. Yeah. Yeah. So, um, uh, \[00:02:11\] **Mike:** I'm, I'm kind of curious to hear the genesis of Open Cage. So where did the need come from and how long ago did you start building it? \[00:02:17\] **Ed Freyfogle:** Well, we, we, I, I originally had a, another business. I was co-founder of a business many years ago. This is, this is really gonna date me, but, um, Around 2005, 2006, Google Maps came out and for the first time this, you had the amazing phenomenon that you could put points on a map, right? And you could move the map around. \[00:02:39\] I don't, I dunno if you remember this. I mean, kids today assume this was all how it always worked, but that wasn't the case. Uh, and so this was a huge innovation. And so we, um, like many other people had the idea that we would put, do, use this in, in the real estate space. And so we built a, a real estate search engine, kind of a, a metasearch engine where we would aggregate all the \[00:03:00\] different real estate listings from different, uh, sites and started the company do that, ran that company for about 10 years based in London. \[00:03:07\] But we served 10 different markets, um, and uh, you know, did quite well. Uh, anyway, one of the key challenges there is, of course you have to geocode. So we typically would get the, the property listings as addresses. And to put them on the map, you need the longit latitude. And so we would start, you know, figuring out ways to do that, using external APIs, trying to use our own, uh, software. \[00:03:30\] Um, and it was challenging. I mean, some countries, some countries it's very easy. Uh, for example, in the UK you, if you have very precise postcodes, uh, some countries have very good address data. So we were also serving markets like Brazil and India. The much less good data up there. Um, I should also mention that while this was going on, those, the 10 years that around the company was also coincided with the exact, the founding of Open Street Map in London, which, which initially started in 2004. \[00:03:57\] Um, and then kind of really took off. And so \[00:04:00\] we were very active kind of in that scene going, going to some of the, the pub meetups and things like that. So got to know this space better and better. Um, anyway, fast forward to 2015\. We eventually sold this real estate business to one of our competitors. Um, but in the years leading up to that, in like from like 2013 or so, some of our customers, our customers were big media companies in these different markets. \[00:04:25\] And some of them had said, how, how do you guys do the geocoding? How do you, you know? And so, um, we said, well, we have this internal technology and, and basically they asked us if they could use it. And so we kind of thought, let's spin this off as a, as its own project. And we did that under the name Open Cage. \[00:04:40\] Um, but it was just really still very much in fledgling form when the acquisition happened. So it was very much a beta product. Uh, so the buyer was not interested in that. And so myself and, and one of my colleagues at the old business, he, he and I came the co-founders of Open Cage and we, we took it over when the real estate business was sold \[00:05:00\] and we've been running it ever since. \[00:05:02\] So, so it's been going now since 2015\. Slow but steady, uh, growth, you know, that there's, there is a lot of demand for geocoding, and we can, we can talk a bit about that, about why that is. But, um, yeah, and, and we've been, it's, it's, it's our, it's a, you know, it, it pays the rent. So that's what we, \[00:05:21\] **Mike:** yeah, sure. No doubt. \[00:05:22\] I think it's really interesting to be a serial co-founder and your, your second, um, company is essentially the brainchild of the first company, uh, kind of like a. A bit that broke off and, and became its own success story. That's super interesting. Um, have you found that along the way, um, the things you learned building your first company, the real estate search engine sort of helped to inform what you're doing now? \[00:05:45\] Well, \[00:05:46\] **Ed Freyfogle:** the two, the two companies were quite different in that, and when they, and they said this is quite common, right? For the second business, you wanted to do the exact y you're so fed up with all the problems of the first business. You wanna do the exact opposite. And, and that was kind of the case with us as well \[00:06:00\] in that. \[00:06:00\] The first business, first of all, it was a much more of a traditional startup in that we, you know, we had external funding. We eventually had about 20 employees. Um, it was kind of more your traditional startup. Uh, but our customers were mainly big media companies. So the people who run the big, uh, real estate websites of the world, which on the one hand was good because they were, they would pass a lot, but, but it was, you know, very one-off bespoke kind of deals, long negotiations, that kinda thing. \[00:06:30\] With, with open Cage. Our business, particularly in the beginning, it was much more self-service. You just come to the website and stick your credit card in and you can start geocoding. Um, much less kind of customer interaction. Um, very tight. I mean, one, the, one of the reasons we liked being an API business is much less emphasis on, um, User interface. \[00:06:53\] I mean, of course the API needs has a good interface, but you don't need to spend, you don't need to have a, spend all your time designing and things like that. \[00:07:00\] Um, much less SEO susceptible. Our first business was very dependent on seo, which when it's great, it's great, and when it's bad, it's very bad. Um, so we kind of went exactly in the opposite direction. \[00:07:13\] Of that. Um, and I, and, and it's a, there's no funding, there's no external funding. It's, it's just purely funded from profits and from our, in the beginning, from our own savings and things, um, from selling the first business. But, um, so kind of the opposite in many regards. Uh, sure. So, yeah, this, \[00:07:33\] **Mike:** by, by now, this may feel like an obvious thing to you and maybe to the people listening to this, but, uh, I'm, I'm curious what your, has your value proposition changed as, um, The years have gone from, hey, this is an alternative to Google Maps, to, hey, this is, uh, you know, maybe the first place you should turn, or something like that. \[00:07:50\] Why, why were people looking at alternatives to Google Maps to begin with? Well, \[00:07:52\] **Ed Freyfogle:** as, so really we have, we have kind of three competitors, so, so one competitor is definitely Google Maps. Google Maps is the giant in the \[00:08:00\] space. There are a few other big, um, proprietary players. There's a company called here. \[00:08:05\] Basically they do kind of car navigation services and typically their model is that they. They collect their own data. You know, Google has their cars driving around and things like that. Um, and it, it, it's, the data is theirs. They, they, and they run the whole system. And then if you want to use their data, you have to of course pay them in some cases quite a lot. \[00:08:26\] But also you have to agree to their terms and conditions, which can be quite onerous. Um, particularly around attribution, about how long you can store the data. Um, can you keep storing the data after you, if you stop being a customer, for example. Um, you know, many, many players make you delete the data if you ever stop, which, um, but a big issue, of course is cost. \[00:08:47\] Um, so Google charges quite a lot. Um, which, which they need to, to justify cuz they have all these teams collecting data, things like that. The other, \[00:08:56\] **Phil S:** and that's not just, that's not just like fender lock-in, that's fender \[00:09:00\] blackmail. It's like if you try and leave us, you have to delete Correct. Your database. \[00:09:04\] Correct. Correct. That's not, \[00:09:05\] **Ed Freyfogle:** that's not cool to, to their credit. Uh, I mean the, technically the product is of course quite good. Um, and, and because of their dominant position in, in the consumer mindset, uh, you know, the database does get updated. I mean, the first thing someone does if they open a new business is update their listing on Google Maps, cuz otherwise they won't be found. \[00:09:24\] So, um, so what I always tell people is like, look, if, if you want, you know, if, if you feel Google's the best and you wanna pay for that, then then go for that, you know? Uh, but, but you're gonna pay a lot and you're gonna, if you want good enough at a radically more affordable price, we are your choice. Okay? \[00:09:45\] Um, and we're able to do that because we use the open data, but that does. Open a new, um, competitor and so, and, and, and that, that competitor's companies doing it themselves in-house. So, so we \[00:10:00\] often, that's kind of what we're competing with. People are like, well, I need a lot of view coding. Let me do it myself. \[00:10:05\] And that's a difficult argument to have with people. Obviously engineers like to build it themselves. They like to have their own trace to train, set to play with. Those people typically become our best customers once they realize it's a freaking nightmare to run it yourself. Um, or also when, when, you know, maybe the, the guy who ran it, um, leaves the company or whatever, you know, and then ev you know, some new person has to maintain it and they're like, actually this is complex. \[00:10:31\] 1, 1, 1 point to make. There that in one way, though, I guess we're different probably in many of the APIs that you've covered here on the show or that people think about, is. And we, there really are two pieces to our system. One is, of course, the software, like any other api, but secondly, it's the data. The data is changing all the time. \[00:10:51\] And because the world is changing all the time, so the O Open Street map gets about four to 6 million edits per day. \[00:11:00\] Um, so, so this is kind of a living creature, uh, and that always needs to stay updated. So the there is, and, and the volume of data, of course, is massive. So this is kind of a DevOps challenge, if, you know, for, for, to stay on top of that. \[00:11:13\] And it's not something where you, you know, it's not, you just, you install it and then you forget it and, um, you know, maybe once every six months you upgrade to a new version or something. It's not like that. Um, so, uh, so that's why many, many people then turn to experts like ourselves to, to handle that \[00:11:30\] **Mike:** for them. \[00:11:31\] Of course. Yeah. So behind the scenes then I get, I would imagine this is me just, uh, wager a guess. That you're probably not living in the world of, um, a AWS lamb does and things like that. Are you running like a mess of servers in data centers all over the \[00:11:44\] **Ed Freyfogle:** place? Yeah, we have our own servers. We don't, we don't use, um, AWS or any other cloud provider. \[00:11:51\] We we're very happy with Hener. I don't know if you're familiar with Hener in Germany. Uh, not, not just in Germany. They have other locations as well, but, but they're very good. Um, but \[00:12:00\] yeah, we have our own dedicated server set, multiple locations. Of course. Um, Just cost wise, it wouldn't make sense. I mean, with that volume of data, yeah, there's no way you can do it cost effectively. \[00:12:10\] And this is actually one of the arguments I make when I, when people are like, oh, we'll do it ourselves. I'm like, if you do the math, it's gonna be very difficult for you to do it cheaper than letting us do it for you. Certainly, yeah. \[00:12:22\] **Mike:** Yeah. So, so when you started building the product, when Open Cage was born, um, I'd imagine there was probably some basic use cases in mind, uh, right. \[00:12:31\] And, you know, beyond maybe just forward and reverse Geocoding, was there, like, were there certain languages or, um, I don't know, was there, um, uh, a Venn diagram of customers you were after versus customers you were not after for the, \[00:12:43\] **Ed Freyfogle:** uh, api? Well, it's very interesting because there are many, many different types of geocoding and many different use cases, and. \[00:12:50\] One of the challenges that we face is people come to our service and they're very kind of fixated on their, what, what they need. And \[00:13:00\] they think, you know, that's the only way to geocode. So many, many people think only of forward geocoding. They don't think of reverse geocoding, which reverse geocoding is actually massive because it's um, kind of the vehicle tracking space. \[00:13:12\] Um, basically the cost of the cost of a tracking device and the accuracy of a tracking device and the performance of a tracking device In terms of. I've gotten so much better. Right? So, um, more and more things have tracking devices, you know, obviously vehicles, but you know, bikes and things. So, um, so, you know, so there's more and more data being, being general there. \[00:13:38\] But the other point is then thinking about, you know, of course if you send us some coordinates or you send us an address, we can return the opposite. But what we also do now is, um, Kind of crosslink this with other data sets. So there are many, many different ways to refer to locations and many different types of statistical codes, you know, uh, uh, ranging from the obvious ones like iso \[00:14:00\] country codes to, but like all these things, like in the eu, there's a thing called nuts codes, which is a way that people refer to locations or, um, in the US you have FIPs codes, um, all kinds of other relevant information. \[00:14:12\] For example, you might ask, you know, what time zone is this location in? And, and so we have tons of different, uh, data sets that we kind of cross reference and thus make it simpler for developers. So that usually geocoding is only the very, very first piece or, or a tiny piece in a data processing chain. \[00:14:32\] Okay? So in some sort of ETL system or whatever. And so the next step then is, okay, now I got the coordinates. Now I need to feed that into this other data set. So, so we're trying to do things like that for the developer so that they don't need to, you know, take the time to do that. So, um, right. \[00:14:49\] **Mike:** Yeah. I was, uh, playing around in your, um, demo for the Geocoding api, uh, earlier and was kind of struck by the amount of information that comes back, like. \[00:14:57\] For, for folks listening to this, I think you should \[00:15:00\] challenge yourself to think of what would happen, what you would expect back from an a p I if you typed in your home address, for example, or your office address, uh, and asked it to return geo code information to you. To be honest with you, a few things came to mind for me, which are probably obvious latitude, longitude, country, and things like that. \[00:15:15\] But the, the shock that I had was that there's a breadth of information that, uh, open cage returns that are like really thoughtful things that, uh, obviously make sense, but once you see them, they make sense, right? Like, uh, time for sunrise and sunset, what flag emoji to use for that particular location, uh, what country it's in. \[00:15:32\] Things like regions and things like that, that are just like, there's so much information about a given, you know, dot on a map. That I can see this, um, fulfilling lots and lots of use cases. And I can imagine that, uh, as time has gone on, you've added lots of features here that help a lot of people get things done without having to go from one API to the next, to the next to chain this \[00:15:51\] **Ed Freyfogle:** altogether. \[00:15:51\] Yeah, we, we've expanded it quite a bit over the years. Um, but yeah, it's the most obvious example. Mike. Uh, uh, you're in the US right? So if someone types in a US \[00:16:00\] address, uh, US addresses do not contain the county. But they very often, you might wanna know which county I'm in. Uh, so like this gets into some of the use cases that we didn't quite anticipate. \[00:16:09\] So for example, um, payment providers. So let's imagine if a transaction happens in a certain county, maybe then they need to assess a different tax rate, for example. Um, so things like that, uh, where a, you know, humans have developed almost an infinite number of ways to divide the world up and categorize it. \[00:16:30\] And, and, and. Um, and talk about it and assign codes to it and whatever, and, and with tons of weird historical anomalies, you know, and, and so that's that chaos, you know, that, that, that's kind of what we try to simplify and present a clean, obvious interface too. Um, I mean, Phil, you're in the uk, which in my opinion, having spent 10 years living in the uk, uh, is. \[00:16:55\] Possibly the most insane country on earth in terms of \[00:17:00\] admin hierarchies. And I, I mean it started, it starts even, yeah. It starts even with the fact pretty, which country are you in? Right? I mean cause Cause you know, you could say, I'm in the UK and they'd be like, no, I'm in England. No or no, I'm in like as well as a country. \[00:17:12\] Is it not a country? Exactly. We're \[00:17:14\] **Phil S:** a country of countries and that right off the bat is fucking ridiculous. So, So, um, yeah, absolutely. There's, I, I used to work, I've, I've worked with, um, geocoding and Reverse Geocoding a bunch of different times for the last couple of jobs. I mean, uh, all the rage doing all the solo mo stuff in New York. \[00:17:30\] Um, and, uh, yeah, it was working for a, a carpooling company where we had these really weird scenarios. Like most people just think, I would like to type in the address that I'm going to for work, and then we'll see if anyone's going along with me. Should be easy. Some of these ridiculously rich corporate offices were like, there was one, there was one road where the road was one the word, so it was like one Pepsi drive or whatever it was. \[00:17:54\] And so people were typing in the number one Pepsi drive, which doesn't exist because it's actually a, A \[00:18:00\] one one. And so the building was number one word, one Pepsi drive, and there's all these weird things. So I, I have noticed that at least, um, at least your product will kind of take a guess at a few different levels of, of likely result. \[00:18:13\] And so if you don't match on the first attempt, it will kind of try and show you some other things like I just put in my address. And one of the suggestions was, did you mean Bristol? It owned my house, but it also knows the city in different types of place. So it gives you different things you can play with for different confidence levels, I'm guessing. \[00:18:27\] Uh, yeah, and \[00:18:28\] **Ed Freyfogle:** I mean, there's a lot going on. I mean, first of all, duplicate place names, typos, um, I mean, in the uk many, many people, they don't use the number zero. They say like, oh, my phone number is oh seven six. They use, oh, so then when they type just subconsciously, They type O instead of zero. Right? And, and I mean, even though intuitively they pray, no, it's a zero. \[00:18:50\] So all kinds of things like this about the way people think about numbers and things. And, and this is before we even get into the absolute madness of postcodes. \[00:19:00\] Um, uh, and then, and then this is before, you know, at the UK to its credit. I mean, you know, many countries are, are at least have postcodes. Some countries don't have postcodes, some countries don't have addresses. \[00:19:14\] Very common. Uh, uh, you know, most of the world does not have an address. Um, they use kind of just landmarks and things. So it, I mean, this is a, from a technical standpoint, this is a project you can work on, you know, easily for the rest of my life. It can work on geo kitting. There's, there's no shortage of, of weird cases. \[00:19:31\] And, and, and the big issue is a lot of the data that people have that they want to have geocoded is just gibberish. You know, it's bad and um, you know, so we have, it's a lot of working on data cleansing, frankly, so \[00:19:47\] **Phil S:** That's awesome. I mean, what, one question I've got for you. I was going back through, I was working on the, uh, survivor and other people's APIs book, um, self Plugging on the podcast cuz Might as well. \[00:19:57\] And one of the things I remember was how, how many \[00:20:00\] different. Geocoding services have shut down. Like every time I find one that I like that's not Google, it ends up vanishing. And I found the list of ones that was, um, uh, or is it, uh, Yahoo Place Finder. That was really great. Um, Algolia Places vanished Simple. \[00:20:13\] Geo Cor bought out by Urban Airship and then just closed down and rolled into their marketing platform. Um, what's keeping yours going when other one seemed to be \[00:20:21\] **Ed Freyfogle:** vanishing? Well, first of all, Phil, I'm not gonna apologize. We're gonna put 'em all out of business one after another. So it was you. Um, you know, you can tell Google we're gonna inform, um, no. \[00:20:33\] Well, first of all, I will say I do think we have a bit of an advantage of not having investors, right? I think many of the business, some of the business that you've mentioned, you know, have taken lots of funding. And so they're then under a time pressure to kind of provide a return on investment on that funding. \[00:20:52\] And it's difficult. It's difficult. I mean, one of the challenges with geocoding. You know, I can, I can sit here and tell you lots of funny stories about \[00:21:00\] geography and weird use cases or whatever, and if you do not need geocoding, you are not gonna buy geocoding for me, right? It doesn't matter whether you like me, you like my website, like it is very difficult to induce demand for geocoding. \[00:21:13\] So, uh, what I have to do is kind of be present in the market and wait until you have a geocoding project. And then hopefully at that point, you're aware of me, you have a good opinion of me, and, and then you come and test out my service and hopefully we do a good job and then you become a customer. But, um, yeah, but the point is, it, it's, it's very difficult. \[00:21:31\] We've been fortunate that we could go slow, so to speak, and we weren't under pressure to, to very quickly, um, provide, provide a return for any kind of investors. Another point is, uh, I mean, Algolia for example, they. I think you're talking about our places search, which shut down a while back. Um, so, so again, the, the, within the realm of forward geocoding, they're kind of different things. \[00:21:58\] There's what we call \[00:22:00\] geocoding, which is you have a complete address and you wanna know the location. But I think what you're thinking, talking about more is kind of, people have like a restaurant name or they want kind of an auto suggest on a page or whatever. And this is really challenging. The challenging part there is not the technology, the priority part is having the database. \[00:22:18\] Having a database of all the restaurants worldwide is almost impossible. And, and how do you keep that up to date? And, and this is where someone like Google shines, of course, because they have the consumer demand, because they have, you know, billion people walking around with Android phones. So they're collecting data off of, it's gonna be difficult for a startup to compete in that space in terms of how do you get this data, um, and keep it up to date. \[00:22:42\] I mean, now with Covid. You know, we saw this huge, uh, uh, dying off of many businesses. So instantly everyone's database was very out of date. Right. And, and, and you know, of course we try to, to, we rely on the, the open data community, particularly open street map \[00:23:00\] and, and we do our best to give back and support that community. \[00:23:02\] But, you know, there are some areas where open street map is excellent. There are others where, certainly with transient things like restaurants and stores that come and go very quickly or. It might be seasonal. It's a challenge. I mean, there are even, there are even businesses, um, that just help big companies kind of keep their location data up to date and get it out. \[00:23:25\] It's, I, I, I'm friends with the guy, he's the founder of a company in London called Loco, and all they do is help these mega organizations keep all their data up to date on all the different platforms. Cuz you've got. You've got Google, you've got Facebook places, you've got TripAdvisor, you've got all of it. \[00:23:40\] And, and you know, someone like Starbucks for example, I don't know how many tar Starbucks are. I mean, they're like worldwide. Probably three or four Starbucks every day. Opening and closing. Yeah. That's \[00:23:50\] **Phil S:** impossible to manage. Entitled \[00:23:51\] **Ed Freyfogle:** by. So someone's gotta get that data cuz someone's gotta push it out in the right format. \[00:23:56\] Uh, ideally, of course, not just like a, a longitudal latitude, but \[00:24:00\] like a nice pitcher, the opening hours, all these kinds of things. Um, it's a lot. It's a \[00:24:05\] **Phil S:** lot. Yeah, for sure. I mean, I remember the, um, a while ago, like 10 years ago, while Simple geo was still a thing, somebody very proudly announcing that, like they'd invented this, this brand new kind of geocoding service that was basically simple geo, but, but bonk. \[00:24:20\] Which if for our international audience is like one, not particularly major city in the uk. And it's like, yeah, I've, I've got this, uh, service this's basically a data set of every single restaurant in Bourmouth that I bought off someone. And so my, my business is that I put that on a map. \[00:24:35\] **Ed Freyfogle:** Yeah. I mean, no disrespect to the great people of Bourmouth, but, uh, the, there are players who take that approach, uh, in, in the terms of they focus only on a certain country and they try to be really the best at just that country. \[00:24:47\] And, and you know, that can work and obviously there isn't a big advantage to having that local knowledge. And, and, but, you know, many people writing software are trying to build platforms that can be used globally \[00:25:00\] or, or in multiple markets. So for them it's probably better to, it can be a challenge to work with all these different small individual players in the different geographies. \[00:25:08\] So, That's also one of the advantages that we try to provide is that we aggregate these different open data sets. So some countries are quite progressive in releasing the governmental data and things like that. Mm-hmm. \[00:25:21\] **Phil S:** Yeah. You've gotta pay quite a lot for UK data, right? Like the Royal Mail has loads of amazing data, but they keep that under locking key, or is Open Street Map replacing that or supplementing that in some way? \[00:25:32\] Yeah. \[00:25:32\] **Ed Freyfogle:** The, the, the situation, the UK has gotten a bit better. There is, um, open data from the ordinance survey. Again, there are restrictions. You know, it's not, it's not always, you know, I think there's a delay. There's a six month delay until it's published or whatever. Um, yeah, again, the, the, the situation in the UK has its own level of insanity. \[00:25:50\] So, um, but, but nevertheless, the, the, the tide is definitely moving in the direction of more openness and more sharing of data and, \[00:26:00\] um, and so we try to ride that, that tide, ride that wave and aggregate it all and make it simple for people to use. It's encouraging \[00:26:07\] **Mike:** to see that products like this can be so well supported and, and obviously so well used. \[00:26:11\] Um, one of the things that really stands out on your site as you kind of browse as a potential user of, uh, open Cage is that, uh, you support maybe the largest list of, uh, client library SDKs that I've ever seen. Uh, I, and on your site, I think it says something like 30 plus languages and it everything you could imagine laying and go and JavaScript and PHP and all these things. \[00:26:33\] Um, that's, that's pretty impressive. H how does that come about? How do you end up supporting all these different languages? \[00:26:39\] **Ed Freyfogle:** Well, we built it up over time. Um, some of them are, obviously when we started we, you know, we had the, the languages that we use internally and, and, uh, some of the more popular languages. \[00:26:49\] Um, but basically we are our standing offer to anyone. So, so to you, dear listener, is if you can write an SDK for our api. \[00:27:00\] Which, which actually our API is really not that complex. You know, we have one endpoint and, and a few optional parameters as a REST api. Um, anyway, if you write an SDK in a language that we don't yet have or we will gladly pay you or, or an integration with, um, you know, different CMS systems or what any type of software. \[00:27:20\] And get in touch and, and as long as you open source it, we will list it. We will pay you. We will, we will do a blog post where we give you full credit. We don't, we don't try to claim any credit for it. Um, we want obviously as many SDKs and things as possible, so, and a lot of people have taken us up on that over the years. \[00:27:38\] Um, and, and yeah, it's worked out well. Yeah. Yeah. There's quite a few, and, and some of those, some of those are, are actually quite, Active projects. I mean, you know, we obviously, it's not possible for us to know every single language, so it it, we rely on the community for some of these other languages and, but, but it, it's worked out. \[00:27:56\] I mean, people are using them. That being said, I do think, as I said, \[00:28:00\] our, our api, we, we make an effort to keep it as simple as possible, have very few changes. Um, sure. And, and so in that regard, it's, it's, it's a more stable API than perhaps, you know, others that, that have been \[00:28:15\] **Phil S:** featured here on your show. \[00:28:16\] Yeah. This is really cool. I was just looking at the list of providers and I noticed that, um, geo Code p h p has built a driver for it. So that's William Duran is, um, that package is a big deal in the PHP community and I've really liked that they have built a provider for it. So it's kind of a generic interface with different, um, plugins on the backend and. \[00:28:34\] I was just looking at it. Go, man. Uh, geo Code PHP is getting 300 downloads a month. No, it's not. Your, your provider is getting 300,000 downloads a month. So just that plugin. That's massive. Well, yeah, \[00:28:47\] **Ed Freyfogle:** so, so there are many of these kind of aggregator libraries in, in almost every language has one like that. \[00:28:53\] Um, it's a double-edged sword. It, it, what I would say is that, uh, on the one hand, obviously there are a lot of \[00:29:00\] people who. Don't, they want geocoding, but they don't wanna pay for geocoding, right? So they, they use a library like that and they just go through them all. But, um, the bigger problem is they usually, when you use an aggregator library like that, you're getting kind of the least common denominator of all this. \[00:29:18\] It has to provide a kind of standard interface. So, so all these different useful data annotations that we provide typically get lost. Like the, the, the, it may not provide a way to access those and interface with those, so, Obviously we try to promote our own SDKs, which we have more control over, and where we can really make sure it's, it's fitting our, our api. \[00:29:38\] But you know, if that's how p we we're not dogmatic and I try to force anyone, you know, anyone who wants to use our api, they're, they're welcome to it whichever way they want. Um, and, and we'll do our best to document that and present that. But, um, Yeah, there, there is a lot of demand for geocoding. The most typical use cases, someone just has a \[00:30:00\] database of addresses, they wanna stick on a out, and that may range from, you know, 50 addresses to, you know, 50,000 addresses. \[00:30:07\] One of the big challenges we face is it is difficult to convert those people into customers. Um, so they, they \[00:30:14\] **Mike:** probably don't want to pay on an ongoing basis or don't think they should pay at all to, you know, put a pin in a map somewhere at that point. \[00:30:20\] **Ed Freyfogle:** I mean, one, one. I, I've made this joke before, uh, but, um, we offer a pretrial so we understand people need to test our service. \[00:30:30\] And so you can sign up for, uh, an API key. It takes about one minute, and then that api, you're on a free trial, and with that you can do 2,500 API requests a day, which I think is more than enough for people to test. One challenge we face is somehow there's some sort of thing that happens to software developers with their vision when they read our documentation, is that they go, they go blind to the word trial. \[00:30:54\] Mm. They think we're offering a free service. Yeah. Yeah. Okay. They, they, they, they don't \[00:31:00\] see that. And, and we do not offer a free service. We offer a free trial. So, and then, you know, so if someone's using my service every single day, after a couple weeks, obviously, you know, we notice that we have scripts that catch it. \[00:31:11\] And I read to 'em like, look, you're obviously depending on my service, so you know, if you're gonna depend on my service, I I need to be compensated in some way. Of course. Yeah. \[00:31:21\] **Phil S:** I think I've been, I've been guilty of that a bit in the past, like the amount of time I've spent doing like really creative cashing and all these other like really cheeky things to keep the rate limiting down on services. \[00:31:29\] When I. Probably would've been better off just paying for it. Like, I've definitely done that. It's just something about the developer mindset that's like, we will not throw 20 bucks a month at it. We will throw one dev, you know, a hundred dev hours at it a month instead. That's much more value for money. \[00:31:44\] **Ed Freyfogle:** Yes, yes. So it's, it is frustrating. I mean, you know, some people, you know, I, I polite Lee contact them, say, Hey look, you've been, you know, every day for the month you've hit the free trial limit. Maybe it's time to customer and say, oh, thanks, you know, and then, but \[00:32:00\] obviously some people upgrade and some people stop or whatever. \[00:32:02\] Those people don't bother me as much as we, there are the people who then, you know, try to sign up a hundred times and things like that, and it's so. I, it's so annoying because it's like, guys, we've been doing this for eight years. Do you think you're the first person we've seen who would like, thought to put a plus on their email address and, and to trick, you know, like, come on guys. \[00:32:21\] Well, I like that you have \[00:32:22\] **Phil S:** monitoring system that's like capable of spotting all this \[00:32:25\] **Ed Freyfogle:** good observability. We have a lot of different system to try to. Nudge people to become customers. Uh, you know, so it that, that is, I have to say, one of the most frustrating parts about the business is that we provide, in my opinion, a, uh, a, a good service at a very reasonable price. \[00:32:42\] You know, and we well documented and, and we do quite a lot to give back to the community, but the, but the open data community and the open source community and sponsor projects and sponsor events and everything. And then you have people who are just like, can't be bothered to, you know, pay us. As you say, $20 to, to help on that project. \[00:32:59\] \[00:33:00\] It's that, that does burn you out after a while. Well, this \[00:33:02\] **Mike:** is a problem too that has existed in Mapmaking, at least for as long as maps have been a thing. Right. And I feel like the, the grand tradition of Mapmaking is to include paper towns in your, uh, your print maps. Right. Which is like a fake location on a map so that if I, uh, someone copies my map and they see this town that doesn't exist on their map, I know they copied specifically from me. \[00:33:21\] Right. So maybe the open cage answer is to just start filling, uh, SDK responses with fake, uh, data after a while. And, you know, slowly, well, \[00:33:29\] **Ed Freyfogle:** well, that, that what we do do eventually, I mean, after many warnings and things, we eventually just start returning random results to people. Oh yeah, sure. There you go. \[00:33:38\] And, and we, we, I mean, we tell them ahead of time, we're like, look, dude, I've asked you politely, you know, five or six times to become a customer, but like, if you. You know, at this point I can't help you anymore. \[00:33:49\] **Phil S:** It's better than my suggestion. I was gonna just start returning 1 23 butt street if everything is, but something, \[00:33:54\] **Ed Freyfogle:** no, we do, we do actually true random results so that they probably won't even notice it until \[00:34:00\] much later when they actually need to rely on that data. \[00:34:02\] Oops. And, and that's, you know, revenge is a dish vest cold. Brilliant. Yeah. That, that's one challenge of running a freemium service. Sure. I, I will say that, that we do, it does get a bit frustrating at times, but, but by and large, many people, um, you know, they see the value in what we're providing and, and become happy, satisfied customers and, uh, and it's great. \[00:34:26\] Yeah. So let's \[00:34:27\] **Mike:** talk about what else you're working on, ed. Uh, we talked a little bit, uh, before the show about geo mob. Do you wanna tell us a little bit about that? \[00:34:33\] **Ed Freyfogle:** Yeah, so, um, you know, we, we, we. Mentioned briefly there, there's a whole world of interesting things going on in the geospatial space. Uh, you know, weird, uh, weird stories and weird anomalies and, um, but also, uh, a whole lot of innovation. \[00:34:47\] I mean, there's a massive amount of innovation, and so for the last 15 years now, I've run an event in London. It started as, it was kind of a meetup of geospatial developers. And it's called Geo \[00:35:00\] Mob. And we, we meet up once a quarter or so in the evening, a couple people give talks about their projects and then we go to the pub and people can hang out and, you know, have a raging debate about whether England is a country or a sub country or whatever. \[00:35:15\] Um, anyway, so that's, that's proven quite popular and, and there's a big community around it. And so now we've also expanded to a few other cities in Europe, um, which is great. And then when the pandemic hit, and we obviously couldn't have our events anymore, we started the podcast as well. So, and usually on the podcast we interview different people who have spoken at the events, who talk about their different projects. \[00:35:39\] Um, and it's really interesting. I mean, as I said, there's, there's, there's just been a wave of innovation unleashed by. I would say a couple things. One, uh, first of all, um, Opus Street Map, we're making data readily available to everyone. Secondly, smartphones, you know, now everyone knows exactly where they are all the time. \[00:35:58\] And, and, and you have a, \[00:36:00\] not just, you know where you are, but you have a, a computer that can do things with that information. And so tons of cool services are being built and tons of interesting, um, things about that. That. And then of course, so we just have all these weird, wonderful, wacky anomalies of the geospatial world. \[00:36:15\] So, uh, people give talks about all this and, and so anyone who is interested should please come along to one of the events or, or listen to the weekly podcast we're on. We're coming up on episode 200 now, so, uh, wow. Congratulations. Yeah. There's a lot going on. Yeah. \[00:36:31\] **Mike:** So Phil is, uh, one particular, uh, interesting user of, of geocaching. \[00:36:35\] I feel like, uh, Or not Geocaching. Geocoding. I feel like at some point, uh, Phil, you can, uh, enlighten us all on your, your Geo journey. Uh, somewhere along the way you might, might be a good guest for the show there. \[00:36:45\] **Phil S:** Yeah, yeah, yeah. I mean, at the moment I'm. At the moment, I'm mostly just putting like trees on a map. \[00:36:51\] I've just got, you know, we're planting a hot thousands of trees. I mean, God, we're doing 8,000 in a week soon. So we're doing a lot of tree planting. \[00:37:00\] Um, and we, for many of those projects have to take a photograph of where the tree is and get that on a map. And some people have told me that I should put all the trees on open street map cuz there are layers for that sort of thing. \[00:37:12\] **Ed Freyfogle:** There are people who do that. Yeah. There are people who tag individual trees, of course, with the, you know, the scientific genus and all that. And, and, um, yeah, \[00:37:21\] **Phil S:** we can do it. Um, yeah, the, um, Trouble with ours is that is kind of like a proprietary data set in that like people pay us for those, for that tree data. \[00:37:31\] So if I then shove it onto open street map, then they, they, those people didn't need to pay me to plant those trees. Am I lose all my funding. So at some point when, when the woodland is a bit grown and like we can kind of, you know, tag the mature trees and, and then the woodland becomes quite lovely, that that's something I'll probably do, but. \[00:37:47\] There's all sorts of other weird stuff we've gotta start doing. Like we're looking at, um, various, um, open weather API to see how much rainfall they're getting so we can kind of preempt which ones are gonna struggle and go water them. And looking at what soil types \[00:38:00\] there are so that we can try and plant, um, plant clay soil earlier in the season, um, before Christmas, so it has more time to like mush closed and all this kind of weird online mapping data stuff we're trying to munge together in our own platform. \[00:38:16\] **Ed Freyfogle:** We had a very, very cool last, uh, cool talk last summer at Geo Mob in London. This guy, he works for, um, an agricultural technology company and check this out. So they fly the drone over a field and the way it works is so they're growing cabbages. And the supermarket will not take the cabbage if it's too big or if it's too small, because if it's too big, it'll clog up the, you know, the, the conveyor belt sort \[00:38:41\] **Mike:** supermarket. \[00:38:41\] It's not small destroying agriculture. Small. \[00:38:44\] **Ed Freyfogle:** Right. So, so while, so, but if the, if the cabbage head is too small, then you know no one will buy it. Okay. So it needs to be exactly within sub parameters. So they fly the drone over the field as the cabbages are, are, you know, growing. And then they used \[00:39:00\] the, the photo analysis to target every individual head of cabbage and, and you, you know, do a prediction of like, this one is gonna, you know, hit the gross size or not. \[00:39:09\] And, and then based on that they target, you know, how much water, how much fertilizer, how, you know, hype, hyper precise. I mean, it was amazing. Um, and, and it really was, you know, each individual, it was about taking all these different technologies and kind of chaining them together. Uh, and, and as a result, they able to be much, much more efficient in, in the watering and the use of, of fertilizers, pesticides, things like that. \[00:39:33\] So it was a great, great talk. Um, and, and that's what I mean as an example of some of the things happening around geospatial. Um, I mean there are of course lots of consumer applications, but, but the real power is coming more and more in the combination with, um, you know, tools like Earth observation, you know, now it used to be in, you know, it used to be a very rare thing that you would get an aerial photograph of a place. \[00:39:54\] Now you can basically target, you know, you can purchase a satellite photo of anywhere, you know, like put in your \[00:40:00\] credit card and have it immediately and start doing analysis on it and things like that. And so this is really quite cool. Yeah, this, there's a lot going, there's really \[00:40:07\] **Phil S:** cool stuff happen in that space. \[00:40:08\] I mean, specifically your cabbage thing. Um, I. It's really funny. I feel two halves of me kind of being pulled in two different directions With that, there's the kind of, I spend a lot of time on farms with farmers and they're just fed up with all the nonsense that happen to be put through with all the grants and all the kind of demand for tech and change, and supermarkets trying to low ball them at every single point. \[00:40:28\] So they have to fire everyone and use massive machines and then everyone shouts at them for using too many emissions. Um, but then I also kind of feel the environmentalist that's like, let's be more efficient with everything that we do and get the best right. Um, you know, results. And also kinda like, yeah, the, the, the techie of just like, we could totally fix this with a drone and some algorithms. \[00:40:47\] Like all of those things are competing in my head right now to be like, this is a genius idea and it's fucking stupid. Get it out of there. Fun in here \[00:40:54\] **Ed Freyfogle:** right now. I mean, one of, one of the big use cases of geocoding is, as I said, is vehicle tracking. Yeah. \[00:41:00\] And a lot of that is cargo, cargo tracking. And you know, I mean, there's this amazing stat that like, you know, we, we produce, I don't, I don't know what it is. \[00:41:07\] It's like two x the amount of food that the world needs. Right. It's just we throw half of it away. Yeah. Right. Because it, because it's harvested at the wrong time, or, you know, it rots on the way to the, into the store and all these kinds of things. So, Maybe by using all these different technologies along the way, we could be much more efficient and, um, yeah, for \[00:41:24\] **Phil S:** sure. \[00:41:24\] I mean the replacing, replacing whole field, spraying of pesticide with, um, drones, some of them by air, some of them are fondling along like some little early prototype styles, robot, but they're just kind of rolling around, spraying weeds that pop up cuz they can see 'em from space and drone instead of just spraying the whole damn thing and killing off the entire river and everything in it. \[00:41:45\] Um, so yeah, that sort of stuff is definitely gonna help us, you know, continue to exist as a, on this planet, \[00:41:52\] **Ed Freyfogle:** which is nice. I, I hope so. I mean, as I really, it is pretty fascinating to be a member of this industry to see, to see it \[00:42:00\] really come together and the, and the, the, the making the data available, making the devices, but have become much more cheap and much more robust. \[00:42:08\] It's really cool. And, and, and then you add things like drones and you add aerial imagery and things like that to it. It's. It's, it's an exciting time in Geo special. Yeah. It feels \[00:42:18\] **Mike:** like there will be no shortage of things to keep you busy for the, uh, immediate and far future, uh, ed, um, it's been super fantastic having you join us on the show today. \[00:42:27\] Um, thank, thanks so much for being here. Um, before we let you go, I'm curious, where is the best place for people to find you online and where can they go to get started with open Cage? \[00:42:36\] **Ed Freyfogle:** Well, anyone, anyone of course who needs geocoding should go to our website, which is, it's not open cage.com. It's open cage data.com. \[00:42:43\] But if you just search for Open Cage geocoding, you'll, you'll find us. Um, yeah, if you're interested in geospatial stuff, come along to Geo Mob or, or listen to the podcast of geo mob.com. For me personally, my, my, my surname is, of course, is a bit difficult for spelling, but \[00:43:00\] let's go for it. My, my website is fry foal.com. \[00:43:02\] Maybe you can get that in the show notes. Absolutely. Yeah. Um, in terms of social media, the best, um, best place with Maed, I've, I've fully made the switch to. Right on. Me too. Um, uh, also LinkedIn. You can, you can ping me on LinkedIn as well if it's, if it's, if you'd like. So exactly how I feel about LinkedIn. \[00:43:20\] Anyone, anyone who wants to get in touch with. \[00:43:23\] **Phil S:** Nice. So yeah, you'll, you'll notice when I'm starting to feel a bit broke because I'll start posting on LinkedIn more that the place I go to when I like, need a job or need to refresh my cv. I forget that people use that as a thing otherwise, but, um, yeah. Cool. \[00:43:35\] Well I'll, I'll find you on Master On Cause I'm on there now doing a bit more over there. Um, it's good. You'll usually see me just post some really like, um, very short bullshit on Twitter, like some very little mini shit post and then mastered on, there's more space, so you actually get a thoughtful piece instead. \[00:43:49\] I quite like that longer \[00:43:50\] **Ed Freyfogle:** form. One, one quick note on that, you know, so Open Cage also has a mastered on account and we, we do a weekly thread about what we call geo \[00:44:00\] weirdness, so hashtag geo weirdness, where we talk about some of the. You know, we profile different countries and talk about all the different, you know, wackiness of, you know, enclaves and exclaves Absolutely. \[00:44:09\] And border disputes. And, um, so actually this week for, for, for our British listeners in honor of the upcoming coronation, we had one about all the remaining British overseas territories like Pit, Karen Island and, and um, Gibraltar and uh, uh, places. \[00:44:27\] **Phil S:** Nice. I'm following that right now. I am very interested in map \[00:44:30\] **Ed Freyfogle:** weirdness. \[00:44:33\] Yeah, no, we have quite a lot. We, we, we've done, I don't know, 20 different countries or so now every country has its own weird edge cases, so. Awesome. Thank you so much. Um, guys, it's been great to be on the show. Thanks for having us. Yeah, ed, thanks for joining. \[00:44:45\] **Mike:** Cheers. Take care. Bye. Bye. ### Security Matters URL: https://apisyouwonthate.com/newsletter/api-and-app-security-really-matters/ Last updated: 2023-06-03T13:59:04.000Z Let's take a break from the usual discourse that pops up on Twitter to talk about security and privacy. We recently had several significant data breaches in Australia. One, in particular, allowed data from more than 11 million customers to be stolen. The culprit an unauthorized and public API endpoint. Seriously what were they thinking? So why am I bringing this up? Security often gets missed, or your manager cares more about pushing a feature and thinks security is something you can add later. Security and privacy should be woven into the planning, design and development process. We should care about data breaches affecting our customers. It's much easier for people in tech to protect themselves when their data is stolen than for everyday people. Let's put our end users first. Hopefully, the security-focused round-up helps. Alexander --- ### Monthly API Round-Up Our monthly round-up of articles, news and updates from the web. ### [What is API security?](https://www.cloudflare.com/en-au/learning/security/api/what-is-api-security/?ref=apisyouwonthate.com) Cloudflare gives us an overview of API security, some common security risks, and how these are exploited. ### **[OWASP API Security Top 10 2023](https://github.com/OWASP/API-Security/tree/master/2023/en/src?ref=apisyouwonthate.com)** A top ten list of security threats for APIs, including example attack scenarios. The 2023 list by OWASP is still in draft, and a request for comments is open, meaning there is still time to give your feedback. ### [Best Practices for Protecting APIs](https://www.nginx.com/learn/api-security/?ref=apisyouwonthate.com) NGINX covers some best practices for protecting your APIs and a short intro into API security. I like how this article covers the differences in threats between SOAP, REST and GraphQL APIS. ### [Container Security](https://snyk.io/learn/container-security/?ref=apisyouwonthate.com) Many APIs are built and deployed as containers, so securing them is essential. I try to avoid using blog posts from a company selling you a tool/service, but Snyk is a great tool, and this post covers the basics nicely. --- ### APIs You Won't Hate The latest content from the team at API's You Won't Hate. ### [Permissions are the gift that keep on giving, with Or Weis from Permit.io](https://apisyouwonthate.com/podcast/permissions-are-the-gift-that-keep-on-giving-with-or-weis-from-permit-io/) On this episode of APIs You Won't Hate (the podcast), Or Weis from Permit.io talks to mike about permissions, authentication, authorization, and the challenges facing developers building out products for real people. There is a *lot* to learn, and getting permissions right is critical to building great products. ### [Securing APIs with the Spectral OWASP Ruleset](https://apisyouwonthate.com/blog/securing-apis-with-spectral-owasp-ruleset) Our writeup on using Spectral to create rules to help prevent the top 10 biggest security threats as found by the OWASP API Security Project 💌 **Updates coming twice a month!** Mike and Phil have been extremely busy with work, and I have been holidaying in Norway. The good news is the newsletter is now coming at you twice a month on the 1st and 15th. Feedback is always welcome, so feel free to contact me in our [Slack](https://apisyouwonthate.com/community/) group. --- ### From Our Community Articles written and shared in our free [Slack community](https://apisyouwonthate.com/community). ### [Driving Improvements in the API Design Process by Harnessing UX Research Techniques](https://www.linkedin.com/pulse/driving-improvements-api-design-process-harnessing-ux-joyce-stack-?trk=public%5Fpost%5Ffeed-article-content&ref=apisyouwonthate.com) Here is a killer of an article from our community member Joyce Stack about how she looked to improve the API design process with her team. Phil was so impressed he shared it himself with the Slack community. ### [Living Out of a Van](https://philsturgeon.com/struggle-bus/?ref=apisyouwonthate.com) Want to know why Phil is not on every episode of the podcast? Well, he is trying to save the world from climate change with an amazing team of volunteers. Phil recently got an electric van and decided to give you an insight into his life and a review of how the van has changed it. Thanks so much for reading! If you found this helpful, consider sharing it with a friend. It helps us out a great deal. Until next time...! ### Permissions are the gift that keep on giving, with Or Weis from Permit.io URL: https://apisyouwonthate.com/podcast/permissions-are-the-gift-that-keep-on-giving-with-or-weis-from-permit-io/ Last updated: 2023-06-02T12:56:17.000Z ## Show Notes - [Permit.io](https://permit.io/?ref=apisyouwonthate.com) \- Never Build Permissions Again - [Opal](https://github.com/permitio/opal?ref=apisyouwonthate.com) \- open-source project: Open Policy Administration Layer - Or Weis - [@orweis](https://twitter.com/orweis?ref=apisyouwonthate.com) - Or's talk about onboarding and complexity - https://youtu.be/1\_Iz0tRQCH4 - [Permit elements](https://www.permit.io/blog/elements?ref=apisyouwonthate.com) \- ready-made UI components for user management and access control[Foaz](https://hello.permit.io/foaz-ea?ref=apisyouwonthate.com) \- front-end only authorization #### Transcript \[00:00:00\] **Track 1:** Hello and welcome back to APIs you won't hate. Uh, as always, I am your host, Mike Fulco. I am sitting down here today to talk with a new friend of mine named or Weiss from permit.io, or it's great to sit down and chat with you. How are you doing today? \[00:00:14\] **Or Weis (Permit:** I'm great. Super excited to talk with you, Mike. \[00:00:17\] **Track 1:** Yeah, likewise. Likewise. Uh, it's, um always interesting to meet people who are working in the developer experience world, especially around, uh, APIs and things like that, which is obviously why you and I are talking today. Um, yeah. So let's start here. Why don't you tell me a little bit about yourself, uh, and your, your sort of history before, uh, uh, permit.io, which is what you're working on right now, uh, and how we got to permit existing. \[00:00:41\] **Or Weis (Permit:** Sure. Sounds like a plan. So, um, my name is, or Weis, you probably gathered that by now. My background starts in the intelligence score in the idf. I was an officer, developer, reverse engineer, engineer team, yada, yada. Basically a cliche, an Israeli entrepreneur. Uh, afterwards I \[00:01:00\] worked in a startup called Antigua, where we built containers before containers were a thing. \[00:01:04\] but with a terrible go-to market. Even worse than Dockers after they ruined their go-to market. Um, I worked a couple startups, founded a startup called React that was acquired by metadata io was a VP of r d and a cyber security company. And, uh, between late 2016 till up, uh, almost three years ago, I co-founded and ran a ceo, a company called Brook Out. Uh, which is another, uh, dev tool, uh, production debugging solution. I didn't go as far as saying the company that created the production debugging space. And, uh, during my time at Rout, I ended up building access control to our product times when the company wasn't even three years old. And I just said, stupid. I don't wanna do it once, let alone five times. reflecting on it. I realized that throughout \[00:01:56\] my career I've been building this crap part of my French for \[00:02:00\] thousands of times, and I get at no point, at no point did I want to. um, I got together with a friend of mine myself. We are now friend. We know each other for 18 years, which makes me feel old. And he, uh, aside from serving with me in the unit, he also worked at Facebook Meta where he worked on their internal developer tools and in infrastructure and authorization. And he saw that they've invested a team of 30 people for half a decade to build the level of access control that they have. So we did the math and we quickly realized that it's not only a huge problem now and a huge annoyance now, it's only gonna get far worse down the road as the complexity of software continues to arise, and as more machine learning agents are mixed in. \[00:02:44\] So if you think it's hard to manage users and their access to your product, wait till you need to manage their machine learnings, uh, machine learning models as they talk to your machine learning models. I, I just got back from con eu. uh, in Amsterdam \[00:03:00\] and if some one thing was apparent is that every company now is plugging in a machine learning model, a chat G p t like thing into their software. Um, and that's gonna make access control there and security there quite a challenge, I'd say. \[00:03:15\] **Track 1:** I'll say. Yeah. Yeah. It's a really interesting, uh, feature set to be working on. I feel like every product team I've worked on, every product I've worked on, every project I've worked on, whether it's for, for a formalized company or not, this feels like one of those things where it's like, ends up being engineered into the product later on, right? \[00:03:32\] Like \[00:03:33\] prime on the list of tech debt is, oh, we'll put in permissions later. For now, we'll just only give logins to the people who need it, that kind of a \[00:03:39\] **Or Weis (Permit:** Yeah. Yeah. You know, everyone starts, like we all talk about permission models, but everyone starts with the most classic one, which is called admin. Not admin, meaning I'm the developer, I'm the admin. All of you guys will stick with whatever else I, I'll leave out. And then requirements start to funnel in, and then your stakeholders want permission, so you go, okay. \[00:03:58\] Okay. Okay. So we'll have. \[00:04:00\] admin, super admin, I'll be the super admin, and then, uh, just a regular user. And then it moves to access control lists, and then it moves into our ARB is like the bread and butter of this world, right? Roll based access control. You have a role that gives you permissions, but then it continues. \[00:04:18\] Like a lot of people think it's, it ends there, but it doesn't. You have attribute based access control, like how do you define a policy like only users that have paid for a feature can use it. Is that like a billing, like a paying roles? What, what, what would be that? And then you have role based access and control and just on and on. \[00:04:37\] This is, I like to say the permissions is the gift, the gift that keeps on giving. And I, I mentioned my story at Rout, so. I, I literally went through the same thing, the same point, same pain point. okay, I'm done with this. Like the fourth time or fifth time I built it, I was like, I have built the best dam access control solution out there. It's perfect. I'm never gonna have to touch it \[00:05:00\] again. And I remember it really shattered me. Uh, one of our, uh, business partners, Cisco came in, they were co-selling out, and they came in and said, at some point, we want our own back office. And I was like, ah, shacks. I didn't think of that one. Okay. Out the window. \[00:05:17\] Start from scratch yet again. And because every product is a snowflake and because every product manager is a unique being, and because your security compliance is always your own thing, there's constantly new things that each product needs to tackle in a proprietary way. \[00:05:35\] **Track 1:** Sure. Yeah. I think that's pretty easy to imagine. Um, especially with these sort of like evolving product scape where, where, um, project teams will add features to their product that add complexity or add, uh, collaboration needs and things like that. Um, \[00:05:49\] one thing that I feel like has been coming up quite a bit lately, which is probably related to what you're working on, is everyone and their brother is building an AI-based tool, which is. \[00:05:58\] Uh, usage based. And so you may pay \[00:06:00\] for credits per use or something like that in advance, and that's sort of a different view of access control, right? You may only get, I don't know, call it a hundred executions of some bit of functionality per, \[00:06:09\] uh, you know, payment that you make. And so that's a very different feeling than do you have access to this thing as a user? \[00:06:15\] It's do you have access to this thing as a user and have you paid for it and you still have, you know, leftover bits of credit. \[00:06:21\] **Or Weis (Permit:** Yeah. These kind of quotas add challenges both in how you manage the data. your \[00:06:27\] authorization layer needs, also in the realtime event driven aspect that you need for it as part of your application, it's no longer something that you can manage statically or you can manage with periodic updates. \[00:06:39\] It's alive with the same pace of change of your application, and it's in a performance critical way. Sometimes, like if you were. Allow them to exceed the quota, or if you allow them, or if you allow the quota to be miscalculated when it's kind of in a race condition, it can even really break things and affect the financials. So you need the authorization layer \[00:07:00\] to not just be well organized and well modeled. You also need it to be extremely performant and event driven, and it's really hard to do if you haven't planned for it in advanced, and if you haven't set the groundwork for that. So a lot of what I try to do when I work with, uh, the community that we've built around our open source project, Opal and around permit as a company, it's not just, um, these tools, it's also providing the best practices and tools to think about this problem. \[00:07:29\] I spend most of my time and also the best, like this is what I like the most about my job, talking to fellow kindred spirits and just geeking out on the tech and helping people like myself solve, solve this problem. So it's we, we really try to also bring in the best practices to the table. The decoupling your policy and code, creating a separate microservice for authorization building in and. \[00:07:51\] Event driven architecture that will enable you to scale it out and, uh, keep it with the pace of the changing requirements of your application and the \[00:08:00\] pace of the application itself. Like a lot of people don't realize this, but like in a average microservice application, you need to handle three authorization queries on average for every incoming request. So unless you have something that is ready for that scale and pace, you're gonna have a bad time. \[00:08:17\] So, and if you plan for it though in advance, even if you don't build. Even if you don't use any tools, if even you don't use the open source, but you just couple out the things that you need to be independent, like the policy from the code, like the data plane from the authorization engine itself, um, you can save a lot of pain just by having a little bit of foresight. \[00:08:39\] **Track 1:** Yeah. You know, it's interesting. I feel like we, we've dove in headfirst here and one of the things that. I find comes up a lot with people who may be new to building applications that have sort of a user facing feature, uh, of any, uh, form is the, the important difference between authentication and authorization. \[00:08:58\] Uh, right. I feel like those things are easy \[00:09:00\] to, uh, confuse. So, um, can you maybe help disambiguate that? What is authentication, what is authorization and what do they have to do with each other? \[00:09:07\] **Or Weis (Permit:** Yes, of course. I, I do that disin variation basically, uh, three times a day. So happy to do so. \[00:09:13\] **Track 1:** Perfect. \[00:09:14\] **Or Weis (Permit:** So maybe first let's start with identity management. So I am identity management and access control. So you have identity management authentication and authorization permissions. These are all three cascading waterfall like tiers that connect to one another. So identity management happens on the organization side. That's just deciding which identities you have, which are part of the organization, which are not, and what data or attributes or claims you have about them. Then as you move towards products or applications, those users or identities want to consume you move through authentication, which is the main responsibility for that. Is to verify the identity. So it's talking to the identity management, talking to other sources of identity, biometric data, um, uh, one time password tokens that you send over \[00:10:00\] SMS multi-factor. And that's allowed to determine if you are who you say you are. usually culminates in creating adjacent web token. That's like, uh, it's a cryptographically signed document, \[00:10:12\] **Track 1:** Mm-hmm. \[00:10:13\] **Or Weis (Permit:** kind like your passport that you take with you either in your cookie or in your http enters or whatever else you use to communicate with your end app. And then you go and you get into the app itself and you have your identity and a lot of times additional claims like. You belong with this department. Your current location or residency is this country. Um, and then the application needs to decide what you are allowed to do or not, which is very different from authentication. That's authorization or, uh, permissions. There. You need to evaluate a policy, take in the relevant data of what's happening now and what's the current context like. \[00:10:50\] Where are we in the world? What this, which service am I, uh, what are they trying to do, et cetera, et cetera, and come up with an answer. Can they do this or not? And the \[00:11:00\] challenge there is that that's deeply ingrained into the code, into the product. Unlike authentication that just happens once at the gateway. Authorization needs to happen for every little request interacting with the system. Otherwise, it'll be vulnerable. \[00:11:13\] **Track 1:** Yeah. Yeah. That is a, uh, great synopsis there. And I think maybe the biggest disservice that new developers run into is that we tend to refer to both of them casually as off, right. Problematic, very, very problematic that those two things have different functions and they're, they're codependent and use the same name. \[00:11:31\] **Or Weis (Permit:** Honestly, in defense of every developer out there, the ecosystem hasn't done us any favors. If you \[00:11:37\] **Track 1:** Yeah. \[00:11:37\] **Or Weis (Permit:** it, you, even in HTTP when you, there are multiple authentication methods. One of them is called the Bear Token, and you literally write in the http headers, it's authorization, and then the bear token, the adjacent web token. So for Authe, we are literally calling the authentication, the authorization. So everyone's confused and you have things like O off. And \[00:12:00\] specifically o uh, oof two, um, which talks about authorization, but it's part of, uh, like in single sign on in, uh, it's how you do authentication. \[00:12:12\] **Track 1:** Yeah. \[00:12:13\] **Or Weis (Permit:** it's really a mess out there. \[00:12:14\] I, I think our predecessors kind of, uh, left a lot of technical debt for us to fix. Um, that's, that's being engineers we \[00:12:23\] built on the shoulder of giants and on the crap they left as well. \[00:12:28\] **Track 1:** for sure. Yeah. Maybe it's, uh, an opportunity for us that the, uh, the language creates such ambiguity in what we've got going on here. Uh, there's also quite a. \[00:12:37\] **Or Weis (Permit:** I try to call this permissions just to avoid the \[00:12:39\] **Track 1:** Yeah. Yeah. Okay. Actually, that's perfect. So my next question is, uh, I, I feel like I hear people use permissions and entitlements, uh, in, in the same context often, and this is probably something I'm guilty of, that I sort of think of those two things as the same. \[00:12:55\] Is that in your world, is that roughly the same thing? \[00:12:58\] **Or Weis (Permit:** So they're very different\[00:13:00\] \[00:13:00\] **Track 1:** Perfect. \[00:13:01\] **Or Weis (Permit:** but I don't, uh, begrudge anyone confusing them or interchanging them. I think it's okay. It's like we don't need to make a big issue out of it. But if you wanna be, uh, if you wanna be nitpicky, um, permissions are what you're allowed to do or not. Entitlements are, um, Claims or values or, uh, information that can be used to make those deductions. So for example, you can have the entitlement of a manager or an admin in a system that can be even coming from the, uh, identity management that would still need to be translated into an application level role. In the applications that you are providing, and that needs to be translated into specific permissions. So I like to think of a permission as the intersection between identity, resource, and action. So, or a principle. So instead of a identity, if you prefer \[00:14:00\] this person with this context, or this entity with this context can also be an automated agent. Obviously this entity with this context. What are they allowed to perform this action against this resource or not? Uh, and in, literally in permit. So we really, we really put a lot of effort in building a simplified ux. Part of the cool things about permit is that we have a low-code, no-code editor that generates policies code for you. So we literally took that concept in tri, translated into a table. So when you, you wanna say, oh, I have the permission. Uh, an admin as their permission to delete a file. You literally click a check box at the intersection between file and delete \[00:14:43\] **Track 1:** Yeah. Yeah, that was a wonderful explanation, and I can tell that this is something that you've done more than once, and clearly this is not a part of the, the stack that I, uh, dip my toes into too often. Uh, this is one that I'm gonna keep in my pocket very, very handily as a hey, even, uh, someone who's been around the block a few times, \[00:15:00\] uh, gets the answer wrong pretty regularly, you know? \[00:15:03\] **Or Weis (Permit:** Yeah. And, uh, by the way, I get these kind of, there's a lot of these little confusions out there, both \[00:15:09\] in the implementations of things and all just the names that you have of things. And, um, so what we try to do with that, like, I really just make it approachable for people as much as possible. So I spend a lot of my time, like, I encourage people listening to us reach out to me and like, uh, or WISE, or w e i s on. \[00:15:28\] Twitter on GitHub, on LinkedIn, whatever, like, and I'm happy to help. I'm happy to answer questions like even silly ones. Um, so \[00:15:36\] **Track 1:** Yeah. Perfect. Yeah. Thank you. Um, so I'm, I'm interested in hearing maybe some of the basics of about permit then. So, uh, \[00:15:44\] **Or Weis (Permit:** mm-hmm. \[00:15:45\] **Track 1:** maybe let's start here first. Uh, how does it work? Is, does permit essentially work as a middleware? Is it something that you integrate into, um, your API side? Uh, does it, is the generated code something that I inherit? \[00:15:56\] How does, how does it work? \[00:15:58\] **Or Weis (Permit:** Terrific question. \[00:16:00\] So at the core of it, we are providing a policy engine for you to work with. So maybe you've heard of open policy agent opa, or maybe you've heard of Cedar from aws, and there a bunch of others. These are engines that are general purpose. You load them with policy. You load them with data and then you can query them. So we provide that agent for you, bundle up all nicely in a container that you can run it as a sidecar or as a cluster next to your app or if you, and soon it'll be available from aws. You'll have Amazon verified permissions. So that's a service that will be running that agent for you. And we can, and we layer on top of that. So we, but we enable you to run this next to your software and that's very important. For two reasons. Two, actually three reasons. Uh, the first one is latency. As I said before, every microservice application on average handles three authorization queries per request. Even if you just go out to an external cloud, uh, for one, for a 50 \[00:17:00\] milliseconds roundabout, uh, you'd be killing the performance of your app. So this needs to be as little latency and ideally zero latency. So by deploying the service next to your app, ideally is a sidecar, or at least on the same physical machine, you remove the latency aspect. Yeah. For part, obviously permissions or security is a critical aspect. If that doesn't work, you, you, you don't want your app to work, let's put it that way. So if you're dependent on the external availability of, for example, our SaaS service or any our cloud, Well, you're gonna have a bad time, so you want everything, that decision engine to be able to answer everything locally. this is where our open source project Opal comes in. Opal Open Policy administration layer is a way to keep policy agents up to date with the policy and data that they need in real time. So you opal subscribes to, uh, policy management from permit or your own use of it as an open source project and it fetches the instructions for data and \[00:18:00\] policies. It needs policies arrive directly from GI repositories cuz we want to manage them as policies, code and data arrives from whatever you needed, including your local database. So this is where the security aspect comes in. You can load data into your authorization layer with permit. Without being dependent on, uh, sharing it with permit at all. Cuz Opal sends instructions on where to get the data instead of the data itself. we now understood the pdp, the container that can, that runs in your app. We also provided hosted, but ideally in production, at least you run it as part of your app. All that's left is for your app to consume the decision point. Uh, via enforcement points. Um, there are multiple ways you can do so. You can, uh, apply it in the code and you can apply it externally, like in a reverse proxy or API gateway. But applying it in the code is the most classic in what, uh, most people do. So we provide a function and SDK function in para basically in every language you want called permit duct check. \[00:19:00\] receives three arguments and I think will sound familiar to you. Identity, resource, and action. you're basically saying this identity, the identity being the Jason Web token that you got from your authentication, by the way, that's how we seamlessly connect your authentication providers and identity management without having to replicate any of them at all. Um, so you have the identity performing this action on this resource. So in your app, you are describing what's happening instead of describing how you should handle it. You, so you don't put the policy there, you're just doing the description. With that permit, that check flow. By the way, even for people who doing the open source option or doing, uh, completely building on their own, I really recommend this pattern decouple your policy and code and work with identity resource action. \[00:19:45\] It simplifies reasoning. It allows you to put this sec the policy separately and it gives you a lot of elasticity with mutating your app or mutating your authorization there afterward. So you put in that permit check, it directly talks to that \[00:20:00\] PDP that lives inside your app. So zero latency, you get authorization decisions and you're good to go. Now we only need to do is manage the policy. So this is where the para permit, uh, control plan comes in. We have a policy editor UI that I'd like to say a monkey can use, or even a product manager if they're smart enough. \[00:20:18\] **Track 1:** Ah, \[00:20:19\] **Or Weis (Permit:** by the way, the product managers love that joke. I'm not sure why, but they just love it. Um, so we generate code for you. The policy editor is something that you can, uh, do some clicks with. It does both Rback, role-based access control and avac. So you can do very complex policies, but with an interface that you can easily use yourself and obviously easily delegate to other people. This is actually one of the most important things that I think. um, we have permit recognize is that you don't want to do this. Like you don't want to build the policy and take care of it all day. Groom it like a bonai tree. You wanna get rid of the scrap, so you wanna bake something in and delegate this to \[00:21:00\] the other people. Product security, compliance, sales, professional services, support, whomever, everyone and everyone needs to connect in the end of the day. \[00:21:08\] Building access control is about connecting people and systems to what you've built in a secure fashion. and any company building a product, that's what you're doing all the time. So you want to enable all the people around the table to do so without turning yourself into a bottleneck. So that's what that, um, policy, uh, low-code, uh, policy generator does for you. But still, you get off the best practices here cuz it's a low code interface that generates code. So, for example, if you're choosing Rego or Cedar, we write that code for you. Uh, by the way, these are complex languages. They're not like, Um, Python or do, uh, derivatives of data log or prologue. So \[00:21:49\] those are basically a rule engine that recursively runs between all of the rules and functions that you wrote. So it's kind of, it's not a run of your me run of the meal, uh, \[00:22:00\] language. So it also can be a lot of, uh, annoyance just to learn this new thing. So we just take that off the table. You can just generate the policies for you, but they get pushed into GI and in Git you can add more code on your own if you want, you can do code review. \[00:22:13\] So if that product manager creates a policy, you can review it. You can do tests on the, you can do benchmarks. So you, while you delegate access to the access control, you don't lose the control yourself. That's something that is very important, by the way, I think with every developer product, every developer, pro product should be very powerful, uh, but easy to use and you can choose on that slider as a developer, how much you want to go deep into the reads, but you never should be forced to. \[00:22:40\] You always should be able to, but never forced to. That's my, in general, my philosophy about, uh, dev tools. \[00:22:46\] **Track 1:** Sure. \[00:22:46\] **Or Weis (Permit:** generate that code for you. And, uh, we provide a lot of other interfaces like user management and the audit logs and, uh, interfaces that you can embed for your end customers. But that's, uh, in just a basic concept. \[00:22:59\] **Track 1:** Yeah, \[00:23:00\] there's, there's a lot to digest there. Um, I think one of the fundamental things that. Um, maybe is a, is a subtle point that I feel like is probably something that permit is using quite, um, elegantly. Is that all authe, um, make sure I'm saying this right, authentication providers who use JWTs, \[00:23:16\] uh, is JWT is a secure but also a mutable thing, right? \[00:23:20\] Like you can send information within that token, uh, which is important. It sounds like you're tapping into that functionality to be able to, uh, you know, provide the permissions information to enable, um, various scenarios for, for whatever the cases may be there. Um, That is, that's a subtlety that I think a lot of, um, uh, author, uh, yeah, authentication providers don't get across super well. \[00:23:40\] Like they tend to say, they dangle it in front of you. We use JW T, but I think the breadth of what that means is, uh, a little trickier than, than maybe not trickier, but a little more, uh, deep than it seems on, on first glance. \[00:23:52\] **Or Weis (Permit:** They, they actually do something that is, that is more insidious. They tell you that they provide rule-based access control for you \[00:24:00\] and they're doing two major sins at once There. One, they tell you that they've done rrb back for you, so you are done. But then you get that Jason web token and you still need to write code. to actually enforce access. And \[00:24:13\] what happens is when the claims change or if the, uh, identity management, uh, for the customer, uh, ends up sending different things, they, you actually haven't built arrb. You've built claims or entitlements, and now you need to process them into your actual policy and you actually need to write code or something to handle that. And the second part is that they're actually confusing the term roles. So there are two kinds of roles here. There's the roles that you have in the organization and the roles that you have in your app, and they're not the same thing. If you were the, for example, the VP of marketing that, what does that mean in the app? \[00:24:51\] Are you an editor? Are you an admin? Are you a monkey? What \[00:24:54\] does that mean? So the translation, a critical application concept, \[00:25:00\] logic, translation, needs to happen and. A lot of times they, they try to kind of minimize the difference there. Um, but that actually creates more work for you as a developer. So recognizing that early, by the way, I'm not saying don't use those claims. \[00:25:15\] Those claims are great. Those are amazing attributes that allows you to connect to the customer side and have, uh, the ability to build complex attribute based access control policy. but you need to recognize that, that there, that thing that you need to manage that complexity and not just assume that some magic ferry would come in and, uh, make it all work for you. \[00:25:36\] **Track 1:** right? Yeah, that's a fair point. Uh, so I wanna talk about onboarding then for permit. What are, what does your typical, um, use case look like when people are coming to permit? Like where, where are they at in the product development lifecycle? Are they brand new? Are they working on something where they're trying to shoehorn in permissions or are they reworking permissions entirely? \[00:25:56\] **Or Weis (Permit:** So most often than not, it's someone that's already has something in place, but \[00:26:00\] a new requirement has come in. As I mentioned before, permissions is the gift, \[00:26:03\] the gift that keeps on giving. So every three, six months. There's new requirement comes in and you either completely refactor or, uh, refurbish your, uh, authorization layer. Um, and so we do get some people starting from scratch, but most people have something in place and something that we recognize is that you don't want to touch it as long as it's working. As long as you don't have another requirement or an issue, you, you should, I encourage you, stick with what you have. but once you do come to that point when you need to upgrade, Recognize that it's probably not the last time. And then it's a question of how do you best utilize your time to both meet the current requirements and make it easier next time you need to upgrade. And that's, and that's where the best practices come in. Decoupling policy in code, working with policy as code, making it event driven, creating the interfaces for the ever stakeholders. There's a talk that I gave at covering all of \[00:27:00\] these in details, so I couldn't just Google it. \[00:27:03\] **Track 1:** Yeah, I'll make sure I put a show note, a link in the show notes as well. \[00:27:07\] **Or Weis (Permit:** Sure. Uh, so I don't, I won't go too much in, in the reads into that, but by recognizing that you can really improve things, a lot of times when people, uh, I get a lot of calls, like we have a LY link on the footer of our website. I get a lot of calls for that and, uh, I tell people like, what do you actually need to do now? You shouldn't like refactor everything at once. Let's be focused here. We need to be cost effective, but let's also recognize what are the easy things that we can implement that will minimize technical debt later. a lot of times it can be even something very simple, like, uh, just decouple it out. Just create the microservice for authorization if you don't have one. \[00:27:48\] It can be even a, like a lamb function that currently just returns true for everything, \[00:27:53\] **Track 1:** Yeah. \[00:27:53\] **Or Weis (Permit:** can gradually add more logic into it and then switch in place, add, replace it with a different container or a different service, or a different \[00:28:00\] open source project. Okay. Um, but, but by starting to have that modularity, you do two things. A, you set the ground for what you actually want to build, and you are communicating to the re to the other engineers, uh, about where this is going. Th this is, by the way, the mirror image of this is where most of the pain comes from. By putting the authorization in the app itself, what do you end up having is organic drift where people just, other engineers just add more things into it. \[00:28:26\] So you find a lot of these if conditions that have logic, both for the app itself and for the \[00:28:31\] authorization there. And then when you wanna change it, you have to cherry pick and kind of go one by one to fix it up. by putting it separate, you are communicating to everyone. This is a separate thing. Don't push your silly if condition here. \[00:28:44\] It, it doesn't go here. Uh, so you are also, um, more, uh, dirt or unrelated stuff accumulating there. Uh, and that will also say you work later. You less cleaning to do, aside from just setting \[00:29:00\] the stage. \[00:29:01\] **Track 1:** Yeah, I have personally done that many times where I've had to go in and, uh, dees up that if statement that I wrote months and years ago or someone else wrote months and years ago, uh, as complexity increases. And I think that's one of those things that, um, you're paying yourself a big favor in the future if you're able to sort of, even just at the early stage, like you said, kind of. \[00:29:20\] Plum plumbing and if true, or whatever, you know, a return true statement there. Um, very, very similar weirdly to, to, um, in a, in a recent past life, I was doing developer advocacy at Stripe, uh, and \[00:29:31\] changing your pricing strategy, uh, you know, months down the line is an infuriating process if you haven't thought of it in advance. \[00:29:38\] Uh, so giving yourself, like decoupling all of the logic of everything so that you can plug and unplug things is really helpful, especially in a world where you may need to test or iterate on things or. Uh, you know, add complexity too. That's, um, that's a bit of wisdom that is not to be taken for granted. \[00:29:52\] Certainly. \[00:29:54\] **Or Weis (Permit:** I, by the way, I really love Stripe. We drive a lot of, uh, inspiration from. from the play there. I really love \[00:30:00\] both the developer alignment and the thought around the how a product should grow and how you, um, not only provide it for the developers, but. For the developers, for everyone else. That's a philosophy that I really subscribed to, \[00:30:14\] **Track 1:** Yeah. \[00:30:15\] **Or Weis (Permit:** one of the features that we ended up creating for permits is something that we call permit elements. And we just reaped up the concept directly from Stripe Elements, right? And so with Stripe Elements, you have readymade UI components for billing, for, uh, like a checkout, for example, that you can abandon your app. And so we've done the same with, uh, With permit elements, these are already made experiences for user management and access control. And actually, when you think about it, these are the main things that you actually do with access control. These are experiences, you know, like things that you've seen a billion times. Uh, user management. With the ability to assign roles, API key management, secrets management audit logs for yourself, for your end \[00:30:57\] customer, multi-tendency management approval \[00:31:00\] flows. \[00:31:00\] One user starts from action, another user approves it, invites permission, requests, emergency access, personation, and this list just goes on and on and on and on. So the philosophy is is not unique to any app. Why? Why in the hell should I build this? Why can't I focus on what actually my apps needs to do? So we just provided, already made, you can always build it on top of our api. Again, that's the same philosophy. You should always have the power to do whatever you want. You're the developer, you know best, but if you don't want to, you shouldn't be forced to. And so we provided off the shelf. \[00:31:37\] **Track 1:** Yeah, I think that's very smart. The, um, UX designer part of my brain also really loves that because you're not reinventing the wheel, uh, and you're taking advantage of presumably permit elements. If, if I went in and looked at the UI library, it would look awfully familiar, and just by seeing it, I would be keyed into what the likely functionality of that bit of interface does, uh, which is. \[00:31:58\] A wonderful thing because you're taking \[00:32:00\] advantage of all the deep, you know, ingrained psychology that we've given ourselves from using, uh, software for all these years. Um, yeah, definitely always a good idea to reuse things and not, uh, create, you know, clever, um, parallel patterns for things when you don't have to. \[00:32:13\] Uh, so, or we, we've talked a little bit about, um, your product from a. I guess from a ground up standpoint. So, um, really interesting to hear, you know, the, the very basics of where the problem came from and then sort of why, uh, it's important to think about this from the beginning of, uh, building out your software, but then also what it's like to take on complexity and why permit can help with that. \[00:32:35\] Um, I, I know, um, There's a lot of developers who will come to this thing with a, okay. I think I fit the scenario you're talking about here. Um, what, what is the first thing that a developer does when they jump in with permit? What is sort of Hello world? \[00:32:50\] **Or Weis (Permit:** Uh, so the yellow world is picking, so per permit is very granular. So you can put pick, if you just apply it to like a single function, a single microservice, a single \[00:33:00\] route, a single middleware, single application, whatever you choose. So it's about picking the right granularity for you to start. Like, what do you actually care about now? \[00:33:10\] What do you actually wanna try? Adding that one permit check and seeing how you change things from the editor, they propagate into get and in real time, propagate. into the, your live application and you are able to change the permission, just toggle one action on and off very quickly. That's the, that's usually where you get it. And, uh, and then you move to like, okay, let's add another role and assign it very quickly. Now \[00:33:36\] let's make the roles dynamic and assign it very quickly. Now let's have different, two different applications or two different tenants with slightly different policies. And it gradually, everyone kind of drifts from there into their own snowflake scenario. Um, and it's, uh, it's very easy to start and the end of the day it's just embedding an SDK or potentially a microservice. Um, you don't have to put in the microservice initially \[00:34:00\] if you just wanna tr test that we hosted for you, but again, please put it in production. It's highly recommended. \[00:34:06\] Um, so yeah, that, that's kind of the basic flow. \[00:34:09\] We try to keep it \[00:34:10\] as simple as possible. \[00:34:12\] **Track 1:** sure. Yeah. When evaluating developer tools, I think that's an important thing, and it kind of keys into what you were saying before, that you should be able to opt into whatever depth of thing you need. Um, but a lot of devs who may be listening to this may need to go sell this to their team as, Hey, this will make our lives better. \[00:34:27\] Uh, and it's really, really genuinely better if you can do that by implementing it in a small place and saying like, this is what this thing does in an atomic version of itself. Uh, Can you, like, either we can see the value in this or not, and by not having to go and rewrite your entire application structure, your entire, uh, API stack, whatever it may be, you at least are able to have the discussion and sort of sell that to your PMs or, uh, business stakeholders wherever the case may be there. \[00:34:52\] That's, uh, a very, very helpful set of features to have. \[00:34:55\] **Or Weis (Permit:** Yeah, you also wanna build confidence in this on your own. Just \[00:34:58\] as your, as a \[00:35:00\] professional, you wanna say, this is something that I trust. This is final thing that has the performance profile that we \[00:35:05\] need. This would actually have the features and, uh, interfaces that we need. So we also encourage our people to run this side by side with their existing, uh, authorization solution. \[00:35:14\] Initially, don't actually gate, don't actually enforce the fossil. Just \[00:35:18\] see that it produces the same result. At the same pace as what you already have in place, then you gradually build confidence. Sometimes people would give it like a couple days to run. Often they don't get to that. Like they see that it works and they start running with it. But it's malleable enough for you to pick the, the level of trust that you have and gradually grow it and see that it doesn't bite your back just because you deployed it. \[00:35:43\] **Track 1:** right? \[00:35:44\] **Or Weis (Permit:** very email thing. Right. \[00:35:45\] **Track 1:** Yeah. \[00:35:46\] **Or Weis (Permit:** and we've actually, when it comes to. making this more relevant for the bigger team. Uh, this is where a lot of features that we have that people don't necessarily think about when they're building \[00:36:00\] authorization come in. \[00:36:00\] And those are the cur features of authorization for authorization or \[00:36:05\] permissions for permissions or roles for roles. for example, we have a concept called a project and a concept called an environment. These are silos for your policy and data. A project would be an application that you run and you can have different applications with different policies. An environment would be a separate deployment of, uh, the same application also can have, you want different policies in staging than you have from production, for example. \[00:36:30\] **Track 1:** Sure. \[00:36:30\] **Or Weis (Permit:** Um, and you can assign specific access to specific people to specific deployments. Um, so for example, and that also enables kind of more complex r and d patterns as opposed to individual comp, uh, contributor patterns. \[00:36:45\] So for example, you can say something like, um, as part of our preview branches, we, every time we deploy a new preview branch, um, we want to create an environment, a deployment environment thread, and we wanna control the access to it as well. So use our API and you create a, \[00:37:00\] uh, permit environment and you assign access to it only to the developer that deployed it. And so suddenly you have these dynamic patterns that are very easy to use, that are not just about these, uh, uh, permissions and access control for your app, for your customers. It's also how you as a developer with your team work on this in a way that you don't st stop on Trevor's toes. And these are the things that, that, hey, it's really hard to build on your own or even realize that you might need, \[00:37:29\] **Track 1:** Right. Yeah, \[00:37:30\] **Or Weis (Permit:** I think, a big enabler for developers to work on the, as a team on these things. \[00:37:35\] **Track 1:** I can certainly imagine why it makes sense to spin up an entire product around that too. That's represents a level of scope creep that started from, Hey, we need two different types of users to, uh, you know, an entire galaxy of permissions and complexity that comes along with that. It's super fascinating. \[00:37:48\] Uh, so tell me about what's next for permit? What are you working on now? \[00:37:52\] **Or Weis (Permit:** Uh, so we're working with something very exciting that is, uh, Um, I think, uh, even a little groundbreaking \[00:38:00\] and, uh, also kind of puzzling. It's called uh, ez, which stands for front end only Authorization. which, uh, I think only by the, that title, like front end only authorization, like that sounds like an oxymoron because how can you have authorization something secure in the front end? Um, and the, the reason the explanation for that is it's front end only, just like, uh, serverless has no servers. So \[00:38:27\] we provide basically the, uh, backend component that will do the authorization for your front end. But the idea is that you'll be able to consume sensitive APIs directly from the front end without writing any backend code. And I'll explain, let's say you want to use, uh, Twilio to send an sms. Or strive to send an invoice, or you wanna talk to a chat bot in Slack, you wanna do this in your front end, just one, click a button. The user clicks a button, and that happens. Currently you can't do that. You have to \[00:39:00\] write backend code. \[00:39:00\] It will add the secrets. We'll check for permissions and we'll actually call that external service. Otherwise, if you put that. In the front end, it's exploitable. Um, first of all, everyone's gonna steal your account key and second, they'll just circumnavigate your permissions cuz they're, they could just edit them in the front end. So what do you usually do is you, if you're a front end developer, you go and ask a backend engineer to write this for you. Or if you're a full stack or you're savvy enough, you'll go and write backend code in addition to the front end code that you actually want to write. Uh, but with FOAs you can just take that off the table cuz Foaz is a generic purpose, um, implementation of that backend component. \[00:39:41\] They need to write. So it includes \[00:39:44\] the, our policy engine that can do any enforcement that you want on when this is okay for the identity running in the front end. Now to consume or use that external service. And you can apply the application logic, not just the end account, like the Twilio logic, the \[00:40:00\] actually what it means in your application, and it injects the secretes that you need for like the token for the, for Twilio or Stripe or whatever. And then proxy is your call. From the front end to the actual end service and returns the answer to you. So you can securely use that from the front end without giving up any of the security, while not having to write glu code, which is like literally, uh, the amount of time people wrote That line of code adjusted proxies things to Twilio is, uh, I dread, I just dread the thought so much wasted time. Um, And by the way, also, this also applies for your own code. Let's say you have a backend service that you want to consume and you want, you want to let people use, but you don't have the granular permissions that your customers want there. ideally, you'll end up embedding permit into that service. But let's say you don't wanna open it up right now, you just want a quick solution. You can apply for as, as a gateway there as well. So you can slap on \[00:41:00\] permissions on whatever you need. and focus on what's important for you now, but gradually already build up, build up that, um, groundwork to scale into a more complex permissions model. So, so that's, that is for as front end only authorization, and we are launching it soon in two ways. \[00:41:17\] One, as a standard, as an open source standard and how you take general purpose decision engines like opa, couple them with Vault solutions to manage the secrets. Um, and with a proxy that will do the actual calling bundle those together with this standard and the schema and APIs we offer. And this, uh, and you can implement foz on your own and obviously is a service that we just provided. \[00:41:40\] We have this proxy component. We have the, uh, management APIs and management UI just get started and consume all of those, um, uh, sensitive services directly from the front end. And I think this is also. very empowering for front end developers. a you're in, you have \[00:42:00\] less dependency on the backend and backend engineers. And I think it finally, and I think this is important and finally brings front end engineers into the actual conversation of security. You're not just tagging along with the rest of the org. You can actually lead that conversation. I think that's important. We need like, Security is something that is really based on the weakest link. And if we \[00:42:25\] have places where the, the conversation around security is not strong enough, this is where those vulnerabilities are going to arise. So we need to embrace everyone into that conversation and hopefully this will enable front end developers to do that. \[00:42:39\] **Track 1:** Yeah, sure. I can think of a lot of devs who will be happy that they don't have to think about splitting things out to the server side. And, um, it's also an interesting problem for, um, devs who are learning the ropes, right? Where it's not obvious when you come up with an idea for like, oh, I wanna build this simple project for whatever it may be. \[00:42:57\] Uh, and then one day you stumble into, oh, I, \[00:43:00\] you know, I, what does the secret key mean? Why is it a secret? Who is it a secret from? And you suddenly realize that you've left yourself open to attack and things like that. Um, generally speaking, right to date, certainly the, um, path to take for that is to go spin up a server somewhere or to use a lambda, or in some cases an edge. \[00:43:17\] Uh, edge deployment to be able to execute that code. Um, why do you think it's, uh, interesting to do this now and why is this, um, an important use case to start tackling? \[00:43:28\] **Or Weis (Permit:** So first of all, we are seeing an emergence in this, the entire IAM space as the complexity of software continues to arise. This entire space with us and regardless of us is constantly evolving. We're seeing new products come in, as also we are seeing more threats come in. Um, applications become more complex, they also store more value and more data, and more people want to, um, take advantage of that. the risks that we are facing are also on horizon. This entire space as a result of that, is also on horizon. So, \[00:44:00\] and then that's basically supply and demand. More engineers need these things. More engineers need to build more of these things. More engineers need more velocity against this, these things. So they wanna just build their software, but they're stuck building all these security features, all of this glue code or, um, security additions that need to that. So, want to simplify that. We wanna take it back to basics. We wanna enable the front end code to focus on being front end code and easily consume the services without having to build the actual security components, but be able to consume them. Um, so I think, uh, at the basics, just supply and demand. And for us, we've already built this general purpose, easy to manage, uh, policy engine that you can do all the policies you want with, uh, as I said, an interface that a monkey can use. Uh, so that really can take that friction off. And this is, I think what you want here is a front end developer. \[00:44:54\] You want the velocity to build your app without a hassle. So that's what we're bringing to the \[00:45:00\] table. \[00:45:00\] **Track 1:** Yeah, maybe it'll allow my fellow front end developers to stop needing to call themselves full stack developers to get any respect around here. Uh, you know, especially now that GitHub co-pilot is, is able to cough up people's, uh, dirty secrets in their code, like, uh, \[00:45:14\] I think that was one thing that was found pretty early on is API keys that GitHub co-pilot knows about from someone else's environment that actually got, accidentally got committed to Git and all that. \[00:45:22\] Um, yeah, it's a super fascinating use case. So you, you're in early access right now? Is that what you said? \[00:45:27\] **Or Weis (Permit:** Yes. So we we're launching it for everyone very soon. Maybe when this is launched, it's already when this, uh, podcast goes live. Maybe we've already launched it. Um, but otherwise it's very, very soon. And, uh, for those that want early access, there's a button on our website. Uh, sign up and you'll, you'll get that access as well. \[00:45:47\] **Track 1:** That's great. Well, I will make sure that we also have a link to Foaz in our show notes here. Uh, and it sounds like, or you're a pretty accessible guy on the internet. So, uh, for for listeners to the show, I would encourage you to reach out to or, \[00:46:00\] um, it's, uh, been a very, very fascinating conversation and I think you're under some really cool stuff. \[00:46:05\] Uh, at permit. Uh, I would be, uh, more than interested in chatting with you again in the future as things develop and as the I am space tends to keep changing too. Um, or Weiss, thank you so much for joining today. It was really a pleasure to have you on APIs you won't hate. \[00:46:18\] **Or Weis (Permit:** Pleasure, Thank you, Mike. Uh, look forward to, uh, staying in touch \[00:46:22\] **Track 1:** Thanks again. We'll do it \[00:46:23\] **Or Weis (Permit:** everyone don't Yeah. Reach out \[00:46:26\] **Track 1:** All right. Thanks Laura. Take care. \[00:46:28\] **Or Weis (Permit:** Take care. Bye-bye. \[00:46:29\] **Track 1:** Bye. \[00:46:30\] ​ ### Laravel is Harmful URL: https://apisyouwonthate.com/newsletter/larvel-is-harmful/ Last updated: 2023-05-22T15:18:56.000Z For the last two weeks, it's felt like I can not escape PHP. The post about [Laravel being harmful](https://www.reddit.com/r/PHP/comments/131t2k1/laravel%5Fconsidered%5Fharmful/?ref=apisyouwonthate.com) made me chuckle. My feeling is that you can make *great* and *bad* things... with any framework. Even as someone who spends a lot of time in JS land, you must respect the web that PHP helped build. To top it off, Vercel and Deno dropped new databases and key values stores. Combine this with updates from React, and the world is aflame with the JS community reinventing PHP. Let's be honest; everything old is new again. I tend to avoid taking sides as we know who will win 😉. JavaScript is now coming full circle to combine the backend and frontend again; it will be interesting to see what happens with API layers. We should show love to all languages and frameworks. Except for Java, seriously, use Kotlin. 🤣 Alexander --- ## Monthly API Round-Up Our monthly round-up of articles, news and updates from the web. ### [Microservices to a Monolith](https://www.primevideotech.com/video-streaming/scaling-up-the-prime-video-audio-video-monitoring-service-and-reducing-costs-by-90?ref=apisyouwonthate.com) The Amazon Prime Video team discusses moving from serverless microservices to a monolith. The move saved them 90% in costs. Before jumping on the bandwagon of bashing serverless or microservices, it is worth taking a moment. In development, there is never a solution, just tradeoffs. If you want a balanced response to the drama that has kicked off around this issue, Kelsey Hightower as always [responded with wise words](https://twitter.com/kelseyhightower/status/1654098279116992513?s=20&ref=apisyouwonthate.com). ### [GraphQL, gRPC or REST](https://speakerdeck.com/robcrowley/graphql-grpc-or-rest-resolving-the-api-developers-dilemma?ref=apisyouwonthate.com) A talk by Rob Crowley from 2019 about choosing the right tool for the job. Everything has a tradeoff, and as much as I love REST, sometimes you might find other approaches to API development a better fit. ### [Should it be a Microservice?](https://goldbergyoni.com/score-game-is-this-a-microservice-checklist-for-microservice-design-verification/?ref=apisyouwonthate.com) With all this talk of microservices vs monoliths, I thought it would be prudent to include this article by Yoni Goldberg. This a great article with a point system on when to move something into its own service. ### [Understanding Distributed Systems](https://understandingdistributed.systems/?ref=apisyouwonthate.com) The previous link and this one might make you think I want you all to build microservices; don't worry, I don't. This book is a great read, even if, for some weird reason, you find yourself drawing sides. As an API or backend developer, you will have to build distributed systems at some point. Nice to be ready when the day comes. ### [How to be a -10x Engineer](https://taylor.town/-10x?ref=apisyouwonthate.com) Taylor Town had me laughing all morning with this insightful article. ### [Google Cloud's Parisian Outage](https://www.theregister.com/2023/05/10/google%5Fcloud%5Fparis%5Foutage%5Fpersists/?ref=apisyouwonthate.com) Google Cloud's outage has extended into its third week and possibly even a fourth. Given the escalating impact of climate change, which triggers heightened weather disruptions and significant temperature fluctuations, it seems likely that we will witness an increase in outages across all cloud service providers in the coming years. --- ## APIs You Won't Hate The latest content from the team at API's You Won't Hate. ### 🎙️[APIs You Won't Hate (the podcast)](https://apisyouwonthate.com/podcast/no-code-apis-and-automation-with-constantin-schreiber-from-fastgen/) Mike has FastGen co-founder Constantin Schreiber on the show to talk about FastGen's no-code API and workflow builder products. ### [APIMatic Review: Developer Experience-as-a-Service](https://apisyouwonthate.com/blog/apimatic-review-dx-as-a-service/) From somewhere deep in the forest, Phil takes some time to review APIMatic. --- ## From Our Community Articles written and shared in our free [Slack community](https://apisyouwonthate.com/community). ### [Getting Started with OpenAPI](https://learn.openapis.org/?ref=apisyouwonthate.com) Not written by our community but an outstanding share by Alex Savage in our Slack community. If you want to start with OpenAPI, this is an excellent place to start. ### [Portman and Contract Testing](https://dev.to/oneadvanced/api-provider-contract-testing-for-all-with-portman-openapi-and-postman-4ll1?ref=apisyouwonthate.com) Alex Savage popped up twice this week with an article he has written covering contract testing. See how you can use Portman with OpenAPI and Postman. Thanks for sharing this one, Alex. Thanks so much for reading. If you found this helpful, share it with a friend - it helps out a great deal. 👾 ### APIMatic Review: Developer Experience-as-a-Service URL: https://apisyouwonthate.com/blog/apimatic-review-dx-as-a-service/ Last updated: 2023-05-16T11:52:21.000Z # *This article is sponsored by [APIMatic](https://apimatic.io/?ref=apisyouwonthate.com), in that they're paying for our time to review their product, but they're not paying for what we write. If it's rubbish, we'll let you know.* I've used APIMatic a fair bit in the past, but never had the time to write about it. I'd like to thank APIMatic for paying for me to sit down and see what they've been up to lately, so I can write it up, and you can see if it will fit into your API toolkit. Since I last used APImatic a few years back they've released v3.0 with a bunch of new features and improvements. - **Overhauled Code Generator:** Support for latest language versions, new immutable client design, timeout and retries, XML, deprecated methods, enhanced OpenAPI support, and more. - **Redesigned API Portal:** A new slick look, new API Explorer, language-specific guides, pop-out code samples, and more. - **New Docs Editor:** Create pages faster with our new Markdown editor and organize your content the way you want. - **DX Analytics:** Analyze the performance of your API through detailed charts per language for metrics like API Portal page visits, SDK downloads, Time to first Hello World and much more. Noice! Let's jump in. ## Getting Started Once I'm done signing up for an account the UI asks me for an API description, like an [OpenAPI](https://learn.openapis.org/introduction.html?ref=apisyouwonthate.com) document or a [Postman Collection](https://learning.postman.com/docs/getting-started/creating-the-first-collection/?ref=apisyouwonthate.com). ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2023/05/Screenshot-2023-04-27-at-10.50.18-AM.png) My OpenAPI usually lives in a Github repo somewhere, but this is more of an "Import to APIMatic" process than a "Tell us where it lives" kinda process. That's a little tricky, because my OpenAPI is usually split up into multiple reusable files with `$ref`, and it was asking for "A File". For a moment I thought I'd need to bundle it up into a single file (`$ swagger-cli bundle api/openapi.yaml > /tmp/bundled.json`), but before going down that route I took a punt at zipping up the folder which contained `openapi.yaml` and schemas/\*.yaml\`, and throwing that into the upload form. Phew, it worked! Immediately I was rewarded with API reference documentation for the [Protect Earth](https://protect.earth/?ref=apisyouwonthate.com) API, with a built in request samples, response examples, and API Playground. ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2023/05/Screenshot-2023-04-27-at-10.58.40-AM.png) I'm also seeing all sorts of buttons about making an SDK, without having to muck about with CLI tools. Where to start! ## SDKs (Software Development Kits) I'll start with the SDKs, because it's one of the more unique offerings. All the popular "API management" tooling vendors like Postman, Stoplight, Kong, Optic, Mulesoft, etc. cover an over-lapping combination of verticals, like API design, mocks, docs, testing, learning, playground, etc., but so far as I know none of them offer to build an SDK for you. If you're new to SDKs here's an excerpt from the new [*Surviving Other People's APIs*](https://apisyouwonthate.com/books/surviving-other-peoples-apis/) book (aimed at API consumers) to get you caught up. > Knowing how to interact with a HTTP API directly is a good skill to have, but sometimes the API team will have created a library/package/module written in your programming language of choice, so that you don't have to bother. This is know as a Software Development Kit, or an SDK for short. > Theoretically, an SDK abstracts most of the transport layer specifics away behind idiomatic code, taking care of authentication, URLs, methods, headers, data formats, caching, retries, even HTTP status codes will be converted into meaningful exceptions. The best SDKs will even turn validation errors into a useful format. If that is the case for the specific API you are attempting to integrate with, then you might be in luck, and perhaps you can skip a lot of this book. > For example, we have not covered how Authentication works yet, and with a good SDK you would not need to know. They will give you somewhere to pop in the authentication details, in this instance an API key. ```javascript import Stripe from 'stripe'; const stripeApiKey = 'sk_test_4eC39HqLyjWDarjtT1zdp7dc'; const stripe = new Stripe(stripeApiKey); const charge = await stripe.charges.create({ amount: 2000, currency: 'usd', source: 'tok_amex', description: 'My First Test Charge', }); ``` Developers consuming an API either love 'em or hate 'em, but you don't need everyone to be excited about using an SDK, you just want to make sure developers trying to interact with your API have the choice available to them. In my experience it's a huge faff to generate them. You have to make some sort of CI/CD pipeline that notices changes to OpenAPI descriptions, rebuilds the SDK with hopefully some sort of breaking change detection, publishes new versions, updates docs, etc. That's all possible but I think most of us would rather be working on the product instead of mucking around with "DocOps" like this, so APIMatic aims to do that for you. ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2023/05/dx-472debc5eca2549ef222197ab02917de.svg) ### Generating SDKs with APIMatic APIMatic lets you make SDKs, and there's various ways to do this automatically but you can just download them ad-hoc to see how they look. ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2023/05/Screenshot-2023-04-27-at-1.36.46-PM.png) When you click download ZIP there's the option to download .Net, Java, PHP, Python, Ruby, or TypeScript, which is a pretty decent spread of options. ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2023/05/Screenshot-2023-04-27-at-1.37.47-PM.png) I'll go with TypeScript because it's going to be familiar to the biggest cross section of you all. ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2023/05/Screenshot-2023-04-27-at-1.43.27-PM.png) There's a whole folder structure made, full of code! That was infinitely easier than mucking about with setting up Java to try and use [OpenAPI Generator](https://openapi-generator.tech/?ref=apisyouwonthate.com). Before I dig into the code, a quick look at the generated README for my new SDK. It's super verbose, and spends a lot of time telling you how to make sure npm is set up, which the average TypeScript user probably knows. I'd have to trim this down quite a lot, and replace a few paragraphs with a single `npm -S install @your-org/sdk`, which is something I'd have to potentially automate. The generated SDK has a good load of options, including [timeouts and retries](https://apisyouwonthate.com/blog/taking-a-timeout-from-poor-performance/), which is great news as they're super important and often overlooked! | Parameter | Type | Description | | ----------- | --------------------------- | ------------------------------------------------------------ | | timeout | number | Timeout in milliseconds. | | httpAgent | any | Custom http agent to be used when performing http requests. | | httpsAgent | any | Custom https agent to be used when performing http requests. | | retryConfig | Partial | Configurations to retry requests. | There's another option that I was a bit surprised to see: | Parameter | Type | Description | | ----------- | ------ | --------------------------------------------------- | | accessToken | string | The OAuth 2.0 Access Token to use for API requests. | I am glad that the SDK generator can support `accessToken` for OAuth 2.0, but... the Protect Earth API doesn't! 😅 I'll ignore that as its probably optional, and I can remove it from my tweaked README and more human documentation later, but I feel like that should be based on the OpenAPI `securitySchemes` and only show up if there's a `type: oauth2` present. ### SDK Documentation The SDK is documented in a few ways. You can look around the hosted documentation online on APIMatic, or you can look at the `README.md` and subsequent `doc/controllers/*.md` files. To find out how to make a particular request can be a little tricky as you have to 1) look at how to do instantiation in the README, 2) see how to work with the controller, then 3) find the example of the method name. Took me a minute to piece it all together but once you know where to look it's ok. The way APIMatic generates the SDK is by turning top-level URL segments into controllers, so you get a `MapController` class for the maps service, and creates methods based on the OpenAPI `operationId`. ``` '/maps/sites': get: operationId: get-project-pins '/maps/sites/{uuid}': get: operationId: get-tree-pins ``` These two endpoints then become: ```javascript const mapController = new MapController(client); mapController.getProjectPins(); mapController.getTreePins(uuid); ``` If you pop all the bits of code examples together then it looks a bit like this: ```javascript import { ApiError, Client, MapController } from 'protect-earth-apilib'; const client = new Client({ timeout: 0, }); const mapController = new MapController(client); const uuid = '61f423e5-3cac-4e82-a520-b233730637bb'; try { const { result, ...httpResponse } = await mapController.getTreePins(uuid); console.log({ result, httpResponse }); } catch(error) { if (error instanceof ApiError) { console.log('api error', error.result); } else { console.log('unknown error', error); } } ``` That looks pretty reasonable to me. Let's "install it" and give it a try. ```bash npm install -S "/Users/phil/Downloads/Protect Earth API-TS_GENERIC_LIB" ``` That'll add this to my `package.json` dependencies: ```json "dependencies": { "protect-earth-apilib": "file:../../Downloads/Protect Earth API-TS_GENERIC_LIB" } ``` I know that looks a bit funny but its because it's a ZIP sat on my hard drive. Just pretend you've published it off to NPM, and go along with it. ```bash $ node index.js api error [Object: null prototype] { title: 'Page not found', type: 'https://protect.earth/probs/page-not-found', detail: 'Could not find anything for the provided URL.', instance: 'maps/sites/61f423e5-3cac-4e82-a520-b233730637bb' } ``` Ah yeah fair cop. You got me. That UUID is nonsense. Let's update that and run it again. ```json { result: { type: 'FeatureCollection', features: [ [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], ... 5600 more items ] }, httpResponse: { body: '{"type":"FeatureCollection","features":[{"type":"Feature","geometry":{"type":"Point","coordinates":[-0.27824000000000004,51.965559999999996]} ``` Huzzah! Thankfully it all worked even without me passing me a `accessToken: 'AccessToken',`, because I... don't have one. 😅 The docs might be a little different to how I'd write them, but it's made some pretty useful code for me, and I can always publish my own `README.md` over the top of theirs. The most important thing is having all that handily written code! When you're ready to publish it, you can have it push to GitHub and use some CI/CD shenanigans, like using `npx semantic-release` to automate releases to NPM, or you can use the built in package management support for Ruby Gems or PyPy. Whatever package manager you use, you won't have to download Zip files or do that funny looking local filesystem install method I did just to play around. ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2023/05/Screenshot-2023-04-27-at-3.10.44-PM.png) SDKs are looking pretty good to me. Let's look at what else they're offering. ## API Reference Documentation There are a lot of options for API reference documentation, so I will try and pop them all on a venn diagram to chart what I think the priorities were as the teams put the tools together, based on some guesswork, nattering with the teams, and the fact I worked on Stoplight Elements as a Product Manager in the past, and had to cut a lot of stuff from the release. 😆 ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2023/05/Screenshot-2023-05-02-at-2.11.59-PM.png) To me, APIMAtic might not be the prettiest API ref docs tool around, but it's got some interesting functionality which makes up for that. One of the first features I noticed was the integration of the SDK into the documentation APIMatic generates. Users can change the dropdown from HTTP to the language of their choice and see the code samples in different languages. ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2023/05/Screenshot-2023-04-27-at-2.59.50-PM.png) Most documentation tools have language selectors to change the sample code, but its usually either a generic HTTP client in that language, like Guzzle for PHP or Fetch for JavaScript, but this is *your* SDK. Those tools which do support inserting a SDK in do it via OpenAPI extensions that need to be kept up-to-date with some hamster-powered nonsense in your CI/CD pipeline. ```cURL x-codeSamples: - lang: 'cURL' label: 'CLI' source: | curl --request POST \ --url 'https://data.apiexample.com/api/example/batch_query/json?format=json' \ --header 'content-type: application/octet-stream: ' \ --data '{}' ``` Having APIMatic already know how your SDK works means you can skip all of that noise, and just publish your API docs knowing that HTTP, PHP, TypeScript, Ruby, etc is all working out of the box. The APIMatic Documentation offering covers more than just Ref Docs too, it can handle other Guide-like content, showing off getting started guides, or allowing you to document workflows instead of just throwing users at the ref docs and telling them to piece everything together themselves. Then there's the dev portal. ## Dev Portal There's a million "API Docs is about more than just API Reference Docs" articles out there so I won't subject you to that. Suffice to say you want to have other guides, getting started docs, help people play around with the API somehow ("API Playground" or "API Console"), and ideally some sort of landing page. APIMatic are helping out there too. They offer a hosted dev portal which can be embedded in an existing portal, or you can generate and host the docs yourself. ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2023/05/Screenshot-2023-04-27-at-3.31.56-PM-2.png) It only took a minute to slap a logo on it, and I could change all the colors, even the favicon, right from the UI without embed anything as a React component or faff around with a CLI. ## API Transformer I have used this tool extensively since 2016 and can tell you it's matured nicely over the years. Being able to convert freely between all these different formats is incredibly powerful. ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2023/05/Screenshot-2023-04-27-at-3.43.01-PM.png) The most convoluted setup I used it in was a homegrown dev portal pipeline that was taking in API descriptions from a myriad of different teams, including two acquired companies, and trying to turn it all into a single consistent API Catalogue and a Dev Portal. My pipeline was taking in RAML, API Blueprint, and OpenAPI v2, then using APIMatic Transformer we converted it all into OpenAPI v3.0 to build the API documentation (and jankily make SDKs using hand-rolled solution). Then I would ironically convert everything that wasn't already a Postman Collection into a Postman Collection, just so people had OpenAPI-based docs and mocks, and had Postman as an API Playground. [APIMatic](https://apimatic.io/?ref=apisyouwonthate.com) does a *lot* of that for you, so it's worth having a play around to see if you can avoid training 100 hamsters to power your own dodgy DocOps setups, and get back to working on whatever Drone/Crypto/AI startup you're being forced to work on now. ### No-code APIs and Automation, with Constantin Schreiber from Fastgen URL: https://apisyouwonthate.com/podcast/no-code-apis-and-automation-with-constantin-schreiber-from-fastgen/ Last updated: 2023-05-08T12:00:04.000Z #### Transcript \[00:00:00\] **Mike:** Welcome once again to APIs you won't hate. My name is Mike Balco. I am co-founder of APIs You Won't Hate and host of the podcast. I'm here today talking to, my new friend, Constantine Schreiber from Fast Gen Constantine. Thanks so much for hanging out with me today. How are you doing? \[00:00:16\] **Constantin:** Thanks for the invitation. Doing very well. How are you? \[00:00:20\] **Mike:** Yeah, I'm good. Thanks. I appreciate it. We've, we've had a long wait to get to actually talk to each other, and I'm really interested in what, what you're building and what Fast Gen is doing. But why don't we start here? Why don't you sort of introduce yourself and tell me a little bit about your I don't know, your, your history what you've done before fast Gen and how you got to where you're at now. \[00:00:37\] **Constantin:** Sure. So I guess my passion for creating things started when I was quite young. as a child, my, my first like green job was to become an inventor. Not necessarily in the digital space. I thought of it more as a, like crafting things and building like robots and stuff, but I think it's this like curiosity that led me to explore coding at a quite \[00:01:00\] young age and web development. \[00:01:01\] So I taught myself how to do pretty basic web dev stuff, with back in the day, like visual basic and all that kind of stuff. And. \[00:01:09\] **Mike:** Yeah. \[00:01:10\] **Constantin:** When it came to the site for college, I went into a slightly different direction. I decided to study economics and management cuz I just wanted to get like a broader understanding of business and the dynamics of various industries. \[00:01:23\] And the time I thought that it would compliment my technical skills and help me to become a more effective entrepreneur, which kind of was the end goal by then. But during my time at business school, I quickly realized that all the other that were entrepreneurial and wanted to have found something were extremely limited because of their lacking tech skills. So I decided instead of being the guy that has to a tech guy, I want to be the tech guy. So that kind of. Led me back to computer science, moved to \[00:02:00\] Berlin enrolled to a new kind of university called Code University which was very practical and allowed us to work on exciting projects with companies like Porsche. \[00:02:11\] So, built autonomous drones with them and really explored all kinds of different science aspects. There are. And it was at that time where that our first company together with two dear friends that I had met at business school, which was a company called Blair. And we were doing income share agreements in the us. Went through yc raised in total over a hundred million in equity, in debt, and sold that business or the assets of that business at the end of last year and, During our time at Blair, we, were kind of looking for a tool like fast Gen to get us off the ground faster and as I essentially like a platform that would use us to friction and \[00:03:00\] allows us to just ship quickly. And yeah, after talking to a couple of hundred users and validating that there might be others that are interested in a similar thing. We went back through a drawing board and started working on Fast John, and that's where we are now. \[00:03:17\] **Mike:** Yeah, very, very interesting. So already I can feel a few things we have in common. I also started my coding journey with Visual Basic once upon a time. , and though I, I did study computer science in undergrad. I also studied mechanical engineering thinking it would make me a more well-rounded. \[00:03:31\] Whatever I am. I don't use mechanical engineering much anymore, but certainly the calculus I learned along the way comes in handy from time to time. Yeah, that's really interesting. So sounds like Blair had a, had a pretty interesting ride. Certainly being admitted into YC and going through that program is really exciting. \[00:03:45\] a second time founder now with Fast Gen, it sounds like you've also learned from sort of the basics of the beginning stages of starting a company, which is talking to lots of people. Validating your idea before you go and hide away for weeks or months or years and build a product that you're hoping people \[00:04:00\] are interested in. \[00:04:01\] That sounds familiar too. I, I probably should have mentioned as well, I'm, I'm a repeat founder myself, so that brings us to fast gen then. What's the, the pitch for fast gen? What do you tell people when you're first introducing it to them? \[00:04:11\] **Constantin:** Yeah. So Fast Gen is a low-code platform for crafting APIs and workflows. And an analogy I like to use for maybe some of the less technical people is we kind of want to become what Webflow did for the front end, for the backend. So we. Are really focused on creating a experience that is less limited than, than other tools. As and I have to mention like we are builders at heart. We have four great engineers apart from me. And when we looked into existing solutions, we were always really annoyed at the lock in or. At the fact that there's just a limited complexity that you can build with it. And that was one of the core reasons why we were always kind of skeptical \[00:05:00\] about employing any kind of low-code tool in our stack. so with fast Gen, it is really at the core of our mission to try to allow people to build as complex things as they want. are definitely not there yet. but yeah, really working hard on incorporating that. \[00:05:17\] **Mike:** Yeah. You've already said a few things that I think are really interesting. Obviously the audience for the show are API developers, so I think I wanna pick your brain about the api generation side of things. But one thing that I've experienced over the past year or so is that a lot of developers seem to wanna lean away from low-code tools because it doesn't validate their desire to be deep in the, the weeds moving around zeros and ones, and, you know, being the smart guy, building smart things. \[00:05:42\] And I feel like there, there's a round of tools that are coming in this sort of revolution of like focusing on DevX and productivity as opposed to focusing on the genius behind the keyboard, so to say. That's, that's a really interesting, like shift in mindset. So what I'm curious to hear from you is, have you found it challenging \[00:06:00\] to explain to, to sort of hardcore developers that low-code tools are something they should play around with? \[00:06:06\] **Constantin:** Yes and no. So, in my experience and after talking to couple of hundred deaths about this, there seems to be a pretty clear distinction between two types of developers in my experience at least. The one type of developer is. More focused on the outcome. So, so more of the like maker type, I would say. They don't necessarily care too much about how they get there. All they care about is getting there. So for this group, it's extremely easy. Usually there's very little skeptics about local tools. But then you have the other group. , which is what I maybe would call the, the coding purists who are really in love with architecting with like thinking about I don't know, how can we refactor it, is to make it like even more efficient and for them it's definitely more, more difficult, I would say. but I think as soon as developers \[00:07:00\] realize that you are like pulling on in the same direction and not trying to work against each other things are becoming much, much easier and. Also like we are not product guys that kind of stumble into this. Like again, we are builders and programmers at heart ourselves. \[00:07:16\] So we are kind of building this tool for ourselves as an audience as well, and wanna build it in a way that yeah, we would've wanted to use it. So, yeah, really trying to, to incorporate that feedback that we get, especially from the second group. \[00:07:30\] **Mike:** Yeah, that's really solid. I think the hardcore sort of coding purists are a really interesting audience too because they are really good at sniffing out tools that are valuable for them and. Maybe it's just getting over that hump of like, you're not just being sold to, but you're being given something that multiplies your powers is a really interesting thing to do there. \[00:07:50\] So can you tell me a little bit about the API generation side of the house? What exactly can Fast Gen do? \[00:07:55\] **Constantin:** Yeah. So with Fast Gen you can essentially create \[00:08:00\] APIs at a click of a button. So the first thing a user sees when he opens our interface is kind of this drag and drop interface with a flow chart. And so you can select from any type of arrest endpoint that you wanna build. You can manage your authentication. We do input validation and then you can start pulling in your what we call actions. And there are different types of actions. So, One are the flow control blocks, conditions, if else switch for whatever. Then we have native integrations. So you can up your own server and put it in there. You can send General HTTP requests to third party services, which is extremely powerful if you want to integrate something where we haven't built out direct integration with yet. And then that's the third part you pull in the direct integrations we have. So say you wanna send an email with SendGrid, you just pop in your API key, pull in the block into the builder and then do what you wanna do. \[00:09:00\] And then all that's left as you hit deploy and you have a life endpoint that will return whatever logic you've built inside it. And I think one thing I want to add here is that again, trying to be really developer focused. Right now we are working on allowing you to and also ride all of this in kind of a conflict file. \[00:09:19\] So similar to how with Terraform you would specify your DevOps logic and code. So yeah, developers are not limited to using the interface, but can do so if they wanna do. \[00:09:32\] **Mike:** Yeah. I see. So there's some sort of, I don't know, edited in your it edit in your ide opportunity here with the config files. I imagine that probably allows you to use source control to manage the state of your workflows, APIs within fast gen. Is that right? \[00:09:47\] **Constantin:** That is exactly right. So there's two options for users. One, we offer versioning directly in the tool itself. So every time you're save your endpoint, you deploy it, it automatically creates a version and so you can easily \[00:10:00\] roll back. But if you want to, you can totally version it yourself as well. \[00:10:04\] **Mike:** a little bit of something for everyone there. I think that works well with the ethos of kind of being low code too. Don't necessarily have to require folks to have a full, you know, get. Support system strung up there. Okay, so you're generating API endpoints for sort of whatever I might need as, as kind of your end user. \[00:10:21\] What are some of the typical use cases you're seeing for this? \[00:10:25\] **Constantin:** Yeah, so they are, we have really diverse use cases, so and that's also worth mentioning. We do have customers that are not even focused on the AP API part, but also on the automation part. So, We have started or launched only I think a month ago now pretty much to the date. And very selectively picked 15 design partners that we let on the platform and that are currently building with it. And there we have from the small startup that are actually building their. MVP with us as a, as their full backend. Two more enterprisey organizations \[00:11:00\] just building not around their car but like for example, notification flows. We have on company doing that right now. And then the third type of, of user is for agencies. \[00:11:10\] This also seems to be pretty interesting, empowering them to. Build stuff that they couldn't before, was more more complex before. Yeah, from APIs to simple automations. Pretty much yeah, wild \[00:11:25\] **Mike:** Yeah, \[00:11:26\] **Constantin:** things right now. \[00:11:27\] **Mike:** certainly. Yeah. That . It's interesting because it feels like that gives you so many different directions you can go in. So what does your onboarding look like? What is the hello world of, of something that has so many potential use cases? \[00:11:39\] **Constantin:** Yeah. so because it has so many different use cases and because it is such an open field currently when onboarding a customer, we talk to them beforehand directly and trying to figure out what are you actually trying to use this for? And we even go a step further and pre-build the first. Flow or the first automation or the first a \[00:12:00\] p i for the customer, with the customer together. So, that way they get to see how things work on the platform. They directly get like, support for whatever they're trying to build, and then they're off on their own pretty much to, to further build out these things. \[00:12:16\] **Mike:** Yeah. Okay. So I want to know a little bit about how you're building this. What, what is sort of powering fast gen? Like what is your engineering team using? \[00:12:25\] **Constantin:** Yeah. So the backend we, this time we chose to go with. Using fiber, which for those not familiar, is super similar to Express, which we were using at Blair. Took a bit of time to get used to go coming from JS, but yeah, I think it's worth it in terms of speed. Then front end we're using View have been using that at Blair. \[00:12:47\] Had a great time there and continued using that. And then in terms of our for the backend architecture, we are. Heavily using Centri, FUO as a messaging server.\[00:13:00\] Rabbit mq In terms of our event service. We have built that on our own. We are thinking about temporal or integrating temporal. Not sure if we'll go for that yet, but yeah, that's like a rough overview of what we are doing in terms of \[00:13:17\] **Mike:** Yeah. Okay. I'm, I'm hearing a lot of folks lately exploring go for performance benefits alone. And I think that's something that a lot of developers are starting to have in their quiver, at least to have some experience with it. For a lot of reasons like that. And I would imagine when you're doing things like powering workflows that need to be both fairly performant and also, you know, like dead reliable go is probably a good option there too. \[00:13:38\] **Constantin:** Yeah, that was exactly our line of reasoning for why we went with that. \[00:13:41\] **Mike:** So switching back then to the API generation side of things you're generating rest endpoints, presumably from your the API builder are the I'm trying to think of how API devs will, will sort of want to validate this, this themselves. But I'm, what I'm curious about, I guess are two things. \[00:13:57\] Is are your APIs sort of compliant with something like \[00:14:00\] Open api? Is there a way to spit out a spec or something like that? And then is there a way to test the APIs that are generated by fast Gen to make sure that they're working correctly? \[00:14:09\] **Constantin:** Yes. So first question we are not able to, users are not able to generate their API specs with it yet, although that is definitely. on the roadmap and has been requested from our design partners already. When it comes to testing we just launched that feature yesterday, which is with what we call the debug mode which allows users to like just toggle into a different mode. \[00:14:32\] And then you can see, you can test run your APIs. You can test it with different input data and you visually see, how it goes through the flow. If one of the actions errors out, you directly see it, you see all the context data of all of the actions within that flow. But if you don't wanna use our tool again there's just one button to test with postman. and so you can, you can do that as well if you're more used to that or preferring that. Overusing our debugger.\[00:15:00\] \[00:15:00\] **Mike:** Okay. That at least feels like you know, the process of building these things, especially on the, the like drag and drop experience for building our APIs. I think a lot of people wanna see kind of, Like viscerally see that the data is flowing through as expected and you know, have things turn red on screen or whatever it may be when things aren't working correctly. \[00:15:18\] Yeah, that's really interesting. So why don't we talk a little bit about the workflow side of things too. What are some of the, I don't know, the interesting things about the workflow product. \[00:15:27\] **Constantin:** Yeah. So with the workflows right now, we offer two different types of workflows and they are differentiated by what kind of trigger they use. So one is a pretty simple chron, you just select whatever interval you want it to run. And the other is a event-based workflow. And how that works is. In all of your APIs or in your workflows, at any point, you can emit custom events. So then in the workflow side, again, you can create a workflow that listens to that kind of event to run a workflow. And \[00:16:00\] that kind of allows users to further modularize their APIs, their workflows, and, kind of, yeah, pull things apart a little bit. But apart from that, It's really similar to the API builder, so you have the same kind of logic available. You have the same kind of actions available. I think right now, honestly, the only difference is that you are not available to send like success or error responses as obviously you don't need it. Yeah, apart from that, really similar to, to the API builder, \[00:16:30\] **Mike:** Yeah. Okay. So when you say event-based, that is essentially saying that I can send an H A T P request to trigger one of these from something else. Is that right? \[00:16:39\] **Constantin:** Well, yeah. Well, yes, for that, I guess I would rather go with an api just have an \[00:16:43\] **Mike:** Right. \[00:16:44\] **Constantin:** app and then just trigger that. Right. But, \[00:16:46\] **Mike:** Of course. That makes sense. Yeah. Yeah. \[00:16:47\] **Constantin:** but yeah, you can, you can emit events within your, your flows and then when that event gets emitted, another workflow can get triggered. It listens to that event that gets submitted. \[00:16:58\] **Mike:** Yeah. Okay. It seems to \[00:17:00\] me that it's probably likely that fast gen is using fast gen for some interesting things on, on your side of the defense. Is that right? Are you using fast gen for automations at home? \[00:17:08\] **Constantin:** Yes. \[00:17:09\] **Mike:** Yeah. \[00:17:09\] **Constantin:** it for, for automations internally. We are not using it to build the core product of course, as I don't think that would be a great idea. \[00:17:16\] **Mike:** It'd be a little challenging. Yeah. \[00:17:18\] **Constantin:** but obviously all of our other automations that we have, like tons of slack bots that we built with fast gen, I don't know, simple things like auto archiving or. PR channels, stuff like that. But yeah, obviously heavily using our own tool. \[00:17:33\] **Mike:** Sure. Yeah, I think that's probably a good sign that you like what you're building, but also you know, forcing yourself to experience both sides of building and using it is, is a good thing for iterating on the product too. So what things are you working on right now for fast gen? \[00:17:46\] **Constantin:** Yeah. So a couple things that that we have on our mind. One, as I already mentioned auto generating your API docs is, is one of these. then I think another feature that would be really interesting to many of our users, Is having \[00:18:00\] templates or recipes, I mentioned before, everything you build with fast gen you have a conflict file in the background. And what we notice is that a lot of our customers are essentially building the same things just adapted and \[00:18:14\] **Mike:** Yeah. \[00:18:14\] **Constantin:** for, for their own company. So why should we force users to build the same thing over and over again? Why don't we just offer it as like a template or a recipe. So they have like a library of different, already existing or pre-built APIs or workflows. \[00:18:28\] They can browse, they can search through, and then just add them to the project, change out API keys or whatever they need to change to make it work for them. And then built with that. And then another feature that I find is really interesting. Is I've been experimenting a bit with or with allowing users or allowing you to generate APIs based on natural language. So obviously with the whole L l m hype and everything going around, \[00:18:56\] **Mike:** right. \[00:18:57\] **Constantin:** that got us thinking as well, and \[00:19:00\] surprisingly it works. Pretty, pretty well already. So that's something to look forward to as well. I think that you can just say, I want an API that does that, and then we give you a pre-generated workflow and you just added and adapt the little things that you need to change. \[00:19:16\] **Mike:** I can imagine a whole lot of people who are listening to this would be very interested in that. That sounds super interesting, especially for. , gosh, I mean, everything under the sun from like developer advocates just wanting to show off. Here's how you would do X with Y platform, right? Like implement a weather thing in view or a to-do list in, you know, laville, whatever. \[00:19:34\] But certainly sometimes it's just easier to say what you wanna do than to envision how you might connect the dots together. Especially for educating people and building you know, fresh products. That's, that's, that's super cool. I'm really interested in seeing that. So at this point, fast gen is in, you said it's a closed beta at the moment. \[00:19:53\] **Constantin:** It is, yes. \[00:19:54\] **Mike:** Yeah. \[00:19:54\] **Constantin:** working towards opening it up. It will take a couple more weeks, but I expect \[00:20:00\] that by probably middle to end of May we will open up to our open beta where people can sign up themselves. \[00:20:07\] **Mike:** Okay. Yeah, I think that'll be really exciting. What sort of feedback are you looking for from developers who may be interested in the product at the moment? \[00:20:14\] **Constantin:** Yeah, so, so that's actually one of the, the key things we're looking for. We've obviously greatly value the feedback of customers and potential customers. and we, we'd love to hear about the specific aspects that fa fast gen that they find most valuable, and obviously any feature that they think would make their experience even better. And additionally to that we're always interested in learning about the unique use cases that customers might want to use it. You mentioned it before a lot of different that you could use fast gen for. But the more we hear about what kind of things people want to, to solve and what kind of specific challenges they have, where fast gen might come in, that usually is really helpful for us. \[00:20:58\] **Mike:** Yeah, I'd imagine things like your, \[00:21:00\] I forget the exact word you used before, maybe direct integrations, things with like your, your coach built integrations with say, slack or whatever it may be. I, I'm guessing you'll probably get a mile long list of those from people, things that they wanted to talk to natively. \[00:21:11\] **Constantin:** Please. \[00:21:12\] **Mike:** I. \[00:21:12\] **Constantin:** I'm, I'm looking forward \[00:21:14\] **Mike:** Yeah. Yeah, I think so too. Oh, what about trying to think of like, people who are building the services that might make sense to be an integration for fast gen. Is there a process for that? \[00:21:23\] **Constantin:** Yes, absolutely. So if you are building a service and would like to see yourself being created to fast gen, please reach out. We have Member that is exclusively focused on, on building out the integration side of things right now. And obviously we do have a backlog already. but yeah, which is prioritizing on, on what gets requested the most and then going through and, and trying to, to build that. \[00:21:46\] **Mike:** Sure. Yeah, that makes a lot of sense. I, I feel like that is, will be an endless list and especially people who find value in this will have pretty good feedback for you on the things that they're interested in or missing or whatever. Yeah. Okay. So let's see what else, what, what what are, what do \[00:22:00\] you see as sort of the long term future of fast gen? \[00:22:01\] What are you trying to, I guess solve in the long run? \[00:22:05\] **Constantin:** the major idea is, is to empower people. And by empowering I I not only mean like and non devs. Which it certainly can use the platform. Obviously it's very helpful for them, but also empowering developers and reducing the friction between members in the team. So I would imagine a future where, both the devs and the maybe the ops team are using fast gen and the same platform. For, for different use cases, but also able to reduce the, the friction and the borders between teams. And what I mean by that, like a common example would be we, we saw before that whenever you limit ability to look into stuff or change stuff bottleneck it on, on the developers, becomes much slower and much more of a drag. \[00:22:55\] **Mike:** Yeah. \[00:22:56\] **Constantin:** allowing a non-techie to just \[00:23:00\] observe or see what a kind of workflow does or an API does, or maybe even allow 'em to make changes directly greatly reduces friction in teams internally greatly reduce friction for us internally. And that's what I hope fast gen will be able to achieve for other companies as well. \[00:23:16\] **Mike:** Sure. Yeah, that makes a whole lot of sense. I, I bring this up fairly often on the show, but I in a past life I worked for Stripe as a developer advocate doing payments things there. And one of the things that I found doing advocacy work for payments is that Payments sound very simple, but there are like a lot of things that can happen in the workflow of a payment that can go wrong. \[00:23:32\] And so it's really helpful to give people a visualization for here's, here's the entire workflow of what, what it takes to swipe a, a credit card and move money from one account to another. And. . The really, the easiest example of that is if you swipe your credit card and it gets declined, what do I do? \[00:23:47\] Right? There's a workflow there that has to happen. And one of the things that I was exploring as part of like the educational stuff I was doing was just showing people a diagram of the workflow of, of here's what your payment process looks like. That can be applied to any other sort of \[00:24:00\] workflow based problem solving, but thinking about your template Recipe product that you're starting to build out here too. \[00:24:08\] Those recipes will also sort of become like a checklist for teams that these are the things that we need to get done to implement X feature. And showing that here's the shape of the workflow to a PM or to a business stakeholder, something like that puts a lot more perspective into the scoping process, into making sure that we know what the dev team is going to need to do to get this all done. \[00:24:27\] Or even just, you know, the low-code integrator team, whoever that may be building this stuff out. . It's very, maybe it's not surprising, but it's very interesting to see how people. Think through problems differently when they have visuals placed in front of them. And for me, that's one of my favorite things about low-code tools is that you know, constantly you and I are fairly technical people. \[00:24:46\] We've probably spent a lot of time yelling at computers and trying to get that console logged to show what you want. But for someone who's never done this, it's really hard to convey what the problem space looks like and what you're actually doing when you're sitting at your computer all day, you know, downing \[00:25:00\] red Bulls and, and scratching your head. \[00:25:02\] . I think that's a really cool opportunity and I think it's something that a lot of folks who've built things can identify with. Often you feel like you're under the gun, you know, to get things done faster than is realistic because it just seems so simple to go take a payment. I think there's a big opportunity for you there. \[00:25:15\] That's, that's really interesting. \[00:25:17\] **Constantin:** Absolutely. I couldn't have said it better. You chose way better words than I would've used to Describe that \[00:25:22\] **Mike:** I. That's right. I won't take credit for it. Actually one of the things that I'll share in the show notes in addition to fast gen contact information and where to find you and things like that at some point in the past I wrote along diatribe on why developers should embrace low-code tools. \[00:25:35\] And I think that's a really valuable thing just to kind of like have in your back pocket as like, it's empowering. It's really cool to see, and it makes all of our lives easier to a certain extent. So Constantine, I believe, did I see that you are currently in YC in the, the winter program for 2023. \[00:25:51\] What, how long does that program last? What, what comes at the end of that? \[00:25:54\] **Constantin:** At the end of the program, demo day comes, which already happened actually last week or \[00:26:00\] the week \[00:26:00\] **Mike:** It did. Yeah. Okay. \[00:26:01\] **Constantin:** So we are, we are through with our, our second wife Iran now. But yeah, as always was a extremely valuable experience and yeah, couldn't recommend it more due to maybe any listeners trying on the edger on the fence about doing we. \[00:26:17\] **Mike:** Yeah. Got it, got it. Well, congratulations. I'm sure that's quite the thing to run through. For the folks listening to the show, what's the best way to find fast gen and what's the best way to find you? \[00:26:25\] **Constantin:** Yeah. So to find fast gen, really simple, just go to fast gen.com. Feel free to reach out to me directly. My email is Constantine with a c fast gen.com. And that's also where you find me. I'm not really that active on social media. You'll find me on LinkedIn. I have a Twitter account, but not really active on there. Yeah, best way would probably be via email. \[00:26:46\] **Mike:** Sure thing. Yeah, that's great. I'll make sure I include all of that in the show notes. And Constantine, it's been super, super cool talking to you. I'm really interested to hear sort of the future of your product and would love to have you come back and chat with us anytime. Especially if you've got launches and things like that to talk about. \[00:26:59\] I \[00:27:00\] think it'll be really interesting to see. And I'm, I'm definitely interested for the folks listening to the show and hearing what you think about the product and uses you might have for an a low-code tool that helps generate APIs as well. So p please feel free to to get in touch with us on the site APIs you won't hate.com. \[00:27:14\] You can email me, Mike APIs you won't hate.com, and I'm also on Mastodon and LinkedIn in a bunch of other places. Constantine Schreiber, thanks so much for being here. It's been a real pleasure. I've had a great time chatting with you. \[00:27:24\] **Constantin:** Thank you Mike. Thanks for inviting. It has been a pleasure and \[00:27:27\] **Mike:** You \[00:27:27\] **Constantin:** a good day. \[00:27:28\] **Mike:** of course. Take care. \[00:27:31\] ​ ### AI is Coming For Your Job URL: https://apisyouwonthate.com/newsletter/ai-is-coming-for-your-job/ Last updated: 2023-05-01T11:47:09.000Z At this point, I am starting to get a little sick of all the AI content in my feeds. From people telling me my job is obsolete to telling me I need to adopt these tools immediately. I always start as skeptical of new technology due to the harm our industry tends to cause. I am excited about some things we can do with AI, but I am already sick of the constant tidal wave of nonsense that comes with the hype. I found this to be a measured response regarding everyone telling me my job is over. Not strictly API related, but a good read, especially with all this AI panic: [The End of Front-End Development](https://www.joshwcomeau.com/blog/the-end-of-frontend-development/?from=newsletter&ref=apisyouwonthate.com). Alexander --- ## Monthly API Round-Up Our monthly round-up of articles, news and updates from the web. ### [AI-Powered API Docs](https://stripe.com/en-au/newsroom/news/stripe-and-openai?ref=apisyouwonthate.com) Stripe is adding OpenAPI GTP-4 to their products. I am excited about training the AI with API docs and commonly asked questions. You can ask it questions about the docs, helping you learn the API docs and build solutions faster. ### [Low Code APIs](https://blog.postman.com/announcing-postman-flows-early-access/?ref=apisyouwonthate.com) Build an API with Postman Flows. Represent your API with a visual canvas showing connections and triggers. Not going to be for all use cases, but an exciting approach to building an API. ### [Designing APIs for humans: Error messages](https://dev.to/stripe/designing-apis-for-humans-error-messages-94p?ref=apisyouwonthate.com) After reading [this article](https://preslav.me/2023/04/14/golang-error-handling-is-a-form-of-storytelling?ref=apisyouwonthate.com) about Go's error handling as a form of storytelling, it jogged my memory on a tweet from Phil about silly API errors. Paul Asjes from Stripe walks us through creating human-readable API errors and why they're essential. ### [AWS Lambda response streaming](https://aws.amazon.com/blogs/compute/introducing-aws-lambda-response-streaming?ref=apisyouwonthate.com) For those of us who have been building serverless APIs using AWS Lambda, they finally dropped response streaming. A lot of us have been waiting a long time for this feature. Providers like Vercel resorted to writing their own streaming response on top of AWS Lambda's as the feature was missing. ### [Load Balancing](https://samwho.dev/load-balancing/?ref=apisyouwonthate.com) Hands down the best post I have ever read on Load Balancing. It covers how it works, tradeoffs with different algorithms and includes visual aids. The author Sam Rose has put a lot of effort into this post, which shows. --- ## APIs You Won't Hate The latest content from the team at API's You Won't Hate. ### 📖 [Surviving Other People's APIs](https://apisyouwonthate.com/books/surviving-other-peoples-apis/) Phil and Mike have been working hard on the next book, "Surviving Other People's APIs", and it's close to being done. They would love your feedback. Pre-order now and email us your feedback or message in our Slack community. Phil's writing style and no-nonsense attitude from his first book are why I am here. So definitely getting this one too. ### 🎙️[APIs You Won't Hate (the podcast)](https://apisyouwonthate.com/podcast/learning-about-webhooks-with-tom-haconen-from-svix) Since the last newsletter, Mike and Phil have been busy cranking out a few podcast episodes: - [In this episode](https://apisyouwonthate.com/podcast/jazzed-about-api-client-library-codegen-with-danny-sheridan-from-fern/), Mike chatted with Danny Sheridan about how easy it is to create client libraries for your API with Fern. - Mike chats to Tom Haconen from Svix [in this podcast episode](https://apisyouwonthate.com/podcast/learning-about-webhooks-with-tom-haconen-from-svix/), covering webhooks as a service. - [Finally, Mike and Phil talk with Steve MacDougall](https://apisyouwonthate.com/podcast/analytics-for-your-api-with-steve-mcdougall-from-treblle/), who has recently started working in Developer Relations at Treblle. A previous sponsor of APIs You Won't Hate, Treblle is like Google Analytics for your API. ### [Turn HTTP Traffic into OpenAPI with Optic](https://apisyouwonthate.com/blog/turn-http-traffic-into-openapi-with-optic/) I am starting to think Phil must have a twin or a clone. While saving the planet and planting more trees than any man alive, he found time to review Optic. Learn how to get your API documented in ten minutes. Phil tests the tool documenting the Mastodon API. ### [APIs You Won't Hate on LinkedIn](https://www.linkedin.com/company/apisyouwonthate/?ref=apisyouwonthate.com) You can now find us on LinkedIn. Follows appreciated! ## From Our Community Articles written and shared in our free [Slack community](https://apisyouwonthate.com/community). We are adding a brand new section to our newsletters sharing articles from our community. If you have written something you think is worth sharing, please get in touch with me via our [Slack community](https://slack.apisyouwonthate.com/?ref=apisyouwonthate.com). We are looking forward to seeing what you all write. ### Analytics for your API, with Steve McDougall from Treblle URL: https://apisyouwonthate.com/podcast/analytics-for-your-api-with-steve-mcdougall-from-treblle/ Last updated: 2023-04-19T05:58:11.000Z ## Show notes Phil and Mike sit down for a chat with Steve McDougall, who has just recently started working in Developer Relations at Treblle, a past sponsor of APIs You Won't Hate. - [**Treblle**](https://treblle.com/?ref=apisyouwonthate.com) \- Mission Control For Your APIs - [https://treblle.com/](https://treblle.com/?ref=apisyouwonthate.com) - **Steve McDougall** on Mastodon: [@juststeveking@php.social](https://phpc.social/@juststeveking?ref=apisyouwonthate.com) - **Steve McDougall** on twitter: [@juststeveking](https://twitter.com/juststeveking?ref=apisyouwonthate.com) - API Quality Score [https://www.treblle.com/features/api-quality-score](https://www.treblle.com/features/api-quality-score?ref=apisyouwonthate.com) - Green Software Foundation [https://learn.greensoftware.foundation/](https://learn.greensoftware.foundation/?ref=apisyouwonthate.com) - Current YouTube playlist for API content: [https://youtube.com/playlist?list=PLKJBt7aeK3k\_v52GHzGgRDwoxWRvweOuA](https://youtube.com/playlist?list=PLKJBt7aeK3k%5Fv52GHzGgRDwoxWRvweOuA&ref=apisyouwonthate.com) - Recording of Livestream with Steve & Phil: [https://www.youtube.com/live/HxxlSQtsYU8?feature=share](https://www.youtube.com/live/HxxlSQtsYU8?feature=share&ref=apisyouwonthate.com) - Building APIs in Laravel talk at Cambridge PHP: [https://youtu.be/CYPLEvJGpPU](https://youtu.be/CYPLEvJGpPU?ref=apisyouwonthate.com) - Testing JSON:API endpoints in Laravel with PestPHP: [https://www.juststeveking.uk/blog/testing-json-api-endpoints-with-pestphp/](https://www.juststeveking.uk/blog/testing-json-api-endpoints-with-pestphp/?ref=apisyouwonthate.com) - Building APIs in Laravel: [https://www.juststeveking.uk/blog/building-apis-in-laravel/](https://www.juststeveking.uk/blog/building-apis-in-laravel/?ref=apisyouwonthate.com) - Eloquent API calls: [https://www.juststeveking.uk/blog/eloquent-api-calls/](https://www.juststeveking.uk/blog/eloquent-api-calls/?ref=apisyouwonthate.com) --- #### Transcript Analytics for your API with Steve McDougall from Treblle \[00:00:00\] **Track 1:** Take it away, Phil. \[00:00:02\] **Phil S:** Oh me, \[00:00:02\] **Track 1:** Oh yeah. \[00:00:04\] **Phil S:** How do you even do an intro? I haven't done it in so long. Uh, Hey everyone. Welcome to APIs you won't hate. I dunno what episode number this is or what day it is. But thank you for coming along. \[00:00:13\] **Track 1:** That's what we like to hear. It's perfect. \[00:00:15\] **Phil S:** Yeah. Do you wanna do the intro \[00:00:18\] **Track 1:** Oh, you better believe I'm gonna use that. \[00:00:20\] **Phil S:** I haven't done it in age years. Matt was doing it forever. And then you did it and yeah, I'm just, I'm just along for the ride. Half the time. \[00:00:26\] **Track 1:** Yeah, I, it's just your velvety narrator voice, Phil, that the people come for. Whoa. Why don't we do it this way? Uh, Welcome to APIs you won't hate for the second time. My name is Mike bco. I am one of the, co runners of APIs you won't hate. I'm hanging out here today with Phil Sturgeon, and our good pale Steve McDougal from Treble. Phil, how are you doing today? \[00:00:45\] **Phil S:** I'm pretty good. I've spent the whole morning out and about in the Woodlands up to my knees in mud, which is my happy place. And now I get to talk about APIs, which is my other happy place. So I'm excited that we got Steve along today. Thanks for coming, Steve. \[00:00:56\] **Steve McDougall:** Thanks. Thanks for having me. So yeah, I'm really excited to be here. \[00:01:00\] Obviously I've been a longtime listener, new in the kind of public facing API world right now. So I'm excited to, to have a chat about APIs. \[00:01:10\] **Track 1:** Yeah. Well, uh, to have you here. Why don't we start this way? Uh, tell us little bit about yourself. Um, live? Where do you work? What have you been working on? How'd you get here? \[00:01:18\] **Steve McDougall:** So I live in South Wales and I believe my house is probably gonna get turned into trees by fill at some point soon. But yeah, I've been in the API space for maybe six, seven years heavily on the PHP side using Lara Valve Slim. Recently you've been using framework X, which is really fast, asynchronous framework. Yeah, I do a lot of, a lot of talks around APIs. I do a lot of talks around integrating with APIs with a focus on kind of quality and scaling out the complexity. And my day-to-day has basically been building lots of tooling around APIs, so I'm in that API head.\[00:02:00\] \[00:02:00\] **Phil S:** So you've just started uh, working at Treble? Right. And I, I've come across Treble a bunch of times. I was on one of their Twitter circles a while ago and I think I chatted to them like right when they were starting out, but, Ages ago, and they do a bunch of stuff. So in a similar way to a lot of the tooling vendors, there's, there's so many different parts of the API lifecycle. Most tooling vendors don't want to just do one bit. They want to try and do a few. And so you guys have got an interesting different set, mostly with some overlap of stoplights. So I kind of bounced around treble a little bit then, but we've gotta mention that uh, Trevor in the past have sponsored the. But I don't listen to the podcast, so I don't know what they said about it. So, , so can you explain to me what's, what's going on with? \[00:02:46\] **Steve McDougall:** Yeah, so we are currently scaling, we're hiring more engineers to help us build out better and higher quality tools, building out more tooling. And we're we're basically what you're. Google Analytics, but for your \[00:03:00\] api, that is the best way to explain what Trevor is to most people. You could go into the observability, into the monitoring and explain all of that aspect of it, but I've always found the easiest way to explain it is just, it's like Google Analytics for your api. \[00:03:15\] **Phil S:** Okay. Nice. And looking at it just bouncing around the homepage, I mean the, the API monitoring and observability stuff looks to me a little bit like New Relic, but kind of API focused. \[00:03:26\] **Steve McDougall:** Yeah, similar. \[00:03:27\] **Phil S:** Okay. \[00:03:28\] **Steve McDougall:** very similar. One of our one of the clients I'm working with at the moment kind of described as, as a log rocket, but for their api, they use Log Rocket on the front end, and. Now currently trying out us on their APIs. So \[00:03:43\] **Phil S:** Interesting. Okay. And then, so you've got the monitoring, but then it kind of moves into auto-generated API docs. How, how does that work? \[00:03:51\] **Steve McDougall:** So basically every request that you make We log, we do ETL script on it to make sure that we're pulling out data, standardizing data formats, \[00:04:00\] and adding it into our database. And while we're doing that, we're triggering lots of events to start building up paths and endpoints and query parameters, emerging query parameters, and we are just kind of merging and mangling data in the background to build out an open API spec. Basical. \[00:04:16\] **Phil S:** Nice. That's pretty cool. There's, there's. People doing that now. Like I, the last post I just did the other day on, on APIs, you won hate blog was about using Optic to do that, which is a slightly different approach. You, you run a CLI script and then basically man in the middle, whatever, a API you feel like, man in the Midling. I was doing it for the master mastered on a api, which is pretty cool. But this, this will be kind of implemented by the API developer team or the CIS ops team or something. You just kind of shove that. No, it, it's not done at the server level. It's not like a proxy. It's done as like an sdk. How does it, how does it get that? \[00:04:52\] **Steve McDougall:** our, well, we we're a, we've got an s STK as well, and basically it's just a, a piece of middleware that \[00:05:00\] will capture the request of response. and then on the kind of termination of the script, we gather the data and send that off to our API to then be processed. So it's, yeah, it's a middleware basically. So one of the benefits that I found is that you can choose what endpoints you want to add it to. You can add it to all of them, add it to some of them, and you can slowly roll it out to make sure that you know you're getting the data you need. And because we are en we're enabling field masking that gentler rollout is sometimes recommended because sending over passwords, credit card numbers, email addresses, all of those sorts of things in a monitoring solution you don't want. So with our field masking every request and response, Goes through this masking process where we turn the value into stars of the same length, roughly. So that you're still getting the understanding of that data, but \[00:06:00\] you're not passing that data, which keeps things a lot cleaner for you. There's no kind of data privacy worries about are they going to know my passwords? And it, it does that with the headers as well. , you, you kind of, we are masking all the different orth headers that you might have and custom orth headers as well. So \[00:06:20\] **Phil S:** Okay, that's pretty smart to find it. I guess it's looking for like X hyphen a P i key and X hyphen authentication token, as well as \[00:06:27\] **Steve McDougall:** yeah, we, we basically got a big list of different ones, uppercase, lowercase, spelt wrong, not spelt wrong, , just to try gather all of them basically. \[00:06:37\] **Phil S:** Access tok. Um, fantastic. Okay, that sounds really handy. I, I like, I like the idea of it being a, a middleware and I like that there's different people doing different approaches cuz I know I wrote about aquita turning your HTP traffic into open API as well and there's a few more steps of indirection involved there cuz Again, Aquita is like a whole platform that does a bunch of different parts of the puzzle. And then you kind of export \[00:07:00\] the open API out and they just added like a engine X module so you can really wedge it in at, at \[00:07:07\] **Steve McDougall:** Oh, nice. Yeah, \[00:07:09\] **Phil S:** that, that seems complicated and something that a lot of people can't necessarily do in certain hosting environments anyway, so. \[00:07:15\] **Steve McDougall:** yeah. We are looking at different options at the moment as well. Like I was talking to one of our other devs the other day about could we create our own custom ingress for Kubernetes? So that we can just capture specific parts and then it's not handled within the web request. It's handled within the, the, the Kubernetes engine itself. We'll then send that data to us. But you know that that's probably something down the road once we've hired a few more people, I think. \[00:07:42\] **Phil S:** Absolutely. Well, there was another approach that I think a previous job I was talking about trying to build this, I think Stoplight was trying to do it and WeWork was trying to hack it. But we were talking about trying to, cause we've got um, the Prism proxy, right? And that in previous versions, prism Proxy was doing some learning. It's not doing learning \[00:08:00\] anymore. It does do a bit of like content validation to go like, oh, that's bad. You did an error, whatever. I'm telling. And, and all of that stuff. We only ever really got it as far as working in development and staging because we didn't really have the confidence in the production performance to say like, Hey, go wedge this into the critical path and I'm sure it'll be fine. And so we were kind of recommending a few bigger customers you can do like Traffic cloning or like traffic shadowing, I can't remember what it's called. So you basically in, in Kubernetes somewhere, say like, look for all these requests coming through. Take one in a hundred of them and just like fire 'em over this way as well. And then you just make sure the database doesn't do anything and you have to build like a separate environment that's just like a, a castrated version of the royal production. \[00:08:43\] **Steve McDougall:** It. It sounds like a lot of juggling just to get that simple kind of logging effort, right? \[00:08:48\] **Phil S:** Exactly. Yeah. But then on the other hand, like with, with your sdk, it's super simple cause it's just in the code there, but then that's still kind of slowing things down. Although if it's after the person got there, Jason, maybe \[00:09:00\] that doesn't matter. So there's, there's like a bunch of different ways of like pros and cons of, of how to get that stuff. And \[00:09:07\] **Steve McDougall:** definitely. And that's one of the struggles that I'm working on this week actually. I was working with the people from React, P h p, working on asynchronous middleware. So what I can do is I can send a H E T P request asynchronously in P H P, and while that's running, I can return that response and then deal with any failures within that asynchronous process. \[00:09:32\] **Phil S:** Yeah, that's brilliant. That's what you want. I mean, yeah, obviously I have no idea how the SDK works, but that , that sounds like a good approach. \[00:09:38\] **Steve McDougall:** currently not like that, but , I'm this refactoring effort at the minute where I'm looking at our p the PHP ecosystem for our SDKs to make sure we support a wide bunch of frameworks as, as well as, you know, psr 15 middleware as well for those kind of build your own. So yeah, I'm, I'm trying to assess all. To try and figure out what the \[00:10:00\] best approach is so that I'm not forcing dependencies on any one framework. So it, it's been a juggle definitely. \[00:10:08\] **Phil S:** guessing, I'm guessing that a lot of the different SDKs in different languages will work differently as well. So I'm, you know, we are probably by default talking about PHP without really mentioning it there, cuz that's our backgrounds. But I'm sure, I'm sure like the node one is asynchronous by default or whatever. So, Just gotta try and find different ways to make it work. I thematically, or whatever the correct word is, I just, I just do tree stuff now. I forget words. \[00:10:30\] **Steve McDougall:** I forget words. I just know. Birch \[00:10:33\] **Phil S:** concepts. Yep. \[00:10:36\] **Track 1:** so Steve, I'm a little bit curious about, um, end user benefits of, uh, adding trouble to your system. So, made the comparison before between treble and uh, right? Sort of the log rocket for your APIs. Um, used Log Rocket before and am a big fan of it for the applications that I build for the web and whatnot. Um, assuming I had never been exposed to Log Rocket before, what the heck does that mean? What do I get? \[00:10:59\] **Steve McDougall:** so \[00:11:00\] what you get is you can have multiple projects, obviously. Um, Maybe you've got internal APIs, external APIs, and you want to kind of keep all of them, you know, separate. We capture the requests, the response, response time, request time, the total low time so that you can start to see. Perhaps you're getting issues in that API lifecycle, in that request response lifecycle. We capture bugs, we capture that full kind of stack trace from the exception and handlers so that you can see bugs that are introduced. And this is like over time, for example. So let's say you do a big feature deployment and it breaks a couple of endpoints. You can kind of pinpoint that back down through. Tracing through the requests, seeing where those errors were back on. Were, were starting to be introduced and we're, we are looking at building out a, a, you know, a fully fledged alerting system as well, and enabling you to ignore certain errors within your \[00:12:00\] api. For example, with with Laravel, for example, if I return a 4 22 from Laravel, it means that my validation's failed in Laravel. That's currently being reported as a problem. However, it's technically not an API problem, it's a user problem. So we're looking at extending rsdk to say that you can ignore specific exceptions \[00:12:23\] **Track 1:** Yeah. \[00:12:24\] **Steve McDougall:** that they aren't reported as an actual problem cuz it's a user problem or an implementation problem, not a API problem. As. \[00:12:32\] **Track 1:** right? Yeah. And I guess when you're trying to assess how your, um, system is running, uh, are working as they should, you don't wanna report that as a problem? Yes, sir. \[00:12:40\] **Steve McDougall:** You don't wanna report that someone's trying to register again, using the same email address as a API problem. Right. Because it's not, it's, it's a human problem at that point. \[00:12:50\] **Track 1:** I think it's interesting too that, um, getting a bunch of information in addition to just whether things are working or failing or crashing, whether you're getting errors and things like that. Especially in a \[00:13:00\] world where people are starting to talk way more about, uh, between. typical lambda serverless functions and edge functions and all that. Uh, sort of like technical debt creep that can happen where your, um, can become slow and slow, can be the problem not failing, right? You're things can start to feel like they're kind of creeping along and having some sort of intelligence behind the scenes to be able to track like, hey, which calls are getting slow, help your engineering teams to, to trace that all down. Um, \[00:13:27\] **Steve McDougall:** massively. I'm, I'm currently working with our engineering team at the minute to implement, a re-architected approach to how we're logging things so that we're gonna have much more intelligence on the reports that we can generate. So, you know, you can pinpoint the exact request when it was, why it was what happened, and a lot more. And we're also jumping on that bandwagon of G P T and saying, okay, so what can we actually. What, what could we, what information could we get from AI on this? What, you know, we, we've obviously got a lot of generating going on \[00:14:00\] for building out an open API spec. Is there any way that we could improve upon that? \[00:14:05\] **Track 1:** of course. And behind the scenes, those G p t uh, are really good at noticing patterns that are probably not, uh, apparent to you and me. \[00:14:13\] **Steve McDougall:** yeah, exactly. \[00:14:14\] **Track 1:** superpower in itself. imagine when adding trouble to your service. Actually, one of the things I saw on your site earlier, I was kind of browsing around, is that I think you get something like 50 data points for every API call, as part of the kind of analytics end of this. Um, that feels like a lot, right? Like that can be imposing at first. If you're, if you're working with a team that's just adopting trouble, like what are some of the first things you tell them to look for or look at or optimize for when they're working? \[00:14:38\] **Steve McDougall:** To be honest, I've not worked with many people onboarding who have asked questions. I've only been here three weeks, the only thing I've dealt with so far is uh, talking about. You know, how much of a bottleneck is it gonna cause with it being middleware and sending a request? Had to explain that. It's terminating middleware. So it's at the end of that PHP CGI process or you know, \[00:15:00\] FPM process. It triggers a terminate command, and then that middleware triggers talked about data leakage and using the masked fields to be honest. The hardest thing is getting people to understand what it gives you. When it's just like, well, I know I'm sending a request. I know I'm getting a response. I, you know, I could probably generate an open API spec using an open source tool. But what that isn't telling you is what's being used, how much it's being used, which is the most popular endpoints, which is the slowest endpoint. Where is where your request body is starting to get a little bit too excessive? Should you. introducing Jason API specification so that you can tweak that payload coming back a little bit more because your response size is getting a little bit too big and taking a little bit too long. So yeah, there's, there's lots of things we have to do when we are onboarding people to explain what you can do and how to get the most out of it. And \[00:16:00\] part of my job is to record and create content around that as well. So, I'm gonna be creating videos around, I'm gonna be able a really bad API for fun and stick it through treble and see how much I can improve it based on just using those insights instead of me using my brain and thinking, oh, I know this is gonna be a problem, I'm gonna take that. I might even get someone else to join me on it and get them to do it . Cause I know that habits can die. And just see how much improvements we can get on an API based off of just that insight. \[00:16:33\] **Phil S:** As a consultant in APIs and as a, like someone who's worked on dev tools, I have this really interesting kind of thing where I'm, we're building these dev tools to try and solve problems and then the people don't necessarily know that they need them cuz they've just got shit breaking. If, if some shit's breaking and some stuff's going slow, then they're like looking at some exceptions and saying, oh, you should probably fix that. And it is very much like a how do we fix the problem that happened? But then. They \[00:17:00\] don't really. I've worked with so many teams that don't really recognize there's so much more than just the code exceptions. Like you get some exceptions or you get some slowness, or you might not even know it's going slow. Like, how do you know that something was slow or someone complained about it? Well, how many customers did you lose before they complained? \[00:17:15\] **Steve McDougall:** yeah, exactly. \[00:17:16\] **Phil S:** they do complain, you're just like, well, I don't know. It seems all right to me. And you get some really weird scenario where \[00:17:21\] **Steve McDougall:** Works on my machine \[00:17:22\] **Phil S:** month or something. Yeah. And so, or there's this, like, we've had this crazy thing where, you know uh, One CR job would run like, you know, once a month and then requests would be really slow then. So someone would complain about the a p i being slow and then you check it out tomorrow, you're like, nah, that's fine. You're mad. So being able to look back in time and see that kind of the performance, just graphs and stuff are super handy. But then people don't know that they need that. Cuz if you are, like I said, if you're getting complaints, you, you, you wanna try and treat the the symptom or the why, no. You wanna try and treat the specific problem at the time and \[00:17:54\] **Steve McDougall:** Yeah, you wanna, you wanna treat the cause, not the symptoms. Right? \[00:17:57\] **Phil S:** Yeah. Yeah, \[00:17:59\] **Steve McDougall:** \[00:18:00\] So, you know, before I started down this journey of API monitoring on observability stuff you know, the typical approach would be, oh, I don't know. I'll install Century and I'll get all my logs and I'll enable performance monitoring, and if anything notifies me, I'll deal with it. But that's, you know, you've gotta, sometimes you've gotta take that proactive approach when you're trying to build a good product, right? You got proactively search for ways to improve what you've got. If, if you're not building new features, you're improving what you've got. So it's just a tool to enable you to do that at the end of the day, isn't it? \[00:18:35\] **Phil S:** Yeah, exactly. I mean, I, I basically lived in New Relic and went at previous jobs and it, it seems like this would be more useful than that cuz it. Targeted APIs, I had to kind of convince it to, to, to tell me what I wanted a lot of the time. And I'm sure they've changed in the years since I've used it. But yeah, like that was my, my job as kind of API governance before API governance existed was just to snoop around New Relic, see what was happening and, and \[00:19:00\] the development teams maintaining those APIs Would, if there was an, an error or, you know, something bad in the logs, they would, they would usually get told about it and they would usually fix it. But it was, it was trying to find all the things that. Weren't problems yet. Things that were on their way to being problems \[00:19:15\] **Steve McDougall:** Yeah. And the problem with things like that is that you're not spotting trends, are you? You're just seeing. Oh, it took 52 milliseconds to do that. It took this long to do that. It was doing this, his a SQL query that was ran and you know, you're not seeing, okay, this response payload's huge or this response load time is increasing. You're not seeing that information when you're just kind of scrolling through performance metrics from Century or New Relic or anything. \[00:19:45\] **Phil S:** Right? And so is that what the treble API score is all about? Is, is trying to remove some of that, like trying to guess at what's good or bad and just basically saying, \[00:19:54\] **Steve McDougall:** Yeah, basically. \[00:19:56\] **Phil S:** that's kind of cool. \[00:19:57\] **Steve McDougall:** And I, I'm currently building out monthly \[00:20:00\] reports at the moment as well, so that you can see, you know, for, cause I used to be an engineering manager. One thing that was important to me was, you know, what is our performance like over the last month? Has it gone up or down? It's a low time going up or down. Is this an issue? Memory consumption being increased to see, you know, sometimes a PR slips through that causes a bug that you don't know about. Maybe use with more memory. Maybe it locks a database for X amount of seconds. You, we, yeah. You don't see that unless you can kind of get that holistic overview on, you know, the last 30 days. Response times gun up by X percent. Your low times gun up by this. \[00:20:43\] **Phil S:** really interesting cuz I think, again, in when I've worked on this sort of thing, people like. Look at the graph, look at the response time. I mean the, the classic New Relic graph of like median time, so it's not even, it's not even showing you the percentiles, it's just like, on average it looks fine. And, and even then you're kind of \[00:21:00\] looking at a graph and depends on how thick your glasses are and whether you can spot the fact that it's gone up by 2% over the \[00:21:06\] **Steve McDougall:** Especially as they don't, yeah, they don't, they don't allow you to zoom on the graph, so it's like you are looking at 30 days and you can't really tell that much. \[00:21:14\] **Phil S:** Exactly. So like if, if your application is getting slower by 2% every month or whatever, there's gonna be some serious trouble in a couple of months and, \[00:21:21\] **Steve McDougall:** Yeah, exactly. \[00:21:22\] **Phil S:** just by looking to graph. So if you can kind of let people know that sort of trend, that's really helpful. \[00:21:28\] **Steve McDougall:** Yeah. It's again, it's taking the metrics and spinning it into something that's a little bit more proactive for someone that they, they can act upon and not just have to search to try and find, cuz they may not think of it. \[00:21:41\] **Phil S:** Yeah, that's cool. On the topic of API scores, it's something that we, we always talked about doing with spectral, and I don't know what they're up to these days. I don't work there now, but the the idea of an API score we were going for we were working on like looking at open API and then judging your API on that and . Saying like um, oh \[00:22:00\] this is inconsistent. This breaks these rules. It's kind of the opposite of Lin. Instead of saying, here are problems, it just kind of says you don't have any problems or you're an a plus plus a api. And it, it really makes me want to get back to it. Cause if you can have a design score and then like a performance score, that would be super cool. \[00:22:19\] **Steve McDougall:** I'm actually, I was working on that not last week, the week before. I spent literally the whole week going through every page of the open API specification, converting that to data objects so that I can pass an open API spec, pass it in, and start to do calculations. you know, data that I actually know and understand. My eyes were, you know, very kind of bleeding by the end of the week once I'd finished building all of it. But yeah, it was um, it's coming. We, we are currently doing, we've got a big list of metrics, which we're gonna check against once we've got that open API spec. And what we're gonna be doing is \[00:23:00\] we're actually gonna release that part as a free. So it'll be under, like, once it's built, it'll be like insights treble.com and you can either pass the URL to your open API spec, paste it in, or upload the file. And what we'll do is we'll build, pass and give you a score and give you, kind of hear other problems. Here is where you're doing well, things like, are you using restful patterns? Yes. Authentication percentage and all those sorts of things. \[00:23:30\] **Phil S:** That's \[00:23:31\] **Steve McDougall:** that will be a, a free tool for people to just use \[00:23:34\] **Phil S:** Do you guys have any sort of do you folks have any sort of like green tech ideas in your, in your roadmap? Because I would love to talk to some of you folks about that, if not. \[00:23:46\] **Steve McDougall:** it. It's something that's on my mind. I'm very much aware that a platform like ours, which deals with analytics, can quite easily increase carbon usage by. You know, a new client \[00:24:00\] coming on board who's got a very busy api, suddenly our carbon footprint goes from X to X plus seven. And definitely looking at what the options are for that, and we are currently having discussions internally about how our application works, how it performs, and what we can do to kind of shrink that a little. \[00:24:18\] **Phil S:** that's really interesting. That's, that's less, less what I meant, but really interesting or the same. What I mean is Knowing and doing something about the carbon footprint of your organization and your application wherever you may work, is a good thing. That's kind of making tech green. But kind of helping other people make their tech green is something that, that treble or any analytics software could really help with. Like, I've toyed around with the idea for ages of coming up with some sort of tool, some sort of proxy that would just look at all the traffic coming through and being like, you could have put a cash header on that. It's been exactly the. 99% of the time, what are you doing? And there's a bunch of other like tweaks and suggestions you could make to people. Like if, if someone's making \[00:25:00\] the same request over and over again, you could even just say like, you know, that could be a head request or, or, there's a lot of different stuff. And I think that that could be really powerful for an analytics company \[00:25:10\] **Steve McDougall:** I mean, we have those sorts of that information already, so, so as it'll be something, you know, I, I'd speak to the designer and say, okay, can you design something cool so that on a project or on an endpoint, I can add a badge and say, you know, you've got a green bonus, which boosts your API score. \[00:25:30\] **Phil S:** Yeah, \[00:25:31\] **Steve McDougall:** That, that sort of thing to say, you know, you are actually using caching, you're limiting your response size. You, you know, you, you're following a practice, which is perhaps more green than I'll just make another API request. \[00:25:46\] **Phil S:** Exactly. Yeah. That's something that I think we're gonna talk about a lot more as a few guests that I'm lining up to, to talk about more green tech stuff in depth. I know that you've kind of looked into it a bit and, and you suffer the fire hose. That is my Twitter profile, so you see me banging \[00:25:59\] **Steve McDougall:** \[00:26:00\] Yep. \[00:26:00\] **Phil S:** there. But , God knows what's coming up on there next, but yeah I think the, the green tech angle is something obviously I'm super passionate about, and I think there's. There's a lot more people starting to notice that we can, we can really do something about this cuz I, I forgot all the numbers, but like the internet is the 2%, 4% of global emissions. Like it's, it is growing. It's the same as flying and getting worse. And of that like 85% is API traffic. And so if we can like put a little dent in that, just by helping people realize they should, they could have cashed something where they weren't thinking about it before. That seems pretty helpful, especially if you can show a graph going down. You can even have people selling carbon credits off the back of it or something. Or at least kind of that, at least their company can pat themselves on the back for doing it. But yeah, \[00:26:42\] **Steve McDougall:** Yeah, exactly. \[00:26:43\] **Phil S:** emissions go down is a good thing. Cool. \[00:26:46\] **Steve McDougall:** I've found is that a lot of developers only understand certain levels of caching. So a lot of the questions I get asked all the time, cuz I do lots of tutorials and live streams and that sort of stuff, is, how can I properly cash? You know, \[00:27:00\] how can I enable cashing on my API or on my application? It's like, well, it depends on what you, what your application does. You know, you need to build out a cash strategy to understand that and understand. Kind of some of the business logic to know what you can cash and for how long, but it seems that there's a big lack of, well, a big gap of knowledge just missing in, in some areas where people don't understand how to cash, \[00:27:26\] **Phil S:** Yeah. \[00:27:26\] **Steve McDougall:** which again, would really help on that green front, right. \[00:27:30\] **Phil S:** Absolutely. I feel guilty cause I've not really talked about it enough. Myself on the blog, we've done a, a blog post about how clients can implement caching with, with client middlewares on their end. But like, we've not really talked about how to do it particularly well for the backend and oth other people have. But Yeah, I feel like I should be doing more there. Ah, I was gonna say something and I totally forgot. God damn it. I'm just looking out the window and I'm seeing an invasive cherry laurel that I want to cut down, but it's not on my property and it's distracting me. . \[00:27:56\] **Phil S:** Something, something green tech. Yeah, that was it. It's the \[00:28:00\] green. Making your APIs like greener is a really interesting topic. I mean, when it comes to the front end, if you see all of the Green Software Foundation and all the Green Tech people, they're for the front end. They're basically giving you all the same advice they would give you to make your website faster. There's nothing particularly different in there, like for the front end. It's literally. You know lazy load images when you're scrolling and try and like, make your requests smaller and more cashable and blah, blah, blah. And it's, you know, not squish 'em all into one request because that just means that ev if you change anything, you have to blow the whole lot away and different pages have more information than they need, blah, blah, blah. But it's all the same advice as making it faster, making it greener, kind of the same thing. And then when it comes to API design, Development. It always seems like making the A API faster just means spinning up more instances all around the world and then like doing, doing things over and over and over and over and over and over and over again. But from like nearer and and so yeah, like \[00:29:00\] anyone who can help people realize that that's not the case is gonna get a big thumbs up from me. \[00:29:04\] **Steve McDougall:** Yeah, we're currently investigating what we can do with CloudFlare work is that in a minute. So you know, part of the, part of the process with Treble is that you send that payload to us. Now, if we can move that us closer to you, that means it's gonna be a quicker response time. It's gonna be less network traffic. Less carbon footprint by sending it closer instead of, you know, where our servers are located, somewhere in America or somewhere else. You know, if I'm doing that from AUR Wales where I am, then you know, it's just quite, quite a distance. I've gotta send that payload, which, yeah, I could reduce that traffic by using Edge locations a little bit more. So we are looking into all of those different ways that we can really reduce. The, the latency and that payload and that what we can do with it. \[00:29:58\] **Phil S:** Brilliant. I look forward to finding \[00:30:00\] out more about that. I was gonna ask, cuz we've, we've asked you a whole bunch of stuff about treble, but I know that you've done lots of other a p I stuff. So you were working on you invited me to a livestream that I think I didn't make it to, but you've also done like a video series on building APIs with Laravel. Can you tell us, tell us more about that. \[00:30:17\] **Steve McDougall:** Yeah, so I'm actually working on another video series at the moment called API Masterclass and Lara Valve. So going from, I've never built not, I've never built an API before, but I've built an api. But let's be honest, I can't really call it an api. I can call it adjacent interface to my database. I'm gonna go from where that would be to building something that is, Better going, you know, going through API design, what's important, actually understanding the process behind why you have an api, what the API's for cashing, you know, building up effective queries around it and that sort of stuff. \[00:31:00\] That is underway. I'm tempted to lead that into a book because, but you know, we, we've spoken a few times, Phil. Me, me releasing an API book as an updated APIs you won't hate specifically for Laal to stop the questions for you. \[00:31:17\] **Phil S:** Not allowed. There can only be one book about APIs. No one else can do any API books. \[00:31:22\] **Steve McDougall:** Well, my, my book's gonna be called APIs Phil won't hate, so it's okay. \[00:31:27\] **Phil S:** Nice. I was joking the other day that the Build APIs you won't hate was the dumbest name for a book, cuz it's not about whether you like it or not, it's about whether your clients do \[00:31:35\] **Steve McDougall:** Yeah. \[00:31:36\] **Phil S:** So I'm like, there's a mistake in my book name. Yeah, that's brilliant. I mean, the, the book that I did was like how to build Rest ish APIs with Laravel \[00:31:44\] **Steve McDougall:** pretty short. Yeah, I've got it here, I think. Yeah. \[00:31:48\] **Phil S:** you had a whole box of him at one point. \[00:31:50\] **Steve McDougall:** Yeah. No, I, I've finally managed to give all of those out. \[00:31:52\] **Phil S:** Fantastic. You just giving 'em away on the street corner. You can put mugs on it. It's \[00:31:57\] **Steve McDougall:** Yep. \[00:31:57\] **Phil S:** it as a door stop. Awesome. But yeah, \[00:32:00\] no, I like the idea of there being a specific for Laravee up-to-date book because yeah, it, it's really hard to like any. Writing about APIs should be polyglot. It should be completely agnostic to the, to the code in question. But people don't just want ethereal concepts wafted in their face for 300 pages. So you need to \[00:32:20\] **Steve McDougall:** cons. Yeah. They want code examples and things they can relate to. \[00:32:24\] **Phil S:** Exactly. And not just, not just shoving it into a a page like, cuz if I'm, I, I remember like when I was like 11 years old, just like trying to type code that was in a book and, and that is dumb. People don't want that anymore. They want the code in GitHub. And so you have to make it a real thing. And then, you know, Laravel has had four or five, probably more. Are they on La LaVale? 10. \[00:32:46\] **Steve McDougall:** Yeah. LaVale 10 currently. \[00:32:47\] **Phil S:** Cool. My book was done with like 4.2, so, um, Yeah, Taylor's been a, Taylor's been a busy, busy boy, but yeah, trying to keep that up to date is hard work. And so, yeah, I think more people should make kind of specific, this is how you do it in this, \[00:33:00\] this sort of language. That sounds good. So the, the video series, the masterclass, you are gonna like build a real API that does actual stuff, or is it, is it a Twitter clone? , what you doing? \[00:33:12\] **Steve McDougall:** No, I'm, I'm gonna I'm gonna build an actual api. And it's gonna be, I'm gonna have, each chapter will be a new branch. So here's where you start. Go to that next branch, follow along, go to the next branch, follow along so you can really understand the journey that you're going on. And you know, by doing it that way, people following along can choose when they want to get off. Like, okay, I'm happy with how far I've gone. I'm gonna stick with this for. , I know I've got four more branches to go, or four more chapters to go. But for now I'm happy. I've learnt a few lessons and then they can come back and revisit it to kind of go those further steps. Cause there's a bit of a joke in Bel community about me being opinionated with my code. And it's not really, it's, it's not a \[00:34:00\] joke. It's true. I'm probably one of the most opinionated developers you'll find, but it's for a. You know, I, I, I like things a certain way. Probably part of my OCDs just rubbing off on my coding standards, so, you \[00:34:16\] **Phil S:** the idea of somebody in the Laravel community saying that somebody else is opinionated, they're like opinionated by default. So anybody, anybody that's sticking out above that has to be really up there. Fantastic though. Opinions do help make things quicker, as long as you can get along with those opinions. Like a lot of the frameworks that try to do, anyone could do anything in any way that they want. Then everyone just goes, how the hell do I do anything? So \[00:34:43\] **Steve McDougall:** yeah, exactly. \[00:34:45\] **Phil S:** frameworks, Julie, grease the wheels on getting things done sometimes, but \[00:34:48\] **Steve McDougall:** Yeah. It's helped us learn a few, learn a few lessons over the past in the industry, you know, by someone formulating an opinion and people adopting that opinion. It means that that's become the new standard. \[00:35:00\] And so that's a little bit how PSRs came about. in the PHP world. \[00:35:04\] **Phil S:** For sure. I mean, I've been using Laravel for the Protect Earth api and that's, that's been a really nice experience. We've got a bunch of live wire, well, no, we've got Nova, but we might be switching to Live wire or something. But the API is really slick and there's an an amazing community of people as well. We've got like five, six volunteers who just throw code at us and, and enable us to do really useful stuff. And I was saying we've launched the we've launched the weather track. Stuff, which is like finding out what the temperature, humidity, and rainfall was getting that from an api and then we're talking to another API and finding out what the soil type is in a certain area. And then we can find out like when, when we do beat up surveys to find out which ones died, we can, we can kind of go, well, we probably shouldn't plant in a, in a place like that. Or if we are going to, we should at least make sure it's more watered or whatever. And so, yeah, like right now, Laravel is literally saving the world by improving forest. So \[00:35:58\] **Steve McDougall:** Yep. \[00:35:59\] **Phil S:** So it's, it's \[00:36:00\] been really been really fun to be back in that, in that kind of in the community, but mostly just using it and, and just it working and me not feeling like I have to get involved with the framework or like bickering cuz it just works really well and I can just get on with it. That's quite nice. \[00:36:16\] **Steve McDougall:** It's definitely, I mean, yeah, I've lost my hair, but you know, that's not Lara's fault. That's, that's me having the amount of children I do. But yeah, if your children went here, I'd definitely have long, luscious looks like you, Phil. \[00:36:28\] **Phil S:** Need to get a cut. \[00:36:31\] **Steve McDougall:** Lara's definitely made development easier, especially in the API space. \[00:36:34\] **Track 1:** Yeah, I'm gonna start blaming Laravel for my, uh, l l lack of coverage here too. I think even though I've been a JavaScript developer for longer than I can think about at this point. \[00:36:43\] **Phil S:** That is \[00:36:44\] **Steve McDougall:** that's just the weight of the NPN modules. Pulling your hair out of your \[00:36:47\] **Track 1:** That's it. Totally. Yeah. Dragging it away from me. One, one thread at a time here. So Steve, what are the things that you're working on now that are coming next? Right? Like, I, I know you have maybe some work things. You have, uh, \[00:37:00\] personal projects that may be coming. Um, what are you thinking about right? \[00:37:03\] **Steve McDougall:** Right now. Ooh, that's a very good question. At the minute I'm, I'm building out SDK tools. I released a new open source package a few days ago called sda SD K tools. I'm trying to, my second attempt at enabling people to build better SDKs in, in P h P. Um, I'm trying to adopt more. PSRs, lots more auto discovery, but still enabling them to keep uh, kinda that friendly API and developer experience I might be used to in something like LaVale, for example. That's my current project, if going well so far. I released it, didn't publish it really, I, I tweeted about it as it's been starred a few times. I've used it twice already. and it, it's, it's definitely helped. It's, you know, that's my current project \[00:38:00\] cuz I like integrating with APIs as well as building APIs. So it's always a bit of a where do I want to focus? \[00:38:07\] **Track 1:** Of course, yeah. It's probably a good sign that you're your own user as well. Uh, you feel some of the pains that you're trying to solve there. \[00:38:14\] **Steve McDougall:** Yeah. I mean that's one of the things that I say to anybody, building an API is, You know, as you're building your api, make sure you're integrating with it so you can feel those pain points that your user will do, because that's gonna point to where you can improve upon. \[00:38:29\] **Track 1:** Yeah, \[00:38:30\] **Steve McDougall:** And it was actually what part of my talk that I gave at Lara EU around I think it was called building APIs or something like that. They wouldn't let me change the name to building APIs. Phil would like so \[00:38:41\] **Phil S:** Good. I'm glad. It's not just me. It's not just me. Freezing. \[00:38:44\] **Track 1:** I've, supported every, uh, every proposal I put in from now on is gonna have Phil's name tossed in there somewhere. Yeah. \[00:38:49\] **Steve McDougall:** Approved by Phil \[00:38:50\] **Track 1:** so Steve, um, appreciate you coming and joining us today. Uh, can you tell us where folks can find you online? \[00:38:56\] **Steve McDougall:** just Steve King, everywhere. \[00:38:59\] **Track 1:** it. \[00:39:00\] Cool. Yeah, and I'll, I'll throw, we have lots of links to you everywhere, uh, which I'll drop in the show notes here. Um, about trouble? Where's the best place to go to get started there? \[00:39:07\] **Steve McDougall:** trevor.com. T R E B L L e.com. \[00:39:10\] **Track 1:** Perfect. Well, \[00:39:12\] **Steve McDougall:** spelling, but we're, yeah, we are. There we're uh, Trevor API on Twitter. We're always tweeting, always \[00:39:20\] **Phil S:** Awesome. Well, thank you so much for telling us about all of those cool things that travel are working on and you are working on. I want to keep an eye on them. And on the subject of s D K stuff, I've, I'm writing the latest book, surviving Other People's APIs. I know I've mentioned it in the past. I am li I am honestly working on it very, very quickly now. I've done I've finished three chapters in three days, and so people should check that out because it's not just if you are a front end developer, but it's any API that talks to other APIs. So anyone that talks to APIs, which should be most of. This, this book should be right up your alley and it talks about SDK's a fair bit because half of the \[00:40:00\] stuff that you have to know in order to successfully interact with an API can be done by the sdk. So you don't have to, otherwise you just have to build an SDK for every API that you ever talk to every time, and that's probably going to be bad. So yeah. Brilliant. I look forward to your, your stuff about SDKs and I'll, I'll shove it in the book if it's out in time. \[00:40:22\] **Steve McDougall:** Yeah. Sounds good. \[00:40:24\] **Phil S:** Sweet. Alright, thank you very much. I'm gonna go and get back out into the woods. I've got invasive species to hit with an \[00:40:30\] **Steve McDougall:** Enjoy the trees. \[00:40:32\] **Track 1:** Right on. Thanks for joining, \[00:40:33\] **Steve McDougall:** Thanks very much for having me. Take care. \[00:40:36\] **Track 1:** Take care. ### Learnin' about webhooks, with Tom Haconen from Svix URL: https://apisyouwonthate.com/podcast/learning-about-webhooks-with-tom-haconen-from-svix/ Last updated: 2023-04-17T12:44:41.000Z ## Show notes - **Svix** \- webhooks as a service - [https://svix.com](https://svix.com/?ref=apisyouwonthate.com) - **Svix play** \- [https://www.svix.com/play/](https://www.svix.com/play/?ref=apisyouwonthate.com) - **Svix is hiring engineers** \- [https://www.svix.com/careers/](https://www.svix.com/careers/?ref=apisyouwonthate.com) - **ngrok** \- [https://ngrok.com/](https://ngrok.com/?ref=apisyouwonthate.com) - **Catch Tom Haconen** online - [tom@svix.com](mailto:tom@svix.com) [svix.com/slack](https://svix.com/slack?ref=apisyouwonthate.com) --- #### Transcript \[00:00:00\] **mike:** Hello and welcome to APIs you won't hate. My name is Mike Fulco, your host . I'm here today chatting with a new friend of mine. And we're gonna dive into something that I think is an interesting topic for lots of devs of various flavors. And it's something that comes up for me I think fairly often when I'm talking to developers who are sort of like thrust out into the wild for the first time and have gone past. I don't know the 100 or 200 level tutorial experience of building things. It's a topic that it comes in handy if you're building anything that uses dynamic information that I'm sure you'll run into from time to time, especially if you're building APIs with information that has any sort of real time component to it. And of course that means the elephant in the room today is webhooks. And so we're here to chat about webhooks. I'm talking today to Tom McConn from S spx. Tom, how are you doing today? \[00:00:45\] **tom:** Yeah, I'm great. Thank you. How about yourself? \[00:00:47\] **Track 1:** I am doing really well. Thanks. Yeah. Thanks for taking the time to talk to me. I am gosh, I would say I'm, I'm the dangerous kind of web hook user in that I've consumed them many, many, many times with reckless disregard for my own health and wellbeing. And I have lots and lots \[00:01:00\] of feelings on using web hooks. But to be honest with you, I've never built the. Event side of web hooks, the side that emits web hooks. So I've got lots of questions for you there, but maybe let's take a step back first, and let's start with this. Can you tell me what Web hooks are? Give me like the, the 100 level description of what a web hook is and why someone might be interested in them. \[00:01:18\] **tom:** Yeah. You know, th this is a question I'm always iterating on, so if someone from the audience has a better description, I'm willing to listen. I, the way I describe it is kinda like a reverse api. So if an API is you making a request to a server, you know this a web hook is the server making a notification to you. So kinda like a server to server po push notification or like any event emitting. Yeah, I give you few versions. I hope choose the best one. \[00:01:42\] **Track 1:** Yeah, sure. I think, I think that's fair. So what are some examples, maybe common examples where people might have run into web hooks or use them whether they realized it or not? \[00:01:50\] **tom:** Oh, so literally everywhere. You know, so like if you've ever used a product that uses Stripe you've used web hooks without realizing it, because that's usually how Stripe notifies people that \[00:02:00\] someone has paid or like an or someone is delinquent or anything like that. You know, when you receive an email on Gmail, it submits a web hook to your c r m or anything like that to let it know that someone's pay. So that sorry that an email was received. I mean, literally everything I can give you every interaction that you, that happens in synchronously. \[00:02:16\] **Track 1:** Yeah, it's true. They are all over the place. And I, I think Tom, actually, you just set a new record for how early in the show. I need to give my disclaimer that I used to work for Stripe and I no longer work for Stripe. So we're, we're not associated with Stripe. Although obviously no negative feelings there. I just feel like it's one of those good transparency things that us people in the world of Derell need to do a little more often. But yeah, certainly in, in my role working at Stripe and as an end user of Stripe, I've, I've used their WebBook stuff quite a bit, so, From my perspective and sort of from my observations of the way devs learn and the way devs get things done, it seems like webhooks may be a topic that's kind of intimidating for a lot of developers. Can you talk a little bit about why using webhooks maybe feels hard, or is hard, or is challenging? \[00:02:56\] **tom:** Yeah. You know, so the people we interact with a lot are the people \[00:03:00\] sending, but let's start with actually from the people consuming, which I think is where you're going with. So I think the first thing is, You know, it's kinda like it's something that's incoming, right? And that means you need to use a tool like En Grok or we have a tool as well, like six play in order to kinda like punch through the firewall and actually get it to your local development server. It's not, it's not one of those things that you can just make a call from your computer and everything just works. It kind of requires this initial setup. The second thing is, you know, we spend, you know, all of us are like day in, day out building like HDP services that kind of like follow this pattern of like getting a. doing some magic. making a response. And that is like a, you know, a paradigm that we're really used to. But like, kinda like with webhooks, we don't even know, like, when are they coming, why are they coming? You know, like we, we have this kinda like lack of visibility of like the trigger that makes it start. And then the last thing, which I think is the most, the most this is another one as well, you know, like when you, you need to validate the signature of web hooks. And a lot of frameworks actually go in the way of that. They kinda like pause and re serialize. The, the payload, which \[00:04:00\] can, you know, for, for encryption or for like anything with cryptography can mean havoc. And then the last part is kinda like event-driven architectures which are just more difficult. It's super hard to reason about you know, knowing when an event actually triggers. I mean, I think they're probably more, but I thought, I think those are the main ones. \[00:04:17\] **Track 1:** Yeah, that covers it fairly well to revisit some of those. I think the validating web hook is an interesting problem and one of those things that like, certainly the first few times I was an end user of web hooks was something that I hadn't even considered. But from the developer's perspective, the way you receive an end hook is you open up an http endpoint, so some URL that some service can send a call to, and you. Accepting information through that endpoint. So they send, you know, to the specific URL that you set up a packet of information. And if you just trust that blindly, what you're not doing is making sure that the right person is sending you that information. So again, it's a public url. Anyone can hit it. Anyone can send any information there. And by coming up with some way to use encryption to validate who is sending you that information. You can reject WebBook calls that are coming from bad actors \[00:05:00\] or crawlers or, you know, people who've just stumbled upon your API endpoint. The event-driven architecture one is really interesting I think as well, because especially in a world where people are Expecting their applications to feel a lot more real time and have this like native mobile app experience as the, the default expectation for just about anything on the web. We're very different from, you know, 10 years ago when you were expected to go back and refresh a page to see changes. That's just not the world we operate in anymore. And driving your UIs updates through events that come in from webhooks on whatever level, however often that may be, can create that dynamic effect quite a bit more. So with all that being said, I think that's some really good background on web hooks, and I think that sets the stage pretty well for, for you, Tom, and sort of for where you are right now. So tell me about fix and tell me about what you're working on. \[00:05:45\] **tom:** Yeah. So we do web hook sending as a service. So essentially we help companies send webhooks. You know, they're kinda like a lot of, you know, we talked about the challenges of receiving webhooks. But there are a lot of challenges with sending them. And even in a way, this challenge of receiving them are also part of the challenges of sending them \[00:06:00\] because you need to understand them well. You need to sign correctly. And all of those things that we just covered, you need to do it on the sender side as well. So we really just turn all of that into a simple API call. Yeah, I think that's that. \[00:06:11\] **Track 1:** Yeah, sure. Yeah. So how, how long ago did you start in on this journey? When did fix. \[00:06:16\] **tom:** Wow. Actually more than two years. I was gonna say almost two years, but more than two years at this point. Yeah. So a while back, \[00:06:22\] **Track 1:** Sure. And so I guess tell me about that. How did you identify the need for, for the, the service to exist? \[00:06:27\] **tom:** yeah, so I had a different you know, project side, project company, whatever you wanna call it at that point that I actually still running on the side. You know, people who kept on asking us for web hooks, like all the time. And we know that A, we don't have the upfront, like all the upfront costs, like all the effort to build it, we're like, we didn't wanna spend it. And then like, the ongoing effort of like maintaining it, that sounded like hell as well. So we kinda like skipped that. A and then kind of like the last part of it, which is, you know, we built this like beautiful, beautiful p i and we spent so much time on it, we're not gonna have a terrible web hook experience, right? So kinda like \[00:07:00\] multiply the first two estimates by like 3 0 5\. And kinda like all of that just meant, meant that we. actually built web hooks. And I wish I could say at this point, like, you know, I was smart enough to realize that this is a problem that n needs to be addressed, but I wasn't. It took a few months later when someone asked a silly question about web hooks in a slack I'm a member of. And I kind of like, well, this person is smart. That doesn't make sense. I kind like dug deeper and I realized she actually didn't wanna understand anything, anything of what I was saying. All she wanted was to send WebBook and not have to think about it. And I'm kinda like, huh. If I build it for you, will you pay me? She's just like, yeah, and then someone else chimed in. I'm like, huh, maybe there is something. I remember I had a problem as well, so like decided to give it a go and everything kind of like went on from there. \[00:07:40\] **Track 1:** Sure. Yeah. I mean, maybe that's, that conversation is exactly the reason that I started this podcast by saying I've never bothered to create WebBook. I've only ever consumed them. The, the thought of having to do that honestly is, is a fairly intimidating thing because of all the architectural, this and that, that goes into it. One of the things that happens, I think with the event driven architecture that is really tricky is like these web hooks are asynchronous communications that \[00:08:00\] you. The emitter send out when something happens. But as the consumer, you need to think about, well, what happens if I miss that? You know, if someone sends a web hook and my service is down or off, or glitches out, or whatever the case may be, has that thing gone into the ether? Like that's, that's a very intimidating. Perspective to have, you know, something like that going wrong, especially when you're dealing with like finances FinTech, things like that missing a deposit or a withdrawal can like fully wrangle the books incorrectly if you want. So there's a lot to think about there. So your first customers then were, were developers, were looking to send out an emit, emit webhooks. What does that look like from their perspective? Are they sending you something like an open API definition? Are they calling a service? Are they. Self-hosting something that's fixed stands up. What, what is the infrastructure? \[00:08:41\] **tom:** Yeah, so it's kind of like all of the above. We really believe in meeting like customers where they want to be. You know, we have very strong opinions on how things should be done, but that doesn't mean, you know, we know we are not always right. So it's better to kind of like help people. So I mean, like what the, the main gateway to our service though is either R SDKs or H T HTTP requests. And \[00:09:00\] people, you know, like every six integration is kind of like two and a half steps. So the first one is creating a destin. So kind of like, you know, whenever a user signs up to your service, you create what we call a consumer application, and then you send web hooks to that location, and then the second API call is just to send that, you know, that web hook to that location kind of saying like, Hey, an invoice was paid. The, these are the details, just go and send it on. As you may have noticed, we actually didn't, I didn't talk about adding the actual HTP endpoint yet. For that step, we offer two things. Either they use the API to do it and they can do it, you know, like from the front end, from the backend, whatever they wanna do. Or we have like an eye frame that they can embed in our ui, in their ui, sorry in order to let their customers kind, like, control all of that to kind of have like a very, a turnkey solution. Two and a half. Two and a half. API calls, \[00:09:48\] **Track 1:** Yeah. Okay. It sounds like it. I think I've actually used that iframe service through a clerk I've been using clerk for authentication for, for my new company. And it's really interesting to consume that, right? Like I get as a user \[00:10:00\] of clerk. There's an embedded interface from fix there. And I think this may be how I first came across Fix in the Wild. but essentially it's an interface that allows me to define URL endpoints that web hooks end up at whether that's sort of my local host environment or you know, pipe two and actual say production or staging U r L. And having worked formally in the past, For Stripe. I think it, it does a pretty good job of mocking some of the best parts of that too, where Stripe has all this tooling available via CLI, via ID integrations and their dashboard and all these other places where you can tell Stripe where you want them to pipe information to when events happen. And I think SPH does that really well in a way that I hadn't really considered as something that a product developer might want to use. So I guess this brings me back to then What's it like to plan for those? Like do you find yourself having to educate your customers? What they, what the ideal sort of user experience looks like for their consumers? \[00:10:51\] **tom:** Oh, a hundred percent all the time. But I think the nice thing is that they come to us already as experts and they realize that we've seen. a ton of integrations, \[00:11:00\] both ones that follow our best practices, ones that existed before us, and then they migrated to VX and just, you know, like market research that we do all the time. So education is, is less painful than it has to be because they really are coming, like willing and like, you know, willing to learn and willing to listen. But we do get every now and then we good get into like let's say healthy arguments about the right thing to do it. Yeah, every now and then, \[00:11:22\] **Track 1:** Yeah. Developers love a good scholarly debate. I don't doubt that \[00:11:25\] **tom:** Yeah, exactly. \[00:11:27\] **Track 1:** that most of your customers tend to have already had web hooks and they're looking to move to a less painful implementation, or they're creating a web hook experience for the first time? \[00:11:35\] **tom:** You know how to say, we see the whole gradient, so we see both. They had a mature implementation for a few years now, and they're like, they've had enough maintaining it and they wanna switch. They've had a terrible implementation that they kind of hacked together, cobbled together for like a few months and now they're looking to switch. And also they just wanna. With something. So really we see the whole gradient, hard to say, which is Yeah, \[00:11:54\] **Track 1:** sure. Yeah. I guess I can see the need for both. So, tell me a little bit about, I guess then the \[00:12:00\] experience of someone who's trying to onboard, onboard to space. What's the I guess the developer story look like? \[00:12:05\] **tom:** yeah. So like six customers. \[00:12:07\] **Track 1:** Yes. Yeah. \[00:12:08\] **tom:** You know, so, the way I look at APIs or like in products in general, you know, like think I'm not an apple fanboy by any means, but like think of like when you buy a new Apple device, like a new iPhone, for example. You get a box, you open it and you get a phone, and then you pick it up, you turn it on, you start using it, right? That is kind of like, that is the experience. You don't read a book before. You don't need any of those kind of things, and we really try to mimic that as much as we can in order to make sure that. You know, our customers don't get frustrated before they need to understand it. And, you know, they can read the docs afterwards. So the first thing when you sign up, you kind of see essentially like a one paragraph that explains what consumer application are, which I explained a moment ago actually. I mean, we, I think we went through the onboarding mentally a moment ago. So like the, you know, Yeah, they, we explained what consumer applications are and then we have a Kell command that they can just copy paste to the. And that's their first API call. It actually works. And \[00:13:00\] then they pro automatically progress to the second page where again, we explain what messages are. They copy paste that telco command and there you go. They send the first message and then kind of like we, we let give them a button to move to that embedded ui, or we have a button to move to the docs and kind of like read more about how to create that. \[00:13:16\] **Track 1:** Sure. And then they can grow from there. And so is your functionality through six, it's not just done via, via like plain CTP calls, right? I think you have quite a few client libraries. Can you talk a little bit about the languages that you. \[00:13:29\] **tom:** Oh, you now I need to remember all of them. So I \[00:13:31\] **Track 1:** there's a lot. \[00:13:32\] **tom:** Yeah. there's a lot. So like Python, we have a version for both Async Python and Sync, Python, JavaScript type script. We have one in Java, one for Colin. That is actually different because it uses like you know, async Colin rust. I think I mentioned that. C. I think that's it. Oh, Ruby. Yeah. a few of them. Yeah. \[00:13:51\] **Track 1:** There's a lot which to me to, to my ear, that sounds like it must be quite a bit of maintenance challenge as well. Right. Do you have a pretty large engineering team working on keeping these \[00:14:00\] libraries functioning and well cared for? \[00:14:01\] **tom:** Yes and no. So no mostly, no. We had like one, two, I think two of the libraries were contributed by the community and we employ a lot of tricks in order to like auto generate. A big chunk of them. But it's still a pain. It's still definitely a pain to like update things, even though a lot of it is altered. Yeah. \[00:14:19\] **Track 1:** Yeah. Yeah. I think this is something that API providers are thinking about more and more is you have quite a few, you know, flavors of consumer these days. And especially like, I'm the first to tell you I'm not a Python expert. I can look at some Python code and, and wrestle my mind through what's going on there. But it was only fairly recently that I, I came to understand that like Async Python was a challenge and was something that was sort of a different, you know, developer experience. So I, I think. , having devs on staff write engineers who, who work for you with you, who are able to produce libraries and all these different flavors is, is going to be an important thing. And definitely like having community support is always a good sign too when you're getting open source contributions because people want to be able to embed themselves in your, your product and your \[00:15:00\] ecosphere. I think that's really good sign. \[00:15:01\] **tom:** Yeah I \[00:15:02\] **Track 1:** yeah. Yeah. That's, that's really interesting. Are there, so what's the world look like for fix right now? Like what, what's the breadth of products that you offer? \[00:15:08\] **tom:** You know, so as I said, it's kind of like an onion. So the first thing that we do, we kinda like, we just make it possible or make it easy to send web hooks. And then we also have a tool that's called Slicks play which is both like a c l I, kind of like enro, like, but just very specific. I mean, if you want anything more robust, use Enro. But for, it's kind of like agro, but you know, like a quick testing thing with a ui. In order to make it easier for consumers, we. A wide range of open source libraries for verifying web hooks. So our customers, and by the way, not our customers as well, like anyone can just use them to sign and verify. And it's kinda like just much better than like trying to reinvent the wheel there. \[00:15:42\] **Track 1:** Sure. \[00:15:42\] **tom:** And then kinda like the Fixx product, you know, does like everything you would want and more. It's kinda like all the reach wise and like manual and automatic and like observability. And we, we also have something really cool that we call like a payload transformation. So we realize like a lot of our customers, what they do, they receive a WebBook, they do something to it, and then they \[00:16:00\] send it somewhere else. And we realize we might as well let them embed a bit of JavaScript, kinda like run it there and, and, you know, move it forward. \[00:16:07\] **Track 1:** Oh, that's really interesting. Yeah. So where does that run? \[00:16:10\] **tom:** On our, our environment. We use Dino for that. Yeah. it's kind of isolated like it's, it is on mini microservice. \[00:16:17\] **Track 1:** Sure. That's a really interesting idea. Then, then you get a little bit of that transformation that's sort of offloaded. Ha. Are you finding yourself wrestling with implementation of Edge services at all? Is that something that you need to think about as a, a provider? \[00:16:28\] **tom:** Not for us but a lot of customers, you know, and, and this is kinda like adjacent to what you asked, but like a lot of our customers use those services, which again, like going back to what we talked about with the transforming the payload and making it very e hard to verify. So we do actually end up, you know, we have actually, I mean we have integrations with Netlify and we're a cell, but we also have like tutorials on how to use how to verify WebBook for all of these environments. \[00:16:50\] **Track 1:** Right. Yeah, actually I saw that. I meant to ask. So I'm, I'm both a Netlify and Versace user for, for various purposes and various little projects I have. I know that they both provide, I forget the language. They use \[00:17:00\] add-ons or plug-ins or extensions, whatever their terminology is. But they, they make it pretty easy to add. Auxiliary services, right? Things that are sort of feature add-ons for your application. And I saw this fix comes up in both cases there. What does that do when, when I click the ads fix to net? \[00:17:15\] **tom:** Yeah. So Netlify is actually not as mature as our reell integration is fairly new. But Inify, it just helps you. I think, actually, you know what? I don't remember. I think it just, at this moment, it just like helps you with like the verification. I don't remember what is the extent is there, but with versa, it also creates an environment, fetches and authentication token puts it in the, you know, in the secrets and all of that. But just Netlify don't, don't have all of that, all of those APIs yet. So we haven't built it fully. \[00:17:41\] **Track 1:** Yeah. There's a surprising amount of manual steps to adding a new provider to, to your application. And usually the one that catches me on my first deploy of anything is I forget the environment variable that needs to go somewhere, whether it's to my hosting provider or to CI or, you know, GitHub actions, \[00:17:56\] **tom:** Yeah. And, and you know, it's kinda like all of those things are like, it's, you know, silly, \[00:18:00\] what I just described is like a very simple integration, right. But it's kinda like, well if you, you save people like a few annoying manual steps that are gonna like, bite them in the ass, like you said that is like a, you know, a total win for society, A total win for the development society, at least. \[00:18:14\] **Track 1:** Yeah, absolutely. I don't, I don't really have a phrase for this yet, and I wonder if it's something you've thought about, but I feel like day-to-day in the life of, of someone building anything, any engineer at any company, I feel like there's a moment where you reach, like your level of, oh man, this is one step too far for me. Like the mental tax I've paid to get there is too far. Going to take me another day to get this done, or I need to step away from this at some point. That, that's something that I've experienced certainly in, you know, like so right now I'm building a company and, and every day is learning something new and applying some sort of new, you know, integration of things together. But I see in a lot of the folks that I work with as sort of mentors and friends who work in the industry as well. And I don't hear developer experience companies talking about a lot, but I think that's the ultimate like problem to solve is if you're able to get rid of those little bits of friction from developers experiences, they sort of love your thing more for being easier and easier to use. \[00:18:58\] **tom:** Yeah. I mean, I, I couldn't \[00:19:00\] agree more. That's, that's why, you know, we don't. that that's why like this, the, the curl command that you copy already has an OR token that's temporary for the onboarding, already has all of those things. Because even just telling people, Hey, replace the OR token here kind of thing. It's like already way too much for someone that doesn't care about your product yet. Yeah. \[00:19:18\] **Track 1:** Yeah. If, if you really break it down, there can be quite a few steps to getting any of these things done. And you know, the ability to compound a couple of steps behind the button that says install on, you know, X provider is a really nice thing to be able to do. So we talked a little bit about what Fixx does currently. What are the things you're thinking about for the. \[00:19:33\] **tom:** Ooh. You know, so like one, one thing that we are very excited about, and this is, it's just like web hooks in general, right? I mean, it's not even just web hooks, you know, like the way, when you think about how, you know, like, kind of like the Unix philosophy of kinda like one thing, one tool that does it well. and kinda like all the tools are separated, you kinda like concatenate them together. I think we reached a point where the internet is almost kind of like the database is the wires kind of thing. And \[00:20:00\] we make an action and like we don't need to persist anything. Like a message is being sent there and they do something and a message is sent there and they do something. And kind of like all of this, like workflows and you know, like even see it when our product, we use off zero for authentication, strive for payment mail, gun for emails and people use us. God ask for web hooks and, you know, clerk for authentication again and max for video and kinda like you have all of those. Interactions and what we just wanna do is we wanna make those reliable, more widespread and exactly what you were referring to earlier. Kinda like removing, even like when the beginning of this chat, like removing those like annoying extra steps that people need to do in order to ingest those events, I think like gonna be a massive win. And we kinda like we all just marching in those directions, making it easier to send, more useful to receive, and easier to receive. \[00:20:46\] **Track 1:** Yeah, sure. I think as people who are on, on teams that build things, I think we all benefit from that too, right? Like it's, it's always nice to see when a company's goals align with the better need of the, the developer community too. You know, you're not putting artificial tiers \[00:21:00\] in place to, to make more money. You are literally making the process of making these implementations better for people. And I really like the idea of also releasing, you know, open source tools to help people do it, whether or not they're embedded in your, your. Stack. I think that's really one of the things that I like, especially about the, the world of sort of API developer community stuff too, is like a lot of the tooling we work on and a lot of the people we work with are all working with each other and sort of allied together to, to make our lives easier. Cuz ultimately, like we are all successful when our end users are successful, we're not successful because we've, you know, stomped on the competition necessarily. I think that's really one of the things I like about the world we get to live in, I suppose. \[00:21:35\] **tom:** Yeah. I mean like the developer. It's also like the open source mentality. Which is kind of like, Yeah. it's great. I agree. \[00:21:43\] **Track 1:** Yeah, yeah, definitely. For sure. So I'm, I'm a couple more questions, I guess, about fix, and I, I want to talk a little bit about sort of your perspective on leading a company as well, and especially in, in this sort of changing atmosphere we've got with economic changes and whatnot. but I, I, before we get there, \[00:22:00\] let's, let's do this. So I'm really interested in I guess who, who you think. Lemme take a step back. Our audience for APIs you won't hate are all API developers of some flavor. Probably they're all API consumers on some level, and many of them are probably writing APIs that require web hooks to be emitted at some point. What do you think is the point where someone needs to realize that, Hey, maybe I should be considering taking advantage of a service like this. Like, how does someone identify, oh, I should be looking at fix, I should be looking at, you know, whatever's out there in the. \[00:22:25\] **tom:** Yeah. So. You know this, it's like the eternal question of like build versus buy. And when I say buy, I don't even mean like pay money. I mean like using an existing product. Could be open source, could be free, you know, develop. There's just like so many. We don't have that much time on Earth in general as individuals. I know this became a bit morbid, sorry, but \[00:22:45\] **Track 1:** philosophical. I \[00:22:46\] **tom:** Yeah. But we, we don't have a lot of time on earth. We don't have a lot of time to finish those goals that we want. And, you know, especially like early stage startups, all we want is to make our customers happy. So when it came like. you know, the point for us, like, do we build \[00:23:00\] X or do we use off zero? We chose off zero, right? Because we, we don't care about those. We wanna just give our customers what they want. So I think like, if you are resource constrained, definitely like you need to look at a third party. And, you know, you know, you can always, like, as I said, like our libraries are open source. You can always switch us out. If we don't make you happy, it's our job to kind of like make sure that we keep constantly, make you happy. So I, I think the second part is, it's kind of like when you reach a certain. Scale where actually the maintenance is becoming annoying. Like if you look at some of our use case, like case studies on the website you see that, you know, like just bringing back, you know, you kinda like, you've spent all of this kind of time convincing these amazing employee, you know, like team members to join the company and work on. you know, payments or work on whatever it is that you're building, and then all of a sudden you are sending them off to like a small team that does infrastructure work. They're not gonna be happy about that. And they're gonna be, you know, you're gonna be like a lot of like movement there. Either they're gonna like leave the team or join a different team. So it's kinda like it becomes this like core infrastructure that no \[00:24:00\] one knows very well and no one cares about. So I think the moment you start realizing it is something you care. That's the time to professionalize it. Professionalizing it either means a dedicated, strong team that has buy-in to this or outsourcing it, that that's the way I look at those things. \[00:24:18\] **Track 1:** Sure. I appreciate that quite a bit. I both didn't expect this answer to be quite so philosophical, but also really appreciate your ability to, to turn that into a framework for recognizing you know, what the opportunity looks like and when to, to jump on top of it. I think that's actually pretty sagelike advice there. Yeah, Tom, I think that's really great. All right, so let's pivot a little bit. I'm interested in hearing what your what the process is, or, sorry, not the process, but what it's like leading in your, your company. Like what are you finding are the challenges of, of growing, of building? What are the things that, that keep you excited about working on a, a product like this? \[00:24:50\] **tom:** Yeah. Wow. That's a big question. I really. I enjoy almost everything. You know, if you look at our slack, I do a lot of support. I enjoy \[00:25:00\] talking to our customers, and I kind of like seeing what they want and what they're asking. I enjoy jumping on calls and helping customers like find out the, the best solution for them and what they need. Engineering, you know, I've been coding since I was nine. It's still like a soft place in my heart. Really just everything. What keeps me up at night. Is making sure that we build the right thing for our customers which is not very obvious, right? I mean, like, there's like one end of the spectrum, which is like just saying yes to whatever they ask for. And this, this is a terrible. Terrible way of building a product. But the other end, which is like say no all the time, also comes with its challenges. And it's kinda like you need to balance both of these. Kinda like knowing when to say yes, when to say no, and while keeping your customers happy, because you know, at the end of the day they're outsourcing this huge chunk of the infrastructure to us. So like how can we make sure that we're. Trusted partner and not this annoying, you know, brat that always says no, whatever. So it's kinda like, it's really like a tough balance to to keep, \[00:25:59\] **Track 1:** Sure, yeah. \[00:26:00\] That's the eternal struggle balancing needs and, and desires and especially demands with like the moment you've started serving customers, you can find out who your louder customers are and the ones with stronger opinions, and that's not always best for them or for the business. But you need to kind of distill that information into good decisions, you know, both for your business and theirs. It's a tricky thing. \[00:26:17\] **tom:** Yeah. I guess another part as well is kind of like going back and fixing things. So kinda like technical debt versus marching on, Which is especially relevant for company like hours because you know, the moment we. Onboard a customer. Like, I mean, Microsoft is not a customer, but when the moment we onboard Microsoft as a customer, we're Microsoft Scale. And then Amazon is a customer with Microsoft and Amazon Scale. So it's kinda like he keeps on like you know, it keeps on like jumping as like a, a, a menacing step function that we have to always be aware of. \[00:26:45\] **Track 1:** You make it sound like a horror movie. Yeah. . Without a doubt. So I, I wanna talk about, about your company's growth then too. Are you how big is your engineering team right now? \[00:26:53\] **tom:** So we are five at the moment, like five engineers. Yeah. . \[00:26:58\] **Track 1:** Yeah, that's that's a, a \[00:27:00\] solidly small team for the amount of output you've had. Especially like looking at this list of climate client libraries that you rattled off before, you know, to 10 or 15 different flavors of, of spic exist. That's really cool. Are you expanding, are you hiring for any \[00:27:12\] **tom:** Yeah, we are hiring. We, so we just recently raised around with Andre Howorth and we have like, you know, a, a big chunk of it is like hiring more engineers. You know, we we're lucky to have, you know, like a high output team. But I think we can even do even more. We just like more people helping out. Yeah, very excited about \[00:27:27\] **Track 1:** Yeah, sure. And if folks listening to the show are looking for a job, where's the place to go to see your \[00:27:34\] **tom:** Yeah. Vx.com/careers. But also if you think we should be hiring you and we don't have an opening email us. Anyway happy to chat. \[00:27:42\] **Track 1:** I like that. That's solid perspective. I will obviously make sure that there's a link in the show notes too to your careers page and, and just, you know, speak and some of the other things we've chatted about here. What's the best place to find you, Tom? If some of our, our listeners want to shout at you about web hooks or, you know, their thoughts on open source or whatever the case may be. \[00:27:57\] **tom:** Yeah, I mean, you can email me \[00:28:00\] directly@tomatvx.com or just we have a community slack at vx.com/slack. So just jump in, have a chat. Happy to \[00:28:06\] **Track 1:** Perfect. Yeah, will do. I'll, I'll include links to those as well. Hopefully we don't fill your spam coffers too \[00:28:12\] **tom:** Yeah. Already filled. Don't worry, .Yeah. \[00:28:16\] **Track 1:** Yeah, me too. I, I can definitely relate to that. Something about having an easy name and then@yourcompanyname.com makes it real easy for people to find you. I think. Yeah. Now that I've said that, I'm sure I made it worse for both of us. Tom, thanks so much for coming to hang out with me today. It's been really interesting talking to you. I am looking forward to you obviously getting the use fix more in the future, and I, I will likely end up be being in your inbox one way or the other, asking questions about web fix as things go on. We'd love to have you come back anytime if you're interested in talking a little bit more. But thanks for joining today. I really \[00:28:43\] **tom:** Yeah. Thank you. It was great. Great being here. \[00:28:45\] **Track 1:** Likewise. Take care, \[00:28:46\] **tom:** You too. ### Back and Better than Ever: APIs You Won't Hate (the newsletter) URL: https://apisyouwonthate.com/newsletter/back-and-better-than-ever-apis-you-wont-hate-monthly-newsletter/ Last updated: 2024-04-05T12:45:06.000Z APIs You Won't Hate is back with a new monthly newsletter. We'll share news, updates, and more on all things APIs. Along with some of our favorite articles and resources from the past month. If you're wondering why you're getting this email, you signed up for our newsletters at some point. It's been a while, so if you want to unsubscribe -- No problem, we still love you! You can always find the resources we make at [apisyouwonthate.com](https://apisyouwonthate.com/). Now, into the good stuff. My socials for the past month were a stream of AI hype and the JavaScript debate about signals. Whatever they are, at this point, I am afraid to ask. However, there has been so much excellent API content floating around. Don't worry if you missed it; that is what newsletters are for, keeping you updated. I hope you enjoy this week's issue. All the best, Alexander | Thanks so much to our [sponsor](https://apisyouwonthate.com/sponsor), **Zuplo**: [![Zuplo](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2023/03/zuplo.png)](https://zuplo.link/open-api-tools?ref=apisyouwonthate.com) [ Zuplo: API Management you actually want to use ](https://zuplo.link/open-api-tools?ref=apisyouwonthate.com) Fully-managed, serverless, programmable, edge API gateway that natively supports OpenAPI, with game-changing developer experience. [Start free](https://zuplo.link/open-api-tools?ref=apisyouwonthate.com) | | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ## News from the API dev world ### [JSON Schema: Another Breaking Change](https://json-schema.org/blog/posts/the-last-breaking-change?ref=apisyouwonthate.com) The well-loved JSON Schema has introduced a breaking change to their next release as they move towards a stable specification. Sometimes you have to break things to improve them, and that's ok. ### [Build as a monolith and deploy as microservices](https://opensource.googleblog.com/2023/03/introducing-service-weaver-framework-for-writing-distributed-applications.html?ref=apisyouwonthate.com) Google released a new open-source framework for building distributed applications called Service Weaver. The cool part is you can make it as a monolith and then deploy it as microservices. Service Weaver also helps with service-to-service communication complexities so you can focus on creating business logic, not networking. We can save the conversation about whether microservices are good for another time 😉. ### [Design and Build Great Web APIs](https://pragprog.com/titles/maapis/design-and-build-great-web-apis/?ref=apisyouwonthate.com) I would be amiss if I didn't include this book in this week's newsletter. I have been reading this book, and it is fantastic. It is an excellent resource for anyone looking to design and build APIs. Mike Amundsen's books are pure gold; he knows his stuff. ### [OpenAPI Tooling Registry](https://www.openapis.org/blog/2023/03/08/tools-that-support-openapi-specification?ref=apisyouwonthate.com) The OpenAPI initiative has put together a registry of tools that support OpenAPI. This registry covers everything from parsers to SDKs, gateways and more. ## APIs You Won't Hate The latest content from the team at API's You Won't Hate ### 📗 [Surviving Other People's Web APIs](https://apisyouwonthate.com/books/surviving-other-peoples-apis) The latest APIs You Won't Hate book is coming along nicely! After a few years in hiatus, the book is getting an update – including 5 completely rewritten chapters and a new section on async APIs by [Mike](https://mikebifulco.com/?ref=apisyouwonthate.com). Preorders now help support the continued development of the book and everything else we do on the site. ### 📐 [Securing APIs with the Spectral OWASP Ruleset](https://apisyouwonthate.com/blog/securing-apis-with-spectral-owasp-ruleset) Seriously – how does [Phil](https://twitter.com/philsturgeon?ref=apisyouwonthate.com) find time to write with all the tree planting he does? Learn how to find security flaws in your API by enforcing style guides on your docs or running Jest tests. ### 🎙 ️[A fresh episode of the APIs You Won't Hate Podcast](https://share.transistor.fm/s/494ca018?ref=apisyouwonthate.com) In this episode, Mike Bifulco chats with Anton Zagrebelny from Stigg. Mike and Anton cover pricing strategy for SaaS and pricing as infrastructure with Stigg. --- Thanks so much for subscribing - we're lucky to have such a strong and supportive community of API devs. If you found this helpful, we'd love it if you shared the newsletter with a friend. It helps out a great deal. Thank you! ### Turn HTTP Traffic into OpenAPI with Optic URL: https://apisyouwonthate.com/blog/turn-http-traffic-into-openapi-with-optic/ Last updated: 2023-12-28T11:20:34.000Z *This article is sponsored by* [*Optic*](https://www.useoptic.com/?ref=apisyouwonthate.com)*, in that they're paying for our time to review the tool, but they're not paying for what we write. If it's rubbish we'll let you know.* Documenting an API with OpenAPI can take a long time when you're starting from scratch. Some API developers use the API Design-first strategy to make their OpenAPI first, then they use that to easily make mocks and docs. API developers who missed that step are now stuck playing catch up, for their OpenAPI-based documentation/dev-portals. Some tool vendors are trying to throw AI at it, but there's a better way: capturing HTTP traffic. I wrote about [creating OpenAPI from HTTP traffic with Akita](https://apisyouwonthate.com/blog/creating-openapi-from-http-traffic), but a new tool has popped up that's making things even easier: Optic CLI. I used this tool for a client of [Green Turtle](https://www.greenturtle.io/?ref=apisyouwonthate.com), to create an OpenAPI of their legacy black box API that nobody understood, then used that OpenAPI to spec out the work for their new API. Super handy, and since then they've made some improvements. Optic wrote a simple *Document your API in 10 minutes (Step-by-step guide)* guide which can get you a long way, but to slightly repeat it: ```bash npm install -g @useoptic/optic ``` Yeah sorry it's NPM, but that's massively popular and a lot of the best [OpenAPI Tools](https://openapi.tools/?ref=apisyouwonthate.com) are built in JavaScript so it makes sense. In this article we're going to learn how the Optic tool works, and use it to build an OpenAPI document for the Mastodon API. You can change the example commands as we go to point to your API, and by the end you'll have a functioning OpenAPI document for your API without having to do any real work at all. ## Making OpenAPI for any API If you just want to practice doing this, you can pick any API, so I'm picking the Mastodon API because it's "new" and interesting, and the Twitter API costs a few bajillion dollars to use now because neoliberal tech bros ruin everything. ### Create an empty OpenAPI document Optic does not generate OpenAPI from scratch, it only patches existing description documents, which means you need to make one to start it off. This seems a bit confusing at first, but it's a huge benefit, because you can run through this process over and over and over, and it will keep improving your OpenAPI document with all the new details as it learns more about the API. The `oas new` command will write an empty OpenAPI document to disk. ```bash optic oas new openapi.yaml ``` ### Setup TLS (Optional) Capturing traffic over `https://` can be tricky, because it's all encrypted. This is good (you don't want every proxy reading everything) but bad if you literally need your API is running on HTTPS and you literally a proxy like Optic to read everything. If the API you want to capture is running on `http://` because maybe you're recording traffic via localhost, docker, or your API is just super insecure, then you can skip this section and move onto capturing that plan text traffic. If the API you want to capture is running on `https://` then you'll need to set up a man-in-the-middle attack. Other tools need a lot of setup to do this, and you have to teach those tools where the other tools are running, but Optic can wedge itself into the users system settings for network proxies. This means you can snoop on any API, even the production API from your computer. You can do it for literally anything, which is how we're going to record the Mastodon API, just by running a command. Once the shock of the convenience wears off, you might be thinking "hang on a minute, I'm not sure I want this tool seeing all my traffic..." but it's only going to be running when you run the command, and it's not reporting anything to any servers anywhere. You'll see how it works as we go. To start off with, setup the TLS logic. ``` sudo optic oas setup-tls ``` If you have permissions to run this command you should see a prompt asking you to click on one of your machines existing users. If not, add a user, and its your usual computer login username and password. ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2023/03/optic-perms.png) On macOS the CLI spawns an account picker interface when you run this command. Once that's going through successfully you should see something like this. > Trusting Cert. This may take a few seconds. If you see a Keychain prompt appear, enter your password > Certificate trusted. 'oas capture' can now see traffic sent to https hosts when Optic is running Let's do it. ### Capture traffic from the API Optic can capture traffic from a service you are running locally, or from staging/production. Point Optic at the hostname you want to capture traffic from: ``` $ optic oas capture openapi.yaml https://mastodon.green/api/ » Proxy running on https://localhost:8000. System proxy updated help Press [ Enter ] to finish capturing requests ⠼ 0 requests captured ``` At this point my computer knows to shove all traffic through localhost:8000, which you can confirm by looking at Settings > Network > Proxies. So long as your browser is detecting system proxy settings instead of using its own configuration then you should be capturing requests. To test its working, head over to `https://mastodon.green/philsturgeon` and start clicking around on every button on the website. Fill out every form. Edit things. Delete things. Try and get as much API functionality involved as possible, so that Optic can build up the best representation of a superset of all the traffic is spots. After a while I felt like I'd done everything I could, so I went back to the Optic CLI tab and hit Enter to end the session. It let me know to run `optic oas verify openapi.yaml`, which lets you see all the traffic its recorded, then as I was happy with that I ran this command to save the output: ``` optic oas verify openapi.yaml --document all ``` This tells Optic to populate the `openapi.yaml` document with everything it knows about your API! Mine looked like this: ```yaml openapi: 3.1.0 info: title: Untitled service version: 1.0.0 x-optic-path-ignore: - "**/*.+(ico|png|jpeg|jpg|gif)" paths: /v1/custom_emojis: get: responses: "200": description: 200 response content: application/json; charset=utf-8: schema: $ref: "#/components/schemas/GetApiV1Custom_emojis200ResponseBody" # ... snip ... components: schemas: GetApiV1Custom_emojis200ResponseBody: type: array items: type: object properties: shortcode: type: string url: type: string static_url: type: string visible_in_picker: type: boolean category: type: string required: - shortcode - url - static_url - visible_in_picker - category ``` Brilliant! That's a whole lot of OpenAPI I don't need to write, and Optic is even making components to [keep your OpenAPI Dry](https://blog.stoplight.io/keeping-openapi-dry-and-portable?ref=apisyouwonthate.com). One problem I had was that I ended up loads of excess URLs, like `/settings` and `/packs/{pack}/153-c1b3066c820d0c5de2e2.chunk.js` which I am not interested in. Getting rid of these is easy enough. For starters I should add my server URL, with the base path of `/api` set to let Optic know it should ignore things outside of that path. I can add that in `openapi.yaml`, then delete all the irrelevant endpoints. ```yaml servers: - description: Production url: https://mastodon.green/api ``` With that servers definition added I removed all the '/api\` prefixes from the paths, and reran capture. Due to the way Optic patches the OpenAPI document, I had no problems rewriting the OpenAPI, and it immediately picked up on what I was trying to do. Optic has inferred a lot of the what, but it cannot infer *why*, and this is where you cannot avoid spending a bit of time getting descriptions written. If you have a big enough team maybe a Technical Writer can handle this for you, maybe using a fancy GUI like Stoplight Studio, or maybe just going right into the YAML. Thankfully however you do it, Optic will happily patch that resulting OpenAPI, and the work will not be overridden. ## Sharing OpenAPI to Optic Cloud Once your OpenAPI is in a state you're happy to share, you can take the `openapi.yaml` and get it working with any of the many [OpenAPI documentation tools](https://openapi.tools/?ref=apisyouwonthate.com#documentation), but Optic has that covered too. ``` optic login ``` This will pop open your browser, and there is a "sign in with GitHub" flow. Then you'll be asked to create a Personal access token for Optic, which you can paste that the CLI. Then run this command to upload the OpenAPI document to their servers: ``` $ optic api add openapi.yaml Adding API /src/openapi.yaml ✔ Untitled service is now being tracked. View history: https://app.useoptic.com/organizations/7727d0f3-dba3-4c52-8b32-046187d60239/apis/sdCx0sapGyPyohJiKdOqB ``` Take a look at the [Mastodon API documentation](https://app.useoptic.com/organizations/7727d0f3-dba3-4c52-8b32-046187d60239/apis/sdCx0sapGyPyohJiKdOqB?ref=apisyouwonthate.com) Optic created. ## Thoughts and Feedback This is still a very new tool, and you might find a few rough edges as you go. I found a few bugs using the tool, but [Aidan Cunniffe](https://twitter.com/aidandcunniffe?ref=apisyouwonthate.com) and the Optic team are on our [Slack channel](http://slack.apisyouwonthate.com/?ref=apisyouwonthate.com) and released bug fixes quickly. Optic has mixed results with [different browsers](https://www.useoptic.com/docs/oas-reference/client-guides?ref=apisyouwonthate.com), and I could not get things working with Firefox. Thankfully Google Chrome works just fine, so I could easily switch browsers for this work. You can also attack the API directly with Postman, Curl, Insomnia, Paw, etc., which is pretty handy if you don't have a frontend you can use to make the requests. My biggest gripe with the quality of the OpenAPI being output at this point is the lack of examples. Optic will happily look at actual JSON values coming through to infer the type might be e.g.: "string", but it forgets the value it saw entirely instead of saving it as an example. There's a lot of ways to [handle examples in OpenAPI](https://phil.tech/2020/openapi-examples/?ref=apisyouwonthate.com), but Optic does not seem to leverage any of them. No media type examples, schema examples, or property examples. Without any examples, the documentation it produced can feel rather lacking, so I really hope that gets added in the future. All in all this is a really handy tool which can quickly get you an OpenAPI document, without having to litter your code with annotations which may or may not be accurate. You'll definitely want to do some work to get it up to scratch before going live, but it reduces the human error of having to manually type thousands of lines of YAML out, or click 1000 buttons in a GUI. I don't think of this as a replacement to the API Design First approach, but it's a brilliant second best when you've got to get some OpenAPI out fast. Beyond the scope of this article, but Optic does a bunch of other cool change detection stuff too, so it's worth having a poke around the [documentation](https://www.useoptic.com/docs/?ref=apisyouwonthate.com) to see if any of that can fit in with your API workflow. ### Jazzed about API client library codegen, with Danny Sheridan from Fern URL: https://apisyouwonthate.com/podcast/jazzed-about-api-client-library-codegen-with-danny-sheridan-from-fern/ Last updated: 2023-04-14T17:47:46.000Z ## Show Notes - Fern - Build APIs Twice as fast - [https://buildwithfern.com/](https://buildwithfern.com/?ref=apisyouwonthate.com) - Fern on GitHub - [https://github.com/fern-api/fern](https://github.com/fern-api/fern?ref=apisyouwonthate.com) - Fern's Profile with YCombinator - [https://www.ycombinator.com/companies/fern ](https://www.ycombinator.com/companies/fern?ref=apisyouwonthate.com) - Danny Sheridan - CEO and cofounder of Fern [danny@buildwithfern.com](mailto:danny@buildwithfern.com) - buf.build - protobuf codegen utility - [https://buf.build/](https://buf.build/?ref=apisyouwonthate.com) ### Stigg: Infrastructure for pricing models with Anton Zagrebelny URL: https://apisyouwonthate.com/podcast/stigg-infrastructure-for-pricing-models-with-anton-zagrebelny/ Last updated: 2024-05-14T11:04:58.000Z ## Show Notes - **Stigg:** [https://www.stigg.io/](https://www.stigg.io/?ref=apisyouwonthate.com) \- API-first pricing and packaging - **Stigg is Hiring:** [https://jobs.lever.co/stigg](https://jobs.lever.co/stigg?ref=apisyouwonthate.com) - Find Anton Zagrebelny online: [Linkedin](https://www.linkedin.com/in/anton-zagrebelny/?ref=apisyouwonthate.com) \- [GitHub](https://github.com/antonzy?ref=apisyouwonthate.com) ### Securing APIs with the Spectral OWASP Ruleset URL: https://apisyouwonthate.com/blog/securing-apis-with-spectral-owasp-ruleset/ Last updated: 2023-03-24T12:19:35.000Z Recently I've been on a run of making new all powerful Spectral rulesets, but so far it's been focused on the concept of automating style guides, with the concrete example of the [APIs You Won't Hate: API Style Guide](https://github.com/apisyouwonthate/style-guide?ref=apisyouwonthate.com) showing how it can be done with NPM and tested with Jest. Here's for a new idea though: instead of just making sure you match an organizations style guide, let's look for security issues! If you're new to the topic of API security there's a lot of good content around. One of the best places to start looking is the [OWSAP API Security Project](https://owasp.org/www-project-api-security/?ref=apisyouwonthate.com), where they have the top 10 biggest security threats in the field, last updated in 2019. I've gone through all the guidance and rules, and put a bunch of Spectral rules together in the [Spectral OWASP Ruleset](https://github.com/stoplightio/spectral-owasp-ruleset?ref=apisyouwonthate.com), which is once again available via [NPM](https://www.npmjs.com/package/@stoplight/spectral-owasp-ruleset?ref=apisyouwonthate.com), or [via Stoplight Platform](https://apistylebook.stoplight.io/docs/owasp-top-10/branches/main?ref=apisyouwonthate.com) if you're using their fancy new [integrated Style Guide](https://docs.stoplight.io/docs/platform/71b92b2b5e388-style-guide-overview?ref=apisyouwonthate.com) functionality. The way this works is by looking at your OpenAPI, and seeing if anything clearly wrong is going on, which has pros and cons. There's loads of things OpenAPI doesn't cover, like "Received payload is blindly transformed into an object and stored." or "Unpatched systems" because we can't look to see what the code is doing, and we can't make any judgement calls on the quality of your web servers configuration or versions. Other software exists for all of those concerns, so instead we're just focusing on what we can see. For example, are there any rate limiting headers defined? Let's have a look inside that sausage: ```ts "owasp:api4:2019-rate-limit": { message: "All 2XX and 4XX responses should define rate limiting headers.", description: "Define proper rate limiting to avoid attackers overloading the API. There are many ways to implement rate-limiting, but most of them involve using HTTP headers, and there are two popular ways to do that:\n\nIETF Draft HTTP RateLimit Headers: https://datatracker.ietf.org/doc/draft-ietf-httpapi-ratelimit-headers/\n\nCustomer headers like X-Rate-Limit-Limit (Twitter: https://developer.twitter.com/en/docs/twitter-api/rate-limits) or X-RateLimit-Limit (GitHub: https://docs.github.com/en/rest/overview/resources-in-the-rest-api)", formats: [oas3], given: "$.paths[*]..responses[?(@property.match(/^(2|4)/))]", then: { field: 'headers', function: schema, functionOptions: { schema: { type: 'object', oneOf: [ { required: ['RateLimit-Limit', 'RateLimit-Reset'], }, { required: ['X-RateLimit-Limit'], }, { required: ['X-Rate-Limit-Limit'], }, ], } } }, severity: DiagnosticSeverity.Error, }, ``` This rule is one of the best I've ever written. It's using some pretty hardcore JSON Path to look at 2XX and 4XX responses (because maybe lets not worry about rate limiting redirects?) then uses JSON Schema decide what combinations of headers are acceptable on that responses headers object. Phwar. Whether you've used the [IETF RateLimit Header Fields draft RFC](https://datatracker.ietf.org/doc/draft-ietf-httpapi-ratelimit-headers/%5Cn%5CnCustomer?ref=apisyouwonthate.com), or copied the Twitter or GitHub ways of rate limiting, this rule is A-okay with it. If you went off and did some totally custom approach to rate limiting then you can turn this rule off or override it to train it to your unique approach, ooooor you could change to use something more standard so everyone isn't massively confused. [Yay standards](https://standards.rest/?ref=apisyouwonthate.com)! Another rule written by Stoplight CTO Jason Harmon nicely requests that you add data validation defintions, because a broken validation response could be leaking backtraces, or implementation details. This rule forces you to define what a validation response should look like, so that then any [OpenAPI-based contract validation](https://apisyouwonthate.com/blog/writing-documentation-via-contract-testing) approach you use will be able to spot the different between your nicely defined "this is what it should look like", and the broken nonsense the API is actually doing. ```ts "owasp:api3:2019-define-error-validation": { message: "Missing error validation response of either 400 or 422.", description: "Carefully define schemas for all the API responses, including either 400 or 422 responses which describe errors caused by invalid requests.", severity: DiagnosticSeverity.Warning, given: "$.paths..responses", then: [ { function: schema, functionOptions: { schema: { type: 'object', oneOf: [ { required: ['400'], }, { required: ['422'], }, ], } } }, ], }, ``` There's also the benefit of improving the quality of your documentation by reminding you to document the unhappy path as well as the happy path. Win win. This OWASP ruleset has a bunch of other security rules, and as its entirely open-source we can all work on it together. There's issues to [enforce CORS](https://github.com/stoplightio/spectral-owasp-ruleset/issues/5?ref=apisyouwonthate.com), make sure there's [no backtraces in the errors](https://github.com/stoplightio/spectral-owasp-ruleset/issues/12?ref=apisyouwonthate.com), and to check theres [no PII in responses](https://github.com/stoplightio/spectral-owasp-ruleset/issues/3?ref=apisyouwonthate.com). If you want to get involved but need some help, check out the last few articles in the series covering [Distributing Spectral Style Guides](https://apisyouwonthate.com/blog/distribute-spectral-style-guides-with-npm#publish-your-spectral-style-guide-to-npm) and [Testing Spectral Style Guides with Jest](https://apisyouwonthate.com/blog/testing-spectral-style-guides-with-jest) then check out the [Spectral Custom Ruleset documentation](https://meta.stoplight.io/docs/spectral/ZG9jOjI1MTg5-custom-rulesets?ref=apisyouwonthate.com). ### Don't roll your own API Management tools, with Josh Twist from Zuplo URL: https://apisyouwonthate.com/podcast/dont-roll-your-own-api-management-tools-with-josh-twist-from-zuplo/ Last updated: 2023-04-14T17:50:36.000Z We also discuss Zuplo's support for APIs You Won't Hate through their recent sponsorship of [https://openapi.tools](https://openapi.tools/?ref=apisyouwonthate.com). Thanks so much to Zuplo for supporting our community! ## Show Notes - Zuplo - [https://zuplo.com/](https://zuplo.com/?ref=apisyouwonthate.com) - Open API Tools - [https://openapi.tools](https://openapi.tools/?ref=apisyouwonthate.com) - Josh Twist - [https://www.linkedin.com/in/joshtwist/](https://www.linkedin.com/in/joshtwist/?ref=apisyouwonthate.com) ### Funding Open Source with Dudley Carr from Stack Aid URL: https://apisyouwonthate.com/podcast/funding-open-source-with-dudley-carr-from-stack-aid/ Last updated: 2023-04-15T12:54:37.000Z ## Show notes - Stack Aid - [https://www.stackaid.us/](https://www.stackaid.us/?ref=apisyouwonthate.com) - Dudley Carr - [@dudley@mastodon.social](https://hachyderm.io/@dudley@mastodon.social?ref=apisyouwonthate.com) #### Transcript 00:00:00.181 --> 00:00:02.611 Mike Bifulco: Hello, hello, and welcome back to APIs you won't Hate. 00:00:02.911 --> 00:00:04.531 My name is Mike Balco. 00:00:04.531 --> 00:00:08.689 I am one of your api co-hosts and guide through the world of designing 00:00:08.694 --> 00:00:13.130 APIs and building APIs, and doing all sorts of good stuff with API tech. 00:00:13.197 --> 00:00:16.959 I am joined today for an interview with a new friend of mine, someone who I met 00:00:16.959 --> 00:00:18.969 at a conference here in North Carolina. 00:00:19.454 --> 00:00:22.409 We're gonna be talking a little bit today about his project and some of the sort 00:00:22.409 --> 00:00:25.079 of mission of open source and supporting open source and things like that. 00:00:25.079 --> 00:00:28.101 So today I'm chatting with Dudley Carr from Stack A Dudley. 00:00:28.101 --> 00:00:28.731 How are you doing today? 00:00:29.696 --> 00:00:30.416 Dudley Carr: I'm doing great. 00:00:30.416 --> 00:00:31.386 Thanks for having me on. 00:00:32.141 --> 00:00:32.711 Mike Bifulco: Yeah, of course. 00:00:32.711 --> 00:00:33.801 Super happy to have you here. 00:00:33.856 --> 00:00:37.156 I have lots of questions for you and I'm, I'm super glad you were able to make it 00:00:37.161 --> 00:00:40.432 because from our initial conversations when we sort of bumped into each other 00:00:40.612 --> 00:00:44.182 all over the place at all Things Open your work seemed very interesting to me. 00:00:44.182 --> 00:00:47.269 And I think a lot of the squad here that is part of the APIs you 00:00:47.269 --> 00:00:48.469 won't hate community will really. 00:00:49.009 --> 00:00:49.639 What you're doing. 00:00:49.668 --> 00:00:51.108 So I wanna talk all about that. 00:00:51.108 --> 00:00:53.658 I wanna talk about how you got to where you are and what you're doing at 00:00:53.658 --> 00:00:57.746 Stack and just kinda get some of the history on, on the project in yourself. 00:00:57.795 --> 00:01:00.285 So tell me a bit about yourself and tell me about Stack. 00:01:01.785 --> 00:01:02.615 Dudley Carr: Absolutely. 00:01:02.615 --> 00:01:06.632 So I've been a, in the software engineering space for the. 00:01:07.437 --> 00:01:08.487 22 years. 00:01:08.577 --> 00:01:13.371 I did my undergraduate in computer science at Stanford and graduated at 00:01:13.371 --> 00:01:16.661 the peak of the dot com bubble burst. 00:01:16.931 --> 00:01:21.452 And briefly did a stint in finance, actually worked at Lehman Brothers on 00:01:21.457 --> 00:01:26.472 their exotic derivatives until I realized that stuff is insane and I got out. 00:01:28.122 --> 00:01:33.132 In the last 20 years, I've spent all of it working with my brother, who 00:01:33.132 --> 00:01:36.972 also did computer science, and so we've gone from one venture to the next. 00:01:36.977 --> 00:01:41.115 So he is not here, but is probably the. 00:01:42.000 --> 00:01:43.830 More important of the duo. 00:01:44.010 --> 00:01:48.547 And anyway, we did our first startup in Rhode Island in my parents' basement. 00:01:48.580 --> 00:01:53.760 I think there was radon in that basement, but we we managed we 00:01:53.760 --> 00:01:59.160 actually built in the, you know, 2002, 2003, we built a product that. 00:01:59.535 --> 00:02:00.705 Became g talker. 00:02:00.705 --> 00:02:03.694 It was flash-based, you know, pre action. 00:02:03.694 --> 00:02:07.522 It was action script, but before it was even before they released all of 00:02:07.522 --> 00:02:09.232 their UI toolkits and stuff like that. 00:02:09.232 --> 00:02:11.032 And back backend was Python. 00:02:11.037 --> 00:02:13.252 It was initially a desktop application, then became a. 00:02:13.672 --> 00:02:14.812 Web-based product. 00:02:15.052 --> 00:02:20.112 And we developed that out and ended up selling that to Google and moving 00:02:20.112 --> 00:02:24.472 to Seattle in 2006 to join the Google Talk team and work on that. 00:02:24.532 --> 00:02:29.418 And we spent about five years at Google going from one project to the next. 00:02:29.435 --> 00:02:32.793 First we were in apps and and then eventually I worked on Google 00:02:32.793 --> 00:02:34.754 Voice and then before leaving. 00:02:35.174 --> 00:02:37.694 So that was super formative for us. 00:02:37.694 --> 00:02:39.764 We learned a lot of things, met a lot of great people. 00:02:39.764 --> 00:02:42.044 I think that was kind of the heyday for Google 00:02:42.974 --> 00:02:47.828 And and then after that we, we did some more startups food, food related things. 00:02:47.828 --> 00:02:52.345 And then we joined a company called Moz that does SEO here in Seattle. 00:02:53.260 --> 00:02:58.330 And we spent another four or five years there, I helped run a large 00:02:58.330 --> 00:03:01.450 portion of their engineering team and then grew some of their product areas. 00:03:01.517 --> 00:03:06.202 That was also really formative for us in terms of, you know, understanding 00:03:06.202 --> 00:03:10.672 that space, growing teams and you know, just going through various product 00:03:10.672 --> 00:03:12.202 life cycles and things like that. 00:03:12.284 --> 00:03:18.862 At the end of our MOS experience, we actually did another startup with a friend 00:03:19.132 --> 00:03:21.229 here in Seattle around crowdfunding. 00:03:21.349 --> 00:03:23.749 And this was actually crowdfunding for sports team. 00:03:23.809 --> 00:03:27.249 So, There was another platform that was really taking off. 00:03:27.370 --> 00:03:30.340 We found out about Stripe Connect and started using that. 00:03:30.850 --> 00:03:34.180 And really the, the basis for it was, you know, you have like 00:03:34.180 --> 00:03:35.440 a high school football team. 00:03:35.740 --> 00:03:38.140 They're selling candy bars and things like that. 00:03:38.200 --> 00:03:41.350 There's a lot of inefficiencies there and there's a lot of price 00:03:41.355 --> 00:03:46.010 gouging actually by merchants who sell products to schools to do that. 00:03:46.580 --> 00:03:47.330 And so there was. 00:03:48.205 --> 00:03:52.195 You know, 2017, 2018, there was a real impetus to you know, 00:03:52.195 --> 00:03:53.485 move all of that stuff online. 00:03:53.605 --> 00:03:56.875 And we have a lot of learnings that I think happy to chat about, but that was 00:03:56.875 --> 00:04:00.635 kind of formative for us in terms of thinking about, you know, how you move 00:04:00.865 --> 00:04:05.365 money from a set of people who wanna support something to, to the recipients 00:04:05.370 --> 00:04:06.925 and what all is involved in that. 00:04:07.735 --> 00:04:11.455 That was also just kind of how we, we transitioned from that into consulting. 00:04:11.455 --> 00:04:12.745 So we've been doing consulting for. 00:04:13.435 --> 00:04:16.223 Four years you know, we're kind of embedded engineers and product 00:04:16.223 --> 00:04:19.823 specialists in inside of organizations and to help them transfer in companies. 00:04:19.823 --> 00:04:24.176 And that's gives us a ton of flexibility and allows us to do cool things like what 00:04:24.176 --> 00:04:25.931 we've done over the last couple of years. 00:04:25.931 --> 00:04:28.436 At the beginning of the pandemic by the way, we launched 00:04:28.436 --> 00:04:29.786 something called Covid Trace. 00:04:30.056 --> 00:04:33.746 So we had the hot idea to do contact tracing. 00:04:33.844 --> 00:04:35.374 We tried to launch an app immediately. 00:04:35.374 --> 00:04:36.694 It was blocked by Google and Apple. 00:04:36.745 --> 00:04:37.345 Mike Bifulco: Oh wow. 00:04:37.470 --> 00:04:40.750 Dudley Carr: you're, you're not doing anything location based and we're gonna 00:04:40.750 --> 00:04:42.520 sort this out first, which is great. 00:04:42.550 --> 00:04:44.800 I think it was totally the right move on their part. 00:04:44.861 --> 00:04:48.429 We ended up adopting their the exposure notification. 00:04:48.435 --> 00:04:53.325 APIs that they have, and we ended up lo, I think we were the second 00:04:53.325 --> 00:04:55.005 app to launch in the United States. 00:04:55.005 --> 00:04:57.885 And so we launched with the state of Nevada and worked with them over the 00:04:57.885 --> 00:05:01.665 course of two years doing exposure notifications, rolling that out for 00:05:01.670 --> 00:05:06.435 iOS and Android, and then eventually moving all of Nevada off of our 00:05:06.435 --> 00:05:11.465 custom app onto IOS's, built-in exposure notification function. 00:05:12.290 --> 00:05:15.170 And at the same time building out other things in terms of getting 00:05:15.170 --> 00:05:16.670 results to people and things like that. 00:05:16.670 --> 00:05:20.872 So really interesting problems around health totally unanticipated. 00:05:20.934 --> 00:05:23.655 So that, that was actually that was all open source. 00:05:23.655 --> 00:05:26.404 We released all of that infrastructure, open source and the apps. 00:05:26.404 --> 00:05:29.584 And then, yeah, about a year ago we started on decade. 00:05:31.399 --> 00:05:31.849 Mike Bifulco: Wow. 00:05:32.119 --> 00:05:34.429 Yeah, that's some in incredible back history there. 00:05:34.457 --> 00:05:34.847 I, I. 00:05:36.002 --> 00:05:37.862 Was not prepared for that, that much. 00:05:37.927 --> 00:05:40.942 Incredible problem solving that you've gotten into in your, your career. 00:05:40.992 --> 00:05:41.532 For sure. 00:05:41.532 --> 00:05:44.442 As someone who lived through an entire pandemic of being, you know, 00:05:44.687 --> 00:05:47.177 Locked in my home and not leaving and being very concerned about 00:05:47.177 --> 00:05:48.377 public health and those things. 00:05:48.430 --> 00:05:51.070 Super, super cool to hear, hear you worked on that and, and 00:05:51.100 --> 00:05:52.660 obviously impacted so many people. 00:05:52.693 --> 00:05:55.491 And also, you know, collaborated with the, the big organizations 00:05:55.491 --> 00:05:56.311 like Apple and Google. 00:05:56.311 --> 00:05:57.630 That's massively cool to hear. 00:05:57.645 --> 00:06:00.178 I also don't think I realized that you and I had some sort of shared 00:06:00.183 --> 00:06:03.733 overlap not overlap, but, but maybe an odd Venn diagram of career stuff 00:06:03.733 --> 00:06:06.420 before working at Stripe, I worked at Google for a couple years. 00:06:06.430 --> 00:06:08.620 Not quite on Google Voice, on Google Assistant, so voice 00:06:08.620 --> 00:06:09.920 related stuff at Google. 00:06:09.920 --> 00:06:13.111 Although I'm no longer there and actually probably worth mentioning for posterity 00:06:13.203 --> 00:06:14.463 since you and I met at All Things Open. 00:06:14.463 --> 00:06:15.573 I'm also no longer at Stripe. 00:06:15.573 --> 00:06:18.922 So I'm, I've left Stripe in the past couple weeks, but I'm very curious 00:06:18.922 --> 00:06:22.204 to hear about your experiences with Stripe Connect and, and all that. 00:06:22.204 --> 00:06:22.732 And so. 00:06:23.227 --> 00:06:26.407 All of this history of all the crazy things you've done and, and like working 00:06:26.407 --> 00:06:30.282 with complex teams and big problems and across devices and problem spaces, 00:06:30.282 --> 00:06:32.802 and I'm sure languages and all the other things that have changed since 00:06:32.862 --> 00:06:36.349 what, 2003 when you first got into the the, the world of, of building things 00:06:36.424 --> 00:06:37.654 has led you to where you're at now. 00:06:37.654 --> 00:06:40.218 So tell me a little bit about Stack Aid and what you're doing. 00:06:40.285 --> 00:06:40.588 There. 00:06:41.923 --> 00:06:45.870 Dudley Carr: Yeah, so Stack stack is a service that allows you to fund 00:06:45.870 --> 00:06:50.610 your second first order and second order dependencies automatically. 00:06:50.673 --> 00:06:54.651 It, the impetus for it came about a year and a half ago when we. 00:06:54.971 --> 00:07:02.080 You know, repeatedly saw articles about people exasperated by their inability to 00:07:02.080 --> 00:07:07.060 sustain their open source project because, you know, the demands have increased on 00:07:07.570 --> 00:07:11.555 what they have to deliver and the reach, you the reach of their open sources 00:07:11.675 --> 00:07:16.445 beyond their wildest dreams, but, you know, they, they basically pay for it in 00:07:16.445 --> 00:07:20.923 their spare time or it takes away from other paying opportunities that they have. 00:07:20.928 --> 00:07:25.363 And so you see a lot of people kind of torn in those situations. 00:07:25.453 --> 00:07:27.273 We, that really resonated with us. 00:07:27.361 --> 00:07:32.131 As I mentioned, you know, we had spent time in the fundraising 00:07:32.461 --> 00:07:35.281 arena and we, you know, we saw. 00:07:36.391 --> 00:07:40.981 Definitely momentum around Get Up sponsors an open collective, but we, we thought 00:07:40.981 --> 00:07:42.841 that there was an opportunity there. 00:07:43.261 --> 00:07:47.461 You know, I think what's super interesting about the software development space as 00:07:47.461 --> 00:07:52.441 opposed to any other space where people are trying to raise money is that we 00:07:52.441 --> 00:07:55.771 know we know what, what you use, right? 00:07:55.891 --> 00:08:00.151 There's sometimes it's imperative, but increasingly it's a declarative. 00:08:02.146 --> 00:08:04.366 Way of specifying all your dependencies. 00:08:04.366 --> 00:08:09.406 And so we can, we can do so many things automatically to determine 00:08:09.473 --> 00:08:13.193 what you use and, and potentially influence how we allocate money. 00:08:13.313 --> 00:08:17.483 And so the, the, the seed of an idea was there and we started exploring, 00:08:17.573 --> 00:08:21.983 you know, the feasibility of it and what that would look like, and is it 00:08:21.983 --> 00:08:24.293 an effective model, things like that. 00:08:24.331 --> 00:08:26.491 And so that's been like the last year and it's, it's. 00:08:27.196 --> 00:08:28.246 Super interesting. 00:08:28.251 --> 00:08:30.856 Kind of flushing that out and we're, we've been super happy with the 00:08:30.856 --> 00:08:34.094 results and the initial reception when we launched a couple of months ago. 00:08:34.931 --> 00:08:35.471 Mike Bifulco: So. 00:08:35.599 --> 00:08:38.269 I've seen it and I'm sort of familiar with the product, but I wanna make sure 00:08:38.269 --> 00:08:41.219 that you know, it's abundantly clear what you mean when you're talking about this. 00:08:41.219 --> 00:08:44.241 So we're talking about funding open source projects in a way that is sort of 00:08:44.241 --> 00:08:48.462 sustainable and based on your dependency graph for projects that you're using. 00:08:48.462 --> 00:08:51.132 So when you say first and second order dependencies, what do you mean? 00:08:52.602 --> 00:08:52.752 Dudley Carr: Yeah. 00:08:52.752 --> 00:08:57.472 So by first order, so let's take a Packers saw JSON in the node E. 00:08:58.527 --> 00:09:04.077 The first order of dependencies are the the dependencies and dev dependencies 00:09:04.077 --> 00:09:07.257 that you list directly in that za js o n. 00:09:08.097 --> 00:09:13.047 Now, those first order dependencies in turn have their own za js o, 00:09:13.137 --> 00:09:14.667 where they list their dependencies. 00:09:14.672 --> 00:09:17.217 That would be the second order of dependencies. 00:09:17.787 --> 00:09:21.417 Now you can walk that tree down all the way down, and there are 00:09:21.417 --> 00:09:22.947 gonna be lots and lots more. 00:09:24.192 --> 00:09:27.912 not unusual for a project to have literally thousands of. 00:09:29.217 --> 00:09:31.047 Dependencies in their dependency tree. 00:09:31.797 --> 00:09:36.027 But you know, from a funding perspective, you have to draw the line somewhere. 00:09:36.117 --> 00:09:39.207 Otherwise, you know, you take a certain amount of money and divide 00:09:39.207 --> 00:09:43.167 it into tiny little pieces and it becomes somewhat meaningless. 00:09:43.254 --> 00:09:47.664 So we wanted to, you know, the, the easy thing was would be to 00:09:47.664 --> 00:09:49.854 just fund first order dependencies. 00:09:50.394 --> 00:09:53.634 But we, we realized, you know, a lot of those open source 00:09:53.634 --> 00:09:54.744 projects also want to give. 00:09:55.599 --> 00:09:57.729 And if we, you know, defaults matter. 00:09:57.789 --> 00:10:02.499 And we realize that if we came up with a mechanism that, you know, when you 00:10:02.499 --> 00:10:06.699 find a first order dependency, it passes some of that onto its dependencies. 00:10:07.179 --> 00:10:09.909 You know, you're doing that automatically for the ecosystem. 00:10:10.089 --> 00:10:13.959 You're bene, you don't have to have everyone opt-in in order to have 00:10:13.959 --> 00:10:15.639 further reach into the ecosystem. 00:10:16.239 --> 00:10:20.242 And so yeah, that was the impetus to fund first and second order depend. 00:10:21.772 --> 00:10:22.042 Mike Bifulco: Yeah. 00:10:22.042 --> 00:10:22.432 Got it. 00:10:22.432 --> 00:10:26.512 So from the, I I, gosh, I don't even know what, what you would consider to be the 00:10:26.512 --> 00:10:30.890 end user, but from the perspective of someone who is doing the funding, doing 00:10:30.890 --> 00:10:32.235 the supporting what does that look like? 00:10:32.235 --> 00:10:33.795 Like what is, what is the process for me? 00:10:33.795 --> 00:10:36.705 Say for a project I'm running, let's say APIs, you won't hate.com, right? 00:10:36.705 --> 00:10:38.145 It's a, it's a no JS project. 00:10:38.211 --> 00:10:40.891 We've got a whole heap of dependencies that are sort of built into this thing. 00:10:40.891 --> 00:10:42.128 What would I need to do to adopt. 00:10:44.528 --> 00:10:45.218 Dudley Carr: a great question. 00:10:45.218 --> 00:10:50.468 So, you know, when you go to Staca us, there's the first step in the 00:10:50.468 --> 00:10:55.199 onboarding process is oh, often thing with GitHub and actually adding the 00:10:55.199 --> 00:11:00.209 GitHub app to either your personal organization or some other organization 00:11:00.209 --> 00:11:06.216 where repositories are we then scanned those repositories for you know, files 00:11:06.216 --> 00:11:09.966 like Bax, J S O N, or you know, others depending on whatever language you're. 00:11:11.091 --> 00:11:16.175 And we use those declarative list of dependencies, we ingest that and 00:11:16.175 --> 00:11:18.815 start looking at that dependency tree. 00:11:19.625 --> 00:11:22.655 Once we have that, we, you know, we, we put you in the dashboard. 00:11:22.655 --> 00:11:26.291 We show you what we had discovered, like which files and which 00:11:26.291 --> 00:11:27.881 repositories we're pulling from. 00:11:28.631 --> 00:11:31.991 And we presume initially that you ne you want to fund all of those. 00:11:31.991 --> 00:11:34.121 You can, you can be selective, right? 00:11:34.121 --> 00:11:36.971 So I wanna fund these repositories and these package digest and things 00:11:36.971 --> 00:11:41.741 like, Based on that, based on the first order and second order 00:11:41.741 --> 00:11:43.421 dependencies we've pulled from that. 00:11:43.631 --> 00:11:46.787 And you can then indicate as a level of support that you 00:11:46.787 --> 00:11:48.287 wanna do on a monthly basis. 00:11:49.097 --> 00:11:52.847 We then calculate how much would go to each of those projects. 00:11:52.852 --> 00:11:56.678 So it's hard to des describe, but there's a tree that we have in the, 00:11:56.858 --> 00:12:01.350 in the dashboard and it shows you, okay, you've got React or low dash, for 00:12:01.355 --> 00:12:04.080 example, as a first order dependency. 00:12:04.080 --> 00:12:07.320 It has these second order dependencies and it shows you 00:12:07.320 --> 00:12:11.430 the amount of your subscription that goes to each one of those. 00:12:11.430 --> 00:12:16.950 And so that breaks down when you're, the next step is to enter a credit card 00:12:16.950 --> 00:12:18.600 and then, then you're off to the races. 00:12:20.685 --> 00:12:21.075 Mike Bifulco: Yeah. 00:12:21.135 --> 00:12:21.495 Okay. 00:12:21.495 --> 00:12:24.863 So from, from my perspective, it is, you know off with GitHub, 00:12:24.863 --> 00:12:28.602 get this thing added to my stack of or to my GitHub organization. 00:12:28.602 --> 00:12:32.174 It'll go and, and I guess introspect and look at, or I guess inspect is 00:12:32.179 --> 00:12:33.224 probably even the right word there. 00:12:33.404 --> 00:12:36.217 Go look at all the projects I have and give me the the first and second order 00:12:36.217 --> 00:12:38.281 dependencies for each is the target. 00:12:38.281 --> 00:12:41.324 Then from there to say like, just using easy numbers I want to 00:12:41.324 --> 00:12:42.344 donate a hundred bucks a month. 00:12:42.964 --> 00:12:44.284 To these various organizations. 00:12:44.284 --> 00:12:47.434 I, I have one fixed cost and Stack Aid kind of does the rest from there. 00:12:48.839 --> 00:12:50.219 Dudley Carr: That's, that's exactly right. 00:12:50.279 --> 00:12:50.759 Yes. 00:12:50.794 --> 00:12:51.154 Mike Bifulco: Yeah. 00:12:51.574 --> 00:12:51.928 Wow. 00:12:51.953 --> 00:12:54.953 So how, well man, I, I feel like I have so many questions. 00:12:54.953 --> 00:12:56.183 How does the money get from A to B? 00:12:56.213 --> 00:12:58.196 Like, how do you track down the the various projects 00:12:58.201 --> 00:12:59.126 that are then being funded? 00:13:00.201 --> 00:13:03.801 Dudley Carr: Yeah, so that's the fun part about building something like 00:13:03.801 --> 00:13:07.491 this is because it's effectively kind of like a marketplace, right? 00:13:07.671 --> 00:13:09.831 I mean, we have, we're engaging with both. 00:13:10.030 --> 00:13:14.908 Individual developers and companies who are supporters and of course have a 00:13:14.908 --> 00:13:16.738 relationship with open source, maintain. 00:13:17.888 --> 00:13:22.317 So we have slowly been reaching out to open source maintainers 00:13:22.407 --> 00:13:28.677 kind of as we drive awareness or if they've receiving funding, we will 00:13:28.677 --> 00:13:30.147 reach out to them individually. 00:13:30.228 --> 00:13:35.178 , but we also have been realizing that, you know, a lot of these 00:13:35.178 --> 00:13:36.468 people don't know who we are. 00:13:36.495 --> 00:13:39.705 There's a lot of things grabbing at their attention. 00:13:40.125 --> 00:13:43.755 So if they have an existing relationship with GI UP sponsors or 00:13:43.755 --> 00:13:47.855 Open Collective, we actually just use our corporate credit card and 00:13:47.855 --> 00:13:49.715 make the donation on those platforms. 00:13:49.781 --> 00:13:53.381 So our, you know, our goal is to get the money in their hands. 00:13:53.430 --> 00:13:57.120 And if they have an existing relationship, we, we lean on that. 00:13:57.150 --> 00:13:59.130 So that, that's worked out well. 00:13:59.130 --> 00:14:03.580 But but primarily over time, I think for for the ease of developers and to 00:14:03.790 --> 00:14:07.690 give them more control in terms of, you know, how those funds are allocated. 00:14:07.690 --> 00:14:09.820 Especially if there are multiple people working on a project. 00:14:09.820 --> 00:14:10.190 Things like. 00:14:10.190 --> 00:14:13.590 You know, we we would like people to, you know, claim their project on stack. 00:14:14.355 --> 00:14:15.700 Mike Bifulco: Yeah, sure. 00:14:15.700 --> 00:14:16.464 What does that look like? 00:14:18.009 --> 00:14:21.013 Dudley Carr: So we use Stripe Connect under underneath. 00:14:21.037 --> 00:14:25.001 So you know, when you log into the dashboard and you owe off you also 00:14:25.001 --> 00:14:27.011 have to oth with GI up at the moment. 00:14:27.087 --> 00:14:27.730 We're working on other. 00:14:27.730 --> 00:14:32.933 Hosting platforms, but you o often we actually verify that you actually are 00:14:32.933 --> 00:14:35.363 a maintainer on those repositories that you're trying to claim. 00:14:35.963 --> 00:14:38.437 We list out those repositories you claim them. 00:14:38.442 --> 00:14:41.957 And then as part of that claiming process, we also need 00:14:41.957 --> 00:14:43.926 to collect the a Stripe account. 00:14:44.136 --> 00:14:46.056 So we send you over to Stripe. 00:14:46.476 --> 00:14:49.162 They get all of the, the details necessary. 00:14:49.162 --> 00:14:52.475 To basically give us a, a stripe account so that we can deposit 00:14:52.475 --> 00:14:54.711 funds into at the end of the month. 00:14:54.711 --> 00:14:55.456 And then that's it. 00:14:55.516 --> 00:14:59.070 Then you're, then you're able to collect money from stack. 00:14:59.845 --> 00:15:00.175 Mike Bifulco: Yeah. 00:15:00.265 --> 00:15:01.075 Wow, that's great. 00:15:01.140 --> 00:15:04.410 So, so I'd imagine there's some population of people who are very pleased to find 00:15:04.410 --> 00:15:07.451 out they can come to Stack Aid, click a couple of buttons and have money being 00:15:07.451 --> 00:15:09.030 funneled into their project every month. 00:15:09.030 --> 00:15:11.419 That, that's gotta feel pretty cool to be able to, I don't know, 00:15:11.419 --> 00:15:12.919 land that dream so seamlessly. 00:15:13.692 --> 00:15:17.862 Dudley Carr: Yeah, I mean, I think it speaks more distract than to us. 00:15:17.862 --> 00:15:22.542 I mean, honestly, that flow is amazing and there's so much complexity abstracted. 00:15:22.563 --> 00:15:26.119 But I think from an end developer perspective, it is surprisingly 00:15:26.119 --> 00:15:27.679 easy to get up and running. 00:15:28.259 --> 00:15:30.885 And yeah, and I think it's, it's pretty great, you know, when you 00:15:30.885 --> 00:15:35.385 show up that a lot of the times there's, you know, a couple of bucks 00:15:35.385 --> 00:15:38.775 at the very least waiting for you there, and you immediately get that. 00:15:38.775 --> 00:15:43.605 I think that has been an important part of stack it, which is, you know, 00:15:43.610 --> 00:15:45.945 you, you don't have to be a developer. 00:15:45.975 --> 00:15:48.615 Like the developer doesn't have to have an account in order 00:15:48.975 --> 00:15:50.475 for money to accrue for them. 00:15:50.578 --> 00:15:55.408 So you know, you have this kind of problem I think on GitHub sponsors 00:15:55.408 --> 00:16:00.028 an open collective initially where people didn't have a relationship 00:16:00.208 --> 00:16:02.578 with those platforms, so there wasn't a way to get money to them. 00:16:03.058 --> 00:16:06.208 A lot of people have set it up, but there's also a large 00:16:06.208 --> 00:16:08.548 portion of the ecosystem that has no relationship with them. 00:16:09.148 --> 00:16:13.138 And so it was important for us to be able to accrue money and, you know, 00:16:13.678 --> 00:16:15.298 show people that you can actually. 00:16:16.238 --> 00:16:19.800 there's money in the open source that they've contributed and have 00:16:19.800 --> 00:16:22.000 that as a carrot for them to sign up. 00:16:23.130 --> 00:16:23.670 Mike Bifulco: Sure. 00:16:23.970 --> 00:16:27.486 Yeah, that's, that's a really interesting model and having been exposed to GitHub 00:16:27.486 --> 00:16:30.456 sponsors a little bit, I know that like one of the nice things that comes along 00:16:30.461 --> 00:16:34.596 with this actually may, might be a Stripe Connect requirement, but to access 00:16:34.596 --> 00:16:38.406 Stripe Connect, you have to essentially have viable tax information, right? 00:16:38.406 --> 00:16:40.836 Like the, the right information to be able to be paid out. 00:16:40.948 --> 00:16:43.318 So that you're not just, you know, sending off money to some 00:16:43.318 --> 00:16:45.088 anonymous bucket somewhere. 00:16:45.155 --> 00:16:48.818 But instead, theoretically it's tied to like an L L C or an individual proprietor 00:16:48.818 --> 00:16:52.031 or, you know, a more complex corporation in the case of vicar businesses. 00:16:52.082 --> 00:16:55.158 But a lot of that is, I would imagine abstracted away from you. 00:16:55.158 --> 00:16:57.651 You just need them to, to, you know, click the button and connect 00:16:57.677 --> 00:16:59.570 to stack with Stripe Connect. 00:17:00.230 --> 00:17:03.964 Dudley Carr: One of the biggest concerns that we had out of the gate was you. 00:17:04.699 --> 00:17:06.679 All open source doesn't happen in the United States. 00:17:06.679 --> 00:17:10.879 There are people across the world, and the United States in particular has a 00:17:10.879 --> 00:17:12.559 requirement called know your customer. 00:17:13.129 --> 00:17:17.989 And so you need a lot of details in order to verify their identity and make, you 00:17:17.989 --> 00:17:23.609 know, make sure that this isn't for money laundering or some other scheme like that. 00:17:23.630 --> 00:17:26.780 And so that is actually all abstracted away for us. 00:17:26.848 --> 00:17:29.638 And that is pretty phenomenal if we. 00:17:31.453 --> 00:17:33.043 A, a two person operation. 00:17:33.043 --> 00:17:34.383 There's just no way you're gonna 00:17:35.953 --> 00:17:36.463 Mike Bifulco: Yeah, 00:17:36.583 --> 00:17:36.863 Dudley Carr: that. 00:17:37.183 --> 00:17:39.733 Mike Bifulco: the, the scope and scale of those money laundering operations 00:17:39.733 --> 00:17:43.941 is far more complex and sophisticated than, you know, I think we might realize 00:17:43.941 --> 00:17:45.441 as, as sort of an average consumer. 00:17:45.493 --> 00:17:48.297 You know, again, I'm, I'm not at Stripe any longer, but during my tenure there, 00:17:48.297 --> 00:17:51.758 like you, you do Financial crimes training and it's pretty astonishing 00:17:51.758 --> 00:17:54.518 in the creative ways people, you know, will, will go to lengths to 00:17:54.518 --> 00:17:57.602 make money disappear or just harder to trace whatever the case may be. 00:17:57.602 --> 00:17:59.402 And nice that you don't have to worry about that. 00:17:59.402 --> 00:18:03.042 There's a lot of mechanisms in place to detect and prevent that fraud as well. 00:18:03.112 --> 00:18:03.147 . Okay. 00:18:03.147 --> 00:18:05.926 So I, I want to know a little bit about when did you what, what 00:18:05.926 --> 00:18:08.496 signals were you given that this was something that was going to work? 00:18:08.496 --> 00:18:11.110 In other words that when you're starting to build stack, because it's only a year 00:18:11.110 --> 00:18:14.557 and change old at this point was there a moment or a series of events that sort of 00:18:14.562 --> 00:18:17.287 made you feel like, oh, this is something that actually has some momentum behind it? 00:18:19.772 --> 00:18:22.712 Dudley Carr: Yeah, I think well, I think we had to prove to ourselves that 00:18:22.712 --> 00:18:28.712 it's viable and, you know, we, we have, there's some nuance to the model in 00:18:28.712 --> 00:18:30.752 terms of how we distribute that money. 00:18:30.752 --> 00:18:33.962 And, and more importantly, what's interesting about this problem is 00:18:34.172 --> 00:18:35.792 that it's not a one-time thing. 00:18:35.852 --> 00:18:38.822 So if no one shows up to collect the money, what do you do with that money? 00:18:38.912 --> 00:18:40.892 So there's a time component to it as well. 00:18:41.377 --> 00:18:41.867 Mike Bifulco: Yeah. 00:18:42.662 --> 00:18:44.342 Dudley Carr: we wanted, so we. 00:18:44.897 --> 00:18:49.217 There's complexity around the model to some degree in terms of implementing 00:18:49.217 --> 00:18:55.912 and doing it right, and we, we knew that the model itself needed to be validated 00:18:55.917 --> 00:19:01.792 and be comparable to things like get up sponsors and, and Open Collective. 00:19:02.512 --> 00:19:05.692 So we actually spent a large portion of the development. 00:19:07.057 --> 00:19:08.407 Building out a simulation. 00:19:09.127 --> 00:19:13.744 And so there's a, like simulation Dots US has. 00:19:13.758 --> 00:19:19.668 It's, it's effectively like the, it's our entire site, but it has 5,000 00:19:19.878 --> 00:19:25.218 made up subscribers at various price points using Pax JSONs that we had 00:19:25.218 --> 00:19:28.348 discovered on GitHub using source graph. 00:19:28.353 --> 00:19:31.473 Source graph was pretty instrumental in terms of d doing that. 00:19:31.938 --> 00:19:35.808 And we, we needed package js os that weren't on n p, right? 00:19:35.868 --> 00:19:39.558 We didn't want to grab load Dash's, patch json accidentally. 00:19:39.558 --> 00:19:43.188 And because that, that's not representative of potential end users. 00:19:44.088 --> 00:19:49.891 So we took those 5,000 subscribers, plugged them in, you know, gave them 00:19:49.891 --> 00:19:54.391 some subscription amount between $25 per month to a hundred dollars per month. 00:19:55.201 --> 00:19:55.681 And we. 00:19:56.371 --> 00:19:58.501 Look to see what happens. 00:19:58.501 --> 00:19:58.891 Right? 00:19:59.161 --> 00:20:01.741 What's the outcome of, of this? 00:20:01.741 --> 00:20:05.791 Like, is it just a couple of projects that get all the money or, you know, 00:20:05.791 --> 00:20:10.231 what does that distribution look like and the, the, the end result is that, 00:20:10.231 --> 00:20:14.558 yeah, you, you still have a power power law curve just like you do on 00:20:14.558 --> 00:20:19.019 Get up sponsors in Open Collective, but it was it was more stretched. 00:20:19.754 --> 00:20:25.484 So we ended up, we ended up funding a larger percentage of the, let's 00:20:25.484 --> 00:20:31.304 say the top 25% of funds included a significantly larger set of projects. 00:20:31.364 --> 00:20:35.954 So even though they're at the tip of this parallel curve, they, you 00:20:35.954 --> 00:20:37.214 know, there's more of them included. 00:20:37.214 --> 00:20:37.784 That's great. 00:20:38.174 --> 00:20:41.054 But the middle, the middle was much broader. 00:20:41.264 --> 00:20:41.534 Right. 00:20:41.564 --> 00:20:46.274 A lot more of the money was going into that, and so that, that was the validation 00:20:46.334 --> 00:20:50.624 that we needed, right, internally to know that, yeah, we can reach more of this. 00:20:51.314 --> 00:20:56.924 I think in terms of the broader like readiness for this type of product, 00:20:57.014 --> 00:21:01.514 I, I think, you know, there's just a drumbeat of vulnerabilities 00:21:01.934 --> 00:21:03.584 and also just individuals. 00:21:04.664 --> 00:21:09.194 Really talking about the lack of funding, the lack of maintenance 00:21:09.194 --> 00:21:10.364 around this, around this. 00:21:10.364 --> 00:21:15.018 And so that is the validation that we continue to look for you know, 00:21:15.018 --> 00:21:18.138 as an opportunity to do something about, I think we're, we're very 00:21:18.138 --> 00:21:21.830 nascent in terms of evangelizing this and, and driving awareness. 00:21:21.893 --> 00:21:27.258 But I think, you know, those two things kind of has given us the confidence that 00:21:27.283 --> 00:21:31.132 you know, the timing is hopefully right and it's the right product for the time. 00:21:31.912 --> 00:21:32.362 Mike Bifulco: Yeah. 00:21:32.422 --> 00:21:32.932 Yeah. 00:21:32.963 --> 00:21:36.743 I, it's an interesting, almost, it's not that you have a chicken and an egg problem 00:21:36.764 --> 00:21:40.214 to, to work with, but I feel like the whole funding nut to crack is that like 00:21:40.219 --> 00:21:44.024 we, we all on some level, developers, engineering teams or organizations 00:21:44.084 --> 00:21:47.886 understand that it's important to Keep these projects funded so that they stay up 00:21:47.886 --> 00:21:51.246 to date so that vulnerabilities get shut down, bugs get addressed, functionality 00:21:51.246 --> 00:21:52.856 gets added, whatever the case may be. 00:21:54.311 --> 00:21:58.200 it seems like a lot of the social pressure lands on individuals to do the 00:21:58.200 --> 00:22:01.683 funding in a lot of ways, and I think that maybe is a law of numbers thing. 00:22:01.683 --> 00:22:04.293 Like people you know, you get a lot more call to action as an 00:22:04.293 --> 00:22:05.703 individual to go fund things. 00:22:05.703 --> 00:22:09.067 But my guess is that the bulk of the volume of money is coming 00:22:09.067 --> 00:22:12.067 from organizations who are willing to fund open source things. 00:22:12.217 --> 00:22:12.997 Is that roughly. 00:22:14.167 --> 00:22:18.296 Dudley Carr: Yeah, so we actually were able to analyze all of the 00:22:18.296 --> 00:22:20.186 Open Collective transactions. 00:22:20.366 --> 00:22:25.826 They do this amazing job of every transaction on Open, the Open Source 00:22:25.826 --> 00:22:27.746 collective, on Open Collective. 00:22:27.814 --> 00:22:30.034 You can literally download all of the transactions and 00:22:30.034 --> 00:22:32.414 so, I did that and I went 00:22:32.434 --> 00:22:33.094 Mike Bifulco: Oh wow. 00:22:33.544 --> 00:22:38.734 Dudley Carr: And yes, you know, organizations like Google and, and 00:22:38.734 --> 00:22:41.584 others, they do put in a ton of money. 00:22:41.944 --> 00:22:47.281 But if I remember correctly, I would say, Over 60% of it are from individuals 00:22:47.281 --> 00:22:49.381 donating at at much smaller amounts. 00:22:49.381 --> 00:22:53.371 So they're, they have a long tail and it is a significant 00:22:53.376 --> 00:22:55.831 portion of the contributions. 00:22:55.831 --> 00:22:59.592 And so it, it's, it wasn't as skewed as you would think 00:22:59.630 --> 00:23:01.022 towards large organizations. 00:23:01.307 --> 00:23:04.011 Mike Bifulco: That that is a, a bigger percentage than I would've guess. 00:23:04.011 --> 00:23:04.881 That's really interesting. 00:23:05.016 --> 00:23:08.801 So what, what is your call to action or maybe your pitch for those who 00:23:08.801 --> 00:23:10.631 might have the capacity to donate? 00:23:10.631 --> 00:23:14.307 Like how, how is the I guess the, is there a sales process for this? 00:23:14.312 --> 00:23:16.647 Is it something that you're going to organizations and people and trying to get 00:23:16.677 --> 00:23:19.047 them to discover and use Stack as donors? 00:23:20.277 --> 00:23:23.847 Dudley Carr: You know, I think, I think there are certain organizations 00:23:23.847 --> 00:23:28.684 that are very attuned to open source and, you know, they have open source 00:23:28.924 --> 00:23:33.904 program offices and they are actively engaging those communities and they are. 00:23:35.239 --> 00:23:38.089 they're looking, you know, they're either doing this themselves. 00:23:38.092 --> 00:23:43.572 So century is a customer of Stack and they did a ton of this by themselves. 00:23:43.572 --> 00:23:47.622 They, they wrote custom things to analyze their dependencies, and they had a big 00:23:47.627 --> 00:23:51.732 spreadsheet and it's super impressive, but it's incredibly time consuming. 00:23:52.452 --> 00:23:56.892 And I think Indeed and others are also analyzing their dependencies and trying 00:23:56.892 --> 00:23:58.752 to figure out where to allocate money. 00:23:59.112 --> 00:24:01.242 So this is something that is happening today. 00:24:01.282 --> 00:24:05.294 So we're looking to engage with those types of organizations and 00:24:05.294 --> 00:24:10.094 understand, you know, how STACK can potentially be a part of that. 00:24:10.137 --> 00:24:13.227 So I think step one is to really engage with organizations 00:24:13.227 --> 00:24:14.457 that are receptive to it. 00:24:14.457 --> 00:24:17.067 I think that's the kind of low hanging fruit. 00:24:17.157 --> 00:24:21.777 And I think beyond that, you know, there's, there's or organizations 00:24:21.777 --> 00:24:23.157 that are certainly consuming large. 00:24:23.982 --> 00:24:29.712 Portion of open source and you know, there's kind of a, a sales, different 00:24:29.712 --> 00:24:33.612 sales process around, you know, here are the ways that you engage with 00:24:33.612 --> 00:24:35.172 open source at those organizations. 00:24:35.592 --> 00:24:37.362 Funding is one aspect of that. 00:24:37.572 --> 00:24:40.781 And so I think over time that's where that conversation's going. 00:24:41.291 --> 00:24:44.921 But I think the organizations that are currently funding open source to some 00:24:44.921 --> 00:24:49.406 degree, You know, they're kind of making the case for that and, and we, you know, 00:24:49.646 --> 00:24:54.026 we're trying to expand that conversation and, and as well as piggyback off of that, 00:24:55.756 --> 00:24:55.991 Mike Bifulco: right? 00:24:56.081 --> 00:24:56.441 Yeah. 00:24:56.471 --> 00:24:59.771 It's nice that it's kind of the zeitgeist is that it seems that support has really 00:24:59.776 --> 00:25:03.161 changed in the past, I don't know, maybe 10 years to, like open source is 00:25:03.166 --> 00:25:06.531 something we can try or should try to, open source is something that, you know, 00:25:06.621 --> 00:25:09.941 I is the infrastructure of the internet in a lot of ways and something that you 00:25:09.941 --> 00:25:14.143 know, almost the, the ethical impetus is to support open source projects and to 00:25:14.143 --> 00:25:15.673 also be a part of that if you're able. 00:25:16.918 --> 00:25:17.728 So, okay. 00:25:17.796 --> 00:25:20.942 I, I guess one more important question then, if I'm an open source developer 00:25:20.961 --> 00:25:24.766 what, what are actions I can take to be proactive about I, I guess making 00:25:24.766 --> 00:25:28.238 sure that I'm, I'm covered by stack or that you know, that I'm doing 00:25:28.243 --> 00:25:29.408 the right things to seek funding. 00:25:30.528 --> 00:25:32.618 Dudley Carr: Yeah, I think you know, one. 00:25:33.503 --> 00:25:41.093 One theory that we have is that, you know, the, there are organizations 00:25:41.093 --> 00:25:44.363 like we were just talking about that are attuned and are willing to donate, 00:25:44.513 --> 00:25:50.243 but I, I actually think a fundamental shift will is dependent on individual 00:25:50.243 --> 00:25:56.513 developers donating and independent of the platform, but actively participating 00:25:56.513 --> 00:26:01.598 in that way of funding open source be it GI UP sponsors, open Collective Stack. 00:26:02.378 --> 00:26:05.888 Thanks, DD Dev, any of those platforms is a good way to start. 00:26:05.888 --> 00:26:11.138 But there, there has, you know, we have to have that expectation that developers 00:26:11.138 --> 00:26:14.528 are doing this just like they do other types of open source contributions. 00:26:14.528 --> 00:26:15.428 And I think that. 00:26:16.523 --> 00:26:21.923 That groundswell of developers participating and educating and kind of 00:26:21.923 --> 00:26:25.313 demanding this in their organizations is what actually turns the tide. 00:26:25.703 --> 00:26:30.803 And so our focus initially is actually to get individual developers to come 00:26:30.803 --> 00:26:34.493 on board and we're, we hope that we're. 00:26:35.168 --> 00:26:37.778 You know, one of those solutions that makes it a lot simpler. 00:26:37.778 --> 00:26:40.778 But if GitHub sponsors is the way that you do it, great. 00:26:40.898 --> 00:26:41.348 Right? 00:26:41.498 --> 00:26:42.638 Go, go on there. 00:26:42.818 --> 00:26:45.938 Fund, fund the people or the projects that you really care about. 00:26:46.478 --> 00:26:48.248 But I think that speaks volumes, right? 00:26:48.333 --> 00:26:51.813 And that I, I think is the thing that actually moves the needle. 00:26:51.813 --> 00:26:54.183 And those platforms have made it simpler. 00:26:54.363 --> 00:26:56.883 We hopefully have made it simpler based on, you know, what 00:26:56.883 --> 00:26:58.113 some set of people care about. 00:26:58.683 --> 00:27:02.523 But, you know, our, our goal is to evangelize individual developers. 00:27:03.673 --> 00:27:04.193 Contributing more. 00:27:06.168 --> 00:27:08.428 Mike Bifulco: Yeah, that's a noble conceit and definitely one of those things 00:27:08.428 --> 00:27:12.154 that I think all of the people listening to the show can probably relate to. 00:27:12.226 --> 00:27:13.636 I certainly identify with it. 00:27:13.641 --> 00:27:16.947 I, one of the things I've been mulling over a lot lately especially, especially 00:27:16.947 --> 00:27:19.527 in the past few weeks that I've been like reconsidering my personal budget and the 00:27:19.527 --> 00:27:22.917 way I allocate money for things is that I, I think I would like to be a little 00:27:22.917 --> 00:27:25.137 more public in sharing and explaining. 00:27:25.387 --> 00:27:28.357 The ways that I spend money in four good ways, right? 00:27:28.357 --> 00:27:31.117 Like charities that I donate to on one side, but open source things that 00:27:31.117 --> 00:27:32.827 I donate to projects that I support. 00:27:33.097 --> 00:27:36.847 And also, this is more on the creator economy side, but like Patreon and 00:27:36.847 --> 00:27:39.067 things like that, where there's like, you know, I love this podcast, so 00:27:39.072 --> 00:27:41.647 I give them a dollar a month, which is, you know, more than they would 00:27:41.647 --> 00:27:43.207 ever get from me clicking on ads. 00:27:43.207 --> 00:27:44.707 I could click ads every day for a week. 00:27:44.707 --> 00:27:45.816 And wouldn't give them a book. 00:27:45.874 --> 00:27:47.283 And it goes a lot further than you would think. 00:27:47.332 --> 00:27:49.702 And it, it's funny, I've been kind of thinking that that's 00:27:49.702 --> 00:27:50.752 something that belongs in. 00:27:51.537 --> 00:27:54.267 Almost public profile, like I should be sharing this somewhere 00:27:54.567 --> 00:27:58.768 and making that a part of the my, my persona, my support for the world. 00:27:58.827 --> 00:28:01.181 And I think that that's something that we have a, great opportunity 00:28:01.181 --> 00:28:04.604 to do with projects like Stack A and with other things that we all 00:28:04.604 --> 00:28:07.934 participate in because it also creates that social pressure and that. 00:28:07.954 --> 00:28:12.310 Impression that expectation that part of being a, a good citizen as a developer 00:28:12.310 --> 00:28:15.251 when you can and if you can, and if you have, you know, the, honestly the 00:28:15.256 --> 00:28:18.400 mountains of privilege that I'm sitting on top of, like, you should be giving back. 00:28:18.403 --> 00:28:19.135 I really like that. 00:28:19.135 --> 00:28:23.115 And I, one of the things that I like about STACK is honestly the, the tree view of 00:28:23.115 --> 00:28:26.775 the dependencies and seeing the amount of impact that, you know, even a few 00:28:26.775 --> 00:28:28.949 bucks a month can have is like visceral. 00:28:29.009 --> 00:28:32.729 You really feel like you, you see that not only are you using this cascade 00:28:32.729 --> 00:28:35.579 of things to power whatever project you're working on, but you can also 00:28:35.579 --> 00:28:37.214 give back to them fairly directly. 00:28:37.263 --> 00:28:39.453 And, there's infrastructure in place to do that for you. 00:28:39.513 --> 00:28:42.735 I think that's really exciting and I think it's a noble cause and I'm hoping 00:28:42.740 --> 00:28:45.909 it's something that a lot of the folks who are listening to the podcast will be 00:28:45.909 --> 00:28:49.128 able to jump into and go ahead long into supporting, but also benefiting from. 00:28:50.523 --> 00:28:50.763 Dudley Carr: Yeah. 00:28:50.763 --> 00:28:51.693 No, I appreciate that. 00:28:51.693 --> 00:28:55.833 I, I think what you're saying really resonates with us in that 00:28:56.223 --> 00:28:58.473 how you spend your money matters. 00:28:58.533 --> 00:29:01.969 You know, we are in a position of privilege where, you know, we 00:29:02.029 --> 00:29:05.239 we have discretionary money that we can funnel towards things. 00:29:05.329 --> 00:29:07.789 And I think, I think you nailed it. 00:29:07.849 --> 00:29:11.704 You know, a lot of these developers are, you know, at the moment 00:29:11.737 --> 00:29:13.417 maybe a couple of bucks per month. 00:29:13.438 --> 00:29:17.668 You know, we're still small, but I think it, it really matters to those 00:29:17.668 --> 00:29:23.098 developers partly because it is a real recognition of what they're doing and 00:29:23.128 --> 00:29:27.118 they know that someone took the time and their money, you know, to do that. 00:29:27.718 --> 00:29:29.248 And I think that's super powerful. 00:29:29.248 --> 00:29:32.368 I think it's easy to dismiss it as, oh, it's, you know, it's a trivial 00:29:32.368 --> 00:29:33.658 sum of money or something of the. 00:29:34.663 --> 00:29:39.103 But you know, when you are working on something, and a lot of times, you know, 00:29:39.103 --> 00:29:42.133 you can look at your MPM install numbers, like, oh yeah, that's through the roof. 00:29:42.133 --> 00:29:45.913 But this is, you know, getting an email from someone saying 00:29:46.063 --> 00:29:47.203 like, I like your project. 00:29:47.203 --> 00:29:49.033 That's really visceral as well. 00:29:49.573 --> 00:29:50.023 Mike Bifulco: Yeah. 00:29:50.023 --> 00:29:51.493 Dudley Carr: like people actually just paying. 00:29:51.583 --> 00:29:53.443 I think that's an incredible way. 00:29:53.443 --> 00:29:57.763 And so hopefully people are not put off by, you know, initially like, oh, the, 00:29:57.763 --> 00:29:59.503 the dollar amounts are not significant. 00:30:00.103 --> 00:30:04.843 It, it, it supports that individual at so many different levels. 00:30:04.843 --> 00:30:09.725 And so yeah, how you spend your money matters and and it has a really great 00:30:09.725 --> 00:30:11.375 upside on the other other side of it. 00:30:12.035 --> 00:30:12.485 Mike Bifulco: Yeah. 00:30:12.735 --> 00:30:14.885 , it's pretty profound and an energizing thing for me. 00:30:14.984 --> 00:30:17.534 Well, Dudley, thanks so much for coming and hanging out today. 00:30:17.561 --> 00:30:20.331 I have two important questions for you before I let you go. 00:30:20.331 --> 00:30:23.731 One is I wanna know how APIs you won't hate listeners can find you and 00:30:23.731 --> 00:30:24.991 talk to you if they're interested. 00:30:25.261 --> 00:30:26.854 And where can they go to get started with? 00:30:28.954 --> 00:30:29.734 Dudley Carr: Absolutely. 00:30:29.808 --> 00:30:30.018 Yeah. 00:30:30.018 --> 00:30:39.618 So you can email me at dudley dod e y stack.us and our website is stack a.us. 00:30:39.619 --> 00:30:42.729 I think if you search for Stack Google, we're number one. 00:30:42.769 --> 00:30:46.519 And you know, as we were chatting earlier, it's, it's super simple to get started. 00:30:46.669 --> 00:30:49.510 If you run into any issues please reach out and we're, 00:30:49.580 --> 00:30:51.220 we're happy to answer questions. 00:30:51.280 --> 00:30:54.213 But yeah, it's pretty self-service at the moment. 00:30:54.393 --> 00:30:56.943 Just click on the button o off and then hopefully you're off to 00:30:56.943 --> 00:31:02.073 the races and, you know, always looking for more feedback and, Yeah. 00:31:02.073 --> 00:31:05.043 No, we, we appreciate every, every person who signs up and 00:31:06.153 --> 00:31:07.293 happy to answer questions. 00:31:07.968 --> 00:31:08.268 Mike Bifulco: Great. 00:31:08.358 --> 00:31:08.838 Wonderful. 00:31:09.168 --> 00:31:10.638 Dudley, thanks so much for hanging out today. 00:31:10.747 --> 00:31:11.827 It's been a pleasure having you. 00:31:11.832 --> 00:31:14.572 And I'd love to catch up again you know, maybe in a few months or ear down 00:31:14.572 --> 00:31:15.482 the line to see how things are going. 00:31:16.327 --> 00:31:16.867 Dudley Carr: Absolutely. 00:31:16.872 --> 00:31:18.067 Thanks so much for having me. 00:31:18.067 --> 00:31:18.607 Really appreciate it. 00:31:19.132 --> 00:31:19.792 Mike Bifulco: Yeah, of course. 00:31:19.792 --> 00:31:20.092 Take care. 00:31:20.887 --> 00:31:21.247 Dudley Carr: Bye-bye. ### Note-taking tools for devs, with Drew White from Stashpad URL: https://apisyouwonthate.com/podcast/note-taking-tools-for-devs-with-drew-white-from-stashpad/ Last updated: 2023-04-15T12:57:45.000Z ## Show Notes - Stashpad - [https://stashpad.com/](https://stashpad.com/?ref=apisyouwonthate.com) - Stashpad Discord - [https://discord.gg/ScxPxcN9fK](https://discord.gg/ScxPxcN9fK?ref=apisyouwonthate.com) - Drew White - [@drucial](https://twitter.com/drucial?ref=apisyouwonthate.com) #### Transcript 00:00:01.860 --> 00:00:04.410 Mike Bifulco: Hello, hello and welcome to APIs you Won't Hate. 00:00:04.410 --> 00:00:05.730 My name is Mike Fulco. 00:00:05.730 --> 00:00:08.660 Your effervescent and ever present host of the show. 00:00:08.663 --> 00:00:12.249 Today I am flying solo and having a chat with actually a friend of mine. 00:00:12.316 --> 00:00:15.939 Locally here in my hometown of Charlotte who I've known for a while now. 00:00:16.034 --> 00:00:18.284 And we're, we're gonna talk a bit about what he is working on, a bit 00:00:18.284 --> 00:00:21.802 about how he got there and you know, some of the backstory of that stuff. 00:00:21.892 --> 00:00:24.472 So I'm very excited to talk to today. 00:00:24.472 --> 00:00:25.102 Drew White. 00:00:25.102 --> 00:00:25.882 Drew, How's it going, man? 00:00:26.587 --> 00:00:27.127 Drew White: Hey Mike. 00:00:27.127 --> 00:00:27.577 How are you? 00:00:27.667 --> 00:00:28.447 Doing good today. 00:00:28.972 --> 00:00:29.422 Mike Bifulco: I'm good. 00:00:29.422 --> 00:00:29.962 I'm good. 00:00:30.035 --> 00:00:31.410 We have a lot of things to talk about. 00:00:31.410 --> 00:00:35.128 I'm really interested to hear your whole story and talk a little bit 00:00:35.133 --> 00:00:38.737 about stash pad where you, you have been laying down your lines 00:00:38.737 --> 00:00:40.627 of code of late among other things. 00:00:40.733 --> 00:00:43.638 Yeah, and I think we, we'll kind of get into all those things. 00:00:43.706 --> 00:00:46.706 In particular, like anything to do with building en engineering teams and all 00:00:46.706 --> 00:00:48.086 that is always interesting around here. 00:00:48.177 --> 00:00:49.823 Drew, tell me about yourself. 00:00:49.823 --> 00:00:50.513 How did we meet? 00:00:50.573 --> 00:00:51.333 Let's start there 00:00:52.073 --> 00:00:55.805 Drew White: this is actually, I feel like it was kismet if I can use that word. 00:00:55.882 --> 00:00:56.152 Yeah. 00:00:56.152 --> 00:00:59.311 So I'm a cyclist as you guys probably know, Mike is as well. 00:00:59.791 --> 00:01:04.447 And I was riding with a buddy on a local Greenway, and Mike was riding one 00:01:04.447 --> 00:01:08.917 of the most esoteric bikes that I feel like only a handful of local cyclists 00:01:09.337 --> 00:01:10.627 probably even know what they are. 00:01:10.632 --> 00:01:13.357 But I saw it was like, Hey. 00:01:13.777 --> 00:01:15.337 Is that a such and such? 00:01:15.847 --> 00:01:20.017 And he was like, Yeah, how did you, like, it was just like a, a sort of thing. 00:01:20.137 --> 00:01:24.457 And so we kind of met on the, the Greenway had a small little 00:01:24.457 --> 00:01:28.381 conversation and then later I had a. 00:01:28.726 --> 00:01:30.136 Set of wheels for sale. 00:01:30.136 --> 00:01:32.917 I, I believe, And you responded to the post. 00:01:32.917 --> 00:01:36.187 I don't think I realized it was you until you came to pick up the 00:01:36.192 --> 00:01:38.347 wheels and bought them and Yeah. 00:01:38.377 --> 00:01:42.932 So like that whole thing and then, yeah, just started riding like morning greenway 00:01:43.082 --> 00:01:45.242 grabbing coffee, that sort of thing. 00:01:45.242 --> 00:01:47.252 And that was a couple years ago now, 00:01:47.487 --> 00:01:49.737 Mike Bifulco: it was during the dark days of the pandemic for sure. 00:01:49.848 --> 00:01:52.248 You know, when, when we were not doing much indoor stuff, 00:01:52.518 --> 00:01:54.018 definitely a bit of kismet there. 00:01:54.085 --> 00:01:57.158 And I, I think if I remember like the space between bumping into each 00:01:57.163 --> 00:02:00.758 other for the first time and then me contacting you on Facebook marketplace 00:02:00.758 --> 00:02:04.802 to buy wheels when I needed them was like days to a week at most. 00:02:04.804 --> 00:02:06.364 Drew White: I think it was two days. 00:02:06.424 --> 00:02:07.544 I think it was two days. 00:02:07.894 --> 00:02:10.909 Mike Bifulco: a very strange back to back set of coincidences that I'm, 00:02:10.909 --> 00:02:12.809 you know, frankly pretty grateful for. 00:02:13.019 --> 00:02:14.039 Drew White: And I am too 00:02:14.261 --> 00:02:15.371 Mike Bifulco: Yeah, of course. 00:02:15.409 --> 00:02:18.039 We've talked about, you know, tons of writing stuff ever since, of course. 00:02:18.039 --> 00:02:21.154 And coffee seems to come up fairly often and you, you have similar tastes 00:02:21.154 --> 00:02:23.074 in design and all that other stuff too. 00:02:23.096 --> 00:02:25.016 So it's been super cool to kind of get to know you here. 00:02:25.060 --> 00:02:27.653 And what's been really cool to see over the past few years is like 00:02:27.653 --> 00:02:29.873 you've done a complete full on career. 00:02:29.878 --> 00:02:32.423 Like I, I, a pivot is not even fair. 00:02:32.423 --> 00:02:35.673 Like you've done an absolute like SUEx to your working world. 00:02:35.715 --> 00:02:37.065 Tell me a little bit about your working history. 00:02:37.065 --> 00:02:38.726 Like what, what have you done and what are you doing? 00:02:39.945 --> 00:02:43.155 Drew White: Yeah, so I've kind of taken a non-traditional 00:02:43.155 --> 00:02:45.975 path into the working world. 00:02:46.069 --> 00:02:49.969 I kind of started in finance for the first two years out of school. 00:02:50.031 --> 00:02:52.161 I did not go to college. 00:02:52.176 --> 00:02:55.220 Just really wasn't my, I attempted, but really wasn't my thing. 00:02:56.000 --> 00:02:59.760 So jumped into finance for a couple of years and then spent the. 00:03:01.515 --> 00:03:04.245 I don't know, decade or so in aerospace. 00:03:04.277 --> 00:03:08.778 Started kind of at the bottom of sort of midsize company and worked 00:03:08.778 --> 00:03:11.085 my way up to marketing director. 00:03:12.045 --> 00:03:17.415 And so from there, pivoted out of that into starting my own marketing 00:03:17.420 --> 00:03:21.633 agency which I did smack Deb in the middle of the pandemic right 00:03:21.633 --> 00:03:23.283 around the time that I met you. 00:03:23.403 --> 00:03:29.688 And what's interesting is I had been, You know, fascinated with the developer 00:03:29.688 --> 00:03:32.295 world for a couple years at that point. 00:03:32.295 --> 00:03:35.941 But really hadn't made it like a high priority on my, 00:03:36.421 --> 00:03:38.198 I tend to accumulate hobbies. 00:03:38.239 --> 00:03:40.996 So it kind of fell to the bottom of the stack. 00:03:40.996 --> 00:03:44.626 And then I met you and we were kind of talking about some of this stuff on 00:03:44.631 --> 00:03:47.086 the bike rides and, and such and such. 00:03:47.266 --> 00:03:50.566 And I had started building a lot of websites and things for. 00:03:51.896 --> 00:03:55.366 and yeah, just with one of your, your previous employers. 00:03:55.366 --> 00:03:59.314 Shown me the, the gymnasium actually which was like sort of like tutorial 00:03:59.314 --> 00:04:03.811 land, educational portal for largely like web dev stuff I feel like. 00:04:03.867 --> 00:04:06.970 But anyways, took every single course available on there 00:04:06.970 --> 00:04:08.500 and got a lot out of it. 00:04:08.500 --> 00:04:12.355 And just like that love of wanting to build stuff just 00:04:12.805 --> 00:04:15.235 ignited from that point forward. 00:04:15.925 --> 00:04:17.155 So fast forward. 00:04:18.070 --> 00:04:23.055 Let's say a year of really focusing on development education, 00:04:23.055 --> 00:04:24.315 particularly with JavaScript. 00:04:24.344 --> 00:04:28.492 I was kind of burn out managing this, this marketing business. 00:04:28.518 --> 00:04:32.457 Found a actual subcontractor that was interested in acquiring it and. 00:04:33.987 --> 00:04:39.657 Bailed and decided that I wanted to take a stab at, you know, working 00:04:39.657 --> 00:04:41.622 for a startup in the tech world. 00:04:41.694 --> 00:04:46.398 And so kind of applied to a couple of places and put my resume out there a 00:04:46.398 --> 00:04:50.101 little bit, However minimal it might have been at that point in time. 00:04:50.701 --> 00:04:55.938 And fielded quite a few inquiries and really landed on I had one conversation. 00:04:56.158 --> 00:04:59.640 Kara Bornstein is Stash pad ceo. 00:04:59.640 --> 00:05:07.136 And really believed in her vision and her as a leader of that company. 00:05:07.166 --> 00:05:10.349 So it was pretty sold and then in the second interview, got to meet with the 00:05:10.349 --> 00:05:14.466 cto the Meron and was even more sold. 00:05:14.496 --> 00:05:17.856 So I had kind of decided at that point that this is really 00:05:17.856 --> 00:05:19.386 where I wanted to be and. 00:05:20.491 --> 00:05:24.451 So took a role there as a developer experience designer, 00:05:24.471 --> 00:05:26.236 , Mike Bifulco: man, you've done so many things in such 00:05:26.236 --> 00:05:27.136 a short amount of time, like. 00:05:27.376 --> 00:05:31.246 Literally from, from finance to being a marketing director, to running an agency 00:05:31.246 --> 00:05:34.392 to figuring out how to find your way into the dev world is really fascinating. 00:05:34.392 --> 00:05:36.882 you know, Along the way, like you, you also had some interesting projects 00:05:36.882 --> 00:05:39.387 that you put out into the world, which, though your resume may have 00:05:39.387 --> 00:05:43.017 been short at the time you had some really cool stuff like your skew 00:05:43.227 --> 00:05:44.797 amorphism project . That, that was cool. 00:05:44.797 --> 00:05:45.637 Do you wanna talk a little bit about. 00:05:46.582 --> 00:05:47.452 Drew White: Yeah, sure. 00:05:47.452 --> 00:05:50.644 So I was just kind of in all of my free time, I was building 00:05:50.644 --> 00:05:53.623 a lot of UI stuff just. 00:05:54.648 --> 00:05:57.768 For learning purposes of my own, but also just cuz there were things that I 00:05:57.768 --> 00:06:01.698 wanted that I, I couldn't find or I didn't think existed or something like that. 00:06:02.208 --> 00:06:06.278 So I was using a lot of like, skew, morphism, glass, amorphism and 00:06:06.278 --> 00:06:08.688 amorphism in some of my designs. 00:06:08.709 --> 00:06:13.209 Primarily because I have a background in 3D design and so it was like 00:06:13.209 --> 00:06:16.359 sort of appealing to me to be able to create some of that stuff. 00:06:16.486 --> 00:06:19.126 Sort of like the in, in the web, which I thought was awesome. 00:06:19.152 --> 00:06:20.535 And so yeah, I created this tool. 00:06:20.655 --> 00:06:26.535 I got tired of like finagling, like, okay, 0.3 pixels, 0.4 pixels, like, like 00:06:26.535 --> 00:06:29.595 all of this stuff, like adjusting 'em to get like the shadows and the highlights 00:06:29.595 --> 00:06:30.955 and all of that stuff just right. 00:06:30.955 --> 00:06:35.823 And so I created a little tool that's basically a, a CSS generator with 00:06:36.003 --> 00:06:39.783 these really nice little sliders that, you know, you can quickly dial. 00:06:40.403 --> 00:06:45.441 The amount of s amorphism amorphism that you want with the right direction of 00:06:45.441 --> 00:06:50.435 light down to like, I think it's 1000th of a pixel or something like that. 00:06:50.435 --> 00:06:51.275 It's pretty crazy. 00:06:51.305 --> 00:06:56.584 But yeah, built that and it's actually gotten quite a bit of use from my, not 00:06:56.584 --> 00:07:00.239 only myself, but like other designers and developers have used it as well. 00:07:00.239 --> 00:07:03.167 And yeah, that was like the first real tool that I built and put out. 00:07:04.077 --> 00:07:07.231 Picked up any traction but it was super fun to build for sure. 00:07:08.336 --> 00:07:08.726 Mike Bifulco: Yeah. 00:07:10.446 --> 00:07:12.776 I appreciate most about you, how understated you are. 00:07:12.793 --> 00:07:17.533 It, this is an insanely cool thing and like to me, the, the perfect example 00:07:17.533 --> 00:07:21.106 of showing , that you're an interesting person who's taking a hands on approach to 00:07:21.106 --> 00:07:22.532 learning and actually building things out. 00:07:22.532 --> 00:07:25.821 I will drop the URL for this tool in, in the browser or sorry, 00:07:25.821 --> 00:07:27.156 in the, in the show notes here. 00:07:27.195 --> 00:07:30.195 And what's interesting for the audience of APIs you won't hate is like a 00:07:30.195 --> 00:07:31.605 lot of the folks we work with here. 00:07:31.702 --> 00:07:35.152 Really into building the data layer, the back end side of things, the connective 00:07:35.152 --> 00:07:36.842 tissue from the front end to the back end. 00:07:36.842 --> 00:07:40.126 But you can imagine in many ways that you could show off your chops as an 00:07:40.126 --> 00:07:44.026 API developer by building out a simple tool that just shows one facet of 00:07:44.236 --> 00:07:47.606 here's how I would, you know, build out these, these knobs and levers to adjust 00:07:47.606 --> 00:07:49.038 the experience of building an api. 00:07:49.106 --> 00:07:49.342 Better. 00:07:49.342 --> 00:07:53.763 New Amorphism is a very touchable like you know, tasty kind of thing to be able 00:07:53.763 --> 00:07:56.327 to go out and use and like as someone who's trying to break into the industry 00:07:56.327 --> 00:07:58.757 or as someone who was trying to break into the industry at the time, it's 00:07:58.762 --> 00:08:02.171 the perfect kind of prism put in front of yourself to say like, yeah, cool. 00:08:02.231 --> 00:08:04.781 I haven't worked in this yet, but I do this kind of work and 00:08:04.781 --> 00:08:05.801 I do a really good job of it. 00:08:05.862 --> 00:08:08.210 And it's gotten some great attention too, which is really cool. 00:08:08.232 --> 00:08:10.512 The, the thing I still need to yell at you about is you need to 00:08:10.512 --> 00:08:11.832 put your name on that webpage. 00:08:12.782 --> 00:08:15.302 In big, bold letters somewhere, minimalism be damned. 00:08:15.302 --> 00:08:16.612 People should know where it came from. 00:08:16.612 --> 00:08:17.212 You know what I mean? 00:08:17.627 --> 00:08:20.777 Drew White: That is sort of like a thing that we've talked about a bit. 00:08:20.807 --> 00:08:27.483 I'm a minimalist through and through like at every phase and yeah, it's, I get it. 00:08:27.543 --> 00:08:28.053 The branding. 00:08:28.053 --> 00:08:29.883 I need to be better about that for sure. 00:08:29.898 --> 00:08:31.647 And maybe someday I'll put it on there. 00:08:32.667 --> 00:08:33.357 Mike Bifulco: Fair enough. 00:08:33.687 --> 00:08:36.177 Yeah, I'll go chase down your code and open a poll request for you. 00:08:36.233 --> 00:08:36.488 Yeah. 00:08:36.488 --> 00:08:36.728 Cool. 00:08:36.728 --> 00:08:39.458 So why don't we talk a little bit about what you're doing now. 00:08:39.458 --> 00:08:40.950 So what is stpa? 00:08:41.810 --> 00:08:46.194 Drew White: So Stash Padd is a notes taking application. 00:08:46.259 --> 00:08:50.639 Kind of aims to flip that concept of notes taking on its head. 00:08:50.755 --> 00:08:54.714 The whole point of what we're doing is reducing the burden of capture. 00:08:54.714 --> 00:08:58.327 I mean from my perspective, notes is not a particularly enjoyable 00:08:58.327 --> 00:08:59.707 experience for most people. 00:08:59.707 --> 00:09:01.867 However, it is a particularly important. 00:09:02.273 --> 00:09:06.290 Part of daily dev life or daily, you know, really work life. 00:09:06.290 --> 00:09:10.202 Being able to get thoughts out of our head, take notes on conversations 00:09:10.207 --> 00:09:13.142 that we've had, meetings standups, code reviews, all that kind of stuff 00:09:13.154 --> 00:09:16.479 very easily, very quickly, and be able to put it somewhere and not 00:09:16.479 --> 00:09:20.090 really have to worry about where you're putting it necessarily and kind 00:09:20.090 --> 00:09:22.340 of give you that feeling and vibe. 00:09:22.670 --> 00:09:26.270 Similar to like if you were dm, DMing yourself in Slack. 00:09:27.630 --> 00:09:31.290 Where it's the, it's the lowest burden of entry for capture. 00:09:31.290 --> 00:09:36.270 And the, the, in my personal experience, I might be biased, but my personal 00:09:36.270 --> 00:09:40.050 experience, it's the, it's the least amount of friction for getting something 00:09:40.050 --> 00:09:44.020 out of my head and into somewhere that I can recall it later when I need to. 00:09:44.020 --> 00:09:48.683 So yeah, we've been working on the app for, oh, probably two 00:09:48.683 --> 00:09:50.053 years now, I guess is when. 00:09:51.278 --> 00:09:55.502 Things kind of started, but we just launched in August on product hunt. 00:09:55.502 --> 00:09:58.261 And reception has been phenomenal. 00:09:58.266 --> 00:09:59.461 It's been so, so good. 00:09:59.598 --> 00:10:00.108 So yeah. 00:10:01.158 --> 00:10:02.658 That's what Stash pad is. 00:10:02.658 --> 00:10:07.285 It's at the helm we have Kara Bernstein and Theo Meron as the two founders. 00:10:07.317 --> 00:10:08.918 And then it's a pretty small team. 00:10:08.918 --> 00:10:11.438 We're located in Raleigh or Durham, North Carolina. 00:10:11.498 --> 00:10:12.518 I keep saying Raleigh 00:10:12.818 --> 00:10:13.058 every 00:10:13.268 --> 00:10:13.868 Mike Bifulco: Middle of both. 00:10:14.138 --> 00:10:14.628 Drew White: Yeah. 00:10:14.633 --> 00:10:14.918 Yeah. 00:10:15.398 --> 00:10:17.048 Mike Bifulco: I mean, most people put 'em right next to each other 00:10:17.048 --> 00:10:17.408 anyway. 00:10:17.510 --> 00:10:20.051 Drew White: At the American Underground there which has been great. 00:10:20.141 --> 00:10:20.681 So, Yeah. 00:10:20.696 --> 00:10:21.116 Mike Bifulco: Yeah. 00:10:21.146 --> 00:10:21.476 Cool. 00:10:21.476 --> 00:10:21.626 Yeah. 00:10:21.626 --> 00:10:24.324 American Underground is kind of like the home of startups in, 00:10:24.324 --> 00:10:25.614 in that part of North Carolina. 00:10:25.697 --> 00:10:27.257 A super cool community created there. 00:10:27.303 --> 00:10:29.643 So note taking is a really interesting thing to me. 00:10:29.736 --> 00:10:34.066 I, I have kind of a, a interesting history with it and actually I remember, I wish 00:10:34.066 --> 00:10:36.796 I could tell you when it was, but I remember a specific conversation I had 00:10:36.796 --> 00:10:38.368 with one of my great friends actually. 00:10:38.429 --> 00:10:42.574 My former employer, Andrew Miller, who is the program director over at Gymnasium 00:10:42.574 --> 00:10:45.100 and his longtime friend of mine, one of the, the smartest people I know. 00:10:45.100 --> 00:10:47.350 At one point I remember having a conversation with him where he asked 00:10:47.350 --> 00:10:49.240 me about how I take notes for work. 00:10:49.240 --> 00:10:50.620 Like how do I keep track of what I'm doing? 00:10:50.860 --> 00:10:53.920 And literally at the time, my response was, why would I take notes? 00:10:53.950 --> 00:10:55.690 Like, I just remember it, you know? 00:10:55.690 --> 00:10:59.399 And like the, the brash, bold statement that I made that 00:10:59.399 --> 00:11:00.512 was just like I don't know. 00:11:00.512 --> 00:11:02.432 My brain's working at a thousand percent all the time. 00:11:02.432 --> 00:11:04.112 Why do I need to write anything down? 00:11:04.179 --> 00:11:07.404 I remember that moment and I remember like literally a month later being like, 00:11:07.674 --> 00:11:08.994 Oh man, I need to write everything down. 00:11:08.994 --> 00:11:10.164 Like I'm starting to forget things. 00:11:10.164 --> 00:11:11.614 They're all falling outta the back of my head. 00:11:11.614 --> 00:11:14.052 And that, that was the moment where I really started to focus on like, trying 00:11:14.052 --> 00:11:17.937 to organize myself, trying to organize my thoughts and have frankly, filtered 00:11:17.942 --> 00:11:19.257 through a lot of tools in the meantime. 00:11:19.257 --> 00:11:19.507 And I think. 00:11:20.202 --> 00:11:22.872 The note taking thing and writing down notes and taking 00:11:22.872 --> 00:11:24.029 notes is a virtuous thing. 00:11:24.029 --> 00:11:24.599 It's very good. 00:11:24.599 --> 00:11:29.009 You want to do it because it, it's less burden for your, your mind, but 00:11:29.009 --> 00:11:30.835 also it helps other people, right? 00:11:30.835 --> 00:11:33.565 So like, Drew, if you and I have a conversation, I'm teaching you 00:11:33.565 --> 00:11:35.065 something one on one, that's awesome. 00:11:35.125 --> 00:11:36.245 You might learn something from me. 00:11:36.271 --> 00:11:39.806 But if I also write it down and one other person reads it, I've doubled 00:11:39.806 --> 00:11:41.526 the efficacy of that conversation. 00:11:41.556 --> 00:11:42.606 And that's why note taking is good. 00:11:42.606 --> 00:11:43.236 It's also helpful. 00:11:43.236 --> 00:11:45.426 If I forget it in the future, I can come back to it. 00:11:45.473 --> 00:11:47.423 What, what I also really like about it too is that. 00:11:48.653 --> 00:11:51.353 Note taking is different for everyone and you kind of have 00:11:51.353 --> 00:11:52.353 to find what works for you. 00:11:52.353 --> 00:11:55.333 And I feel like people may feel like the market is kind of floated, 00:11:55.393 --> 00:11:57.073 flooded with note taking tools. 00:11:57.105 --> 00:12:01.138 But I think that's because people's style of thought and their style of organization 00:12:01.143 --> 00:12:04.398 is very, very different from one another and like, Some people are good with 00:12:04.398 --> 00:12:08.589 just a notepad, you know, txt file and, and the chaos that that may bring on. 00:12:08.589 --> 00:12:11.829 Some people might like the iOS, you know, note app for their own thing. 00:12:11.832 --> 00:12:15.387 But truly finding something that is like broadly applicable and 00:12:15.387 --> 00:12:18.717 easy to use and easy to understand is a challenging problem space. 00:12:18.838 --> 00:12:21.530 Drew White: Yeah, and I think actually your experience that you 00:12:21.535 --> 00:12:23.300 just described is fairly common. 00:12:23.360 --> 00:12:25.730 You know, I had the same. 00:12:27.005 --> 00:12:31.235 Greater than do attitude towards notes in the early days, like 00:12:31.295 --> 00:12:33.245 I have a pretty solid memory. 00:12:33.245 --> 00:12:35.045 I can remember a lot of things. 00:12:36.830 --> 00:12:42.110 But what I think a lot of people who do take notes now understand, and people who 00:12:42.110 --> 00:12:46.490 don't take notes will ultimately figure out is that the more you keep in your 00:12:46.495 --> 00:12:49.850 head, Yeah, you may be able to keep it in there, but you got limited space up there. 00:12:49.910 --> 00:12:53.960 So the more you take in, eventually some of that stuff's gonna start falling off. 00:12:53.960 --> 00:12:56.870 And then there's like the stress of, you know, some of that data may be 00:12:56.870 --> 00:12:59.150 important and then you may not have it. 00:12:59.150 --> 00:13:02.343 So I've definitely adapted a practice and you're absolutely right, there's 00:13:02.343 --> 00:13:03.693 a lot of options out there and. 00:13:05.043 --> 00:13:10.177 Varying degrees of Complexity, which is the interesting part to me. 00:13:10.477 --> 00:13:14.587 But I think what is so interesting is just the fact that there are so many, 00:13:14.617 --> 00:13:18.727 like different note taking applications speaks to a larger problem, right? 00:13:18.967 --> 00:13:20.707 No one has kind of sorted this stuff out. 00:13:20.712 --> 00:13:23.317 Usually, particularly in the dev the development world, 00:13:23.322 --> 00:13:26.707 engineering world dev tools tend to be winner take all, I mean, vs. 00:13:26.707 --> 00:13:31.237 Code by far and away owns the market and in ide, maybe with JetBrains or 00:13:31.237 --> 00:13:33.127 something coming in right behind them. 00:13:33.877 --> 00:13:34.477 You've got. 00:13:35.962 --> 00:13:38.152 Basically issue tracking tools and all these other things. 00:13:38.152 --> 00:13:43.702 There's usually like a winner take all sort of situation and in so sort of 00:13:43.702 --> 00:13:48.442 personal notes that sort of space that really isn't something that is landed on. 00:13:48.442 --> 00:13:51.022 People are kind of all over the board from, you know, untitled text 00:13:51.022 --> 00:13:55.144 files, just flooding their desktop to any combination of different 00:13:55.234 --> 00:14:00.358 apps, big ones, no notion Evernote obsidian, all of those things and. 00:14:01.138 --> 00:14:06.478 Where we like to think that we can fit in and, and, and why we're building 00:14:06.478 --> 00:14:10.708 this thing in the first place is to kind of have this defacto, we'll 00:14:10.713 --> 00:14:13.078 do whatever you want it to do. 00:14:13.768 --> 00:14:15.578 Lightweight and very speedy. 00:14:15.642 --> 00:14:19.006 I've used some of the other big name apps out there particularly. 00:14:19.921 --> 00:14:21.581 Like Apple notes and things like that. 00:14:21.639 --> 00:14:26.619 And there always seems to be a little bit of friction between, I just got 00:14:26.619 --> 00:14:30.789 told some information that I need to remember in four hours from now, or two 00:14:30.789 --> 00:14:32.619 days from now, or two months from now. 00:14:33.459 --> 00:14:34.659 Where do I put that? 00:14:34.929 --> 00:14:36.569 How do I organize that in my. 00:14:37.194 --> 00:14:41.574 Hierarchy or whatever and how am I gonna find that later? 00:14:41.645 --> 00:14:42.815 And that has always been my challenge. 00:14:42.815 --> 00:14:46.025 I've bounced around from, from app to app long before I even 00:14:46.025 --> 00:14:48.035 knew that stash pad was a thing. 00:14:48.076 --> 00:14:52.306 And so that's the problem we aim to resolve. 00:14:52.336 --> 00:14:56.176 And the reality is if we can bring a little bit of joy to something 00:14:56.176 --> 00:15:01.173 that is often like a mundane sort of experience yeah, I mean, all the better. 00:15:01.383 --> 00:15:01.623 That's. 00:15:02.418 --> 00:15:03.048 The goal 00:15:03.888 --> 00:15:04.368 Mike Bifulco: Sure. 00:15:04.518 --> 00:15:04.998 Yeah. 00:15:05.088 --> 00:15:09.408 It's a, it's a hard thing to describe the way, the value of having a 00:15:09.408 --> 00:15:10.818 good note taking system feels. 00:15:10.818 --> 00:15:13.488 But like, when you come out on the other side of it and you start writing 00:15:13.488 --> 00:15:17.298 things down, the task of recall suddenly doesn't become, I need to 00:15:17.303 --> 00:15:18.738 remember every detail about this thing. 00:15:18.798 --> 00:15:21.318 All you need to remember is that you wrote it down and you can find 00:15:21.348 --> 00:15:21.528 it. 00:15:21.805 --> 00:15:24.273 And that's something that, the scale that comes with that is pretty 00:15:24.273 --> 00:15:25.768 tremendous and also really helpful. 00:15:25.768 --> 00:15:28.168 Like in three years when I wanna look up what you and I talked about 00:15:28.168 --> 00:15:29.854 today I certainly won't remember. 00:15:30.419 --> 00:15:33.479 Right in my brain, but I will remember that we had this talk and I can jump 00:15:33.479 --> 00:15:34.919 back into my notes and chase it down. 00:15:35.664 --> 00:15:36.084 Drew White: Yep. 00:15:36.934 --> 00:15:38.944 Mike Bifulco: It's, it's super cool and I feel like there's a lot of 00:15:38.944 --> 00:15:43.534 psychology that goes into it, like both the people's hesitance to take on note 00:15:43.534 --> 00:15:47.046 taking, but then like the personal style, the workflow, the things that trigger 00:15:47.046 --> 00:15:49.986 peoples like, I need to take a note about this, or I need to keep my list 00:15:49.991 --> 00:15:54.115 of tasks in this versus you know, am I summarizing an article or, or writing 00:15:54.115 --> 00:15:57.865 down a note about, I dunno, some hack I wrote in my code, Whatever the case may. 00:15:58.000 --> 00:15:59.410 Yeah, I, I like all of that stuff. 00:15:59.410 --> 00:16:02.170 It's really interesting to think about and like you must be building a very kind 00:16:02.170 --> 00:16:03.970 of generic tool set to do that, right. 00:16:04.960 --> 00:16:05.410 Drew White: Yeah. 00:16:05.410 --> 00:16:09.962 I mean, like our whole concept is, is giving Users, people a 00:16:09.962 --> 00:16:12.452 default place to write to that they don't have to worry about. 00:16:12.452 --> 00:16:15.222 Like, it's, it's essentially a log, you know, it's. 00:16:15.692 --> 00:16:17.342 Date timestamped log. 00:16:17.762 --> 00:16:19.202 That includes everything that you've got. 00:16:19.202 --> 00:16:23.642 So if you even remember roughly what happened during the day, you should 00:16:23.642 --> 00:16:27.338 be able to find the note that, that you took down which is pretty awesome. 00:16:27.398 --> 00:16:31.885 And so sort of the next big thing for us is further removing we'll call 'em 00:16:31.885 --> 00:16:35.995 barriers to capture cuz we believe that that's the most important thing. 00:16:36.055 --> 00:16:40.123 And so as we continue to expand, Develop the product. 00:16:40.123 --> 00:16:44.683 One of our, our major items on our roadmap is like integrations and our api. 00:16:44.818 --> 00:16:47.368 So the whole idea of being able to. 00:16:48.163 --> 00:16:53.984 Send content from somewhere into stash pad or even have that content 00:16:53.984 --> 00:16:58.784 automatically be imported into Stash Padd as a note in the right place when 00:16:58.784 --> 00:17:02.007 you need it is really exciting for me. 00:17:02.075 --> 00:17:05.675 I don't know what it was like, you know, at any of your, your previous employers. 00:17:05.732 --> 00:17:09.962 But like one of the biggest things moving to the tech world that kind of knocked 00:17:09.962 --> 00:17:12.252 me off my socks is the tech stack. 00:17:12.258 --> 00:17:15.198 I was not prepared for that whatsoever. 00:17:15.198 --> 00:17:18.678 Like even coming from like my own business where I was using quite a 00:17:18.678 --> 00:17:21.528 few different tools for different purposes and managing those things. 00:17:21.984 --> 00:17:29.574 Like my bookmark folder for like just dash padd tools is, is, is pretty big. 00:17:29.574 --> 00:17:34.644 Like we, we've got at least 12 separate tools that we use for different purposes. 00:17:35.004 --> 00:17:39.174 And while that's great and all of them work really well, sometimes it's hard. 00:17:39.221 --> 00:17:42.803 Particularly in my position, it's difficult. 00:17:43.538 --> 00:17:47.048 Hey, remember where that comment that someone made that you 00:17:47.048 --> 00:17:48.488 need to reference came from? 00:17:48.968 --> 00:17:51.398 Or like, was it in Slack? 00:17:51.398 --> 00:17:54.908 Was it in, was it a conversation, Was it a thread in Slack, like going 00:17:54.908 --> 00:17:56.228 back and doing all of that stuff? 00:17:56.618 --> 00:18:00.854 Or was it a slab or any, any number of, of different locations 00:18:00.884 --> 00:18:01.874 it could have come from. 00:18:02.564 --> 00:18:07.835 And so the ability to have this sort of automated notes dashboard which 00:18:07.835 --> 00:18:09.545 is, you know, the ultimate goal here. 00:18:10.655 --> 00:18:15.365 Really, really appealing to me to be able to create some smart stacks that 00:18:15.425 --> 00:18:18.695 give you the information you need from the resources that you use, the 00:18:18.695 --> 00:18:22.802 tools that you use and combine that with capturing your own notes from 00:18:22.802 --> 00:18:27.631 one-on-ones meetings, code reviews, all of that stuff is really just feels 00:18:27.691 --> 00:18:30.541 like I would like to have that today. 00:18:31.426 --> 00:18:32.956 Mike Bifulco: Sure . Sure. 00:18:33.376 --> 00:18:33.946 Yeah. 00:18:34.396 --> 00:18:37.547 I what I'm really interested to hear about too is like, this is, this 00:18:37.547 --> 00:18:41.987 is one of those great cases where almost certainly you will be using 00:18:42.047 --> 00:18:43.307 Dash pad as you're building it. 00:18:43.322 --> 00:18:45.990 You know, probably both personally and as, as a company, as a 00:18:45.990 --> 00:18:46.950 team, whatever that looks like. 00:18:47.920 --> 00:18:51.250 Can you tell me a little bit about what, what your, like what your, I dunno. 00:18:51.610 --> 00:18:53.770 Your dog fooding process is like, and some of the things that 00:18:53.770 --> 00:18:54.880 your team does with Stash pad. 00:18:55.800 --> 00:18:59.086 Drew White: Yeah, so our dog fooding process is pretty strong. 00:18:59.086 --> 00:19:03.485 Everybody on our team is very opinionated and also very thorough and not afraid 00:19:03.490 --> 00:19:08.375 to speak up, which is hugely beneficial both from like a development standpoint, 00:19:08.375 --> 00:19:11.855 but honestly from a design standpoint, which I spend a lot of time in. 00:19:12.725 --> 00:19:17.620 And so we all use stash pad very differently. 00:19:17.687 --> 00:19:19.813 It's actually pretty fascinating. 00:19:20.863 --> 00:19:25.273 Often, like, we'll go into like a spec review or something like that and this 00:19:25.273 --> 00:19:28.753 person will say, You know, I use this this way, that makes perfect sense to me. 00:19:28.753 --> 00:19:31.693 And then like I'm looking at 'em like, I don't use it that way at all. 00:19:31.693 --> 00:19:35.653 Like I, my mindset, my brain map is, is different. 00:19:35.653 --> 00:19:37.123 My mental model is different. 00:19:37.783 --> 00:19:42.223 And so what's fascinating is we've, we've kind of engineered 00:19:42.493 --> 00:19:47.468 the flexibility to match different mental models into the app which has. 00:19:48.643 --> 00:19:52.913 I don't know, kind of just eye opening for me, but I use it all the time. 00:19:53.023 --> 00:19:56.974 Primarily with code reviews, design reviews, that sort of thing. 00:19:56.979 --> 00:19:57.994 Spec reviews. 00:19:58.049 --> 00:20:00.769 I have several, one-on-ones every week. 00:20:00.829 --> 00:20:05.359 I like to use it for them so I can both remember what we talked about, but also 00:20:05.449 --> 00:20:08.676 kind of measure my own progress and be able to go back and look at some 00:20:08.676 --> 00:20:09.966 of the things that we talked about. 00:20:10.236 --> 00:20:10.806 I also do it. 00:20:11.604 --> 00:20:13.704 Basically things that I want to bring up. 00:20:13.704 --> 00:20:16.254 I also use it as a drafting tool, believe it or not. 00:20:16.275 --> 00:20:20.977 Cuz it does support markdown and so I can do some longer form notes if I need to. 00:20:21.044 --> 00:20:23.114 So I do like it as a drafting tool. 00:20:23.114 --> 00:20:24.663 They render really, really nicely. 00:20:24.664 --> 00:20:29.241 And then I also use it as like a lockbox for data. 00:20:30.501 --> 00:20:32.451 I know I'm gonna need in perpetuity. 00:20:32.573 --> 00:20:36.383 I can keep a place for quick, quick info that I just need to access all the time. 00:20:36.387 --> 00:20:39.689 And I can know that everything in there is always gonna be there 00:20:39.869 --> 00:20:42.029 forever in the shape that it needs. 00:20:42.029 --> 00:20:43.388 So and that's how I use it. 00:20:43.393 --> 00:20:45.068 I also use it as a task manager. 00:20:45.146 --> 00:20:49.520 We've got a great sort of to-dos system and hierarchical 00:20:49.670 --> 00:20:51.480 todos, which is super awesome. 00:20:51.507 --> 00:20:53.897 So like you can create a stack of todos. 00:20:54.837 --> 00:20:58.167 Which is within another stack of, to-dos, that stack itself can 00:20:58.167 --> 00:21:00.447 be a to-do so on and so forth. 00:21:00.476 --> 00:21:00.999 So Yeah. 00:21:00.999 --> 00:21:03.199 it works really well for keeping me organized. 00:21:03.954 --> 00:21:06.474 Mike Bifulco: I can imagine as an engineer or someone working on a product 00:21:06.474 --> 00:21:09.564 team, whether you're an engineer or a designer or a product manager, whatever, 00:21:09.594 --> 00:21:12.456 whatever your role is there's a lot of value in keeping yourself organized 00:21:12.516 --> 00:21:14.296 and, and making this thing work for you. 00:21:14.367 --> 00:21:18.428 Can you tell me a little bit about the storage plan for for Dash pad? 00:21:18.428 --> 00:21:20.189 So right now, is it local only? 00:21:20.189 --> 00:21:20.969 Is it cloud synced? 00:21:21.059 --> 00:21:24.179 Is it something you use with like Dropbox or Google Drive or something like that? 00:21:24.239 --> 00:21:24.599 How does it work? 00:21:25.424 --> 00:21:27.524 Drew White: Yeah, right now it is local only. 00:21:27.591 --> 00:21:31.084 That was a decision we made based on some, you know, early feedback that 00:21:31.084 --> 00:21:34.850 we had from engineers and, you know, companies being very, we, we want people 00:21:34.850 --> 00:21:39.148 to be very have the option to be very private about their, their data and 00:21:39.148 --> 00:21:40.576 not be sinking to and from the cloud. 00:21:40.637 --> 00:21:41.327 But as. 00:21:42.212 --> 00:21:45.119 Right now we are I don't wanna put an actual date on it. 00:21:45.119 --> 00:21:48.662 We do have a date for release, but just in case things get pushed, you know 00:21:48.662 --> 00:21:51.524 plus a couple of days, minus a couple of days, whatever the case is, we are 00:21:51.529 --> 00:21:55.742 rolling out sync in the very near future which will give users an opportunity 00:21:56.102 --> 00:22:00.795 to not only have data on multiple computers, but also we'll be rolling 00:22:00.795 --> 00:22:03.145 out our mobile app about the same time. 00:22:03.164 --> 00:22:05.384 So yeah, we'll have access to. 00:22:06.584 --> 00:22:11.514 Again, the whole idea is further reducing that, that, that friction capture. 00:22:11.521 --> 00:22:17.021 So yeah, we'll, we'll have cloud sync available for a pretty small monthly fee. 00:22:17.141 --> 00:22:19.301 I don't know exactly what it is off the top of my head. 00:22:19.302 --> 00:22:20.350 But it's very reasonable. 00:22:20.414 --> 00:22:23.724 And I think there will be a, a certain number of. 00:22:24.984 --> 00:22:27.894 Um, like free sync sort of things. 00:22:28.164 --> 00:22:30.684 And then the community version, which is non sync will be 00:22:30.684 --> 00:22:32.364 free forever in perpetuity. 00:22:32.694 --> 00:22:32.904 Mike Bifulco: Yeah. 00:22:32.904 --> 00:22:33.531 Very cool. 00:22:33.531 --> 00:22:37.517 Is there, so is Stpa taking the perspective that notes are a sort 00:22:37.517 --> 00:22:40.451 of personal trove of information or is there collaborative features? 00:22:41.471 --> 00:22:41.651 Drew White: Yeah. 00:22:41.651 --> 00:22:45.911 So I mean, our whole thing through this has been, there are so many 00:22:45.911 --> 00:22:47.781 tools out there for teams, right? 00:22:47.830 --> 00:22:47.920 And. 00:22:49.555 --> 00:22:53.075 There's very little for managing your own daily work. 00:22:53.129 --> 00:22:57.569 And so we have taken this stance that Stash pad is for you, not for 00:22:57.569 --> 00:23:01.409 your team, not for your manager, not for even necessarily the 00:23:01.409 --> 00:23:06.124 enterprise, although I'm sure we will have enterprise level customers. 00:23:06.124 --> 00:23:09.404 The idea is it's for the engineer, it's for the user and. 00:23:10.589 --> 00:23:14.681 That being said, we actually do, we used to have a a web app version, 00:23:14.681 --> 00:23:18.381 which was like version negative 0.1 or whatever you wanna call it. 00:23:18.381 --> 00:23:23.854 That does have a collab feature that we still to this day use for retro. 00:23:23.854 --> 00:23:29.104 And it is easily the greatest platform for something like that 00:23:29.104 --> 00:23:31.054 that we have experimented with. 00:23:31.069 --> 00:23:32.422 We've tried basically everything else. 00:23:32.422 --> 00:23:34.942 We always end up coming back to the old web app. 00:23:36.427 --> 00:23:39.127 So, yeah, there may be plans for, for adopting some of that 00:23:39.127 --> 00:23:40.880 functionality in the future as well. 00:23:41.390 --> 00:23:41.870 Mike Bifulco: Sure. 00:23:42.290 --> 00:23:44.668 Yeah, I think it's, it is a good angle to take or an interesting 00:23:44.673 --> 00:23:45.658 angle to take, certainly. 00:23:45.719 --> 00:23:48.349 I think a lot of folks gut response might be that like having a 00:23:48.349 --> 00:23:51.259 team collaborative tool is maybe the, the table stakes for them. 00:23:51.259 --> 00:23:55.807 But in practice, all of the companies I've worked at that have reached any like. 00:23:56.227 --> 00:23:59.437 Reasonable team size of, call it five people or greater, tend to 00:23:59.437 --> 00:24:01.147 standardize on like, what is easiest. 00:24:01.147 --> 00:24:03.593 So and, and by that I mean like things that they've probably already 00:24:03.593 --> 00:24:04.973 paid for within the enterprise. 00:24:05.043 --> 00:24:08.848 So that may be Google Talks or Jira or GitHub or like the things that 00:24:08.848 --> 00:24:10.559 are sort of built into that process. 00:24:10.679 --> 00:24:13.859 But what I also like about this is that by keeping it local and for yourself, 00:24:13.859 --> 00:24:17.609 like it, it, it's a way for you to keep your information, to grow your own sort 00:24:17.609 --> 00:24:21.442 of stack of knowledge and, and to build upon your own set of notes in a way. 00:24:21.842 --> 00:24:22.862 That is you flavored. 00:24:22.862 --> 00:24:23.702 I think that's really interesting. 00:24:23.702 --> 00:24:25.922 And obviously you can still collaborate with your team right there. 00:24:25.922 --> 00:24:28.132 There are you know, ways to get information out of this thing. 00:24:28.132 --> 00:24:29.227 It's not a one way valve. 00:24:29.227 --> 00:24:30.264 Yeah, yeah, 00:24:30.349 --> 00:24:32.539 Drew White: And I think just based on our experience using the web 00:24:32.539 --> 00:24:36.439 app, I can't see that not making it in like the collaborative use 00:24:36.444 --> 00:24:38.239 case, not making it into the app. 00:24:38.239 --> 00:24:40.811 It's just, it's too good to like pass on. 00:24:40.811 --> 00:24:43.327 I just don't know where it lives on our roadmap today. 00:24:43.419 --> 00:24:44.919 Mike Bifulco: The perpetual startup challenge. 00:24:44.979 --> 00:24:45.249 Yeah. 00:24:45.579 --> 00:24:47.589 When, When is it the most important thing to build? 00:24:48.129 --> 00:24:48.549 Drew White: That's right. 00:24:48.554 --> 00:24:52.599 And I think a lot of people like, I mean, we're a team of seven, so 00:24:52.604 --> 00:24:54.339 like we're, we're pretty small. 00:24:54.449 --> 00:24:57.929 And so we've gotta kind of pick and choose our priorities, particularly 00:24:57.929 --> 00:25:03.419 this close to our launch, you know, And so we're trying to deliver one thing, 00:25:03.569 --> 00:25:06.209 but a perfect one thing, and then we'll 00:25:06.269 --> 00:25:06.659 Mike Bifulco: of course. 00:25:06.929 --> 00:25:07.679 Drew White: the next thing, you know? 00:25:08.339 --> 00:25:08.699 Mike Bifulco: Yeah. 00:25:09.269 --> 00:25:12.621 So I'm, I'm curious to probe in a little more about the sort of API 00:25:12.621 --> 00:25:15.923 layer that you teased, cuz I know that the, the team listening to this 00:25:15.923 --> 00:25:17.303 will definitely be interested in that. 00:25:17.513 --> 00:25:18.233 What does that look like? 00:25:18.233 --> 00:25:19.923 What are the sort of hooks you're thinking about? 00:25:19.965 --> 00:25:21.765 You know, opening up APIs for. 00:25:22.275 --> 00:25:26.672 Drew White: Yeah, I mean, primarily the initial sort of main function of the API 00:25:26.672 --> 00:25:29.716 is intended to expand capture essentially. 00:25:29.716 --> 00:25:35.776 So the ability to send information to stash pad from basically any tool or 00:25:35.776 --> 00:25:39.590 any product, any project that you're working on would be the primary function. 00:25:40.460 --> 00:25:43.967 You may have some other functionalities that come after that. 00:25:43.989 --> 00:25:47.169 But yeah, I mean our whole thing is that the easier you can make capture, the 00:25:47.169 --> 00:25:49.869 more likely people are gonna take notes and the better they're gonna retain 00:25:49.929 --> 00:25:53.209 information and then ultimately the better they're gonna be able to work. 00:25:53.209 --> 00:25:56.327 So yeah, the, that, that'll be the, the primary function there. 00:25:56.327 --> 00:25:59.477 We're still kind of working through the details on this. 00:25:59.477 --> 00:26:01.097 This is on our current roadmap. 00:26:01.164 --> 00:26:04.039 And I know it's coming probably way quicker. 00:26:05.344 --> 00:26:09.050 We're gonna be . It feels like we're doing a lot of things right now. 00:26:09.050 --> 00:26:12.413 But they're all very good things and we're executing at a pretty high level. 00:26:12.413 --> 00:26:14.963 And so we're trying to maintain that, that momentum. 00:26:15.413 --> 00:26:18.855 So I, I'd be surprised if this wasn't out early first quarter next year. 00:26:18.855 --> 00:26:18.915 Yeah. 00:26:20.025 --> 00:26:20.445 Mike Bifulco: Yeah. 00:26:20.475 --> 00:26:20.875 Cool. 00:26:20.893 --> 00:26:21.848 I, I know your team. 00:26:21.848 --> 00:26:23.558 So you said it's a seven person team. 00:26:23.601 --> 00:26:25.371 And I, I know you've done some of the engineering work. 00:26:25.371 --> 00:26:28.182 I'd imagine there's a few engineers that, that work on the product. 00:26:28.217 --> 00:26:30.183 Can you talk a little bit about what dpad is built with? 00:26:31.413 --> 00:26:31.863 Drew White: Yeah. 00:26:31.980 --> 00:26:35.522 Stash Padd is built with react type script in El. 00:26:36.447 --> 00:26:40.256 Has our primary shippable form, and then the mobile app 00:26:40.256 --> 00:26:42.566 will be React native actually. 00:26:42.617 --> 00:26:45.707 So yeah, it's been, it's actually been quite a joy to work with. 00:26:45.712 --> 00:26:46.277 I know. 00:26:46.330 --> 00:26:51.508 Our one of our engineers who kind of does a lot of the electron work 00:26:51.538 --> 00:26:53.794 definitely has some grapes about it. 00:26:53.803 --> 00:26:58.200 He just wrote a blog post that'll be up on our website probably at the end of today. 00:26:59.280 --> 00:27:01.740 But yeah, it's, it's, it's a great tool and there's a reason 00:27:01.800 --> 00:27:03.120 that it's so widely used. 00:27:03.143 --> 00:27:08.045 And so even with some of the, the push and pull I think it's still a 00:27:08.045 --> 00:27:09.695 good option, particularly for desktop. 00:27:09.905 --> 00:27:15.035 And it allows us to ship to Linux and Windows and Mac kind of all in one go. 00:27:16.250 --> 00:27:16.595 Mike Bifulco: Sure. 00:27:16.689 --> 00:27:19.904 Yeah, I feel like the electron's perpetual thing is that as it 00:27:19.904 --> 00:27:21.194 does more people want more. 00:27:21.464 --> 00:27:24.614 And you know, early on the conversation was mostly around performance. 00:27:24.614 --> 00:27:27.595 You know we can't ship a Chrome browser for everything. 00:27:27.607 --> 00:27:30.297 But to be honest, I think that's become less of a problem in recent 00:27:30.297 --> 00:27:33.147 years as computers have gotten better, as electronic self has gotten 00:27:33.147 --> 00:27:35.607 better, as Chrome has gotten more lightweight and all those things. 00:27:35.727 --> 00:27:37.317 Or chromium, I guess not quite chrome. 00:27:38.142 --> 00:27:38.412 Drew White: Right? 00:27:39.222 --> 00:27:41.472 Mike Bifulco: And it's interesting to pair that with React Native too, 00:27:41.472 --> 00:27:44.586 which historically has had similar things and has gotten tremendously 00:27:44.586 --> 00:27:45.996 further along in the past few years. 00:27:45.996 --> 00:27:49.546 Like building for React native now is so much easier than it was in 2016. 00:27:49.557 --> 00:27:51.327 It's, it's a much, much more capable thing. 00:27:51.382 --> 00:27:52.495 It's cool to see that coming around. 00:27:53.260 --> 00:27:57.010 Drew White: Yeah, I did some stuff with React native, just personal projects 00:27:57.031 --> 00:28:00.859 a couple years ago, and I haven't had an opportunity to work on any of 00:28:00.859 --> 00:28:06.261 the mobile stuff Now my role is, is pretty widely split between design, 00:28:06.261 --> 00:28:09.850 engineering, dev, re and then some higher level stuff, product stuff. 00:28:09.909 --> 00:28:13.774 So, but any chance I, I get an an opportunity to, to work in app 00:28:13.798 --> 00:28:17.596 I relish those opportunities cuz that's sort of what drove me to 00:28:17.596 --> 00:28:19.306 this place in the first place. 00:28:19.786 --> 00:28:21.727 But yeah, the we're, we're pretty excited. 00:28:21.732 --> 00:28:24.690 We've got some, some really good things coming out and I think they're 00:28:24.750 --> 00:28:26.460 happy with React native today. 00:28:26.512 --> 00:28:29.824 The engineers are don't, I haven't heard much in the way of complaints, 00:28:29.829 --> 00:28:31.074 so that's always a good sign. 00:28:32.304 --> 00:28:33.847 Mike Bifulco: Yeah, I'll say certainly. 00:28:33.847 --> 00:28:34.320 Cool. 00:28:34.320 --> 00:28:37.657 So Drew what other things haven't we touched on with Stash pad that, 00:28:37.657 --> 00:28:39.834 that folks might be interested in if they haven't tried it yet? 00:28:40.894 --> 00:28:44.351 Drew White: Yeah, I think for me it's the, it's really the speed of the 00:28:44.351 --> 00:28:46.301 thing that makes it so much better. 00:28:46.301 --> 00:28:49.721 Like I, I've been a long time, I, I kind of bounce, I mentioned it earlier, 00:28:49.721 --> 00:28:54.516 I bounced around from app to app for years notes app that is and ultimately 00:28:54.516 --> 00:28:59.546 landed on Apple Notes just because of its, Sort of nativity as it were. 00:28:59.546 --> 00:29:02.355 But it was always kind of like somewhat of a compromise for me. 00:29:02.422 --> 00:29:06.093 But I've actually just, I mean, within the last six months have 00:29:06.093 --> 00:29:10.233 like fully transitioned into stash pad as a whole, primarily 00:29:10.233 --> 00:29:11.553 because of the speed of the thing. 00:29:11.583 --> 00:29:14.393 It's just uncanny, like I think all of our. 00:29:15.078 --> 00:29:18.438 Basic actions are sub hundred milliseconds or something like that. 00:29:18.438 --> 00:29:23.673 Like even like loading a massive list of notes is just ridiculously fast. 00:29:23.926 --> 00:29:28.519 And the other real concept behind it, like particularly if, if you're like a 00:29:28.519 --> 00:29:32.119 developer and you know, the importance of keeping your hands on the keyboard, 00:29:32.124 --> 00:29:37.264 like the thing is, is well set up you can navigate everything create, 00:29:37.294 --> 00:29:40.894 delete, you know, whatever you want to do without ever leaving the keyboard. 00:29:40.894 --> 00:29:44.879 And like, Super familiar, sort of key bindings that make a lot of sense. 00:29:44.902 --> 00:29:48.277 And so that's like another huge thing for, for me in particular. 00:29:48.277 --> 00:29:51.534 We also have like a shortcut, like a global OS shortcut. 00:29:51.534 --> 00:29:54.804 So you can open it up while you're, so you're working in BS code or your 00:29:54.809 --> 00:29:57.234 ide and you gotta take a quick note. 00:29:57.414 --> 00:30:00.444 You can just open it up without ever touching the mouse and bounce over to 00:30:00.444 --> 00:30:04.764 it, dump your note, go back to work, and just basically eliminate that context 00:30:04.769 --> 00:30:07.734 switching sort of moment right there. 00:30:08.864 --> 00:30:12.178 Yeah, I think if anybody hasn't tried it that's listening. 00:30:12.178 --> 00:30:12.988 It's certainly worth it. 00:30:12.988 --> 00:30:15.928 It's free, so no harm, no foul. 00:30:16.025 --> 00:30:19.132 You can download it, our website wws-padd.com. 00:30:19.180 --> 00:30:19.930 And yeah, give it a try. 00:30:19.935 --> 00:30:20.380 Let us know. 00:30:20.385 --> 00:30:23.371 And we're super active on our Discord server. 00:30:23.371 --> 00:30:26.952 We love getting feedback from, from users even when they hate it. 00:30:27.012 --> 00:30:29.382 Like we got railed the other day by some guy. 00:30:29.382 --> 00:30:32.142 He just didn't like the interface like whatsoever. 00:30:32.202 --> 00:30:32.562 And. 00:30:33.177 --> 00:30:37.167 He was, he must have sent like 10 emails yesterday, I think. 00:30:37.206 --> 00:30:38.826 But that's good stuff for us. 00:30:38.826 --> 00:30:40.446 Like, it's, it's good feedback. 00:30:40.446 --> 00:30:42.003 Like we don't mind it at all. 00:30:42.004 --> 00:30:46.888 So yeah, I, I definitely think everybody, if you're using Evernote or Notion or 00:30:46.888 --> 00:30:50.690 Apple Notes or Ulysses or any of the other ones it's worth giving a try. 00:30:50.708 --> 00:30:52.273 It's a different experience for sure. 00:30:52.276 --> 00:30:54.736 You may like it, you may not, but we hope that you. 00:30:55.831 --> 00:30:56.131 Mike Bifulco: Yeah. 00:30:56.191 --> 00:30:56.431 Cool. 00:30:56.521 --> 00:30:59.198 I'll, I'll make sure to drop a link in the show notes here too. 00:30:59.250 --> 00:31:01.580 And if people wanna chase you down, Drew, where's the best place to find you? 00:31:01.592 --> 00:31:04.352 Drew White: Usually you can find me at the Whitewater Center in Charlotte, 00:31:04.352 --> 00:31:06.558 North Carolina or at Fonta Flora. 00:31:07.248 --> 00:31:08.139 Also Shta no. 00:31:08.139 --> 00:31:09.309 Yeah, you can find me on Twitter. 00:31:10.059 --> 00:31:10.519 Atul. 00:31:10.559 --> 00:31:14.159 I don't, I, I, I spend a lot of time there observing, but I'm 00:31:14.159 --> 00:31:16.259 not like a huge content creator. 00:31:16.259 --> 00:31:17.099 I like watching. 00:31:17.699 --> 00:31:19.649 Mike Bifulco: there's a lot to observe on Twitter these days too. 00:31:20.179 --> 00:31:20.669 Drew White: Yeah. 00:31:20.669 --> 00:31:21.069 Yeah, yeah. 00:31:21.193 --> 00:31:23.893 And then, yeah, that's probably the easiest way to get ahold of me, 00:31:23.943 --> 00:31:24.303 Mike Bifulco: Cool. 00:31:24.363 --> 00:31:24.723 Right on. 00:31:25.023 --> 00:31:26.523 Well, Drew, thanks so much for hanging out today. 00:31:26.523 --> 00:31:28.293 It's been really cool talking about STA pad. 00:31:28.325 --> 00:31:31.123 Yeah, come back anytime, especially once you're starting to talk about 00:31:31.123 --> 00:31:34.484 like opening up the API taps we'll have lots of people with very interesting 00:31:34.484 --> 00:31:37.364 opinions for you, and I'm sure you'll get a, a bit of an onslaught in your 00:31:37.364 --> 00:31:40.124 discord for people with feature requests and things like that in the near 00:31:40.364 --> 00:31:40.904 Drew White: Perfect. 00:31:40.904 --> 00:31:43.484 We'll create your own channel just for you guys. 00:31:43.889 --> 00:31:44.339 Mike Bifulco: Right on. 00:31:44.699 --> 00:31:45.549 Thanks so much, Drew. 00:31:45.549 --> 00:31:46.121 We'll talk soon. 00:31:46.616 --> 00:31:47.066 Drew White: thanks Mike. 00:31:47.351 --> 00:31:47.931 Mike Bifulco: See ya. ### Testing Spectral Style Guides with Jest URL: https://apisyouwonthate.com/blog/testing-spectral-style-guides-with-jest/ Last updated: 2023-03-24T12:20:28.000Z Readers of this blog will be familiar with the concept of [Automating API style guides](https://apisyouwonthate.com/blog/automated-style-guides-for-rest-graphql-grpc) implemented with programable API linters like [Spectral](https://github.com/stoplightio/spectral/?ref=apisyouwonthate.com). When combined with the power of [API Design-First](https://apisyouwonthate.com/blog/api-design-first-vs-code-first), these API style guides can power the core of your API design reviews, improving consistency and reducing the frequency of bikeshedding, rewrites, and security issues. As the rulesets that power these API style guides are getting increasingly powerful, and they're being deployed at bigger organizations, it might make sense to add some test coverage to them to make sure they actually work, right? In the last article we talked about distributing an API style guide as a Spectral ruleset via NPM, which is a great solution for the techies. For the less technical (or non-Node savvy) there are loads of other ways to distribute and manage your Spectral rulesets, including the new "Style Guide Projects" from Stoplight, which give you a shiny GUI editor on top of the same Spectral engine. For this article we'll pick up where we left off with the [APIs You Won't Hate: Style Guide](https://github.com/apisyouwonthate/style-guide?ref=apisyouwonthate.com), which was written as an NPM module *with no tests* 😱. Before getting started I thought I'd google around to see what the community was up to, and I found [Test-Driven Development for Spectral with Jest](https://www.jvt.me/posts/2021/12/22/spectral-jest/?ref=apisyouwonthate.com). The idea was great, but the implementation felt like it could be improved. I decided to go for something closer to what Spectral does to test its core OpenAPI and AsynAPI rulesets. A little core package reuse, and a little copy paste, and I have Jest test suite I am quite pleased with. ## 1.) Declaring Dependencies We'll be using Jest, and as I'm using TypeScript theres a few other packages to grab. ```shell npm install -d --save jest ts-jest @types/jest ``` The ruleset we created in [Distribute Spectral Style Guides with NPM](https://apisyouwonthate.com/blog/distribute-spectral-style-guides-with-npm#publish-your-spectral-style-guide-to-npm) was already using the following two packages, but for those following along at home lets make sure we've mentioned this: ```shell npm install --save @stoplight/spectral-formats npm install --save @stoplight/spectral-functions ``` Finally, because I switched to TypeScript since the last article, we'll need to grab a package full of various types Stoplight uses across multiple packages. ```shell npm install --save @stoplight/types ``` Now lets put these dependencies to work. ## 2.) Adding Test Helpers Lets make a folder for our tests, and pop a helper in there to make them work. ``` mkdir -p __tests__/__helpers__/helper.ts touch __tests__/__helpers__/helper.ts ``` Copy and paste the code below into that new `helper.ts` using whatever editor you prefer. ```ts // Author: Jakub Rozek, Stoplight.io // License: Apache License 2.0 // https://github.com/stoplightio/spectral/blob/develop/packages/rulesets/src/__tests__/__helpers__/tester.ts import { IRuleResult, Spectral, Document, Ruleset, RulesetDefinition } from '@stoplight/spectral-core'; import { httpAndFileResolver } from '@stoplight/spectral-ref-resolver'; import myRuleset from '../../src/ruleset'; export type RuleName = keyof Ruleset['rules']; type Scenario = ReadonlyArray< Readonly<{ name: string; document: Record | Document; errors: ReadonlyArray>; mocks?: Record>; }> >; export default (ruleName: RuleName, tests: Scenario): void => { describe(`Rule ${ruleName}`, () => { const concurrent = tests.every(test => test.mocks === void 0 || Object.keys(test.mocks).length === 0); for (const testCase of tests) { (concurrent ? it.concurrent : it)(testCase.name, async () => { const s = createWithRules([ruleName]); const doc = testCase.document instanceof Document ? testCase.document : JSON.stringify(testCase.document); const errors = await s.run(doc); expect(errors.filter(({ code }) => code === ruleName)).toEqual( testCase.errors.map(error => expect.objectContaining(error) as unknown), ); }); } }); }; export function createWithRules(rules: (keyof Ruleset['rules'])[]): Spectral { const s = new Spectral({ resolver: httpAndFileResolver }); s.setRuleset({ extends: [ [myRuleset as RulesetDefinition, 'off'], ], rules: rules.reduce((obj, name) => { obj[name] = true; return obj; }, {}), }); return s; } ``` This big chunk of code should *just work* so long as the path to your ruleset is correct, which by default is `src/ruleset.ts` with `src/` and `__tests__/` living next to each other. My `src/ruleset.ts` looks like this: ```ts import { enumeration, truthy, undefined as undefinedFunc, pattern, schema } from "@stoplight/spectral-functions"; import { oas2, oas3 } from "@stoplight/spectral-formats"; import { DiagnosticSeverity } from "@stoplight/types"; export default { rules: { // Author: Phil Sturgeon (https://github.com/philsturgeon) 'paths-kebab-case': { description: 'Should paths be kebab-case.', message: '{{property}} should be kebab-case (lower case and separated with hyphens).', given: "$.paths[*]~", then: { function: pattern, functionOptions: { match: '^(/|[a-z0-9-.]+|{[a-zA-Z0-9_]+})+$' } }, severity: DiagnosticSeverity.Warning, }, // ... snip ... // Author: Nauman Ali (https://github.com/naumanali-stoplight) 'no-global-versioning': { description: 'Server URL should not contain global versions', message: 'Using global versions just forces all your clients to do a lot more work for each upgrade. Please consider using API Evolution instead. More: https://apisyouwonthate.com/blog/api-evolution-for-rest-http-apis.', given: "$.servers[*].url", then: { function: pattern, functionOptions: { notMatch: '\/v[1-9]+' } }, formats: [oas3], severity: DiagnosticSeverity.Warning, } } }; ``` You can just copy paste these for now, or write something new if you're familiar with [writing rulesets](https://meta.stoplight.io/docs/spectral/ZG9jOjI1MTg5-custom-rulesets?ref=apisyouwonthate.com), up to you. Either way lets move onto writing the tests. ## 3.) Writing Tests for Spectral Rules I create a different test file for each rule in the ruleset, e.g.: `__tests__/no-global-versioning.test.ts`, then use the new `testRule()` method defined in the helper. ```ts import { DiagnosticSeverity } from '@stoplight/types'; import testRule from './__helpers__/helper'; testRule('no-global-versioning', [ { name: 'valid case', document: { openapi: '3.1.0', info: { version: '1.0' }, paths: { '/': {} }, servers: [{ url: 'https://api.example.com/' }] }, errors: [], }, { name: 'an API that is getting ready to give its consumers a really bad time', document: { openapi: '3.1.0', info: { version: '1.0' }, paths: { '/': {} }, servers: [{ url: 'https://api.example.com/v1' }] }, errors: [ { message: 'Using global versions just forces all your clients to do a lot more work for each upgrade. Please consider using API Evolution instead. More: https://apisyouwonthate.com/blog/api-evolution-for-rest-http-apis.', path: ['servers', '0', 'url'], severity: DiagnosticSeverity.Warning, }, ], }, { name: 'an API that got massively out of control as usual', document: { openapi: '3.1.0', info: { version: '1.0' }, paths: { '/': {} }, servers: [{ url: 'https://api.example.com/v13' }] }, errors: [ { message: 'Using global versions just forces all your clients to do a lot more work for each upgrade. Please consider using API Evolution instead. More: https://apisyouwonthate.com/blog/api-evolution-for-rest-http-apis.', path: ['servers', '0', 'url'], severity: DiagnosticSeverity.Warning, }, ], }, ]); ``` This is not standard Jest, and I'd really appreciate help making it better and releasing it as a package, but it gets the job done. There's the happy path, and a few examples of API description documents that will return "errors", which includes warnings, info, etc. It's using `expect.objectContaining` under the hood so you can use other Jest assertions in there and specify as many properties as you like. Check out the type definitions in your IDE if you need more guidance there. One last thing before we can run Jest is creating a `jest.config.js` file: ```js module.exports = async () => { return { preset: 'ts-jest', testPathIgnorePatterns: ['__helpers__'], testEnvironment: 'node', globals: { 'ts-jest': { useIsolatedModules: true, }, }, }; }; ``` With this done you shoud be able to use the `$ jest` command to run the test suite. I was having some trouble with PATH so had to use `npm exec jest` but that got annoying so I updated `package.json` to make `$ npm test` work: ```json "scripts": { "test": "jest" }, ``` When I run the test suite ``` $ npm test > @apisyouwonthate/style-guide@0.0.0 test > jest PASS __tests__/no-global-versioning.test.ts Rule no-global-versioning ✓ valid case ✓ an API that is getting ready to give its consumers a really bad time ✓ an API that got massively out of control as usual Test Suites: 1 passed, 1 total Tests: 3 passed, 3 total Snapshots: 0 total Time: 1.114 s ``` ## 4.) Continuous Integration To make sure nobody is contributing bugs we can make the test suite run for all pull requests. I use GitHub Actions for simple packages like this to avoid having "another SaaS" involved, but there is no special GitHub Action magic happening. ``` mkdir -p .github/workflows/ touch .github/workflows/test.yml ``` In that new file add something like this: ```yaml name: Run Tests on: [push] jobs: test: name: Test runs-on: ubuntu-latest steps: - uses: actions/checkout@v3 - uses: actions/setup-node@v3 with: node-version: 18 - run: npm ci - run: npm test ``` That will install all the dependencies needed, and then run `npm test`. And there we go! Go forth and make amazing rulesets. Shape how your APIs will be built *before* they are built. Improve things over time by adding more rules after they're built. Make your API design reviews easy, and focused on useful domain specific knowledge instead of bickering about style. When they're published add a link to the [stoplightio/spectral-rulesets](https://github.com/stoplightio/spectral-rulesets?ref=apisyouwonthate.com) repository so we can keep track of them, and eventually make a marketplace where people can mix and match styles and standards, like picking a "JSON:API" ruleset, a AWS Gateway ruleset, and the Acme Corp style guide (because you like their style). ### API Environmentalism with Alexander Karan of Climate Clever URL: https://apisyouwonthate.com/podcast/api-environmentalism-with-alexander-karan-of-climate-clever/ Last updated: 2023-04-15T13:00:25.000Z ## Show Notes - Climate Clever - https://www.climateclever.org/ - Alexander's recent Article on APIs You Won't Hate: [Modern API deployment options in the cloud](https://apisyouwonthate.com/blog/modern-api-deployment-options-in-the-cloud) - [Scope 1 and Scope 2 Inventory Guidance](https://www.epa.gov/climateleadership/scope-1-and-scope-2-inventory-guidance?ref=apisyouwonthate.com) \- US EPA - [Alexander Karan (@alexanderkaran\_) on twitter](https://twitter.com/alexanderKaran%5F?ref=apisyouwonthate.com) Thank you so much to our sponsors: - **Lob:** [https://lob.com/careers](https://lob.com/careers?ref=apisyouwonthate.com) - **Treblle**: [https://treblle.com/apisyoulove](https://treblle.com/apisyoulove?ref=apisyouwonthate.com) #### Transcript \[00:00:00\] **Mike:** Hello friends, we're here for another episode of API. So you won't hate, my name is Mike, your cohost. I'm here with the ghost of Phil who was supposed to join me for this call. But he is in the middle of the woods somewhere using an Atari for a modem. And we were unable to get a good enough connection to actually be able to chat. So you'll have to accept my word that he's still alive and kicking and doing well and . Planting things all around the UK there and it seems to be in good Nick. And in the meantime, we will have a chat with my new friend Alex, Karen from climate clever. Alex is working on a product that is super near and dear to my heart and certainly our hearts and kind of the mission with APIs. We won't hate here. It is a online platform for carbon neutrality. It's an API first platform that has quite an interesting story in that. For being built. And as many of the folks who listen to the podcast are probably familiar with carbon neutrality and climate aware projects is something that we really love talking about here. So I'm really keen to hear from and learn about climate clever from Alex, \[00:01:00\] \[00:02:00\] \[00:02:27\] **Mike:** alex, how are you doing? \[00:02:28\] **alexander\_karan:** I'm good. I'm good. Thank you for having me, Mike. \[00:02:31\] **Mike:** Of course. Yeah. You're very welcome. So why don't you tell me a little bit about yourself kind of before climate clever the story of, of leading up to how you got there. And then, then maybe we can talk a little bit about climate clever itself. \[00:02:43\] **alexander\_karan:** Okay. Cool. So yeah my name's Alexander I've been a software engineer for a very long time. Now time has lost all meaning. I'm mostly a JavaScript engineer, so I pretty much play with JavaScript all the time. I love it and hate it. It's a love, hate relationship. Yeah, full climate. Clever. I \[00:03:00\] worked for a dev agency for quite a few years, heading up the development and design teams there. Which was actually how I met the founder of climate clever because we built. first iteration of the product at the dev agency I worked for. Yeah. And then I guess before then, you know, a few years of freelancing doing no development and iOS development, actually, I used to do that. Don't really touch iOS anymore. Got quite frustrated. Yeah, And then rewind even further. I guess I used to do development inside a few like big old school companies back home in the UK, which is where I'm originally from. And then pre that I wanted the musician, you know, so. That's like sort of the brief condensed history. So, you know, accidentally fell into programming slowly took over my life. You know, and then here I am as a co-founder of a startup. \[00:03:51\] **Mike:** Yeah, I think that's probably a familiar story for many of us. Funny enough, you, you obviously, the listeners can't see this at all. You can't see it because of the angle of my \[00:04:00\] camera here, but if I. Toss my door aside. I'm I may a reformed musician, a and M current programmer myself. And I've spent a lot of time in the JavaScript ecosystem, building things and products and companies and doing consulting and all that. So it sounds familiar. It's one of those things where I feel like you stick your head up and you say, "oh yeah, I know JavaScript," and then suddenly you fast-forwarded 10 - 15 years. And you've been doing it for a living and solving other people's problems for quite a long time. Very cool that you were able to get new position to actually work on something that is meaningful to you. And, and in particular, probably coming from more of a consultative sort of studio role. It's really nice to be able to own the thing you're working on and have some you know, personal value tied up in the thing that you're building too. So, yeah, that's really cool. \[00:04:44\] **alexander\_karan:** 100. \[00:04:44\] **Mike:** So what, what yeah, so tell me about climate clever itself. What's what's the elevator pitch. \[00:04:51\] **alexander\_karan:** Ooh, the elevator pitch. I'm the wrong person to do the elevator page. I guess \[00:04:54\] **Mike:** elevator pitch. \[00:04:56\] **alexander\_karan:** they weren't allowed to speak to the edge of D is \[00:05:00\] the elevator pitch. Okay. So we are a API first platform, but we have a pretty cool. Front end. We help businesses in particular measure and report on their comp and remissions and take action to reduce their footprint. We also do help schools and homes because that's where we originally started. But our main focus is sort of SMEs and the business sector. \[00:05:23\] **Mike:** Yeah. And so how has that taken shape over time? What was your initial sort of first version of climate? Clever? What did. \[00:05:31\] **alexander\_karan:** Okay. So rewind to 2012 where I did even know about climate clever and didn't really care about climate change all that much. So the founder of climate clever and also my partner Vanessa Allan's helps certify the first carbon neutral school in Australia. She came in with a group of other people as well. And, you know, they help to measure the emissions of the schools. They helped reduce their footprint, so saved them save them some money as well. And then also offset \[00:06:00\] what was left for. And that got certified as the very first carbon neutral school in Australia. And my partner and the CEO is a, an amazing academic. So obviously that's where it all came from. And then she spent the next few years teaching and lecturing and researching doing all those things that I don't quite understand the academic Stu. And eventually she decided. Bone a pilot where she had like a bunch of schools. I think it was like 10 schools in total. And they ran through the same program. If it was this around 2016, 2017, and you know, all on Excel sheets and Google docs. Right? Like it was so, you know, which I just think you can't knock Excel sheets. They're bloody fantastic. You can build full fledged apps in there. That's great. So, and they sort of took all these schools on the reduction program. And then she got some grad money together and that's when she approached the devil and see, I worked at, and so we built the first iteration of the climate clever platform, which was just targeted at the schools and \[00:07:00\] we would help them measure their electricity, gas, water in LPG. And. Usage and then we'd help them audit their buildings. Like there's some really good stories from that period. Like I think there was this, one of the schools was like, you know, why is our gas bill so much higher than all the other schools when they were comparing themselves? And it turned out because they had seven gas leaks, you know? So like they just didn't know. So it wasn't until he could see all the data and they could compare. And that's sort of what led rise to this little mantra that we have inside the company, which is you can't manage what you don't measure. So yeah, so we sort of, you know, so we grew you know, the platform to more schools. And then I think after about a year I left the dive agency and I had this nice, cushy, easy job lined up. And somehow Vanessa convinced me to come work for And then obviously fast forward to now we're in a relationship, we've run the company together. We have a baby together, so it's turned out pretty, pretty fun. And yeah, so, I mean, obviously COVID came out of nowhere in 2020, so that sort of put a little, you know, hold all plans because all the \[00:08:00\] schools started shutting down. So we pivoted into homes to help homes cause we could do the same. And yeah, and so we've managed to save a lot of emissions and money for homes and schools across Australia, which has been amazing. But as we sort of came through COVID we realized, you know, being a startup need that more sustainable business model and schools and homes aren't necessarily the best way to do. So we sort of moved more into the business sector because as you know, laws change, especially around the EU and their laws that they're looking at that does affect supply chains globally. We've started looking at businesses and helping businesses, not just measure and reduce, but actually report in a really transparent way. And so that's really, what's important because we want to help people. But we called the emissions accurately and also be able to report them in a really transparent way. So there's none of this greenwashing nonsense and they can say, Hey, look, this is my footprint. This is what we're doing. And this is how it was calculated. And so all of that. Is, although it's like \[00:09:00\] this wonderful kind of pretty UI that's, to be honest, I don't build most of that anymore. There's other engineers in my team that are far better at front end development than I am. But it's all built on an API, so it's an API first platform and there's quite a lot of reasons for that. But yeah, so that's kind of a little bit of the journey, what we do and stuff. \[00:09:17\] **Mike:** Yeah, I can imagine all of the listeners of the podcast peeking up like Prairie dogs. Once you start mentioning API is towards the end there. I'm, I'm a bit interested in the product side of this too. It sounds. The value of something like this, it speaks for itself. What I'm kind of interested in is where the actual connection between the technology and measuring happens, like, are you using off the shelf hardware to measure emissions? Is it something where you have nice access to energy and, and gas intake for individual buildings through services in Australia? Or are you doing custom installations on, on buildings? How does. \[00:09:55\] **alexander\_karan:** Sorry, I just got out the left. Would you like easy access to bills that would make \[00:10:00\] my life a dream? That's that's not how utility companies play bowl in Australia. I don't know about the rest of the world, but they're like, no, It's our data. \[00:10:09\] **Mike:** Yeah. \[00:10:11\] **alexander\_karan:** So there's, there's quite a few different ways that we approach this because as we move into. There's more than just simple emission streams, like electricity, gas, and water. You've got like, you know, for software companies, server usage, you've got equipment, you've got liters of diesel. You know, all this stuff has to be recorded. So always, always the first method of data entry is manually. And we create really easy to use endpoints and UI forms to achieve this. So that is always the first thing, no matter what emission we're talking about, we always create a manual entry to deal with it. And the reason we do that is because, you know, some people don't have access to digital. Recordings. They just track things. There's so many different things, especially when you're \[00:11:00\] dealing with a country like Australia as well, where internet isn't always the best, you know, regional Australia is very different to Metro Australia. So you've got to always have these simple methods of doing things. The second way we do is CSV uploads. And that's because of. Tons of businesses, you know, just record stuff and CSV, you know, in Excel sheets. So we have some CSV mapping that we do inside the API end points. So we have API end points that take CSVs and they do that. W our next options, we have quite a few options. Our next option is for utility companies that have easy to use websites, we've built scraping services that go in log in on your behalf and grab all your data. They are incredibly fun to maintain because like every week there's like another bug like, oh, they've slightly changed. The only reassuring thing is noticing when utility company websites go down more than your own product, that is quite oh, that's just the utility companies just down again. That's all it is second time this week. \[00:12:00\] So that's quite interesting. And then the final method, that's still in development at the moment that we've been working on for a while with the help of Microsoft, actually, Microsoft has been really helpful here. in Perth. Well, ahead of off. We've been working on a PDF data extraction. So we've got a few different methods, one we'll log into your utility company and grapple the PDFs. A two, we'll give you a special email where you can drop the PDFs into the email or three. You can just upload it. And we abstract the data out of there. So Microsoft got this wonderful. Sort of PDF form recognized station AI that really just, you know, is the icing on the cake. Yeah. So those are kind of the different methods we use for collecting data. Now we can use life meters, like people like, oh Yeah. live meters. They're really good, but, well, that's only a few utilities, but also like we don't need to know your usage on an hourly basis to work out your footprint. So it's, you know, why collect all that data is, \[00:13:00\] is sometimes the answer to that. \[00:13:03\] **Mike:** Sure. Yeah. It's funny as you're describing this, the thing I, I I've had a couple of thoughts, but one thing that I'm thinking is that forcing manual entry for people on some level is also a really nice way to get people, to recognize that they're using something that by, by forcing you to sit and write down something, you're enforcing a good habit that, Hey, I've actually burned a liter of diesel fuel, or what I've done here is I've turned on the air conditioning for 12 hours and I'm writing that down every day. It makes it. A lot more internalized at that point. I'm sure that's creating some good behaviors. \[00:13:30\] **alexander\_karan:** Yes, it does. I mean, like I have to be honest, like when I first joined climate clever full time, I was blown away by how complex Bill's work, because I'd never even looked at my own bills. And it's amazing how many people have never looked at their own bills as well. And when they actually start looking at them, like, especially when we're working in schools, the amount of schools that just paid them and ever even looked. You know, it was quite alarming. Cause then that's when you find these inconsistencies, then when you find things that are wrong is by actually looking at them. So Yeah. \[00:14:00\] it does, it doesn't force good habits. I mean, it's the same thing with the actions that we suggest to people. I think like 60% of the things that we suggested. Right. So they're not actually, they're just behavioral change. And I always used to think to myself, I was like, there's no way, but just that, that people don't know about this stuff, you know, like it's no way that they're not turning off the lights, you know, like this is crazy, but the thing is, they're not, you know, like schools in Australia, you know, they, they have a six week. Over summer, which is like the end of December and the whole of January for us. And like schools just don't switch everything off, you know? And so we started developing these switch off campaigns where the schools would go and switch everything off. Like all the students were going involve and they'd save like five grand over that six week break, you know? And it's just like, And it's so crazy how like, just behavior change is just enough to get you going, you know, like, not even like, oh, switch to renewable energy \[00:15:00\] solar. So, Oh, get solar panels, like just, just simple stuff that people, and I always thought that people must know about this and they don't, you know, they just don't even think about it. \[00:15:11\] **Mike:** Sure. Yeah. It's not like you're going into office buildings and convincing everyone to become vegan. One by one, you can get a lot done by using a little bit less water and fuel and electricity here and there. The other thing that came to mind for me too, is something in my current job. I talked to a lot of founders of startup. And it's really interesting to hear commonalities between the success stories of growing startups and how they got to where they are. And one thing that seems to be pretty common with a lot of startups that are successful is that they've done a lot of the work manually for a very long time before they started automating it. Because that lets you get familiar with the pain that you're trying to solve. And so as you mentioned before, having this whole thing within Excel spreadsheets for a long time, probably. Your self and Vanessa are really good perspective on what the actual thing is that needs to be automated here, where you look for easy optimizations and things like that, and good ways to make recommendations. And it's, it's been my experience, at least from talking to \[00:16:00\] folks that that tends to be something that helps people design a product. That's actually solving a real problem as opposed to a perceived problem. \[00:16:08\] **alexander\_karan:** Yeah. 100%. I'm always like, give me the low code. Look, it pains me to say, cause I'm a developer, right. And I don't want people to come from. But I'm like, you know, always go for the load code option, never built something straight away, you know, it's, it's, it's always like, you know, Excel sheets. You can do a lot with Excel sheets and Google docs. You can do a lot just whacking up a static website with a survey and it, you know, it's always good to get people using stuff about building a whole heap of stuff first, because you do discover pain points. You know, that there's, there's totally and also you see if it's worthwhile. And then when you hit the limits of the Excel sheets, then you can start converting it. But, but even then I think. It's always keep it simple. You know, I think I've, there's so many times where I think, you know, that sort of program or part of my brain has gone, oh, let's be clever and do this and we can add this. \[00:17:00\] And I'm just like, and then I get the inspire and I'm like, man, we just keep it. Every, every iteration should be simple, just a slightly simpler, you know, small improvements rather than like big, crazy things, you know? \[00:17:14\] **Mike:** Yeah yourself and your team in the future. We'll both be happy with you if you've done it that way, too. \[00:17:21\] **alexander\_karan:** 100%. I always, I always notice, like when you're coding on something, you know, you past self has done a really good job. Like, you know, past Alex, he did a great job today or like other days you're like, seriously, what was passed out. Cause they gain. If I ever get ahold of him, you know, like. \[00:17:39\] **Mike:** Yeah, there are days where I wish I could buy my past self a beer. There are also days where I wish I could knock my past self over the head and write a little more documentation, something like that. I've I've been in the middle of resurrecting, a very old project that's years and years old. And I'm amazed by that dichotomy. Like some of the things that I did are so clever and well-documented and thoughtful and helpful for me now. And some of the things are just like \[00:18:00\] assumptions, assertions, whatever you wanna call. I just assumed I would remember this little trick forever or the way that things were set up and interconnected forever. And let me tell you from eight years since I definitely don't remember how things worked way back when the, the, the simple path would have been helpful in a lot of those cases, too. So can you tell me about climate clever now and maybe what's next? What are, what what's happening in your world now and what are kind of the goals that you might have going through? \[00:18:24\] **alexander\_karan:** Okay. So, well, obviously our biggest focus is becoming businesses. So sort of our focus for the next year or two is to one grow the emission streams that we measure. So every time we measured just in case I haven't explained it already, an emission stream is like electricity or gas or diesel, or, you know, taxi rides to the airport, anything that produces emissions. So sort of, sort of, you know, goals add more ambitions. Make reporting easier. So like one of the cool things that we do is we generate these really cool, like, you know, custom reports that reflect all your data in a really \[00:19:00\] easy to read and consumable way to expand those reports and just get more businesses on the platform and also, you know, help look at supply chain emissions, I think more than anything else. So not just like, Hey, I'm at business. Here's my emissions, but Hey, I'm a business inhale all the scope, three emissions of, you know, like all my suppliers and how they affect me as a business. And what action were taken on those, because now obviously I'm not a carbon scientist, so never quote me 100% accurately. On all the definitions, but like most people care about scope, one scope, two missions and scope three, not so much, but scope three is really starting to become a big focus. So yeah, we really want to help tackle that issue as well through? supply chains. \[00:19:47\] **Mike:** Yeah. So for the uninitiated give me the non climate scientists definition of what scope one, two, and three are roughly. You don't have to be exactly correct. \[00:19:58\] **alexander\_karan:** Oh my God. Okay. I \[00:20:00\] really suck at this. \[00:20:02\] **Mike:** I do too, which is why I put it on you. \[00:20:08\] **alexander\_karan:** Oh, man. I really do Scott. Okay, so scope one and two are sort of like emissions that, you know, you use directly you know, from burning, right? So like they're direct emissions. So scope one is 100% direct. So that would be like, you know, company vehicles company facilities, things that you are 100% responsible for. But Bernie scope to kind of like indirect emissions. So like, like, like a really good example would be like electricity, the transport emissions of getting it to you. That would be. Scope to right. Cause it's, it's in direct emissions from you using something and then scope three, a like completely indirect emissions. So like, you. know You know, if, if you're a business measuring your emissions and you're counting your staff's commute to work, right. \[00:21:00\] Because that's not a company. It's not a scope, one emission, it's a scope three emissions, right? So it's, it's another form of indirect emissions, but it's like once that you're less responsible for, and then this is where all the weird carbon accounting stuff comes in and like what emissions fall under what scope and who's responsible for them and all this other stuff. And that's where it becomes really, really complicated. And that's why we have an API \[00:21:24\] **Mike:** Yeah, that's, that's really cool. That's one of those things where I don't think we were having discussions about this, you know, 10 years ago this kind of thoughtfulness and sort of introspection about where emissions are coming from. And, and the fact that these terms, you know, scope one, two, and three are becoming. Normal in the, in the vernacular around the world is, is a good sign in itself. Even if it's not something we've completely conquered yet. So let's talk a little bit about the API side of what you've built in climate clever. Obviously don't divulge any trade secrets, anything like that, but I'd love to hear a little bit about how you've built climate clever, some of the tools you used, some of the API building strategies that your team uses, that sort of a thing. \[00:21:59\] **alexander\_karan:** \[00:22:00\] Okay. Cool. Alright, cool. So we have, it's a node API because of course I love Java script. And it's, it's ho it's currently hosted on AWS, but it is currently being moved to Azure. So there's an interesting story bound behind that as well. So we use ECS on AWS, which is like serverless containers. So we've got like five microservices all running in Docker containers. You know, simple node API and each, and that just all sits behind API gateway and load balancer, nothing crazy fancy, you know, it's not like, well, you know, we're not a huge, massive company. That's got requests coming in from all over the world, so we don't need any of that crazy stuff. But we take a very sort of design driven approach to API development. Become more and more and more important as we've moved on. So the reason there's an API is because when we started the platform, when we started building the very first iteration, we were like, well, this is all great, having this nice UI for. Things like smaller business, smaller schools, but as we get bigger, you know, bigger \[00:23:00\] companies, they're not going to be like great another app to use. They're going to be like, give me the API so we can just integrate it into our systems. And as we work with supply chains, they're going to say the same thing. So we decided to build it on an API. Level. And this means that I could work really closely with the carbon specialists and sort of, you know, bash back and forth API documentation on how this thing was going to work and all the terminology that went into it. And also with, you know, the other developers and designers in the team, Hey, look, we're thinking about doing it like this. This is all the information is going to include. We actually use a tool, which I believe Phil, you still work for, which is. So we design, you know, we, you know, we designed a mock all our API end points in there first, before we even touch any code. And we sort of, you know, circle that around the team. And it's also a really transparent way of staying open with our customers because we have documentation in there about where all our carbon cows come from and how they. \[00:24:00\] Because we're all about trying to be transparent. Like it's, it's complicated and you might not understand all of it, but we want to try and be as transparent as possible. And then there's kind of two sides to the API. So there's the external side, which our app uses and which other customers use. So we D we do have an integration into a few other platforms which is pretty cool. But, yeah. so that's the external side and then there's the internal side. Because one of the most complicated things. About carbon factors or carbon, you know, calculations and methodologies is that there's all these different factors that you use. So what's a factor. A factor is kind of like a number that you times your consumption or usage of something by to work out the carbon footprint. And there's all these different like rules and. If and else and switch things, statements that decide what factor gets used. So what we ended up building was actually, we built a system that allowed the, you know, \[00:25:00\] carbon specialists to enter the factors and then I not have to write much code. And then it sort of used logic to help assign that to people's emissions data. And that's what we spend a lot of time building over the last. Is a system that does that and it, and it's a really transparent system as well. So like when anybody enters an emission, it tells you who entered it and then it can't be deleted once it's entered, it can only ever been updated. And it's, so it's, it's kind of like, you know, full transparency, but without blockchain, because we can be transparent without it, you know? So, So \[00:25:33\] **Mike:** like you've, you've come up with a fairly elegant approach to an accountability system that that will scale also. It's a really delicate point and something that I feel like is almost even understated that you decided that the API was going to be important from early on. I can imagine a world where you happened on that realization. Maybe a little later in the process or perhaps too late in the process where you'd have to go and re-engineer. Probably much easier to have done that, to begin with and also to start it with a \[00:26:00\] design first API implementation to. \[00:26:03\] **alexander\_karan:** Yeah, it definitely had its pros. I must admit, I think it's, it's not 100% of conscious decision by those definitely that okay. Your business is. Two it'll be easier. But it's also because I like building API is right. There's a little bit of that in there as well. I'd love to just be like, yes, I foresaw everything, but yeah, it's it really does help. It does help cause the API documentation as well. It's like sort of Bible about how this whole thing works. Right? Cause carbon accounting is complicated. Like I thought development was complicated, but like carbon accounting is like, it's, it's just development on steroids. Like there's no format standard. Everything can be different. There's this. Th there might be some fast and hard rules for a few types of emissions streams, but then once you move to another one, that completely goes out the window. It's different in every country as well. It's not the same. So there's like to try it. You have to \[00:27:00\] develop like a very simple generic system in the API that can handle these different factors. So then when the other side of the API presents that the client or I, or our app, you know, it's easy to understand the consume, you know, Yeah. It's it's, it's tricky. It's I it's, it's it's funny. Like, I always think what it does is actually quite simple, but it's been the hardest thing I've ever had to build. You know, it just times is numbers together, but like architecture wise, it was so difficult. \[00:27:33\] **Mike:** Yeah. Yeah. I suppose if it was an easy problem to solve, we'd have it nicked already, you know, \[00:27:38\] **alexander\_karan:** Yeah. And testing was definitely a big problem too, as well, but that was the. \[00:27:44\] **Mike:** Sure. Yeah. Yeah, certainly. I wanted to ask you a couple of questions about an article you wrote recently for APS. You won't hate for people who are listening, you may recognize Alexandra Keller Karen's name because he wrote an article for us called modern API deployment options in the cloud where we talked a bit \[00:28:00\] about just what it's like to spin up an API in 2022, as a, as a sort of a Javascripty developer, like Alexandra and I happened to be. I would love if, if you don't mind just talking a little bit about that too. It seems that we've come a very long way from setting up an FTP account on a server somewhere and, and setting up, you know, your $5 GoDaddy, VPN and setting things up that way. From a very, very basic standpoint, what do you see as the reasons to go with using some of these more modern deployment options now, as opposed to setting up and managing the server? \[00:28:31\] **alexander\_karan:** Well ease, right? Like I feel, you know, this, so there's something called the full stack fallacy. Isn't that an engineer can do everything. And well I think we can, if we use the tools, right. Like, but also I actually let me rewind a little bit, I think looking at front end development, right. Which is undoubtedly hard, has all these really amazing tools now where it's like, Don't worry about this. You focus on solving the problem \[00:29:00\] that you're trying to solve and we'll deal with everything else, right? Like, you know, tools like Netlify and versatile. I'm going to be honest. I don't actually remember how to deploy a website or buy a bat without those tools. I know file share was involved in an AWS thing. And I honestly, I honestly completely forgotten, you know, and that is fantastic, right. Because at the end of the day, as an engineer, I'm not in. To be super clever with my code or like, oh, I've used this tech stack over that tech stack, I'm there to solve business problems. Right? I mean, I mean, yes, I get to do some fancy stuff with code one and they're solving business problems, but that is my main point for existing. And so that's why I, I wrote the blog about the new modern tools is like, we need this. Thinking it's like really great. And I'm taking off points because I can do all this stuff with the server? I can manage all this stuff myself. You know, I don't want to have to do that. You know, at the \[00:30:00\] end of the day, I wanted to deliver value to the company I worked for and for its customers. Right. That's the most important thing. And I can do that quicker if I don't have to manage everything about a server or manage everything about the database. \[00:30:14\] **Mike:** Yeah, definitely. And you don't have to have all that expertise and I'm sure there's a plethora of fall on effect. Like you don't have to be a security expert for some vague Unix version that your server is. In some FTP somewhere or manage those passwords, all that stuff for, for the uninitiated, the Netlify and versatile and I guess other other services like that, that use similar deploy options. The big difference between the old FTP world and this world is. With those FTP servers and the file share stuff that you're talking about with AWS. Like it literally felt in a lot of cases, like you were just crafting some files on your local machine, maybe merging them in, into a get repost somewhere and then dumping all those files onto that machine almost manually. In some cases, maybe add some nice automations. Can you talk a little bit about the deploy pipeline for \[00:31:00\] Netlify or sell what that works or what that feels like as a developer, as opposed to the FTP \[00:31:04\] **alexander\_karan:** Well, it's it's it's it feels great. So I guess so we, we, we used to use Netlify, we're actually on, for sale at the moment. But I can talk to you a little bit about how, like we use for Purcell for our front end, you know, like basically we do a PR into the main branch and all the tests get run, and then it deploys a previous. In the PR that you can go check out and mess around with, right. So that's not even the deployment, right. That's just, just a PR. So you can go and see the code just by clicking the link in the PR. So that's the first thing there's just gold and it's the best thing in the entire world. The second thing is you want to deploy a website, a web app on versatile. You create your account, you connect the get hub repository, you select the branch and then that's it. Every time you push changes to that branch it deploys. And then you can have deploys to other branches in your code. It's so fast and seamless. \[00:32:00\] It's fantastic. And, and because it integrates, since you get hub repository, you get feedback on, you know, every PR you do every get, you, push you to whatever millions of different settings, but it's just, it's fast. It's. It's great. It's I don't even think about it. And I don't have to think about scaling. I don't have to think about delivery networks and all this other stuff. And I, I know my environmental variables are safe because I put them inside for cell, you know, don't need to store them anywhere else. It's just this so many upsides. \[00:32:32\] **Mike:** Yeah, definitely. I think in a past life, if you found yourself working on a, something that became enterprise grade and you needed to have deployments in three times zones or in a bunch of different geographies, you would have suddenly needed to like have a relationship with a bunch of data centers and figure out where your literal physical computer is that you're deploying code to and, you know, devise some really clever way of activating one versus the other. If, if the HTTP request was coming by via some IP route versus the other, and now you're. \[00:33:00\] Merging a pull request. You're even, you're not even emerging, you're opening a pull request and this thing is getting opened and distributed in ways that scales so far beyond just the team that you manage, just you writing code you are just, like you said before, taking advantage of all these things that are making your life easier as the person building the API or the front end too. It's really, really cool to see. And so your, some of your product, at least the API layer is that built using serverless. \[00:33:28\] **alexander\_karan:** So obviously, like I said before, it's currently on ECS, which is serverless containers, but we are moving it to Azure which we're using serverless functions for. So, you know, we, we started the migration about two months ago. We'll be finished in a few more months. But yeah, we are moving it all to serverless functions for quite a few different reasons. Actually. The first one is sustainable. Because although serverless containers are sorta serverless, they still do run a little bit all the time. So even though they're really, really tiny, they still have a footprint, right? So there are times, you know, cause our app, isn't \[00:34:00\] something that you log into every day, every hour. So it does have quiet periods. So it, it makes sense to use serverless. functions. And also, you know, the sustainability goals of Microsoft are. Very admirable and they align with our sort of company ethos. Right. So that's another reason for the move. And then the final thing is there just as your functions are just really, really great. They're fantastic. They got TypeScript support. They integrate into the S code so well, like when they came in and demo day for us and showed us how easy it was. I think we set up the four function app. Like 30 seconds integrated automatically into get hub for automated deploys and another 30 seconds without two button clicks. And then like the local testing is just another click because it's plugged into BS code, you know, it was an extension. So it was really fantastic. \[00:34:51\] **Mike:** Yeah, that's really impressive. That's very cool. That's something I need to spend some time playing with and I can think of one particular team mate of mine. Who's going to really want me to, \[00:35:00\] to give Atlas Azure serverless stuff ago. Okay. One more question for you. And before we'll kind of wrap things up here I've heard a lot of talk recently about serverless functions, but also the hype seems to be building up around edge functions. I'm under the impression that edge functions are not always the right sledgehammer for problems that we're solving. What do you know about edge functions? Maybe? When are they the right choice for something? What's the difference between edge and serverless, especially from a a sustainability and sort of climate and energy use focus thing. Does edge make more sense than serverless? Is there complications there that we might not be aware of? \[00:35:35\] **alexander\_karan:** So I think, I think edge does make more sense until you throw database into the mix. Right? Like, I, I th I think as I think as developers, we're always like, oh yeah, it's a new thing. It's hit, let's go for it. Right. Which I do feel guilty too sometimes, you know, but sometimes it's the right decision. Like, you know, spelt was like this really cool hip thing. And then we tried it and it made so much sense and now we all love it. Right. But like, so. I \[00:36:00\] wouldn't necessarily build a full fledged API on serverless on sorry, on edge functions just yet. Because like, if my database is, you know, only in one location, you know, th the, the edge functions still need to make a big round trip cool to that database. So it would make much more sense for me to just have an API on serverless from. Right in the same data center that my databases. But then the thing is if you will serving lots of static content or. Intercepting request to your API, then it might make more sense to use edge functions. And this is where like also complications come in, right. Because the way I'm starting to see it is it's probably a good thing to have a mix of the two, right. Probably a good mix to have, you know, maybe this. Your main API on serverless functions in a data center next to your database. Although there are serverless and edge databases now as well that you can use and then like edge functions \[00:37:00\] for intercepts and stuff like that, you know, there's, you could protect images. On your server or, you know, on a CDN making sure you're all, if you could intercept other requests and, you know, cash the response in a much harder way or chat, or like, I think a real good example is like intercepting a request, checking the location and returning a different response. You know, things like that, which can be done on the edge, which we much better suited than being done on your API. So maybe a mix I think is the two. But I think that might change, you know, it, it would definitely be. More efficient to have stuff closer to your user. But then, but then this is the question, right? So this is always, this is always the million dollar question. I remember when I was talking about react and graph QL, right. And people said like, we ask great for everybody because we all have similar front end issues to Facebook, but graph QL. Isn't great for everyone because not everyone is Facebook and not everyone has the same backend problems that face. And I think that kind of? applies right. \[00:38:00\] Cause, okay. It's super cool. Having your API and database on the edge network all over the place, but are your users all over the place? Are you global? Are you will wide? Is that kind of like a waste of time? I mean, I guess it doesn't run, you know, if, if it's not being used, but you've still got to deploy it and send it there. Right. So, you know, th th there's always those questions. I think those things that we. W will you always forget about it back over like, oh yes, we need to prepare for infinite scale. And I'm like, yeah, but there's probably no more than 20 to 50 companies that need that crazy scale in the entire world. Right. So always would never scale up at the start, you know? \[00:38:42\] **Mike:** Yeah, that's a brilliant answer that, that Is the sum of all of my fears. I think summarized pretty well. If I had to guess having seen some of the history of serverless stuff, my thought currently is that eventually we'll be able to deploy serverless slash edge functions without thinking \[00:39:00\] about whether it's a serverless or edge function and have the. Hosting do some introspection and decide what's the most appropriate for the code that's being run, but you are right. Like my little blog that I post about designing things and philosophical earth things probably doesn't need to be deployed for instant use in Sri Lanka and Cleveland and Paris and Miami and New York. Because if I'm being honest, most of my readership comes from the U S Canada, Australia and the UK. And you know, I would love it if everyone in the world. It's an access to the words that I have to write, but it's absolutely not the case. And there is a real cost to deploying, you know, my nonsense to a thousand servers around the world. Every time I do a change to my site to \[00:39:38\] **alexander\_karan:** Yeah, 100%. I was just going to say like the, the also the more distributed things become, you know, like just, although microservices are a good thing, you know, for distribution makes testing harder, it makes explaining things harder. Like it makes all these things, other things that we don't think about harder that are also really important, you know? And. Extra layer of distribution. I'm \[00:40:00\] sure it will create some interesting bugs. It works fine for me, but on that edge function over there, it's a little different, you know, so Yeah, \[00:40:08\] **Mike:** Yeah, I'm just looking forward to the day where I create a software bug. That means I have to go to Fiji to test it out and see what's going wrong. \[00:40:18\] **alexander\_karan:** that's, that's a real reason. Edge functions exist, you know, like, oh, well I have to go to Hawaii to, to figure this out either. \[00:40:26\] **Mike:** Yeah. They're all on the edge of an island somewhere that also happens to have drinks served with umbrellas. And, um, Alexander, what's the best way for people to find you and climate clever online? Where can they go? \[00:40:37\] **alexander\_karan:** Okay, so you can find me on Twitter. It's just Alexander cran under school at the end. They can find climate clever, which is just www climate clever.org. We officially only in Australia at the moment but always willing to hear from people around the world. Yeah. And that's why they can find. \[00:40:54\] **Mike:** Fantastic. Well, thanks so much for chatting with me today. I really had a great time talking to you looking forward to \[00:41:00\] hearing more from you, hopefully in the future out Cron. Thanks for your time. Have a great. \[00:41:04\] **alexander\_karan:** Thank you so much for having me, Mike, it's great to be here. \[00:41:07\] **Mike:** Right on. ### Distribute Spectral Style Guides with NPM URL: https://apisyouwonthate.com/blog/distribute-spectral-style-guides-with-npm/ Last updated: 2023-03-29T14:10:58.000Z At some point in the growth of an organization there will come a time where you've got too many different APIs with random different data formats, naming conventions, security schemes, and generally it feels bizarre and frustrating trying to call any API because you cannot share any code between API consumers. If you're not there right now, good, you're reading this in time. If you're already there, yeah it sucks doesn't it, and this article will help you work your way out of that pit over time. API Style Guides have always been a crucial part of API Governance, but they usually come in the form of a giant manifesto, written by the most ~~opinionated~~ experienced person at your organization. Often they're ignored, and when they are read it's impossible to remember all the decisions, especially as some may change over time. Now [API style guides can be automated](https://apisyouwonthate.com/blog/automated-style-guides-for-rest-graphql-grpc), and if you're describing APIs with [OpenAPI](https://apisyouwonthate.com/blog/theres-no-reason-to-write-openapi-by-hand) you can use excellent tools like [Stoplight Spectral](https://github.com/stoplightio/spectral/?ref=apisyouwonthate.com), which is the natural successor to the previously popular [Speccy](https://github.com/wework/speccy?ref=apisyouwonthate.com) (a tool I built at WeWork but was promptly abandoned when I left). Both tools let you define all the rules you need for your API Style Guide in YAML or JSON, and distribute them via publicly via URLs, or smush them into the filesystem somehow, but Spectral offers a fantastically powerful new way of building and distributing your style guides: you can build them in JavaScript, write tests for the rules, use whatever libraries you feel like, and distribute via NPM so that you can version your API Style Guides. To demonstrate I've created the [APIs You Won't Hate: Style Guide](https://github.com/apisyouwonthate/style-guide?ref=apisyouwonthate.com), which will help everyone make their APIs better using tips we've been sharing on this blog and in the books for the last decade. ## Create an API Style Guide in JavaScript Let's use the JavaScript format for the ruleset instead of the JSON or YAML formats that people have traditionally used when working with Spectral. ``` mkdir style-guide cd style-guide ``` Then we will need to create some dependencies. ```shell npm install --save @stoplight/spectral-functions npm install --save @stoplight/spectral-formats ``` If you use Spectral already, you may be familiar with the [core functions](https://meta.stoplight.io/docs/spectral/ZG9jOjExNg-core-functions?ref=apisyouwonthate.com) and [core formats](https://meta.stoplight.io/docs/spectral/4dec24461f3af-open-api-rules?ref=apisyouwonthate.com) it bundles with. Don't worry if not, we'll get into that in a moment. Create a JS file that will hold your ruleset, possibly called ruleset.js or spectral.js, whatever you feel like calling it. ```shell touch ruleset.js ``` In that file we're going to start building the structure for the ruleset. ```js import { enumeration, truthy, falsy, undefined as undefinedFunc, pattern, schema } from "@stoplight/spectral-functions"; import { oas2, oas3 } from "@stoplight/spectral-formats"; export default { rules: { // ... your rules go here } } ``` This is a bit more bootstrapping than if you just start writing random YAML into a file, but seeing as it's just plain old JavaScript you get all the benefits of working in your favourite IDE, which means type hinting, auto-complete, it'll let you know if your function is invalid, and you can import your own functions to power whatever rules you can image. Let's make some rules. ```js import { enumeration, truthy, falsy, undefined as undefinedFunc, pattern, schema } from "@stoplight/spectral-functions"; import { oas2, oas3 } from "@stoplight/spectral-formats"; export default { rules: { 'api-home': { description: 'APIs MUST have a root path (`/`) defined.', message: 'Stop forcing all API consumers to visit documentation for basic interactions when the API could do that itself.', given: "$.paths", then: { field: "/", function: truthy, }, severity: 'warn', }, 'api-home-get': { description: 'APIs root path (`/`) MUST have a GET operation.', message: "Otherwise people won't know how to get it.", given: "$.paths[/]", then: { field: "get", function: truthy, }, severity: 'warn', }, // Author: Phil Sturgeon (https://github.com/philsturgeon) 'no-numeric-ids': { description: 'Please avoid exposing IDs as an integer, UUIDs are preferred.', given: '$.paths..parameters[*].[?(@property === "name" && (@ === "id" || @.match(/(_id|Id)$/)))]^.schema', then: { function: schema, functionOptions: { schema: { type: "object", not: { properties: { type: { const: "integer" } } }, properties: { format: { const: 'uuid' } } } } }, severity: 'error', }, // Author: Nauman Ali (https://github.com/naumanali-stoplight) 'no-global-versioning': { description: 'Using global versions just forces all your clients to do a lot more work for each upgrade. Please consider using API Evolution instead.', message: 'Server URL should not contain global versions', given: "$.servers[*].url", then: { function: pattern, functionOptions: { notMatch: '/v[1-9]' } }, formats: [oas3], severity: 'warn', } } }; ``` The format of these rules might look familiar to anyone used to [creating rulesets](https://meta.stoplight.io/docs/spectral/e5b9616d6d50c-custom-rulesets?ref=apisyouwonthate.com) in JSON/YAML, it's just the JavaScript version. Functions are actually referenced as functions instead of string names for functions, which removes a lot of the confusion around where custom functions live and how they're used. Shove some callbacks in. It's JavaScript afterall! These rules are making sure you have something showing on your API "root route" `https://example.com/api/`, making sure it's got a `GET` defined (consider using [IETF Draft DRFC: Home Documents for HTTP APIs](https://datatracker.ietf.org/doc/html/draft-nottingham-json-home?ref=apisyouwonthate.com)), and you're [not using global versioning which is a giant pain in the ass](https://apisyouwonthate.com/blog/api-evolution-for-rest-http-apis). Your milage may vary, but that's the point of writing your own style guide. This is mine. There's also a rule in there pointing out that using auto incrementing IDs in an API are generally [a pretty bad idea](https://phil.tech/2015/auto-incrementing-to-destruction/?ref=apisyouwonthate.com). There's loads of other rules in there which you can feel free to copy and paste into a ruleset for your API Style Guide. Head over to [ruleset.ts](https://github.com/apisyouwonthate/style-guide/blob/main/src/ruleset.ts?ref=apisyouwonthate.com) and have a root around to see if you want any, just make sure you are attributing the content (I'd have used the [DBAD license](http://dbad-license.org/?ref=apisyouwonthate.com) if I didn't think I was gonna get moaned at about it). ## Publish Your Spectral Ruleset to NPM Before I can publish to the NPM public repository I'll need to create a `package.json` that contains a few specific options. ```shell npm init ``` This will ask a whole bunch of questions via CLI prompts, and you can answer them a little something like this: ``` package name: (style-guide) @apisyouwonthate/style-guide version: (1.0.0) description: Make your HTTP APIs better, faster, stronger, whether they are still being designed (API Design-First) or your organization has flopped various mismatched APIs into production and now you're thinking some consistency would be nice. Using Spectral and OpenAPI. entry point: (index.js) ruleset.js test command: git repository: https://github.com/apisyouwonthate/style-guide keywords: openapi, openapi3, openapi31, api-design author: Phil Sturgeon license: (ISC) MIT ``` With that `package.json` created, the next step is to grab the dependencies we know it'll need. ```shell npm install --save @stoplight/spectral-functions npm install --save @stoplight/spectral-formats ``` Time to yeet this up to the public NPM repository so *everyone* can use it in all their APIs, and as a basis for their own organization-wide API Style Guides! Read the full documentation on how to [publish modules to NPM](https://docs.npmjs.com/creating-and-publishing-scoped-public-packages?ref=apisyouwonthate.com) (and you'll need an account, etc) ```shell npm login npm publish --access public ``` If that's worked, you'll be able to see it up on NPM. My package is up on NPM [over here](https://www.npmjs.com/package/@apisyouwonthate/style-guide?ref=apisyouwonthate.com). ## Using an NPM published package Using a package that's been published to NPM works the same as any using other Spectral ruleset. IIf you're working with Spectral somewhere you can install an NPM module (e.g.: [Spectral CLI ](https://meta.stoplight.io/docs/spectral/ZG9jOjI1MTg1-spectral-cli?ref=apisyouwonthate.com) or [Spectral JS](https://meta.stoplight.io/docs/spectral/eb68e7afd463e-spectral-in-java-script?ref=apisyouwonthate.com)) you can install the package with NPM in the CLI and point Spectral to the rulesets you want to use. ```shell cd ~/src/ npm install --save -D @stoplight/spectral-cli npm install --save -D @apisyouwonthate/style-guide echo 'extends: ["@apisyouwonthate/style-guide"]' > .spectral.yaml ``` Now when you run `spectral lint openapi.yaml` you'll get all my opinions being shouted at your API, and if you're using API Design First maybe it will help you avoid wasting time writing a bunch of code that you'll just have to change later. ``` /Users/phil/src/protect-earth-api/api/openapi.yaml 18:7 warning api-health Creating a `/health` endpoint is a simple solution for pull-based monitoring and manually checking the status of an API. paths 18:7 warning api-home Stop forcing all API consumers to visit documentation for basic interactions when the API could do that itself. paths 36:30 warning no-unknown-error-format Every error response SHOULD support either RFC 7807 (https://tools.ietf.org/html/rfc6648) or the JSON:API Error format. paths./v1/orders.post.responses[401].content.application/json 96:30 warning no-unknown-error-format Every error response SHOULD support either RFC 7807 (https://tools.ietf.org/html/rfc6648) or the JSON:API Error format. paths./v1/orders/{order}.get.responses[401].content.application/json 112:30 warning no-unknown-error-format Every error response SHOULD support either RFC 7807 (https://tools.ietf.org/html/rfc6648) or the JSON:API Error format. paths./v1/orders/{order}.get.responses[404].content.application/json ``` Thanks for making Phil's API better, Phil! ## Using NPM Rulesets in Stoplight Studio & Platform If you're using Stoplight Platform, Stoplight Studio Desktop, VSCode Spectral, etc. then you can still use the NPM-based API Style Guides, but as there's no way to get NPM involved you can load it with this *One Amazing Trick Doctors Don't Want You To Know About*. ```yaml extends: - https://unpkg.com/@apisyouwonthate/style-guide@1.1/ruleset.js ``` Using Unpkg or similar means you can load the ruleset with whatever version specificity you want (`@1`, `@1.1`, `@1.1.2`) and it'll work anywhere that Spectral does. Looking at this, you might be wondering about versioning, and that naturally brings up the question of testing. Both possible, and both better off as their own blog post. ## Other API Style Guides & Rulesets More and more companies and organizations are publishing their API Style Guides as Spectral rulesets, so I've been putting them together in a [simple little repo](https://github.com/stoplightio/spectral-rulesets?ref=apisyouwonthate.com) which will one day become a "marketplace". Here are some of the best: - [Adidas](https://github.com/adidas/api-guidelines/blob/master/.spectral.yml?ref=apisyouwonthate.com) - [Box](https://github.com/box/box-openapi/blob/main/.spectral.yml?ref=apisyouwonthate.com) - [DigitalOcean](https://github.com/digitalocean/openapi/blob/main/spectral/ruleset.yml?ref=apisyouwonthate.com) - [Team Digitale](https://github.com/teamdigitale/api-openapi-samples/blob/master/.spectral.yml?ref=apisyouwonthate.com) Use these rules as a basis for your style guides, and make sure your APIs are useful before you waste time building something rubbish. ## Host API Style Guides on Stoplight If mucking around with NPM isn't up your alley, as always Stoplight have a convenient hosted solution you can use that requires a bit less duct-tape and string. Create a [Style Guide Project](https://meta.stoplight.io/docs/platform/71b92b2b5e388-style-guide-overview?ref=apisyouwonthate.com) on Platform, then you can build and share a ruleset in a GUI experience instead of writing JavaScript. You can even [import your existing Spectral rulesets](https://meta.stoplight.io/docs/platform/364e30f79daa0-create-a-style-guide?ref=apisyouwonthate.com#imported-style-guides) to be a Style Guide Project, and leave your awkward JSON/YAML rulesets behind for good. Either way, all the APIs at your organization can use these style guides to get feedback as people are designing their APIs, improving API Governance, and wasting less time worrying about small fry so you can talk about far bigger and more important things in your API Design Review sessions. ### The Best Content to Supplement Your API Documentation URL: https://apisyouwonthate.com/blog/supplementing-documentation/ Last updated: 2023-03-24T12:21:09.000Z I started learning to code over ten years ago. As a self-taught developer, I picked up most of my knowledge from programmers who left breadcrumbs on Stack Overflow or wrote tutorials online. Early in my career, I often found documentation overwhelming. Without enough contextual knowledge, I struggled to turn raw endpoints into useful applications, so I looked for more complete solutions. Documentation has come a long way since then, as have the supporting resources commonly released along with it. While [following an accepted standard](https://apisyouwonthate.com/blog/theres-no-reason-to-write-openapi-by-hand) and having [comprehensive, honest documentation](https://apisyouwonthate.com/blog/keeping-documentation-honest) are essential first steps, users of public APIs expect more: tutorials, videos, forums, and SDKs among others. This piece will share why more API builders are going beyond simple documentation for their applications. I’ll share some of the types of supplementary content I see most often and my insights into which content you should start with. ## Why Go Beyond Documentation? > "Good documentation is only the starting point. Your users' expertise and industry knowledge can vary greatly, so you should also create educational content like guides or video tutorials to bring users up to speed on the concepts around your product or walk them through complex processes.” - [Anna Tsolakou](https://dev.to/tsolakoua/how-devrel-makes-your-api-irresistible-56cf?ref=apisyouwonthate.com), Developer Advocate at [Amadeus for Developers](https://developers.amadeus.com/?ref=apisyouwonthate.com) It takes a lot of time and effort to produce content to supplement your docs, so the first question is, "Is it worth it?” First, consider your API’s userbase. > "Reference documentation is important, but it is probably the last thing a new user cares about. Building docs that understand the progression from new user to advanced user are the best.” - Aidan Cunniffe, CEO at [Optic](https://www.useoptic.com/?ref=apisyouwonthate.com) Many APIs need to be accessible to junior developers, developers inexperienced in REST/GraphQL conventions, or non-developers. With the number of [low and no-code tools proliferating](https://stackoverflow.blog/2021/06/09/using-low-code-tools-to-iterate-products-faster/?ref=apisyouwonthate.com), there’s a whole new class of "API hackers” out there using APIs without a ton of experience in software development. Even if you only intend for your API to be used by experienced, senior developers, there’s a strong case for supplementing your documentation with other content. First, not everyone takes to a written medium. Some of us learn better from videos or interactive content. Second, API documentation rarely tells a user what to do when something goes wrong. Should they contact support? Post on Stack Overflow? Rant on Twitter? This is where a forum or community can help supplement your docs. Next, API documentation is often [focused on the "how” and not the "why”](https://stackoverflow.blog/2021/06/08/podcast-345-a-good-software-tutorial-explains-the-how-a-great-one-explains-the-why/?ref=apisyouwonthate.com), but the "why” is really important if you want to help developers see creative use cases for your API. Finally, content that supplements your documentation can also help with discoverability. "Don't forget,” Dan Moore, Head of Developer Relations at [FusionAuth](https://fusionauth.io/?ref=apisyouwonthate.com) told me, "most developers are going to use one interface for your documentation: Google.” Whether it’s [long-form technical content written for search engines](https://draft.dev/learn/seo-keyword-opportunities-in-developer-marketing?ref=apisyouwonthate.com), videos on YouTube, or workshops at conferences, the more content you have surrounding your API, the more likely users are to find and try it. ## The Best Content to Supplement Your API Docs If you buy my argument that you should create more content surrounding your API, the next obvious question is, "Where do I start?” I’ve worked with over 80 developer tools companies to help with their content creation efforts over the past two years. While every company is different, I have strong opinions as a developer and API builder about which types of content are most valuable to prioritize first. So, let’s look at some of the best types of content to supplement your API docs. If I were starting a new API today, here’s how I would prioritize my efforts: ### 1\. SDK/API Client > "Releasing your demo apps and SDKs as open-source is a helpful way to build transparency and trust with the community.” - Anna Tsolakou A good SDK allows users to bypass your API docs entirely and start building right away. While abstracting some of the lower-level operations away might remove a bit of flexibility, an SDK or API client is table stakes for a public API. For a good example of how you can integrate your SDK into your documentation, look at [Stripe’s API Library section of their documentation](https://stripe.com/docs/libraries?ref=apisyouwonthate.com). The biggest challenge here is the time required to build and maintain a comprehensive set of SDKs. You’ll probably have to support at least 5-10 languages to cover the majority of users, but you may not have expertise in-house for all of them. Tools like [OpenAPI generator](https://github.com/OpenAPITools/openapi-generator?ref=apisyouwonthate.com) can help, but it’s also a good idea to make your SDKs open source so your users can report issues as they find them. ### 2\. Tutorials and Guides > "API documentation is similar to a cookbook. Your API enables a huge variety of applications which are like prepared dishes enabled by combinations of endpoint ingredients…You can shorten the on-ramp by providing practical guides and simple examples.” - [Julia Seidman](https://blog.stoplight.io/your-api-reference-is-more-than-a-list-of-ingredients?ref=apisyouwonthate.com), Technical Content Consultant Tutorials are probably the next most important supplement to your docs. As I mentioned above, [a good technical tutorial](https://draft.dev/learn/technical-tutorials?ref=apisyouwonthate.com) will focus on the "why,” helping users understand how to stitch your API together in a useful way. Nate Lee, the Founder of [Speedscale](https://speedscale.com/?ref=apisyouwonthate.com) agreed, "Most API docs all include example calls, but following a specific use case can be great, especially if it matches what most users will want to do.” Keeping tutorials up-to-date can be a challenge. You’ll need writers who can cover many languages and frameworks if you want to build tutorials around all the situations your users are likely to encounter. One company doing tutorials well is [DigitalOcean](https://www.digitalocean.com/community/tutorials?ref=apisyouwonthate.com). "It's always great to find one of their articles,” Dan Moore told me. "The instructions have rarely failed me.” ### 3\. Community Forum or Slack Group Once you have more than a handful of users, you should move from one-on-one to group support. Typically, this is done through a forum or chat group. I prefer forums because the content isn’t as transient, and as Dan Moore pointed out to me, "\[a forum\] is full of timestamped questions and is searchable by Google.” So while his team at FusionAuth spends time moderating and answering questions, their forum also draws new users to their site. Building an active, engaged community is a lot of work. Anna Tsolakou pointed out that, "launching a community channel is easy. The real trick is to keep it lively and engaging.” Having a community that’s a ghost town is as bad as having no community at all, so plan to invest time seeding questions and answers, asking members to participate, and incentivizing participants. ### 4\. Interactive Docs or Postman Collections Complete tutorials are a great way to showcase your API’s use cases, but interactive documentation (or [Postman collections](https://learning.postman.com/docs/getting-started/creating-the-first-collection?ref=apisyouwonthate.com)) will help users see how your API works in practice. I especially like docs that allow you to manipulate calls and retrieve real data from your web browser. Interactive documentation like this requires effort to maintain, and it might not be right for certain APIs. If your API contains sensitive information or OAuth tokens for authentication, you might need to be a bit more careful, but your users will appreciate how quickly they can test your API. ### 5\. Architecture Explanations Most API documentation jumps straight into "Getting Started.” The authors expect you to be familiar with the product and domain model, but that might not be the case. I remember a former boss asking me to integrate an obscure product whose poorly-documented API gave me no insight into how their data model was constructed. I had to contact their support to figure out the structure before I could even use the service. Dan Moore agreed with me on this point. "Higher-level *why* docs are helpful for folks who want to dig into the choices that were made,” he told me, adding that making these architecture documents visual has helped their users. "We use \[sequence diagrams\] extensively and have heard praise from customers and users.” If you’re having trouble deciding what should be included in your public architecture documentation, try explaining your API to a software engineer completely outside your industry. You’ll find that many of the things you find obvious are not quite so clear to them. ### 6\. Videos I’m bullish on video for developers. While my generation learned through tutorials, books, and StackOverflow, things are changing. The next generation of developers is video-native, turning to YouTube or course platforms to learn how to use new tools and build apps. For API developers, video is a great way to show users what they can do with your app. "Video and demo apps help developers choose your API,” Aidan Cunniffe told me. "This content should highlight what your API is capable of doing…it's like the product demo for your API.” AWS does a great job using [video content to demonstrate its services on YouTube](https://www.youtube.com/user/AmazonWebServices?ref=apisyouwonthate.com). "Their re:invent videos have info that you simply won’t find in the written docs,” Dan Moore added. That said, video has its limitations. Users can’t copy/paste code from videos, and as Nate Lee pointed out, "Attention spans for videos are rather limited.” This leads video creators to favor brevity over completeness. ### 7\. Well-Maintained Demo Applications > "Demo applications give developers a taste of what your APIs can do and can help promote new or underperforming APIs in the process.” - Anna Tsolakou Another great way to help your API’s users get started quickly is to provide working demo applications. Developers can use them as a starting point or reference when they’re not sure about the best way forward. [Auth0](https://github.com/auth0-samples?ref=apisyouwonthate.com) and [Vercel](https://github.com/vercel/vercel/tree/main/examples?ref=apisyouwonthate.com) both do an excellent job in their GitHub repositories. Of course, maintaining complete demo applications is even more work than keeping a few code samples in your tutorials up-to-date. But, if you can manage it, they will provide a lot of value to new users. ### 8\. "Nice to Have” Content Finally, if you get to all the points above and you’re *still* looking for content to help supplement your documentation, here are a few more ideas. I wouldn’t recommend these to companies with a brand new API, but they might be worth pursuing as your API stabilizes or if you have a large enough user base. - **Error glossary** \- Dan Moore mentioned that having a page with solutions to all your API’s error messages is great because developers are very likely to search for these in search engines. - **Workshops and training** \- These might be nice to have as videos, but if you’re working with enough customers, you might even offer on-site workshops. - **Case studies** \- While typically used for making the business case for your API, technical case studies can be a good way to show users the end results of adopting your API. - **Knowledge base** \- Typically, you can share important terms in your docs as they appear, but some companies break them out into glossaries or knowledge bases. While having accurate, up-to-date documentation is vital, it’s not enough for public-facing APIs. Creating content to support your documentation is a great way to help users solve their own problems, get started faster, and have a better experience when working with your application. Is there anything I missed? What’s working for your API? I’d love to hear your thoughts, so find me [on Twitter to continue the conversation](https://twitter.com/KarlLHughes?ref=apisyouwonthate.com). *Article cover photo by [Patrick Fore](https://unsplash.com/@patrickian4?utm%5Fsource=unsplash&utm%5Fmedium=referral&utm%5Fcontent=creditCopyText) on [Unsplash](https://unsplash.com/?ref=apisyouwonthate.com)* ### Modern API Deployment Options in the Cloud URL: https://apisyouwonthate.com/blog/modern-api-deployment-options-in-the-cloud/ Last updated: 2023-03-24T12:21:20.000Z How do you deploy your API, and what's the best way to structure it? Front-end frameworks have come a long way in recent years, making it easy to spin up and deploy a website/web app. However, I find the noise caused by many strong opinions sometimes clouds the equal and fantastic progress made on letting developers quickly deploy an API. I'm going to cover a few ways you can build and deploy an API to get your MVP into the hands of your users as quickly as possible. I will approach these deployments from the mindset of building a REST API, as I find this is the most accessible format when thinking about structure and deployments. ## Don't build one from scratch Getting features, demos, and MVPs into users' hands is more important than the tech stack, language or processes you use. Users don't care how you built it; they care about the features and how well it works. [Firebase](https://firebase.google.com/?ref=apisyouwonthate.com) has come on in leaps and bounds over the last few years and allows you to knock together a backend for your API quickly. Time and time again, I see people building out an API for simple CRUD operations. Don't make your life difficult. Why build and maintain a whole backend for simple operations? Firebase also allows for a more complex setup giving you the ability to write cloud functions which respond to changes, and run code based on data changes in the database made by your front-end. If you're looking for an open-source alternative to Firebase, [Supabase](https://supabase.com/?ref=apisyouwonthate.com) is a great companion. Plus, they have a great tag line: `Create a backend in less than 2 minutes.` Another tool to deploy backends with hardly any code is [AWS Amplify](https://aws.amazon.com/amplify/?ref=apisyouwonthate.com). You are given tools for authentication and data storage, serving web pages, and connecting to other AWS services. Amplify also comes with a nifty setup they call Studio, allowing you to manage app users and edit content. There are some great tools out there for putting together backends. Managing dev ops, servers, and other backend infrastructure can be a pain sometimes. Assess if there is a need to put together a full-fledge backend/API before starting. We don't all need to be Stripe 😉 ## Deploy on the Edge Edge computing has me excited; so many cool things are happening in this space. Deploying an API on edge is like writing serverless functions, with the main difference being you deploy to the edge network, and they tend to run on top of the V8 JavaScript runtime. I love this approach as you get to write endpoints as functions and forget about everything else. There are a few providers in this space. One of the most notable is [Cloudflare Workers](https://workers.cloudflare.com/?ref=apisyouwonthate.com), and you can write in Rust, C, and C++, not just JavaScript. They have some great examples of starter projects to get you going, They also have an excellent course on [building a serverless API with Cloudflare Workers on EggHead](https://egghead.io/courses/build-a-serverless-api-with-cloudflare-workers-d67ca551?ref=apisyouwonthate.com). I also love [Deno Deploy](https://deno.com/deploy?ref=apisyouwonthate.com), a one-click deploy service for [Deno](https://deno.land/?ref=apisyouwonthate.com). It allows you to instantly deploy JavaScript on the edge every time you push code to Github. Forget servers, forget vendor lock-in and push JavaScript all the livelong day. This is an example of how simple it can be to deploy a Rest API. I have set up a single route using [Oak](https://oakserver.github.io/oak/?ref=apisyouwonthate.com) (an HTTP framework for Deno) that returns an estimated carbon footprint for the provided electricity consumption. ```js import { Application, Router } from "https://deno.land/x/oak/mod.ts"; const port = 3011; const router = new Router(); router.get("/electricity", (ctx) => { const factor = 0.759; const kwh = Number(ctx.request.url.searchParams.get("kwh")); if (kwh) { const carbon = kwh * factor; const inTonnes = carbon / 1000; ctx.response.body = JSON.stringify({ carbon: inTonnes, unit: "tCO2e", }); } else { ctx.response.status = 400; ctx.response.body = JSON.stringify({ error: { message: "kWh have not been supplied in the query", }, }); } }); const app = new Application(); app.use(router.routes()); app.use(router.allowedMethods()); app.addEventListener("listen", () => console.log(`Listening on http://localhost:${port}`) ); await app.listen({ port }); ``` [Full Deno Code Example](https://github.com/AlexanderKaran/electricity-calculator?ref=apisyouwonthate.com) To deploy with Deno Deploy, you need to sign up here and make sure you have committed your API to Github. Then connect Deno Deploy to your Github account once you sign up, select the correct repository, select the entry file and then you're all good to go. Any changes, push code to the main branch, and they go live. All PRs created in the repository will come with a deploy preview URL. You can check out my demo API in full here. Change the query to your electricity consumption in kWh to get your estimated footprint. The question of how to structure your projects is the next hurdle, and this is where breaking them down into microservices would be essential. For example, for this project, I could point to a subdomain, "calculator.alexanderkaran.com", and build out more functionality for working out the footprint for gas or water usage. Each function could be at a different subroute, i.e. "/gas" or "/water"; however, If I wanted to build functionality for testing appliance efficiency. I would create this as a whole new project and point a new subdomain at it, for example, "appliances.alexanderkaran.com". Another great part of Deno Deploy is keeping all the services in one repo is easy; you point each project to a specific folder only the code imported in that folder gets deployed. Deno and Deno Deploy make it easy to share code, create a shared folder in the mono-repo and import the code; that's it, no third party sharing system and no private npm modules. ## Serverless Functions Serverless functions share a lot with edge functions, but you usually code in Node rather than a V8 browser runtime. However, they can have a cold start, meaning the code does not run the second it's called, unlike edge functions which happen straight away. Many providers offer serverless functions, but we will focus on [AWS Lambda](https://aws.amazon.com/lambda/?ref=apisyouwonthate.com) as it is one of the most common. Lambda functions are used for many different solutions, but they need to be paired with an API Gateway when building an API. It turns out AWS have a service for that, too 😉. When creating Lambda functions in the AWS console, they offer options to connect an endpoint in a new or existing [API Gateway](https://aws.amazon.com/api-gateway/?ref=apisyouwonthate.com), making setup a breeze. If you are used to building monoliths and everything in one place, this can seem confusing. However, it simply boils down to your API Gateway containing all your routes, and each Lambda function is the controller for the route. Before we dive deeper into how this works, let's look at how Lambda functions are structured. I have stolen one of the AWS HTTP templates for updating [DynamoDB](https://aws.amazon.com/dynamodb/?ref=apisyouwonthate.com): ```js const AWS = require('aws-sdk'); const dynamo = new AWS.DynamoDB.DocumentClient(); /** * Demonstrates a simple HTTP endpoint using API Gateway. You have full * access to the request and response payload, including headers and * status code. * * To scan a DynamoDB table, make a GET request with the TableName as a * query string parameter. To put, update, or delete an item, make a POST, * PUT, or DELETE request respectively, passing in the payload to the * DynamoDB API as a JSON body. */ exports.handler = async (event, context) => { //console.log('Received event:', JSON.stringify(event, null, 2)); let body; let statusCode = '200'; const headers = { 'Content-Type': 'application/json', }; try { switch (event.httpMethod) { case 'DELETE': body = await dynamo.delete(JSON.parse(event.body)).promise(); break; case 'GET': body = await dynamo.scan({ TableName: event.queryStringParameters.TableName }).promise(); break; case 'POST': body = await dynamo.put(JSON.parse(event.body)).promise(); break; case 'PUT': body = await dynamo.update(JSON.parse(event.body)).promise(); break; default: throw new Error(`Unsupported method "${event.httpMethod}"`); } } catch (err) { statusCode = '400'; body = err.message; } finally { body = JSON.stringify(body); } return { statusCode, body, headers, }; }; ``` [AWS Lambda Snippet](https://gist.github.com/AlexanderKaran/666d7a80e92b7f87c4f81f47211a6145?ref=apisyouwonthate.com) As you see here, the function responded to an event and accessed the HTTP method inside to see what type of request it was. Personally, the biggest hurdle I had when using serverless functions or edge computing was how there were no req, res and next functions like I was used to in ExpressJS. Thankfully, though, there are many ways to import frameworks you're used to into Lambda. You also do not have to code in Node; other languages such as Python and Java are on offer too. Setting up serverless functions with an API Gateway can be done super quickly using the AWS console, but a more extensive API with many endpoints would be time-consuming to set this up way. Enter tools like [ClaudiaJS](https://www.claudiajs.com/?ref=apisyouwonthate.com) and [Serverless](https://www.serverless.com/?ref=apisyouwonthate.com). Claudia allows you to easily set up, deploy, and update code to AWS Lambda, making development and automating releases a breeze. If you want even more capabilities, check out Serverless. It makes deploying serverless functions to AWS easier, allowing you to access and deploy complete AWS services outside of Lambda. Serverless goes even further and has setups for Google Cloud Functions and Azure Functions. Setting up an API made of serverless functions can be tricky the first time, so remember API Gateway for your routes and serverless functions for your controllers. ## Serverless Containers Lastly, we come to serverless containers. I have used Docker containers myself for five years, and I love them. It's great to build something locally, and it runs the same on the server or any other computer that uses it. Need something to be running all the time? Can't afford to deal with cold starts on your API requests or be limited by serverless functions memory or timeout limits? Then containers are for you. Most people tend to reach for Kubernetes or spin up their servers, install Docker, and deploy their images onto the server. These options will tend to be overkill unless you are running extensive backend infrastructure, which, let's be honest, is not all of us. Though, if you are deploying across multiple cloud providers, Kubernetes might be for you. I like to focus on features and quality, not infrastructure, config and deployments. AWS services ECS and ECR are great for this. ECR is a repository for uploading your Docker containers, and ECS spins up each container. You connect them to API Gateway or an Application Load Balancer to expose your API to the outside world. To set up your ECS deployment, you create a task for the docker image, defining setups such as memory and ports. You then create a cluster in ECS and create a Service for each Task. Services are responsible for auto-scaling Tasks based on traffic or memory usage, spinning up new versions when replacing the docker image and exposing the image to the outside world. Setting up ECS does take a while the first time, but after the setup, you add in code pipelines from AWS to auto-deploy updated code when pushing to your Git repository. After setting up your Cluster, you connect it to the outside world in a few different ways. One approach is to connect it to an Application Load Balancer and set up rules in Load Balancer to connect different URL routes to each service. I usually have a few services running in a cluster. For example, my last Cluster contained the following services: **Measure Service:** My service for handling carbon calculations and tracking consumption and cost of utilities. It was connected to the load balancer, and any request that came through "myapi.com/measure" was sent here. **Image Service:** My service for handling image uploads and processing. It was connected to the load balancer, and any request that came through "myapi.com/image" was sent here. **Notifications Service:** The service for sending emails and push notifications but not connected to the Load Balancer. The other services can call it as they're all on the same VPC (Virtual Private Network), which means I did not need to write the same notification code in the Image or Measure Service. AWS has some overviews on setting up ECS and what you can connect it to. If you are looking for a more in-depth tutorial on deploying a Node app from scratch, you can check out this [tutorial by Raphael](https://dev.to/raphaelmansuy/deploy-a-docker-app-to-aws-using-ecs-3i1g?ref=apisyouwonthate.com). ## In Summary While each option here deserves its own blog post to cover its entire setup, benefits, and quirks, you at least have a good overview of what is available. I love the advancement of serverless and not having to think about servers at all. Serverless functions, and new kids on the block like Cloudflare Workers, means we are close to never having to think about servers or backend configs again. It's great to focus on code to fix problems, which gets tested, reviewed and deployed the second your PR gets approved in Github. If you have not tried serverless or edge functions, you should try them for your next API. ### Contract Testing a Laravel API with OpenAPI URL: https://apisyouwonthate.com/blog/contract-testing-apis-laravel-php-openapi/ Last updated: 2023-03-24T12:21:31.000Z Your API does a bunch of great stuff, and your OpenAPI document tells everyone about all the great stuff that your API can do, but making sure those two sources of truth agree can be a bit of a struggle at first. Whether you followed the [API design-first workflow](https://apisyouwonthate.com/blog/api-design-first-vs-code-first) and want the developers to stick to your design, or whether you are trying to [retroactively make documentation for an existing API](https://apisyouwonthate.com/blog/creating-openapi-from-http-traffic) and want to make sure its accurate, you'll want confidence the code and description match. Then over time, there's the chance for the API or OpenAPI to diverge, with a change being made in the code and not in the docs, or vice versa. Don't worry, this is a well solved problem. There are various dedicated tools dedicated which we wrote about way back in [Keeping Documentation Honest](https://apisyouwonthate.com/blog/keeping-documentation-honest/), but these days we love the simplicity of adding some [OpenAPI-based contract testing assertions](https://apisyouwonthate.com/blog/writing-documentation-via-contract-testing) to your existing API test suite. Don't have a test suite? Well, never a better time to start. Writing tests sounds scary to some, but seeing as there are a lot of assertions already written into your OpenAPI document, you will have some basic testing done rather quickly. There are infinite tools for infinite languages and frameworks, but today we're going to focus on this combination: - [Laravel PHP](https://laravel.com/?ref=apisyouwonthate.com) \- A ridiculously popular PHP framework. - [Pest](https://pestphp.com/?ref=apisyouwonthate.com) \- Elegant PHP testing tool that feels like Jest, RSpec, etc. - [Spectator](https://github.com/hotmeteor/spectator?ref=apisyouwonthate.com) \- Light-weight OpenAPI testing assertions for Laravel. This article will assume you're familiar with Laravel PHP, and if you're not there are many good articles out there about getting started. Their documentation is fantastic too. The concepts of this will still be interesting to many who are not familiar or in a rush to learn right now. So, you've already got Laravel running, and you want a test suite. Pest is great, it reminds me of RSpec, Jest and various other tools that I loved using for my last 8 years in Ruby/Go/Node/TypeScript land. I was a little worried it would be confusing trying to get Laravel and Pest to play ball, but Pest has a [Laravel plugin](https://pestphp.com/docs/plugins/laravel?ref=apisyouwonthate.com) which takes care of that. ```bash composer require pestphp/pest-plugin-laravel --dev php artisan pest:install ``` Laravel lets people generate various bits code just like Rails generators, so you can generate a Pest test. ```bash php artisan pest:test OrganizationsTest ``` This will create a very basic test in `tests/Feature/OrganizationsTest.php` that looks like this: ```php get('/organizations'); $response->assertStatus(200); }); ``` Pest is using the [HTTP Tests](https://laravel.com/docs/8.x/http-tests?ref=apisyouwonthate.com) functionality in Laravel to ping the `/organizations` endpoint, and then make sure you get a 200 back. This HTTP Test functionality will simulate a proper network interaction, meaning the test is more realistic than unit testing your controllers. This test is not talking about code, it's testing HTTP interactions. Perfect. Trying to run this test with `php artisan test` or `./vendor/bin/pest` will possibly work if you've got your database server running directly on your machine, but if you're using docker you will probably get failures at this point. [Sail](https://laravel.com/docs/8.x/sail?ref=apisyouwonthate.com) is another Laravel tool which can help interface with Laravel inside docker, so tests can be run with `sail artisan test` instead. Either way, your ping-tests should be passing now. Let's make the test a bit more useful by creating some data before the tests are run. Afterall, we wont be able to contract test the data if there... isn't any data. ```php create(); $this->uuid = $organization->organization_uuid; }); it('returns a 404 for invalid record', function () { $non_existent_uuid = "53d4faeb-e046-4ab1-91ff-6b6e35c4c052"; $this ->getJson("/orgs/{$non_existent_uuid}") ->assertStatus(404); }); it('returns a valid record', function () { $this ->getJson("/orgs/{$this->uuid}") ->assertStatus(200); }); ``` Run `sail artisan test` and hopefully this is working. It might fail complaining you've not got any factories set up, which are a handy feature for setting up fake data to be tested with. Head over to the Laravel Documentation to learn [how to set up model factories](https://laravel.com/docs/8.x/database-testing?ref=apisyouwonthate.com#defining-model-factories) if you've not got them already, this article is getting lengthy and we need to get onto the contract testing bit. Great. But we're still just doing pings on these endpoints. Time to give contract testing a go! ## Grab some OpenAPI If you have an OpenAPI document already, you can skip this step. If you don't have an OpenAPI document, make one with an editor like [Stoplight Studio](https://stoplight.io/studio/?ref=apisyouwonthate.com) or [Postman](https://www.postman.com/?ref=apisyouwonthate.com), or you can nab an example document from APIs Guru's [OpenAPI Directory](https://apis.guru/openapi-directory/?ref=apisyouwonthate.com) to play with. Alternatively, shove this into a file called `openapi.yaml`. ```yaml openapi: "3.0.3" info: title: Example API version: "1.0" paths: /orgs/{id}: get: description: Get an organization parameters: - name: id in: path required: true schema: type: string format: uuid responses: 200: description: OK content: application/json: schema: type: object properties: id: type: string format: uuid ``` ### Using Spectator Armed with some OpenAPI we can now try installing [Spectator](https://github.com/hotmeteor/spectator?ref=apisyouwonthate.com), a tool which will make Laravel's HTTP Tests aware of OpenAPI to help sniff out mismatches. ```bash composer require hotmeteor/spectator --dev php artisan vendor:publish --provider="Spectator\SpectatorServiceProvider" ``` Now let's tweak our tests: ```php create(); $this->uuid = $organization->organization_uuid; // Add Spectator 👇 Spectator::using('openapi.yaml'); }); it('returns a 404 for invalid record', function () { $non_existent_uuid = "53d4faeb-e046-4ab1-91ff-6b6e35c4c052"; $this ->getJson("/orgs/{$non_existent_uuid}") ->assertValidRequest() # 👈 new ->assertValidResponse(404); # 👈 new }); it('returns a valid record', function () { $this ->getJson("/orgs/{$this->uuid}") ->assertValidRequest() # 👈 new ->assertValidResponse(200); # 👈 new }); ``` Those new assertions are being made available to Pest and the Laravel HTTP Test logic by Spectator, which is looking at the openapi.yaml and then figuring out which "path" to compare to the URL in getJson(). Very smart, and it immediately pointed out that my OpenAPI was missing definitions for how the 404 errors should look, along with a few other mistakes in my OpenAPI. Here's an example of the API response mismatching data typed for a property defined in OpenAPI. I've added newProperty to OpenAPI but forgot to add it to the HTTP Resource (what Laravel calls their serializer class). ```yaml type: object required: - id - name - orders - newProperty properties: newProperty: type: string # existing properties ... ``` Now when the test suite is run, Spectacle is going to throw up red flags. ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/posts/contract-testing-apis-laravel-php-openapi/spectacle-errors.png) Done! Docs and code will never be out of sync again. There are a few quirks to watch out for with Spectacle, like expecting my path parameters to have a very specific name, but changing those is fairly low stakes and will not damage the quality of your OpenAPI. ## Summary What I love the most about this simplicity is that it can integrate into an existing applications test suite, and you definitely want to have a test suite. It's not a brand new second test suite, or some hosted tool that is hard to keep up with changes in PRs flagging the "one true cloud test suite" as broken... it's just a few lines of assertions in a standard PHPUnit, Pest, etc. test suite, and run on whatever existing CI/CD you're already using. Other folks use [Dredd](https://github.com/apiaryio/dredd?ref=apisyouwonthate.com), which is a whole other tool to maintain with its own database seeding and state management - no handy DB resets like in Laravel/Pest. It's not able to check multiple responses (like 404's) so you're just kinda hoping those are correct when using Dredd. Then there's [Prism](https://stoplight.io/open-source/prism/?ref=apisyouwonthate.com), which is good for contract testing real traffic and spotting issues, but that's not something you can control from code. There's loads of other fantastic tools on [OpenAPI.Tools](https://openapi.tools/?ref=apisyouwonthate.com) for contract testing, and pretty much any [JSON Schema validator](https://json-schema.org/implementations.html?ref=apisyouwonthate.com#validators) can be used now that [JSON Schema and OpenAPI Schemas are actually the same thing](https://apisyouwonthate.com/blog/openapi-v3-1-and-json-schema), so if you've not got something specifically OpenAPI orientated then hack one together yourself, and maybe release that to make something as simple as Spectator! ### Eight of the Biggest Lies in APIs URL: https://apisyouwonthate.com/blog/eight-of-the-biggest-lies-in-apis/ Last updated: 2023-09-19T10:47:39.000Z Have you ever been told "Everybody does it this way" when arguing about API design? Or have you heard, "We use HTTPS, so our API is secure"? In the API space, numerous statements are highly context-dependent, misunderstandings, or outright lies. To find some of the most egregious of these, I asked the [APIs You Won't Hate Slack](https://apisyouwonthate.com/community) for those phrases that stand their neck hair on end and their eyes to see red. If you find yourself about to use one of these jargon phrases unironically in your next architectural debate, please think twice. These are **The Eight Greatest API Lies**, as sourced from the community: 1. ## It's just a PoC/MVC/Experiment; we'll tidy up the design once we have feedback. *Submitted by Frank Kilcommins ([@fkilcommins](https://twitter.com/fkilcommins?ref=apisyouwonthate.com))* Iterative development is a beautiful thing and a cornerstone of modern software processes. The problem is applying the same approach to an API design. Unlike code, an interface is a promise between one and many individuals. And no matter how an API producer might imply that the relationship has no strings attached, breaking promises undermines trust and hurts feelings. It is tough to get otherwise busy developers to spend their precious time creating code for an interface *the first time*. Imagine their excitement discovering their work was invalidated and that they could now re-implement what was previously working? Long story short, most don't. 1. ## It's internal and on-prem, so we don't need to worry about Auth for a V1. *Submitted by Frank Kilcommins ([@fkilcommins](https://twitter.com/fkilcommins?ref=apisyouwonthate.com))* I get it; developers are under a deadline and looking for some way to jettison features later. However, relying on *implicit trust*, or clients doing the right thing because "they're one of us", is a no-no. In light of increasing security breaches, more companies are moving to a **zero trust** cybersecurity approach, continuously validating at every stage of the digital interaction. Implicit trust means that users – including malicious actors – can move laterally to access or exfiltrate sensitive data once inside. Even if your API wasn't how that threat got into the building, you *do not* want to be how they downloaded the payroll records. With an increasingly remote and distributed workforce, continued migration to other people's servers (the cloud) taking a zero-trust approach has never been more critical. 1. ## Caching does not belong in an HTTP API. *Submitted by Phil Sturgeon ([@philsturgeon](https://twitter.com/philsturgeon?ref=apisyouwonthate.com))* As stated [by RESTfulAPI.net](https://restfulapi.net/caching/?ref=apisyouwonthate.com): > "Caching is the ability to store copies of frequently accessed data in several places along the request-response path." Leveraging caches improve the API experience by: - Reducing bandwidth - Reducing latency - Reducing server load - Bridging network failures To ignore caching's usefulness to an API is like eschewing indoor plumbing; sure, you can make things work, but at what cost? 1. ## I found this software tool that generates OAS3 automatically!" (Joe Malin) *Submitted by Joe Malin* Harold "Hal" Abelson is a Massachusetts Institute of Technology (MIT) professor, a fellow of the Institute of Electrical and Electronics Engineers (IEEE), and a founding director of both Creative Commons and the Free Software Foundation. In other words, he knows he stuff. When it came to the communicative power of software artifacts, he had this to say: > "Programs must be written for people to read, and only incidentally for machines to execute." Auto-generating an OpenAPI description from an existing API, whether by code or traffic introspection, is a fine place *to start*. But no generalized algorithm will create a nuanced, detailed reference designed for a specific audience context. Endless, ongoing support happens when people mistake the starting line for the end destination. 1. ## We'll future proof the design by returning everything we have, and the consumer can sort it out. *Submitted by Matthew Reinbold ([@libel\_vox](https://twitter.com/libel%5Fvox?ref=apisyouwonthate.com))* Designing an API requires making decisions. Many of these decisions are made with imperfect knowledge in the time available. A common set of compromises lies around which information is necessary to fulfill the desired use case. Getting one of these decisions wrong can, in some cases, require a version bump as the design team incorporates new learnings. Done well, the API can grow and evolve with its intended audience. Sometimes, however, teams attempt to defer critical decision-making. By doing so, they instead pass on the cognitive load to the client. Returning everything doesn't bypass the evolution certainty, either. It just demonstrates contempt for the integrator's time. If an API user wants the digital equivalent of a cake, then give them a cake. Don't foist the raw ingredients upon them with documentation wishing them the best of luck. 1. ## We're done when the API is in production. *Submitted by Matthew Reinbold ([@libel\_vox](https://twitter.com/libel%5Fvox?ref=apisyouwonthate.com))* Your clients' mental model of a problem space may be different from the one your team shipped to production. Further, the behaviors demonstrated in the real world may differ from what you were told when discussing idealized hypotheticals. It is like this: I might tell you, in a casual conversation, that I'm eating healthier. However, at 7 pm and under pressure to put dinner on the table, there I am, ordering pizza. The same applies to what behaviors clients aspire to versus what they actually implement. Getting working software into a production environment is a significant milestone for any product. However, when it comes to APIs, that is where the real lessons begin. 1. ## If you use GraphQL, you don't need to design an API. *Submitted by Phil Sturgeon ([@philsturgeon](https://twitter.com/philsturgeon?ref=apisyouwonthate.com))* A *different* set of design decisions should not be conflated with *no* design required. Whether it is [calculating allowed query complexity](https://shopify.engineering/rate-limiting-graphql-apis-calculating-query-complexity?ref=apisyouwonthate.com), [determining appropriate namespaces](https://medium.com/@wundergraph/namespacing-for-graphql-conflict-free-merging-of-any-number-of-apis-a6b81b61dc5f?ref=apisyouwonthate.com), or putting in the effort to [create version-less schemas](https://yelp.github.io/graphql-guidelines/schema-design.html?ref=apisyouwonthate.com), a "good" GraphQL design *still* necessitates several design decisions. 1. ## We'll Update the Docs Later. *Submitted by Mike Bifulco ([@irreverentmike](https://twitter.com/irreverentmike?ref=apisyouwonthate.com))* Documentation debt is a form of technical debt. It is a lie told for momentary expediency. Over time, any initial gains will be dwarfed by the time, effort, and maintenance costs required for ongoing support - costs that, ironically, detract from addressing the documentation omissions. Furthermore, the longer the documentation is left unaddressed, the more effort will be required to repair the damage done. *note: The cover photo for this article is from [Siora Photography](https://unsplash.com/@siora18?utm%5Fsource=unsplash&utm%5Fmedium=referral&utm%5Fcontent=creditCopyText) on [Unsplash](https://unsplash.com/s/photos/frustrated?utm%5Fsource=unsplash&utm%5Fmedium=referral&utm%5Fcontent=creditCopyText)* ### Creating OpenAPI from HTTP Traffic URL: https://apisyouwonthate.com/blog/creating-openapi-from-http-traffic/ Last updated: 2023-03-24T12:22:01.000Z Around this time of year we're thinking about things we're going to do differently, new practices we've been putting off for too long, and mistakes we want to avoid continuing into another year. For many of us in the API world, that is going to be switching to [API Design-first](https://apisyouwonthate.com/blog/api-design-first-vs-code-first), using standards like [OpenAPI](https://openapis.org/?ref=apisyouwonthate.com) to plan and prototype the API long before any code is written. More organizations are switching to [API Design-first with OpenAPI](https://stoplight.io/case-studies/schneider-electric/?ref=apisyouwonthate.com), thanks to huge efforts from tooling vendors - from the bigger folks: [Stoplight](https://stoplight.io/?ref=apisyouwonthate.com) and [Postman](https://postman.com/?ref=apisyouwonthate.com), to the smaller open-source [OpenAPI tools](https://openapi.tools/?ref=apisyouwonthate.com) \- making it far easier to do. Sadly, there's an awkward position many of us are stuck in. We have an API that we built years ago, and now our DevRel team want OpenAPI-based API Reference documentation, the API governance team want OpenAPI to be included in the pull request for any code change, the testing team want our OpenAPI to set up end-to-end contract testing, but we don't have any OpenAPI... AGH! We wrote before about some [slightly hacky ways to create OpenAPI](https://apisyouwonthate.com/blog/creating-api-specifications-from-bullst) from things like Postman Collections, using JSON to JSON Schema converters, and a whole lot of mucking about, but thankfully these days there are far nicer solutions around. One especially smooth tool is [Akita](https://www.akitasoftware.com/?ref=apisyouwonthate.com)! Akita is an observability tool, which can sniff HTTP traffic, and build models of your data. Once it's done that, it can create a graph of all your APIs to give insight into a system, intelligently catch and communicate breaking changes, and various other handy things. We're going to use just part of it's power to create OpenAPI for an API after it's already been deployed to production, so that we can use API Design-first for any new functionality going forwards. Looking for a example wasn't hard. I'd made this mistake myself earlier in the year. We rushed an API for [Protect Earth](https://www.protect.earth/?ref=apisyouwonthate.com). There was no need to design the API because it had to match a contract defined by an existing tree-planting partner, so we just copied some of their JSON, and coded to that rough shape hoping for the best. Of course this rush blew up in our face immediately. The first consumer integration was a lot of awful trial-and-error which took ages, and when the second consumer they didn't have any documentation. I know I know. The mechanic’s car is always broken... So let's get on with it. We could install the Akita Client anywhere, maybe pop it on a staging/production servers to detect that traffic, but installing on a laptop is easier for this workflow: running a proxy, sniffing requests/responses for `https://api.protect.earth/`, and importing into Akita. This is [documented nicely on Akita's docs site](https://docs.akita.software/docs/watch-traffic-with-a-proxy?ref=apisyouwonthate.com), but lets focus on the specific bits for this workflow. ## Step 1: Setup Akita Client locally Head over to [akitasoftware.com](https://www.akitasoftware.com/?ref=apisyouwonthate.com) and click Join Beta. Maybe it's already out so click Register, just get yourself an account somehow. Now we can install the akita-cli client. On macOS that'll be a [brew](https://brew.sh/?ref=apisyouwonthate.com) install, and for [everything else theres docs](https://docs.akita.software/docs/install-akita-client?ref=apisyouwonthate.com). ```bash brew tap akitasoftware/akita && brew install akita-cli ``` When that's installed, use the `akita login` command to log in. You'll want to go fishing for your API Key which is in Settings on the Akita dashboard. ``` akita login API Key ID: apk_0000000000000000000000 API Key Secret: ****************************** Login successful! API keys stored in ${HOME}/.akita/credentials.yaml ``` ## Step 2: Man in the Middle Proxy In order to intercept the HTTP traffic going to an encrypted website (`https://`) we can use the free tool [mitmproxy](https://mitmproxy.org/?ref=apisyouwonthate.com), which is another brew install. ```bash brew install mitmproxy ``` Then, we'll want to grab the har\_dump.py script from mitmproxy which will turn intercepted traffic on their proxy into a [HAR (HTTP Archive format) file](https://w3c.github.io/web-performance/specs/HAR/Overview.html?ref=apisyouwonthate.com). ```bash wget https://raw.githubusercontent.com/mitmproxy/mitmproxy/master/examples/contrib/har_dump.py ``` Ready for action. ## Step 3: Using the proxy In one terminal session, run the proxy server with the har\_dump.py script loaded up, and `dump.har` set so the HAR file will be saved locally. ```bash mitmdump -s ./har_dump.py --set hardump=./dump.har ``` If it's working, the proxy will run on `localhost:8080` so you can use that as a proxy in whatever http client. Maybe you're one of those folks who can remember how curl works. ```bash curl -D - -k --proxy localhost:8080 https://api.protect.earth/v1/orders/c36916f7-7591-47e5-b069-f983b9c0f320 ``` That will make requests to the `https://api.protect.earth/v1/orders/{uuid}` endpoint of the Protect Earth API, pass the request and response through mitmproxy, and write the output to `dump.har`. Doing all of this in curl was a bit of a mess so I grabbed [Insomnia](https://insomnia.rest/?ref=apisyouwonthate.com) and clicked around the API a bit, hitting as many resources and collections as possible, so the OpenAPI is based on a superset of all the data it's seen, instead of just the one JSON representation. ### Step 4: Converting HAR to OpenAPI There are a lot of tools out there to convert a HAR to OpenAPI, but some of them are old, some of them are bad, and most of them are both. Akita is fantastic at doing this, and can handle all nullable, optional, polymorphic, and generally funny shaped data! It'll take a stab at noticing formats of strings, all of which saves you time from filling all this in manually. The `akita apispec` command can import `dump.har` to your service, and give it a name. The service was `protect-earth` and the spec was just called mySpec because that's what the docs said and it doesn't seem to matter. ```bash akita apispec --traces dump.har --out akita://protect-earth:spec:mySpec ``` The Services page in Akita should now be aware of the service you just uploaded. ![Once logged into Akita dashboard, click Services on the navigation bar, and you should see your new service showing up.](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/posts/creating-openapi-from-http-traffic/akita-services.png) Click on that and there will be a list of endpoints its aware of, with parameters used to avoid duplicating endpoints for different UUIDs or other parameters as other tools often do. ![A list of endpoints, with an expandable panel showing metadata for each body property, objects, arrays, even headers its noticed.](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/posts/creating-openapi-from-http-traffic/akita-endpoints.png) Those endpoints have all their metadata associated in Akita, which means it's ready for exporting as OpenAPI through the web interface. ![The "Download Spec" button will download an OpenAPI file in YAML](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/posts/creating-openapi-from-http-traffic/akita-export.png) *The OpenAPI document will be created as YAML, and at time of writing is producing OpenAPI v3.0\. Ideally it would soon be updated to OpenAPI v3.1, but [the differences are not huge and can be changed manually](https://www.openapis.org/blog/2021/02/16/migrating-from-openapi-3-0-to-3-1-0?ref=apisyouwonthate.com).* Once you've got this OpenAPI YAML document you can shove it into your Git repo to live alongside your code. It might not be perfect, but you can hook that Git repo up to a web-based OpenAPI editor like [Stoplight Platform](https://stoplight.io/welcome/create/?ref=apisyouwonthate.com), or a local file editor like Stoplight Studio, or just manually wrangle the YAML in your favourite text editor. ![Connect a project to Git in Stoplight via the Add Project > Add from GitHub/GitLab/etc wizard.](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/posts/creating-openapi-from-http-traffic/stoplight-git.png) However you go about it, you can tidy up the OpenAPI document according to your preferences, and publish the docs when you're done. ![Tidied up and improved OpenAPI document hosted on Stoplight via Git.](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/posts/creating-openapi-from-http-traffic/pretty-docs.png) How you might chose to tidy up the OpenAPI is another article for another day, but getting *some* OpenAPI without having to manually wrangle it all by hand is a huge timesaver. More importantly it's likely to help API teams get on board with any organization-wide push for API Design-first, or any other API Program or workflow that requires OpenAPI. Now, I'm off to plan out a new endpoint for the [Protect Earth API](https://protect-earth.stoplight.io/docs/protect-earth-api/?ref=apisyouwonthate.com) using the design-first approach, so I can give multiple consumers a [mock endpoint](https://meta.stoplight.io/docs/prism/ZG9jOjk1-http-mocking?ref=apisyouwonthate.com) to hit to see if it'll work for them, before I bother writing up a bunch of code I'll only have to change later based on their feedback. ### JSON Schema Bundling Finally Formalised URL: https://apisyouwonthate.com/blog/json-schema-bundling-finally-formalised/ Last updated: 2023-03-24T12:22:21.000Z Here we have a special treat, a guest post written by *[Ben Hutton](https://twitter.com/relequestual?ref=apisyouwonthate.com)*, JSON Schema Specification Lead at [Postman](https://www.postman.com/?ref=apisyouwonthate.com). They are here to talk about the new way in which "bundling" works in OpenAPI, which is the art of turning multiple JSON Schema / OpenAPI documents - linked together with `$ref` \- into a single document. This is done for portability and compatibility, as some tools do not understand "remote references". To support these various needs a few different approaches popped up - often called ["bundling" and "dereferencing"](https://blog.stoplight.io/keeping-openapi-dry-and-portable?ref=apisyouwonthate.com) \- with various pros and cons. These terms were not standard, and their effects were not standard either as various tooling vendors implemented things differently. An end to that hell! Learn about the new `$id` keyword, and what it means for users and tooling vendors. **\-- Phil** ## Bundling has renewed importance OpenAPI has long since [put the spotlight on JSON Schema,](https://apisyouwonthate.com/blog/openapi-v3-1-and-json-schema) and the release of OpenAPI 3.1 has huge implications for the future of both projects. I'm truly excited. Developers of platforms and libraries that use OpenAPI haven't had such a shake up before, and my feeling is it may take more than a few releases to correctly implement all the new shiny features full JSON Schema has to offer. While the number of changes from JSON Schema draft-04 to draft 2020-12 are vast and the subject of more blog posts than are likely interesting, one of the key "features" of draft 2020-12 is a defined bundling process. (draft-04 is the version of JSON Schema that OAS used prior to version 3.1.0; or rather, a subset/superset of it.) Indeed, bundling, if anything, is going to be more important to get right than ever. OAS 3.1 ushering in full JSON Schema support dramatically increases the likelihood that developers with existing JSON Schema documents will use them **by reference** in new and updated OpenAPI definitions. Ultimate source of truth matters, and it's often the JSON Schemas. Many tools don't support referencing external resources. Bundling is a convenient way to package up schema resources spread across multiple files in a single file for use elsewhere, such as an OpenAPI document. ## Existing solutions? New solutions! There are several libraries which offer bundling solutions, however they all have caveats, and I haven't seen any to date which are fully JSON Schema aware. The most popular of these libraries is called `json-schema-ref-parser`, however it [reports](https://github.com/APIDevTools/json-schema-ref-parser/issues/97?ref=apisyouwonthate.com#issuecomment-415963663) that it was not intended to be JSON Schema aware, and is only intended to cover the JSON Reference specification (Which has been bundled back into the JSON Schema specification now). We are hoping to provide you with a canonical implementation, and enough information to get started building your own in your language of choice. (Although, it's always best to read the full specification when developing implementations.) ## Bundling fundamentals Firstly, let's visit some key definitions in JSON Schema draft 2020-12. The `$id` keyword is used to identify a "schema resource". In the example below, the `$id` is `https://example.org/schemas/mixins/integer` for the resource. ```json { "$id": "https://example.org/schemas/mixins/integer", "$schema": "https://json-schema.org/draft/2020-12/schema", "description": "Must be an integer", "type": "integer" } ``` A "Compound Schema Document" is a JSON document which has multiple embedded JSON Schema Resources. Below is a simplified example of one we’ll unpack a bit later. ```json { "$id": "https://example.org/schemas/examples/non-negative-integer-bundle", "$schema": "https://json-schema.org/draft/2020-12/schema", "description": "Must be a non-negative integer", "$comment": "A JSON Schema Compound Document. Aka a bundled schema.", "$defs": { "https://example.org/schemas/mixins/integer": { "$schema": "https://json-schema.org/draft/2020-12/schema", "$id": "https://example.org/schemas/mixins/integer", "description": "Must be an integer", "type": "integer" }, "https://example.org/schemas/mixins/non-negative": { "$schema": "https://json-schema.org/draft/2020-12/schema", "$id": "https://example.org/schemas/mixins/non-negative", "description": "Not allowed to be negative", "minimum": 0 }, "nonNegativeInteger": { "allOf": [ { "$ref": "/schemas/mixins/integer" }, { "$ref": "/schemas/mixins/non-negative" } ] } }, "$ref": "#/$defs/nonNegativeInteger" } ``` Last, let's look at the carefully crafted definition of "bundling" according to the JSON Schema specification: "The bundling process for creating a Compound Schema Document is defined as taking references (such as "$ref") to an external Schema Resource and embedding the referenced Schema Resources within the referring document. Bundling SHOULD be done in such a way that all URIs (used for referencing) in the base document and any referenced/ embedded documents do not require altering." With these definitions in mind, now we can look at the defined bundling process for JSON Schema resources! We will only cover the ideal situation in this article. The goal here is to have no external Schema Resources. Note, this article does NOT cover "total dereferencing", which is removing all uses of `$ref` from a schema. This is not advised, and is not always even possible, such as when there are self references. ## Bundling Simple External Resources In our first example, we have an ideal situation for bundling. Each schema has an `$id` and `$schema` defined, making the bundling process simple. We'll cover various other situations and edge cases in further examples, but having each resource define its own identity and dialect is always preferable. Our primary schema resource references two other schema resources using the in-place applicator `$ref` with the value being a relative URI. The relative URI is resolved against the base URI, which in this instance is found in the primary schema resource's `$id` value. By combining "integer" and "non-negative" schemas, we create a "non-negative integer" schema. ```json { "$id": "https://example.org/schemas/mixins/integer", "$schema": "https://json-schema.org/draft/2020-12/schema", "description": "Must be an integer", "type": "integer" } ``` ```json { "$id": "https://example.org/schemas/mixins/non-negative", "$schema": "https://json-schema.org/draft/2020-12/schema", "description": "Not allowed to be negative", "minimum": 0 } ``` ```json { "$id": "https://example.org/schemas/examples/non-negative-integer", "$schema": "https://json-schema.org/draft/2020-12/schema", "description": "Must be a non-negative integer", "$comment": "A JSON Schema that uses multiple external references", "$defs": { "nonNegativeInteger": { "allOf": [ { "$ref": "/schemas/mixins/integer" }, { "$ref": "/schemas/mixins/non-negative" } ] } }, "$ref": "#/$defs/nonNegativeInteger" } ``` Should "non-negative-integer" schema be used as the primary schema in an implementation, the other schemas would need to be available to the implementation. At this point, exactly how that implementation loads in the schemas doesn't matter, as they have fully qualified URIs as their identity defined in `$id`. It could be a bunch of files in a directory, archive, database, wherever, so long as they are all loaded into an implementation somehow. Any implementation that loads in schemas should build an internal local index of schema URIs defined in `$id` to schema resources. Remember, any schema which provides a value for `$id` is considered a Schema Resource. Let's resolve (dereference) one of the references in our primary schema. `"$ref": "/schemas/mixins/integer"` resolves to a fully qualified URI of `https://example.org/schemas/mixins/integer` by following the rules for first determining the base URI and then resolving the relative URI against that base URI. The implementation should then check its internal index of schema identifiers and schema resources, finding a match, and using the appropriate previously loaded schema resource. The bundling process is done. The previously externally referenced schemas are copied into `$defs` in our primary schema, as is. The keys for the `$defs` object are the identifying URIs, but they can be anything, as those values won't be referenced (They could be UUIDs if you like). Looking at our final bundled schema… I mean "Compound Schema Document", we now have multiple Schema Resources embedded in a single Schema document. ```json { "$id": "https://example.org/schemas/examples/non-negative-integer-bundle", "$schema": "https://json-schema.org/draft/2020-12/schema", "description": "Must be a non-negative integer", "$comment": "A JSON Schema Compound Document. Aka a bundled schema.", "$defs": { "https://example.org/schemas/mixins/integer": { "$schema": "https://json-schema.org/draft/2020-12/schema", "$id": "https://example.org/schemas/mixins/integer", "description": "Must be an integer", "type": "integer" }, "https://example.org/schemas/mixins/non-negative": { "$schema": "https://json-schema.org/draft/2020-12/schema", "$id": "https://example.org/schemas/mixins/non-negative", "description": "Not allowed to be negative", "minimum": 0 }, "nonNegativeInteger": { "allOf": [ { "$ref": "/schemas/mixins/integer" }, { "$ref": "/schemas/mixins/non-negative" } ] } }, "$ref": "#/$defs/nonNegativeInteger" } ``` When the bundled schema is initially loaded and evaluated, the implementation should create its own internal index of schema identifiers and schema resources, just as before. The relative URIs used to reference those schema resources need not change. The simplest way to see this bundled schema working as expected is to paste it into [https://json-schema.hyperjump.io](https://json-schema.hyperjump.io/?ref=apisyouwonthate.com) and then try different values for the instance. I'm hopeful to bring several updates to [https://example.org](https://example.org/?ref=apisyouwonthate.com) over the next few months, but times are busy as we continue to elevate JSON Schema as an organisation. It’s worth remembering that the example in this article shows the ideal situation, when best practices have been followed. The JSON Schema specification does define additional processes for non-ideal situations and edge cases (such as when `$id` or `$schema` are not set), however, some solutions may be indirectly related to Compound JSON Schema Documents. For example, establishing the base URI follows the steps laid out in RFC3986, which JSON Schema does not redefine. ## OpenAPI Specification Example Let’s look at an example of how this might work with an OpenAPI definition. ```yaml openapi: 3.1.0 info: title: API version: 1.0.0 components: schemas: non-negative-integer: $ref: 'https://example.org/schemas/examples/non-negative-integer' ``` We start with our input OpenAPI 3.1.0 specification document. For brevity, we’re only showing the components section with a single component, but let’s assume some other part of the document uses the component schema "non-negative-integer". "non-negative-integer" has a single reference to a JSON Schema resource. The reference URI is an absolute URI, including domain and path, meaning there’s no need to do any "resolve the relative URI against the base URI" dance. All the schemas required to resolve and bundle the reference are provided to the bundling tooling. After the schemas are loaded into the implementation, their originating physical location no longer matters. ```yaml openapi: 3.1.0 info: title: API version: 1.0.0 components: schemas: # This name has not changed, or been replaced, as it already existed and is likely to be referenced elsewhere non-negative-integer: # This Reference URI hasn't changed $ref: 'https://example.org/schemas/examples/non-negative-integer' # The path name already existed. This key doesn't really matter. It could be anything. It's just for human readers. It could be an MD5! non-negative-integer-2: $schema: 'https://json-schema.org/draft/2020-12/schema' $id: 'https://example.org/schemas/examples/non-negative-integer' description: Must be a non-negative integer $comment: A JSON Schema that uses multiple external references $defs: nonNegativeInteger: allOf: # These references remain unchanged because they rely on the base URI of this schema resource - $ref: /schemas/mixins/integer - $ref: /schemas/mixins/non-negative $ref: '#/$defs/nonNegativeInteger' integer: $schema: 'https://json-schema.org/draft/2020-12/schema' $id: 'https://example.org/schemas/mixins/integer' description: Must be an integer type: integer non-negative: $schema: 'https://json-schema.org/draft/2020-12/schema' $id: 'https://example.org/schemas/mixins/non-negative' description: Not allowed to be negative minimum: 0 ``` The schemas are inserted into the `components/schemas` location of the OAS document. The keys used in the `schemas` object have no importance for reference resolution, although you will want to avoid potential duplications. References need not change, and a processor of the resulting bundled or Compound Document, should look for the use of embedded Schema Resources within the OAS document, keeping track of the `$id` values. --- So there you have it! Do the tools you use support `$id`? Poke around and find out! You can reach out to the JSON Schema folks on Twitter [@jsonschema](https://twitter.com/jsonschema?ref=apisyouwonthate.com) or [Slack](https://json-schema.org/slack?ref=apisyouwonthate.com). This article was first published on the JSON Schema Blog and is canonically located at: [https://json-schema.org/blog/posts/bundling-json-schema-compound-documents](https://json-schema.org/blog/posts/bundling-json-schema-compound-documents?ref=apisyouwonthate.com) *[Header photo created by vanitjan - www.freepik.com](https://www.freepik.com/vanitjan1?ref=apisyouwonthate.com)* ### Why Show Users Garbage API Errors? URL: https://apisyouwonthate.com/blog/why-show-users-garbage-api-errors/ Last updated: 2023-03-24T12:22:34.000Z A million things can go wrong when an API client talks to an API, from the expected to the unexpected, from business logic and validation failures, to dropped connections and race conditions. The server could run out of RAM, the request could time out, an automatic retry might trigger a weird validation message for something that actually worked fine but failed on the retry and now is in a conflict state, or maybe the API developers pushed a breaking change without telling anyone and now the API client cannot comprehend the current state. When things go wrong, what do you do? Do you vomit random API error codes and exception class names all over your end users screens, pointlessly confusing the hell out of them? Probably! ![Error box on the trainline iOS app showing Oops, and an unintelligable error we'll walk through now.](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/posts/why-show-users-garbage-errors/trainline.jpg) Trainline barfed this error message at me when I was trying to book a train ticket from London to Bath. "Oops" ok cute. An error has happened. What went wrong. "This operation could not be completed." Thank you yes that was clear from the Oops but good to be explicit at least. "APIService.APIServiceError" Yes ok I see the word error in there I am aware there was an error, but what does that tell me that wasn't covered with "Oops" and "This operation could not be completed." Finally "error 5" That clears it up perfectly! 🙃 I'm sure that means something to somebody, but it doesn’t mean anything to me. I'm just trying to buy a train ticket, and this doesn't give me any information I can use to change the outcome. Should I try again? Should I give up trying to use the app? Should I phone customer support to try and get a ticket? Has the train been cancelled?! Tell users something relevant, or don't tell them anything. If there was a network error, the Trainline app could have said "There’s a problem with your connection, please check your connection settings." If there was a 500-599 error the application could have said "Sorry, our servers are struggling. Please try again in a few minutes, or call customer support on 555-555-5555." ![Bunq iOS app showing error message: Invalid status ACCEPTED_PENDING_RESPONSE for SecureCode with status ACCEPTED.](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/posts/why-show-users-garbage-errors/bunq.jpg) My bank is shouting at me while I try to book a hotel room. Bunq, [the tree-planting Dutch bank](https://www.bunq.com/benefits/trees?ref=apisyouwonthate.com), started shouting at me today. It seemed like my first attempt to approve a payment failed with no UI update. So I clicked approve again, and it hurled these CONSTANTS at me, which if I was NON\_TECHNICAL would be SUPER\_CONFUSING! I guess the server had approved the payment but not told the client somehow, and now I'm stuck out of state, getting moaned at because it didn't like being accepted a second time There's two main problems here: 1. On the client-side, it's weird that they are displaying errors suggesting status is invalid. This makes it look like Bunq doesn't know how Bunq works. Why would an end user ever want to see this? If I am sending ACCEPTED\_PENDING\_RESPONSE to a resource that has ACCEPTED, the human equivalent of that is "This payment was already accepted." so why not say that. instead of spitting out constants and leaking implementation details to the users eyeballs? 2. On the backend, their API could make life a lot easier for the client by using "idempotency keys" (e.g.: [Stripe](https://stripe.com/docs/api/idempotent%5Frequests?ref=apisyouwonthate.com)). If they had assigned a `Idempotency-Key` value of `PHILTUES123MONKEYS` and I approved that request, when it failed to update the UI for whatever reason and I tried again, I would be making a second request with that same idempotency key. This would let the API know that instead of shouting "AGHHHH YOU CANNOT DO THAT, 409 Conflict, bad status, BLARP!" that it could instead say "You got it, job done!" even if it had already been done. Either way, the client and API are failing to let the user do what they want to do: all I want is to mark that payment as accepted, so I can stop standing in the rain and check into my hotel. If it's already been accepted then happy days, we're out of here, move that UI along. ![Error message on twitter iOS app: The operation couldn't be completed. (com.twitter.TFSTwitterAPIError error 120.)](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/posts/why-show-users-garbage-errors/twitter.jpg) Twitter slapped [Alex Garrett-Smith](https://twitter.com/alexjgarrett?ref=apisyouwonthate.com) with this nonsense whilst trying to update their bio. This looks incredibly similar to the Trainline error. It's probably a default in some common iOS HTTP library. Again, we've got ourselves an utterly useless exception class name, and it's not even saying com.twitter.ClientError, just "Somefing dur brok3n!". Then yep, a random "Error 120". This is not a HTTP status code, but something Twitter specific in the JSON body of an error. ![An error tossed by twitter, which was not human-readable](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/posts/why-show-users-garbage-errors/twitter-explains.png) Twitter explain this random API error code to developers but not to users of their mobile app. What is slightly more infuriating about this one, is that Twitter have gone to the effort of [documenting this error code](https://developer.twitter.com/en/support/twitter-api/error-troubleshooting?ref=apisyouwonthate.com) on Twitter Developers, but their iOS team didn't fancy implementing a quick check for code 120? ```js switch error.code { case "120": alert( "These changes are not valid, why did our client" + "application let you get this far, we are as " + "confused as you are, and we have contacted the " + "engineering team to improve their client-side " + "validation. Sorry for wasting your time." }) case "130": alert( "Twitter servers are absolutely melting right now. " + "Take a break. Go outside. Seize the day!" ) case "323": alert("Only one animated GIF may be attached to a single Tweet.") default: if (error instanceof NetworkError) { alert("Please check your internet connection.") } else { alert( "Something wild has gone wrong. We have " + "automatically reported this error to the engineers." ) sendErrorToRollbar(error); } } ``` All client applications should have code like this. Every single one. Bake it into your SDKs as middleware, to kick off after your [auto-retry logic and circuit breakers](https://apisyouwonthate.com/blog/taking-a-timeout-from-poor-performance) have done their job. This lets you decide which known errors to send to error reporting software like Rollbar, BugSnag, Sentry, etc. and which would be a massive waste of time to report. Oh, did Gary just get a 404 trying to get page 2347 from the website when he meant to type 234? Who cares. Most importantly, why is documentation telling me more than what the end user is seeing? It's maddening. Nobody cares about your API, they just want to use the software to do the thing they're trying to do. ## High Quality API Errors API clients cannot work with what they don't have, so API developers should always try to explain what has gone wrong. We've written about [creating high quality API errors](https://apisyouwonthate.com/blog/useful-api-errors-for-rest-graphql-and-grpc) for humans and computers in the past, and the short version is: use [RFC 7807](https://tools.ietf.org/html/rfc7807?ref=apisyouwonthate.com), or maybe leverage [JSON:API Errors](https://jsonapi.org/examples/?ref=apisyouwonthate.com#error-objects) properly if you're already using JSON:API. Something like RFC 7807 is absolutely brilliant, because not only does it suggest you write a useful error message for humans, but it allows other metadata to be provided which mean API clients can provide all sorts of useful information, by knowing that `https://example.com/probs/out-of-credit` is going to have a `balance` property. ```http HTTP/1.1 403 Forbidden Content-Type: application/problem+json Content-Language: en { "type": "https://example.com/probs/out-of-credit", "title": "You do not have enough credit.", "detail": "Your current balance is 30, but that costs 50.", "instance": "/account/12345/msgs/abc", "balance": 30, "accounts": ["/account/12345", "/account/67890"] } ``` If the API only needs to serve English speakers, then maybe that `detail` can be shoved straight out to the client. If suddenly there is a need for i18n, maybe that can be supported on the API side with `Accept-Lang: fr`, or maybe the client can write their own messages for `https://example.com/probs/out-of-credit` and plop the variables in. There are a lot of ways to make good error messages show up to the end-user, with some trade-offs around simplicity vs cusstomizability, but don't just barf exceptions at your users like they know how to read code and fancy googling your API error codes. That's just rude. ### There's No Reason to Write OpenAPI By Hand URL: https://apisyouwonthate.com/blog/theres-no-reason-to-write-openapi-by-hand/ Last updated: 2023-03-24T12:22:46.000Z Some API developers use API descriptions to plan the interface of an API before building it, which is known as the "API design first" workflow. Others build the API then generate (or manually produce) API descriptions later, which is the "code-first" workflow. We wrote *[API Design-first vs Code-first](https://apisyouwonthate.com/blog/api-design-first-vs-code-first/)* recently to help get you up to speed on the differences, but how do you actually create these API descriptions? Many of these API descriptions ([OpenAPI](https://www.openapis.org/?ref=apisyouwonthate.com), [JSON Schema](https://json-schema.org/?ref=apisyouwonthate.com), or [GraphQL Schemas](https://graphql.org/learn/schema/?ref=apisyouwonthate.com)) involve writing out a bunch of special keywords in YAML, JSON, or another text language. Neither the design-first or code-first crew enjoy writing thousands of lines of that by hand. Why? Well, say you've got a list of integers: ```yaml schema: type: array description: List of IDs items: type: integer description: Pet ID ``` Designing an API with 100 endpoints like that will get you through at least one keyboard (maybe two if it's a Macbook). Designing a few APIs could drive you bonkers. Let's look at some alternatives to hand-rolling your own homegrown artisanal YAML, and how these approaches fit into design-first or code-first. More importantly, let's see how they can be used to achieve the ultimate goal for API designers: one source of truth for API descriptions that power mocks, documentation, request validation, shareable design libraries, and more. ## Annotations Some programming languages support a syntax-level feature called "Annotations", for example [Java Annotations](https://www.baeldung.com/java-default-annotations?ref=apisyouwonthate.com). An OpenAPI annotation framework provides a bunch of keywords that help the API developer describe the interface of the HTTP request and response, and hopefully it's telling the truth. ```java class UserController { @OpenApi( path = "/users", method = HttpMethod.POST, // ... ) public static void createUser(Context ctx) { // ... } } ``` Some languages do not have any support for annotations, and they achieve this with docblock comments. ```php /** * @OA\Get(path="/2.0/users/{username}", * operationId="getUserByName", * @OA\Parameter(name="username", * in="path", * required=true, * description=Explaining all about the username parameter * @OA\Schema(type="string") * ), * @OA\Response(response="200", * description="The User", * @OA\JsonContent(ref="#/components/schemas/user"), * @OA\Link(link="userRepositories", ref="#/components/links/UserRepositories") * ) * ) */ public function getUserByName($username, $newparam) { // implementation logic ... } ``` In JavaScript it looks a bit like this: ```js /** * @swagger * /users: * get: * description: Returns users * produces: * - application/json * responses: * 200: * description: users * schema: * type: array * items: * $ref: '#/definitions/User' */ app.get('/users', (req, res) => { // implementation logic ... }); ``` This approach is the oldest around, and is primarily used by the code-first people. It's easy to understand why: you wrote the code already, so now lets get some docs! Sprinkle some keywords around the code, and the annotation system will integrate with your framework to emit a `/docs` endpoint, and boom, you've got some documentation. Great, if all you want is documentation. Design-first people also sometimes use this approach. They design the entire API (writing YAML by hand or with one of the other approaches we're going to mention), then use Server Generators like [openapi-generator](https://openapi-generator.tech/?ref=apisyouwonthate.com) or [swagger-generator](https://github.com/swagger-api/swagger-codegen?ref=apisyouwonthate.com) to create their API code. This API code is created from the machine-readable documents that were made in the design process, and the code that is generated is chock full of annotations already, which in turn can generate documentation. Throw those machine-readable documents away, the annotated code is the source of truth now... right? One downside to annotations is that they don't confirm the code is doing what it says. I've heard the argument "Annotations are closer to the code they describe, so developers are more likely to keep it up to date". Do not confuse proximity with accuracy. Developers can forget to make the changes, and developers can make mistakes. Annotation users need to find a way to contract test the actual output against these annotations, which we've written about before in [Keeping Documentation Honest](https://apisyouwonthate.com/blog/keeping-documentation-honest). The tools in this article generally involve the machine-readable OpenAPI / JSON Schema files around, so you need to export them back to a machine-readable format in order to compare them to the code... This means running a command in the command line which pulls the annotations out into a machine-readable file, then running a tool like [Dredd](https://dredd.org/?ref=apisyouwonthate.com) or a JSON Schema validators, which is a pretty awkward step. Design-first is incompatible with all this, unless you chose to design the API, then generate code with annotations, then figure out how to keep the code, the annotations AND the machine readable designs up to date. If there's anything worse than two sources of truth it's three... For this reason, folks who like design-first run and hide from annotations, but the folks who like annotations generally really really love them because to them their code is the source of truth and if they can crowbar one of these test suites in to confirm that then they're perfectly happy. This mindset can lead to API clients being a bit of an afterthought, but that's another topic for another article. ## DSL (Domain Specific Language) A few DSLs popped up over the years, created by people who wanted to create API Descriptions, but didn't fancy writing it out in that specific format, with articles like *[Making OpenAPI / Swagger Bearable With Your Own DSL](https://developer.squareup.com/blog/making-openapi-swagger-bearable-with-your-own-dsl/?ref=apisyouwonthate.com)*. DSL's can be used in code-first or design-first. You have your code, you have your DSL-based descriptions, and whatever format they were written in doesn't make much difference here. ``` ;;; ENTITIES (define pet-entity (entity "Pet" 'race (string "What kind of dog / cat this is (labrador, golden retriever, siamese, etc...)" "Labrador") 'origin (string "Country of origin" "Egypt") 'birthday (datetime "Birth date of the pet" "2017-10-20T00:14:02+0000") 'species (string "What kind of animal is this" "dog" #:enum '("dog" "cat")))) (define $pet (schema-reference 'Pet pet-entity)) ;;; RESPONSES (define list-pets-response (jsonapi-paginated-response "List of pets" ($pet))) ;;; REQUESTS (define pet-request (json-request "Pet Request Body" ($pet))) ;;; MAIN DOC (define swagger (my-service-api-doc "Pet Store" "Per store pets management" (path "/pets") (endpoint-group 'tags pet-tags 'parameters (list store-id-param) 'get (endpoint 'operationId "listPets" 'summary "Retrieve all the pets for this store" 'parameters pagination-params 'responses (with-standard-get-responses 200 list-pets-response)) 'post (endpoint 'operationId "createPet" 'summary "Create a new Pet record" 'requestBody pet-request 'parameters (list xsrf-token) 'responses (with-standard-post-responses 200 single-pet-response))) ``` This is shorter than the OpenAPI YAML it replaces, but it's also another format for people to learn. People who know how to write up OpenAPI will need to learn this format, and the folks familiar with a different DSL will have to learn this format too. Your editor will also need to learn about this format if you want autocomplete, syntax highlighting, linting and validation. The "native" description formats all have this, but these DSLs usually do not. This approach, just like annotations, do not help you ensure that what you're writing in the DSL is actually correct. The contract written down in the description could be completely incorrect. Some DSLs like [Rswag](https://github.com/rswag/rswag?ref=apisyouwonthate.com) aim to solve this by having their DSL be written as integration texts. The source of truth for how you create OpenAPI is literally integration tests: ```ruby describe 'Blogs API' do path '/blogs/{id}' do get 'Retrieves a blog' do tags 'Blogs' produces 'application/json', 'application/xml' parameter name: :id, :in => :path, :type => :string response '200', 'blog found' do schema type: :object, properties: { id: { type: :integer }, title: { type: :string }, content: { type: :string } }, required: [ 'id', 'title', 'content' ] let(:id) { Blog.create(title: 'foo', content: 'bar').id } run_test! end end end end ``` This approach is pretty handy for [Test-driven development (TDD)](https://semaphoreci.com/blog/test-driven-development?ref=apisyouwonthate.com) advocates, but you're just writing OpenAPI in another form which isn't particularly any shorter, just more Ruby-ish. RSwag was a big favourite at my last job, but it's had a rough time getting updated onto OpenAPI v3.0 (still a work in progress 3 years after OpenAPI v3.0 was released). Writing in a DSL or annotations means you're at the mercy of that maintainer to support functionality that you could already use if you could just... edit the OpenAPI yourself. ## Graphical Design Editors What is the alternative to editing the files but now having to wrangle YAML? Visual thinkers and non-technical people might want a wizard mode, the ability to create arrays of objects with a few buttons, and selection boxes for [shared models](https://stoplight.io/blog/keeping-openapi-dry-and-portable/?ref=apisyouwonthate.com) without having to think about the filepath. Graphical design editors are pretty new in the world of OpenAPI and GraphQL, with a few popping up over the last year or two. ![Stoplight studio's form editor view](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/posts/annotations-dsl-editors/editor.png) [Stoplight Studio](https://stoplight.io/studio?ref=apisyouwonthate.com) main form editor view, showing some the \`$ref\` selector for picking OpenAPI models. Two years ago I was looking around for a beautiful, effective graphical editor to satisfy some code-first sticklers pushing back against [API design-first at WeWork](https://apisyouwonthate.com/blog/weworks-api-specification-workflow/). I figured a GUI would help them convert, and Stoplight told me they were planning a new GUI. Shortly after seeing their amazing prototype I joined the company to help roll it out to even more folks, and now my job is gathering feedback from the API community to make [Studio](https://stoplight.io/studio/?ref=apisyouwonthate.com), our open-source tools, and the upcoming SaaS platform even better. 🥳 Modern GUI editors have mocks and docs publishing built right in so you no longer have to figure out your own "DocOps". Editors like Stoplight Studio add "Design Libraries", where you can manage shared models between multiple APIs in an organization. These editors support organization-wide [style guides to have the editor lint](https://apisyouwonthate.com/blog/automated-style-guides-for-rest-graphql-grpc) during editing (and/or in continuous integration) to enforce consistency, and a bajillion other things. ![Stoplight studio](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/posts/annotations-dsl-editors/editor-lint.png) [Stoplight Studio](https://stoplight.io/studio?ref=apisyouwonthate.com) with [Spectral](https://stoplight.io/open-source/spectral?ref=apisyouwonthate.com) linting results from a custom rule written to stop folks designing a GET endpoint with a body. There are a bunch of different OpenAPI-based graphical editors around, check out our list on [OpenAPI.Tools](https://openapi.tools/?ref=apisyouwonthate.com#gui-editors): - [Apicurio](https://www.apicur.io/?ref=apisyouwonthate.com) - [Apitive Studio](https://www.apitive.com/?ref=apisyouwonthate.com) - [Reprezen API Studio](https://www.reprezen.com/?ref=apisyouwonthate.com) - [Stoplight Studio](https://stoplight.io/studio/?ref=apisyouwonthate.com) GraphQL fans, who are having a lot of [the same conversations](https://blog.logrocket.com/code-first-vs-schema-first-development-graphql/?ref=apisyouwonthate.com) right now, can check out these: - [GraphQL Designer](http://graphqldesigner.com/?ref=apisyouwonthate.com) - [GraphQL Editor](https://graphqleditor.com/?ref=apisyouwonthate.com) Some editors will help you with part of the API design life-cycle, but make a lot of difficult assumptions about what order you're going to do what in. They might help you *create* a design, then they'd assume you wanted to export it and generate some code, leaving your machine-readable description documents floating around to become obsolete, and giving you no way to edit them even if you wanted to. Other tools let you import an OpenAPI document, but convert it to their own internal format and provide no way to pull the OpenAPI back out again. Using tools where the format changes entirely at different points locks you into whatever workflows they support, instead of letting you plug-and-play your own tooling at every stage of the process. For me the ideal solution is supporting a git-based flow, where they live in the repository (maybe before the code exists), and regardless of how these API descriptions were created you can edit them and send a pull request back to that repo with the changes you made. Leaving the machine-readable source of truth in the repo means any integrations are possible. Continuous integration processes can deploy documentation to any documentation provider, you can use use [any code-generators](https://openapi.tools/?ref=apisyouwonthate.com#sdk) to build and publish SDKs, sync with popular HTTP clients like Postman or Insomnia instead of maintaining API descriptions *and* bookmarks, and fully take care of contract testing in-repo or end-to-end repositories. If your editor is backed by a design library then the repositories will be analyzed on push, allowing others to use these updated models instead of having 1000 different versions of a "User", "Company" or a "Flight". ![The main Stoplight Explorer dashboard](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/posts/annotations-dsl-editors/explorer.png) The main Stoplight Explorer dashboard, providing search across all APIs in an organization. This has only been available for [Enterprise](https://stoplight.io/enterprise?ref=apisyouwonthate.com) users, but is available for everyone this April. But, how does having an editor help you catch mistakes? Stoplight Studio has a [built in HTTP Client](https://stoplight.io/p/docs/gh/stoplightio/studio/docs/Design-and-Modeling/05-request-maker.md?ref=apisyouwonthate.com), which amongst other things is watching for mismatches between the OpenAPI defined for the API and the actual HTTP requests you send. It will also notice mismatches between OpenAPI and the responses coming back, so you'll see mistakes popping up like this: ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/posts/annotations-dsl-editors/rm-mismatch.png) Noticing contract mismatches in a HTTP client is all well and good for spotting mistakes in requests you're making, but you'll want to automate this checking too. Instead of having all your model validation rules and header checking written in code, and then also writing it down in the API descriptions, use the existing machine-readable descriptions for [validating incoming requests](https://apisyouwonthate.com/blog/server-side-validation-with-api-descriptions). If your request logic is powered by API descriptions, there is no need to check that it matches the code, because it... is the code. Framework middlewares for every framework and every language implement this. NodeJS has about 100. - **PHP:** [league/openapi-psr7-validator](https://github.com/thephpleague/openapi-psr7-validator?ref=apisyouwonthate.com) - **Node.js:** [fastify](https://github.com/fastify/fastify/blob/master/docs/Validation-and-Serialization.md?ref=apisyouwonthate.com) / [express-swagger-ajv-validator](https://github.com/Zooz/express-ajv-swagger-validation?ref=apisyouwonthate.com) / [express-openapi-validate](https://github.com/Hilzu/express-openapi-validate?ref=apisyouwonthate.com) - **Ruby/Rails:** [committee](https://github.com/interagent/committee?ref=apisyouwonthate.com) - **Python:** [connexion](https://github.com/zalando/connexion?ref=apisyouwonthate.com) - **Perl:** [Mojolicious::Plugin::OpenAPI](https://metacpan.org/pod/Mojolicious::Plugin::OpenAPI?ref=apisyouwonthate.com) The Rails one, for example, takes a single line to set up the Rack middleware: ```ruby use Committee::Middleware::RequestValidation, schema_path: "./openapi.yaml" ``` That covers incoming requests, but how to ensure the responses are doing the right thing? Some of those middlewares will implement response validation too, which can confirm the response coming through it matches the code. This is a great thing to enable in dev and staging, but turn that off for production. 🤣 Another approach to checking responses is contract testing. Instead of having some DSL-based integration testing suite specifically for checking the responses, or using some other tool where you have to write out the contract again, you can can just [use the API descriptions as contract tests](https://apisyouwonthate.com/blog/writing-documentation-via-contract-testing). So easy, and all it takes is a few lines of code to mush the response into a data validator for the API description format of your choice. Another approach is using [Prism Proxy](https://stoplight.io/p/docs/gh/stoplightio/prism/docs/guides/03-validation-proxy.md?ref=apisyouwonthate.com) in end-to-end testing to blow up if any requests or responses are invalid throughout the test suite. This can be implemented with little to no buy in from the folks producing the APIs, because you can just funnel existing cross-API traffic through the proxy in the testing environment without modifying any code. Code first and editors do not jive at all, because the editors do not understand the annotation system in the Java/PHP/Python/etc source code. According to an extremely scientific [poll on my Twitter](https://twitter.com/philsturgeon/status/1234455612265725952?ref=apisyouwonthate.com), 35% of teams are battling through with a mixture of code-first and design-first. ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/posts/annotations-dsl-editors/design-code-poll.png) *[](https://twitter.com/philsturgeon/status/1234455612265725952?ref=apisyouwonthate.com)* So long as the code-first folks add a build step (pre-commit or in CI) to generate a machine-readable file in the filesystem (like `openapi.yaml`), then hosted solutions like Stoplight Platform can analyze repo contents to give the same hosted docs, mocks, and design libraries, to all the projects. No editing for them of course, but those who want editors are on the same remaining workflow as those who don't. 🥳 ## Annotations-as-Code Web Frameworks Many web-frameworks third-party support for request/response validation, which we've mentioned above. This is usually in the form of middlewares or just baked right in, and they read API descriptions from the filesystem. Other frameworks have first-party or third-party support for annotations, which are purely descriptive repetitions of the actual code they sit above at best. At worst they're just lies. There is a new category of API description integration popping up in some web frameworks which is somewhat like Annotations or DSLs, but instead of being purely descriptive it's actually powering logic and reducing code, giving you one source of truth. Effectively they do the same thing as the machine-readable powered validation middlewares, but instead of coming from `openapi.yaml` the logic is coming from the annotations. Here's an example from [tsoa](https://github.com/lukeautry/tsoa?ref=apisyouwonthate.com), which is a TypeScript and NodeJS framework for building OpenAPI-compliant REST APIs. ```js import { Body, Controller, Get, Header, Path, Post, Query, Route, SuccessResponse } from 'tsoa'; import { User, UserCreationRequest } from '../models/user'; import { UserService } from '../services/userService'; @Route('Users') export class UsersController extends Controller { @Get('{id}') public async getUser(id: number, @Query() name: string): Promise { return await new UserService().get(id); } @SuccessResponse('201', 'Created') // Custom success response @Post() public async createUser(@Body() requestBody: UserCreationRequest): Promise { new UserService().create(request); this.setStatus(201); // set return status 201 return Promise.resolve(); } } ``` This is a brand new approach. Instead of descriptive annotations or comments shoved in as an afterthought, the API framework has been designed around the use of annotations. Beyond simple things like request validation, [TSOA handles authentication](https://tsoa-community.github.io/docs/authentication.html?ref=apisyouwonthate.com) quite nicely. Numerous times I've seen API documentation say bearer tokens are required, or an OAuth token needs a certain scope, only to find out the developer forgot to register that in the API controller. Free sensitive data anyone? TSOA solves that by having you register security definitions, then reference them in your annotations, and have middlewares created to handle the actual logic. This way the annotations are all the actual source of truth for authentication, instead of just being lies in comments or YAML. Then, OpenAPI can be generated from a command: ```shell tsoa spec ``` Whilst I definitely have a preference for design-first development for all the prototyping benefits it brings (changing a few lines of YAML in an awesome GUI is easier than rewriting a bunch of code every time you get feedback on a prototype), this new approach for making annotations useful is very much closing the gap. If you're going to use a code-first approach, you should absolutely try and find a framework like TSOA to power your API and reduce the chance of mismatches. Ideally the file-based middlewares and these new annotation-driven middlewares would share a bunch of dependencies. As I mentioned before there's a million of these file-based validation middlewares out there, and some get more love and attention than others. I convinced three PHP request validation middleware authors to combine efforts and make [one amazing one](https://github.com/thephpleague/openapi-psr7-validator?ref=apisyouwonthate.com), so it'd be great if some of these other middleware developers could team up with some annotations-as-code framework people to allow them as inputs to their existing middleware. With [OpenAPI v3.1.0 coming out soon](https://apisyouwonthate.com/blog/openapi-v3-1-and-json-schema), it'll be a lot better for us tooling vendors to start collaborating on a smaller number of higher quality tools, instead of everyone battling through the upgrade process individually. Whatever you're up to: code-first, or design-first, make sure you're doing what you can to avoid maintaining two sources of truth. Of all the options possible, try and stick to: a) awesome editors like [Stoplight Studio](https://stoplight.io/studio/?ref=apisyouwonthate.com) or [GraphQL Designer](https://graphqldesigner.com/?ref=apisyouwonthate.com) to maintain API description documents, then reference them in the code, or b) frameworks which support annotations-as-code that knows how to express itself as API descriptions Just don't maintain code and descriptions separately, because having two sources of truth just means waste time trying to find out which one of them is lying. ### OpenAPI v3.1 and JSON Schema URL: https://apisyouwonthate.com/blog/openapi-v3-1-and-json-schema/ Last updated: 2024-07-03T17:52:26.000Z > **Updated 2019-05-29:** OpenAPI v3.1 has dropped SemVer, so a few things that were going to be deprecated are now just straight up removed. This post has been updated to reflect that. > **Updated 2021-01-10:** JSON Schema released Draft 2020-12 which clarified a few edge cases, it's not a meaningful change to the content of the article but we've replaced references to 2020-12 to avoid future confusion. OpenAPI using an "extended subset" of JSON Schema has caused confusion in the API space since before I had even heard of OpenAPI. Thanks to a lot of hard work from a lot of people over the last 6 months, OpenAPI v3.1 will in fact solve this problem, and there was much rejoicing. For anyone who's not read a million words on the topic from me on this topic already, I was talking about the divergence in [2017](https://phil.tech/api/2017/07/20/my-vision-for-a-perfect-world-in-api-specification/?ref=apisyouwonthate.com#json-schema-divergence), [2018](https://apisyouwonthate.com/blog/openapi-json-schema-divergence), creating [workarounds](https://github.com/openapi-contrib/json-schema-to-openapi-schema?ref=apisyouwonthate.com), [suggesting functionality](https://github.com/OAI/OpenAPI-Specification/tree/master/proposals/Alternative%20Schema?ref=apisyouwonthate.com), [complaining about that functionality](https://github.com/OAI/OpenAPI-Specification/issues/1943?ref=apisyouwonthate.com)... Everyone in and around OpenAPI was trying to get something done to solve this problem. For all this time, often unaware of any of the discussions, the divergence was a constant pain in everyone's ass: tooling vendors and end-users alike. Well, good news. OpenAPI v3.1 has brought along some substantial change. For a while we thought we'd see a few of OpenAPI-special keywords deprecated, and the JSON Schema keywords and functionality supported, but this "two ways of doing something" was not looking good for tooling vendors. So, after much consideration, OpenAPI v3.1 has dropped SemVer in order to more quickly and easily properly align itself with JSON Schema. Here's an overview: 1. OpenAPI Schema is a vocabulary of JSON Schema 2020-12 2. type arrays are now supported 3. nullable has been removed 4. arbitrary keywords are now supported without needing an `x-` on the front 5. exclusiveMinimum and exclusiveMaximum take a numeric value instead of being a boolean 6. discriminator is deprecated, and miiiight still be removed before OpenAPI v3.1 final happens Let's look at some of this a bit closer. ## OpenAPI Schema is a Vocabulary of JSON Schema 2020-12 Check out this new wording from the [OpenAPI Specification](https://spec.openapis.org/oas/v3.1.0.html?ref=apisyouwonthate.com). > The OpenAPI Schema Object is a JSON Schema vocabulary which extends JSON > Schema Core and Validation vocabularies. As such any keyword available for those > vocabularies is by definition available in OpenAPI, and will work the exact same > way. Booyeah. OpenAPI no longer defines a complex list of discrepancies, it simply says that the Schema Object properties are defined in [JSON Schema Core](https://json-schema.org/draft/2020-12/json-schema-core.html?ref=apisyouwonthate.com) and [JSON Schema Validation](https://json-schema.org/draft/2020-12/json-schema-validation.html?ref=apisyouwonthate.com). **There are no more "missing" JSON Schema properties. If a keyword exists in JSON Schema Draft 2020-12 (Core or Validation), then it's good to use in OpenAPI v3.1.** How about keywords which worked... differently? ## `type` can now be an array The most common source of confusion for many people was that `type` could be an array of strings in JSON Schema, but had to be a single string in OpenAPI v2-3\. In OpenAPI v3.0 you could at least use oneOf to select multiple types, meaning that three different ways to do the same thing were possible, with some tooling supporting one, two, or three of them... **OpenAPI v2.0** ```yaml type: string # no integers allowed... 🤷‍♂️ ``` **OpenAPI v3.0** ```yaml oneOf: - type: string - type: integer ``` **OpenAPI v3.1** ```yaml type: [string, integer] ``` OpenAPI v3.1 is now happy to take a type array! Users of languages like Java and C++ will still wonder why anyone would ever do this, but tooling vendors building tools for this can just treat it as they did a `oneOf`. Users of languages like JavaScript, Ruby and PHP are happy it's now much easier to describe common functionality. Maybe you used [API Evolution](https://apisyouwonthate.com/blog/api-evolution-for-rest-http-apis/) to change a string to an array of address components, maybe it could even be an object! Go nuts mate. ## `nullable` is GONE Nullable was deprecated for a while, which was... [my favourite pull request I ever sent](https://github.com/OAI/OpenAPI-Specification/pull/1977?ref=apisyouwonthate.com). Henry Andrews, author of JSON Schema and now OpenAPI contributor, got to take the cake. It's [**removed**](https://github.com/OAI/OpenAPI-Specification/pull/2246?ref=apisyouwonthate.com). **OpenAPI v3.0** ```yaml type: string nullable: true ``` **OpenAPI v3.1** ```yaml type: [string, "null"] ``` This is probably the biggest breaking change, but it's such an easy find and replace. You can even use tools like [openapi-schema-to-json-schema](https://github.com/openapi-contrib/openapi-schema-to-json-schema/?ref=apisyouwonthate.com) to upgrade your schema files (so long as you've been [keeping them DRY](https://stoplight.io/blog/keeping-openapi-dry-and-portable/?ref=apisyouwonthate.com)!) ## Any `format` from JSON Schema is fine JSON Schema Draft 2020-12 added a few new formats so go wild with those. There are some changes to how formats are interpreted now, and they are being relaxed from validation, to annotations which could be used for validation if configured to do so. Basically, trying to validate that data follows the rules for what a specific validation tool things is valid for that format is... awful. Different tools disagree about what an email address is, despite there being [a standard for what a valid email address is](https://json-schema.org/draft/2020-12/json-schema-validation.html?ref=apisyouwonthate.com#RFC5322). Tool vendors can feel free to keep existing format validation, but probably move it behind a config switch. Maybe the next major version of that tooling could have it default off. ## Arbitrary Keywords Every now and then somebody would add arbitrary keywords to their JSON Schema files to keep track of information that special (maybe in-house) tooling understood, but were not part of the JSON Schema spec. Then they'd try and use those JSON Schema files in OpenAPI, only to find it blows up. This was also fixed. > In addition to the JSON Schema properties defined in the vocabularies defined in the JSON Schema Core and JSON Schema Validation specifications, any properties can be used from any vocabularies, or entirely arbitrary keywords. OpenAPI v3.0 would only allow "extensions", which were arbitrary keywords starting with `x-`. Now, no need for the `x-`, unless you like it, in which case carry on. 👍 ## example vs examples JSON Schema has `examples` which is a bare array, which means you can have a string with two possible examples like this: ```yaml type: string examples: - squirtle - charmander ``` Documentation tools could chose to show the first one, or both, doesn't matter they're both good choices. OpenAPI v3.0 was a little different: ```yaml type: string example: squirtle ``` JSON Schema based tooling would not know what `example` was, and OpenAPI tooling would choke when it saw `examples` because it is not a valid keyword inside the Schema Object. In OpenAPI v3.1 the Schema Object `example` keyword is deprecated, so you should start using `examples` in your API description documents. *I will probably add a [Spectral rule](https://stoplight.io/p/docs/gh/stoplightio/spectral/docs/reference/openapi-rules.md?ref=apisyouwonthate.com) for that to help nudge people in the right direction.* ## examples are not examples The change above does solve a discrepancy, but it does unfortunately introduce its own little bit of confusion. Hold onto your butts. OpenAPI v3.0 Schema Objects do not have `examples` as a keyword, but there is a totally unrelated `examples` which shows up elsewhere in OpenAPI v3.0 (in the [Media Type Object](https://spec.openapis.org/oas/v3.0.3?ref=apisyouwonthate.com#media-type-object), for request and responses). Don't mix them up, they act differently. Let's take a look at this not-in-the-schema approach to examples which OpenAPI offers: ```yaml parameters: - in: query name: limit schema: type: integer maximum: 50 examples: # Multiple examples zero: # Distinct name value: 0 # Example value summary: A sample limit value # Optional description max: # Distinct name value: 50 # Example value summary: A sample limit value # Optional description ``` *Example from [Swagger.io talking about Examples](https://swagger.io/docs/specification/adding-examples/?ref=apisyouwonthate.com).* For any `parameters`, `requestBody` or `responses`, outside of the Schema Object, there is a special OpenAPI keyword called `examples`. This is not a bare array like the JSON Schema `examples`, but an object where the keys are names, and the example has to go inside the `value` key. This confusion of two types of `examples` is going to persist, but at least this problem can be solved with documentation and tutorials, instead of being a roadblock making files be unusable in some tooling but fine in others. 😅 ## exclusiveMinimum and exclusiveMaximum are different JSON Schema made an improvement to how exclusiveMinimum and exclusiveMaximum work. In older versions, they were a boolean switch which would suggest that the minim value defined was also an acceptable value. **OpenAPI v3.0** ```yaml # Value has to be 21 or over minimum: 21 # Value has to be greater than 21, not 21 minimum: 21 exclusiveMinimum: true ``` **OpenAPI v3.1** ```yaml # Value has to be 21 or over (same) minimum: 21 # Value has to be greater than 21, not 21 (new!) exclusiveMinimum: 21 ``` This is a small change, and has be done to simplify things. Keywords in schema objects generally should not effect other keywords in unexpected ways, so this keyword now just handles its own business. ## More to Come The work for JSON Schema alignment is pretty much done, but there are [a few licks of paint](https://github.com/OAI/OpenAPI-Specification/issues/2099?ref=apisyouwonthate.com) until its completely sorted. These tweaks might come in over the course of the release candidate process, the first of which is supposedly going to be the end of February (this month?! 🤯) OpenAPI v3.1 has a bunch of other cool stuff coming too: - [Webhooks](https://apisyouwonthate.com//blog/openapi-callbacks-webhooks) which do not have to be documented as callbacks underneath a specific operation - `paths` is now optional so an API description could be Webhooks only - Maybe [overlays](https://github.com/OAI/OpenAPI-Specification/blob/master/proposals/004%5FOverlays.md?ref=apisyouwonthate.com) will be a thing We'll talk about all this in future articles in the future. Until then, make sure your tooling is ready for JSON Schema 2020-12 and OpenAPI v3.1\. Many tooling vendors are hard at work on this as we speak, with a few validation tools already done: - **.NET** \- [Manatee.Json](https://github.com/gregsdennis/Manatee.Json?ref=apisyouwonthate.com) - **Java** \- [Snow](https://github.com/ssilverman/snowy-json?ref=apisyouwonthate.com) - **JavaScript** \- [Hyperjump JSV](https://github.com/hyperjump-io/json-schema-validator?ref=apisyouwonthate.com) Head over to the official [JSON Schema Implementations](https://json-schema.org/implementations.html?ref=apisyouwonthate.com) list of all sorts of tooling as they're released. Hopefully soon this will all be behind us. ### Easy to Follow Hypermedia Controls with Ketting URL: https://apisyouwonthate.com/blog/ketting-v5-hypermedia-controls/ Last updated: 2023-03-24T12:23:09.000Z Hi! My name is Evert, and for a while now I've been working on a project called [Ketting](https://github.com/evert/ketting?ref=apisyouwonthate.com), which is a generic Hypermedia/HATEOAS client for Javascript. I just released version 5.0, and thought it might be interesting to share the new features that have been added since earlier this year. These are the highlights. ## Support For Siren Ketting now features support for [Siren](https://github.com/kevinswiber/siren?ref=apisyouwonthate.com). This means that the client now understands this format and can transparently traverse Siren links. ```typescript const newRes = await res .follow('author') // this could be a HTML5 link .follow('me') // This might be a Siren link ``` Ketting now has support for HTML, [HAL](https://tools.ietf.org/html/draft-kelly-json-hal?ref=apisyouwonthate.com), [JSON:API](https://jsonapi.org/?ref=apisyouwonthate.com), [HTTP Link headers](https://tools.ietf.org/html/rfc8288?ref=apisyouwonthate.com) and [Siren](https://github.com/kevinswiber/siren?ref=apisyouwonthate.com) links. [Collection+JSON](http://amundsen.com/media-types/collection/format/?ref=apisyouwonthate.com) is also in the works and will likely follow soon after in a point release. ## Advanced Follow Features Usually if you want to fetch a collection of things, you might do it using this API: ```typescript const articles = await res .follow('article-collection') .followAll('item') // This might be a list of links to each individual article ``` In HTTP/1.1 style APIs, it was common to actually embed the responses of every 'item' in a collection using for example's HAL `_embedded` property or JSON:API's `included`. This is kind of an [anti-pattern](https://apisyouwonthate.com/blog/lets-stop-building-apis-around-a-network-hack), but a necessary evil because HTTP/1.1 requests have a large amount of overhead. Ketting will take all the items in `_embedded` and store the items in its cache, so that future `GET` requests are not needed. In HTTP/2 APIs requests are less expensive, and it's often desirable to make more smaller HTTP requests and responses. To help with this, the follow/followAll have these new features: ### Prefetching ```typescript const newRes = await res .follow('article-collection') .followAll('item') .preFetch(); ``` The `preFetch()` on followAll will cause Ketting to follow every `item` link, do a `GET` request for each in parallel in the background and store it in its cache. We did some testing with an internal application and were able to prefetch an entire collection of 1700 items (so 1700 `GET` requests) in about 7 seconds. While this could benefit from some paging, it was great to see how fast HTTP/2 servers can be. ### Sending Prefer-Push Very similarly, there's also a chainable `preferPush()` on `follow` / `followAll`: ```typescript const newRes = await res .follow('article-collection') .followAll('item') .preferPush(); ``` This will cause Ketting to send a [Prefer-Push](https://tools.ietf.org/html/draft-pot-prefer-push?ref=apisyouwonthate.com) header when fetching the articles collection: ``` GET /articles HTTP/2 Prefer-Push: item ``` If a server supports this header, they can optimize future requests by sending HTTP/2 pushes for each linked item. ## Support For `rel="invalidates"` Ketting now understands the 'invalidates' link relationship from [draft-nottingham-linked-cache-inv](https://tools.ietf.org/html/draft-nottingham-linked-cache-inv-04?ref=apisyouwonthate.com). A use-case for this is that a client might do a `POST` request on some resource, and the result of this resource causes other cached responds to invalidate. This is handy, because an operation on one resource can alter the state of other, unrelated resources. Using this HTTP header, a server can tell a client which resources those are and control invalidation. ## New OAuth2 Library Older versions relied on the [client-oauth2](https://www.npmjs.com/package/client-oauth2?ref=apisyouwonthate.com) for everything OAuth2, but this library is pretty bulky when WebPacked. This has since been changed over to [fetch-mw-oauth2](https://evertpot.com/fetch-mw-oauth2/?ref=apisyouwonthate.com), which is a library that wraps `fetch()` as a middleware and decorates it with OAuth2 features. This caused the final Ketting Webpack distribution to drop over 30KB. While making this change, support for OAuth2 `authorization_code` was also added. ## Per-Domain Authentication One of the nice advantages of using a hypermedia-style API, is that if multiple APIs support links, one way to integrate these APIs is simply by pointing links from one API to another. A real life example was that one of our APIs needed some integration with the Github REST api, which uses HTTP Link headers for some stuff. Our API could simply point to Github endpoints and the client was able to just traverse the graph and didn't have to be aware that some endpoints were served by Github. However, our API and Github's each have their own authentication mechanisms. In previous Ketting versions authentication could only be set up once globally, causing the same credentials to be sent everywhere. With the new authentication layer, it's possible to set Ketting up to use different authentication mechanisms based on the specific domains you're accessing, including wildcards. ```typescript const ketting = new Ketting(bookmark, { match: { '*.github.com': { auth: {type: 'oauth2', /*...*/ } }, 'api.example': { auth: {type: 'basic', /*...*/ } }, } }); ``` ## Typescript Improvements It's now possible to add Typescript annotations to various Ketting methods. Consider the following example ```typescript type Article = { title: string, body: string }; const itemRes = await res1.follow
('item'); ``` This tells Typescript that the result of this `follow()` function is actually a `Resource
`. If a user later calls: ```typescript const body = await itemRes.get(); ``` body will now automatically have the `Article` type. Similarly, `item.put()` will also now require a parameter of type `Article`. There's many more examples of this, which you can read on my [previous post](https://evertpot.com/ketting-generics/?ref=apisyouwonthate.com) on this subject. ## 201 And 205 Responses On POST If a user calls `.post()` and the response to the HTTP POST request was a [201 Created](https://evertpot.com/http/201-created?ref=apisyouwonthate.com) and a `Location` header, it would automatically return a new resource. ```typescript const newArticle = await articlesCollection.post(newArticle); ``` Now, when a server returns a HTTP [205 Reset Content](https://evertpot.com/http/201-created?ref=apisyouwonthate.com), the `post()` function will return the resource itself (`this`). This is a small change that I hope is useful for APIs with strong hypermedia controls and folks that want to create clients that heavily rely on the entire application state being managed by the server. ## Conclusion I hope you like the changes. My goal is to create the ultimate client for Hypermedia APIs, and I feel these are some good steps in that direction. If you're an existing user, I would be very curious to hear what's currently painful and help adjust the future of this project. If you're a new user and interested in using it, check out the [Github project](https://github.com/evert/ketting?ref=apisyouwonthate.com) and [documentation](https://github.com/evert/ketting/wiki?ref=apisyouwonthate.com) (The latter of which also got a big overhaul). ### Automated Style Guides for REST, GraphQL and gRPC URL: https://apisyouwonthate.com/blog/automated-style-guides-for-rest-graphql-grpc/ Last updated: 2023-03-29T14:28:42.000Z Ask 100 developers where a semicolon should go, and you'll either get 100 answers, or a all-on-all fist fight. To save this from happening at work, most folks implement a style guide, which beyond helping with consistent style to avoid new developers getting shouted at for "doing it wrong". Linters can advise best practices, shout about things which are technically allowed but likely to cause trouble, and shape the API of code as it's being written (snake\_case that method!) This is always done for code, and is becoming increasingly popular for API descriptions. JavaScript users have [eslint](https://eslint.org/?ref=apisyouwonthate.com), PHP users have [PHP Code Sniffer](https://github.com/squizlabs/PHP%5FCodeSniffer?ref=apisyouwonthate.com), and Ruby has [rubocop](https://www.rubocop.org/?ref=apisyouwonthate.com). These linters don't just check that the user is writing valid syntax. They check against existing sets of rules, sometimes written by a company, like the almost defacto-standard [eslint-airbnb](https://www.npmjs.com/package/eslint-config-airbnb?ref=apisyouwonthate.com). Sometimes the rules are made by standards bodies like PSR-12 by the \[PHP-FIG\] ([https://github.com/squizlabs/PHP\_CodeSniffer/tree/master/src/Standards/PSR12](https://github.com/squizlabs/PHP%5FCodeSniffer/tree/master/src/Standards/PSR12?ref=apisyouwonthate.com)), and the tools create a ruleset to match, like the [PSR-2 ruleset for CodeSniffer](https://github.com/squizlabs/PHP%5FCodeSniffer/blob/master/src/Standards/PSR12/ruleset.xml?ref=apisyouwonthate.com). Today I even had to use [awesome-lint](https://github.com/sindresorhus/awesome-lint?ref=apisyouwonthate.com) in order to make sure [awesome-earth](https://github.com/philsturgeon/awesome-earth?ref=apisyouwonthate.com) was conforming to their rules, which were built using the [Remark](https://github.com/remarkjs/remark-lint?ref=apisyouwonthate.com) Markdown linter. When it comes to API descriptions, most companies of a certain size end up with a "Style Guide", "Style Book", "Design Guide", etc. These are often on a Google Doc, wiki, or some other sort of docs/content management system. I've seen loads of these, and written plenty. Many companies [even publish them](http://apistylebook.com/design/guidelines/?ref=apisyouwonthate.com). ## Text-based Style Guides are a Time Suck The trouble with these text-based documents is that they are large, terse, unexciting documents, which developers rarely read. If developers *do* read them cover to cover, *and* remember everything in there, that knowledge becomes partially out of date when new rules are added because they won't know about them until they re-read everything cover to cover again. At *[API the Docs](https://apithedocs.org/amsterdam2019?ref=apisyouwonthate.com)* I saw a talk from *Kelsey Lambert* at Salesforce, and their style guide is an example OpenAPI description document which they ask people to check now and then when they are working on something to get ideas of the sorts of things they should use. Salesforce... The giant company with 238479347 APIs who maintain 40 major versions per API, their style guide enforcement approach is eyeballing and memory. Agh I feel for you folks! I have been here and it was bad. No developers can be blamed for any of this mess. API developers are busy, and the folks writing style guides are just trying to figure it out as they go along. This mess is an industry problem, but thankfully tools have popped up which can enforce these same style guide concepts through automation. - [api-linter](https://github.com/googleapis/api-linter?ref=apisyouwonthate.com) by Google - [graphql-doctor](https://github.com/cap-collectif/graphql-doctor?ref=apisyouwonthate.com) by [Cap Collectif](https://cap-collectif.com/?ref=apisyouwonthate.com) - [graphql-schema-linter](https://github.com/cjoudrey/graphql-schema-linter?ref=apisyouwonthate.com) by [Christian Joudrey](https://twitter.com/cjoudrey?ref=apisyouwonthate.com) - [Spectral](https://stoplight.io/open-source/spectral/?ref=apisyouwonthate.com) by [Stoplight](https://stoplight.io/?ref=apisyouwonthate.com) Each one of these projects sets out to do relatively similar things, but for different types of API. ## Spectral for HTTP APIs Spectral is a JSON/YAML data linter, with built in rules for OpenAPI v2/v3 and JSON Schema. Running the default OpenAPI ruleset on the average document will find plenty of suggestions, which can be helpful for developers not entirely familiar with OpenAPI. Something as small as reminding people to add `parameter-descriptions` can help make [human-readable docs more useful](https://apisyouwonthate.com/blog/turning-contracts-into-beautiful-documentation), and they might not have even realized that was possible. ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/posts/automating-style-guides/so-many-errors.jpg) You can use Spectral to create [rulesets](https://docs.stoplight.io/docs/platform/71b92b2b5e388-style-guide-overview?ref=apisyouwonthate.com), and these rulesets can have custom rules, and even custom functions! These custom rules can look a bit like this: ```yaml rules: schema-names-pascal-case: description: Schema names MUST be written in PascalCase message: '{{property}} is not PascalCase: {{error}}' recommended: true type: style given: '$.components.schemas.*~' then: function: casing functionOptions: type: pascal ``` This one is a popular one. OpenAPI does not care how you capitalize your models, but a lot of code generators will use the model names for code, and having inconsistent class names will upset people. Let's take it a step further: ```yaml rules: paths-kebab-case: description: Should paths be kebab-case. message: '{{property}} is not kebab-case: {{error}}' severity: warn recommended: true given: $.paths[*]~ then: function: casing functionOptions: type: kebab separator: char: "/" ``` This rule is actually looking beyond the metadata of your API descriptions, and is looking at the actual API design itself. This is saying that the "paths" (endpoints) must be hyphenated, so `/recent-files` is good but `/recent_files` is not ok. You can get really creative with regex patterns if that floats your boat. ```yaml rules: no-x-headers: description: "Please do not use headers with X-" message: "Headers cannot start with X-, so please find a new name for {{property}}. More: https://tools.ietf.org/html/rfc6648" recommended: true given: "$..parameters.[?(@.in === 'header')].name" then: function: pattern functionOptions: notMatch: '^(x|X)-' ``` I don't know why but at some point during the lifecycle of any given API, some developer will suggest adding an `X-Foo` header, despite over [a decade of it causing issues](https://www.mnot.net/blog/2009/02/18/x-?ref=apisyouwonthate.com). Well, we can keep them outta here with this rule. Done early enough, this will shape the actual API as it is being developed. If you are doing code first then ok, you have to go back and change a bunch of code. Hopefully you didn't ship it, because now you need to go and make a bunch of redirects for `/recent_files /recent_files`. If you use an [API design first workflow](https://apisyouwonthate.com/blog/api-design-first-vs-code-first), then you notice this early on when you've just got some YAML, and your API gets built right in the first place. Seeing as Spectral is a CLI/JS tool, enforcing this style guide can be done in all sorts of ways. - in [a git hook](https://docs.stoplight.io/docs/platform/a13c113eb9988-common-git-tasks?ref=apisyouwonthate.com) - in a JS test suite - on [continuous integration](https://docs.stoplight.io/docs/platform/6twy9ttde8pan-style-guides-and-git?ref=apisyouwonthate.com) to fail builds with errors If you're using [Stoplight Studio](https://stoplight.io/studio/?ref=apisyouwonthate.com) then it's baked right into the editor, so people designing APIs just do it all correctly straight away. No need to alt tab away to the CLI or wait until a PR is made. ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/posts/automating-style-guides/studio-spectral.png) I am trying to find time to take a style guides from [Heroku](http://apistylebook.com/design/guidelines/heroku-http-api-design-guide?ref=apisyouwonthate.com) or [PayPal](http://apistylebook.com/design/guidelines/paypal-api-style-guide?ref=apisyouwonthate.com) and turn them into a huge example ruleset. At the very least, I can take some inspiration for a new ruleset I'm putting together: The [OpenAPI Contrib > Style Guide](https://github.com/openapi-contrib/style-guide?ref=apisyouwonthate.com). This should be an interesting community effort. Spectral also has a [GitHub Action](https://github.com/stoplightio/spectral-action?ref=apisyouwonthate.com) and a \~GitHub Bot\~ (RIP) which we are working on improving. Commenting on ranges and suggestions coming soon! 😎 ## GraphQL Doctor & Schema Linter GraphQL has it's own \[built-in type system\] ([https://graphql.org/learn/schema/](https://graphql.org/learn/schema/?ref=apisyouwonthate.com)), which has some of the same sort of keywords as OpenAPI / JSON Schema based stuff. GraphQL makes some design decisions easier, like how you handle relationships. No need to pick between nesting/embedding related resources, inlining everything with a compound documents, or using hyperlinks to link to related data, GraphQL decides that for you. Still, there is a lot of inconsistency that can occur outside of the default decisions GraphQL makes. GraphQL people do not escape the need to lint, but luckily a great linter exists: [GraphQL Schema Linter](https://github.com/cjoudrey/graphql-schema-linter?ref=apisyouwonthate.com). ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/posts/automating-style-guides/graphql-schema-linter.png) [Custom rules](https://github.com/cjoudrey/graphql-schema-linter?ref=apisyouwonthate.com#customizing-rules) can be written for this one too, so you can automate your style guide in CI. No bot or GitHub Action that I can see, but they aren't too tough to knock together. One schema tool in GraphQL land with a great bot is GraphQL Doctor. It seems like it wants to help with a lot more linting in general, but so far it is focused on detecting breaking changes. Like any type system, there is a fine line between [careful evolution](https://apisyouwonthate.com/blog/surviving-deprecations-to-resources-and-properties-on-other-apis) and recklessly changing stuff, and GraphQL Doctor will spot the latter. ![A preview of graphql doctor bot, commenting on a github pull request showing the line where a failure happens](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/posts/automating-style-guides/graphql-doctor.jpg) It would be nice to see the two tools merge, or maybe the GraphQL Doctor bot can bake in support for GraphQL Schema Linter, but for now it's a two-stop shop. ## Google's "API Linter" Google is doing some pretty interesting work in the API space. They were one of the first big players in the API space consistently explaining "Sometimes you want REST, sometimes you want RPC", and they're keeping at it with a general tool that works for [gRPC and HTTP-in-general too](https://aip.dev/127?ref=apisyouwonthate.com). API linter operates on the protobuf surface layer, but can be set up to work with HTTP endpoints: > When using protocol buffers, each RPC must define the HTTP method and path using the google.api.http annotation: ``` rpc CreateBook(CreateBookRequest) returns (Book) { option (google.api.http) = { post: "/v1/{parent=publishers/*}/books/*" body: "book" }; } message CreateBookRequest { // The publisher who will publish this book. // When using HTTP/JSON, this field is automatically populated based // on the URI, because of the `{parent=publishers/*}` syntax. string parent = 1; // The book to create. // When using HTTP/JSON, this field is populated based on the HTTP body, // because of the `body: "book"` syntax. Book book = 2; // The user-specified ID for the book. // When using HTTP/JSON, this field is populated based on a query string // argument, such as `?book_id=foo`. This is the fallback for fields that // are not included in either the URI or the body. string book_id = 3; } ``` The [core ruleset](https://googleapis.github.io/api-linter/rules/core/?ref=apisyouwonthate.com) for API Linter is rather impressive, and focuses a lot on awkward bits in the HTTP specification which are a bit unclear. Like, should GET have a body? The answer is a very squishy kinda maybe it can, but probably don't, depends on the tool you are building, ugh. Help. Google decided to just answer that with: nope. ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/posts/automating-style-guides/no-http-body.png) They also decided to persuade teams upgrade from proto2 to proto3. ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/posts/automating-style-guides/proto3.png) ## Rule Ideas You can automate pretty much anything with this stuff, and I've been thinking a lot of rules that go beyond the common use cases of enforcing naming or pluralization. ### Security - Ban HTTP Basic entirely - Make sure every endpoint has some sort of security (OAuth 2, API Key, but not both) - Every response should support `application/vnd.api+json` (JSON:API) not just plain-old JSON - ID's as integers let people [crawl your API](https://phil.tech/http/2015/09/03/auto-incrementing-to-destruction/?ref=apisyouwonthate.com) incredibly easily, switch to UUID ### Errors - Your 20X response seems to have errors in it, why do you hate your consumers - There are no URLs in your errors, how can anyone find out more information about what went wrong - Error format should be [RFC 7807](https://tools.ietf.org/html/rfc7807?ref=apisyouwonthate.com) ### Versioning - Keep [version numbers out of the URL](https://apisyouwonthate.com/blog/api-versioning-has-no-right-way/) - Version numbers in headers please - Ban all versioning and [demand evolution](https://apisyouwonthate.com/blog/api-evolution-for-rest-http-apis/) (prepare for battle) Many of these rules are HTTP API specific but you get the idea. Over time I'll be working on some of these and adding them to OpenAPI Contrib's [Style Guide](https://github.com/openapi-contrib/style-guide?ref=apisyouwonthate.com), and if you'd like to contribute I'll be happy to guide you through the process over on GitHub. ## Summary If you've heard the term API Governance, this is pretty much what most people are talking about. Currently a lot of the people trying to do governance are eyeballing API descriptions on every single PR, and training people to memorize all the quality rules they've come up with. Manual API training is a thankless, inefficient, never-ending task, and it can be replaced (or drastically streamlined) with a linter baked into an editor, git hook, CI pipeline, GitHub Action, or a bot. Don't waste customers time forcing them to try and figure out your inconsistencies. Don't waste all API developers time learning to memorizing style guides. Don't waste the API governance teams time reviewing APIs manually. Don't waste everyone's time fixing inconsistencies in production later. ### OpenAPI Callbacks and Webhooks URL: https://apisyouwonthate.com/blog/openapi-callbacks-webhooks/ Last updated: 2023-11-06T19:01:42.000Z [OpenAPI](https://www.openapis.org/?ref=apisyouwonthate.com) is thoroughly mainstream now with most modern API providers producing a description of their API for both internal and external use. There are a few features that are still without widespread traction, and the [callbacks](https://spec.openapis.org/oas/v3.0.2?ref=apisyouwonthate.com#operation-object) feature is one of those. It's a feature with wide appeal that hasn't seen much support from tooling yet - but that is starting to change. With more tools implementing the `callbacks` keyword, a wider selection of API providers are looking to adopt OpenAPI. One common roadblock is for APIs that are truly two-way - they include incoming HTTP requests that aren't callbacks but instead result of an external event. Luckily there's already support for this at proposal stage so these APIs should be able to put OpenAPI adoption on their roadmaps! Let's take a look at the two features and the use cases they serve. ## Existing Feature: Callbacks for Async Responses The `callbacks` field can be added to any path object, at the same level as `parameters`, `responses`, etc. Here, you can describe any number of HTTP requests that may arrive in response to an earlier HTTP request. This is super useful if an API response will include something slow or expensive or simply not time-critical! Examples could include: - Reporting APIs where the user can make an API request to generate a report and include a URL to send the report data to once it has been generated. - Status updates for a request that won't complete instantly (such as a payment transaction or message delivery receipt). If your API has features like this, then the OpenAPI `callbacks` feature enables you to describe them along with the rest of your API. ## Proposed Feature: Webhooks for Event-Driven HTTP The proposed `webhooks` field is a top-level element (warning: naming is very much under discussion and is probably the most important part thing to get right in this proposal!) that behaves exactly like a `callbacks` element within a path entry does. This is very cool because it means that most tools that support `callbacks` will be able to support `webhooks` without a lot of work from the maintainers. The main difference is that instead of being the result of an earlier HTTP call, these incoming requests are the result of an event happening elsewhere. This makes it ideal for APIs that offer features like: - Notifications of events, my favourite example is the GitHub push event. - Delivery of data that would otherwise be obtained by polling, such as warehouse stock level updates after a purchase - or even a lorryload of stock delivery! Increasingly our machines talk to one another in ways that are more sophisticated that the overnight batch import. Being able to describe HTTP requests in either direction will be a huge boost for OpenAPI. With the implementation agreed (although still at proposal stage), APIs and tools can move together to adopt this feature. ## Callbacks, Webhooks ... or Both? There are a few situations where `callbacks` is a valid way to specify an incoming HTTP request but `webhooks` may be preferable. For example, an API that allows subscribing to events under a configuration-related endpoint: it may make more logical sense (for outputs such as documentation) to put the description of the event-driven HTTP requests independently under `webhooks` than nested under an admin endpoint. If you want to know more about the `webhooks` feature, then you can find the [proposal in the OAS repo](https://github.com/OAI/OpenAPI-Specification/blob/master/proposals/002%5FWebhooks.md?ref=apisyouwonthate.com) and general discussion (all voices welcome, even if you only drop by to share a +1) is [on the github issue](https://github.com/OAI/OpenAPI-Specification/issues/1968?ref=apisyouwonthate.com). OpenAPI is an open standard and it benefits from community participation so do please all share your thoughts! ### API Design-First vs Code First URL: https://apisyouwonthate.com/blog/api-design-first-vs-code-first/ Last updated: 2023-03-24T12:24:03.000Z With API descriptions rising in popularity, the main question I hear folks asking about is "API Design-first" or "code-first". This is a bit of a misleading question because these are not two unique things, there are a few variants. ## Code-First, Write Docs "When We Have Time" ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/posts/api-design-first-vs-code-first/wf-1.png) This is how I came into contact with API description documents like OpenAPI or API Blueprint in the first place, and it is how our first book suggested API developers do things. This may have made sense at the time, but I quickly discovered it to be an immature workflow. One issue here is that "code-first and meh docs later" treats API descriptions like a fancy way of making API reference documentation, which is one of 100 things API descriptions can do. API descriptions are machine readable files with a plethora of data and metadata, which can used to gather feedback from early stages to improve the quality of the API before it's even written through [mocking](https://stoplight.io/prism/?ref=apisyouwonthate.com), offer [client-side validation](https://apisyouwonthate.com/blog/json-schema-client-side-validation) and [server-side validation](https://apisyouwonthate.com/blog/server-side-validation-with-api-descriptions/). Writing a bunch of code first, deploying the thing, getting clients onboarded with special hands-on treatment, etc. is a whole lot of work. When this whole phase is done, spending a month writing up documentation which will "only get out of date" can feel like a giant chore, one that most businesses struggle to prioritize so the task just never gets done. This was the excuse I heard regularly for why WeWork, a company with \~50 engineers in 2016 managed to build \~30 APIs with zero documentation at any point. The lack of documentation lead to some of the most bonkers time-and-money-wasting I've ever come across, with folks building out [new versions of endpoints and APIs](https://apisyouwonthate.com/blog/api-versioning-has-no-right-way/) because nobody could remember how the code worked. Even reading the code was almost impossible due to API A dynamically returning mushed together chunks of JSON from API B and API C [without any serializers involved](https://phil.tech/api/2015/05/30/serializing-api-output/?ref=apisyouwonthate.com). "We'll write documentation later" means "We will not write documentation", and by the time you discover you need it, it'll be too late. On the off-chance you are one of the few who get it done quickly, keeping these documents "in sync" with the code is the biggest problem most developers faced. At my talk on this subject at [API the Docs](https://apithedocs.org/amsterdam2019?ref=apisyouwonthate.com), the entire room of \~80-100 people put their hand up when I asked "Who here struggles with keeping code and docs in sync"? There are [a few approaches](https://apisyouwonthate.com/blog/keeping-documentation-honest), but even if you absolutely nail using [Dredd](https://dredd.org/?ref=apisyouwonthate.com) or similar tooling to keep things synced up, there is the other rather large problem we've not covered yet: the fact that you built the whole API before giving your customers a chance to play with it. [Mocking](https://stoplight.io/mocking/?ref=apisyouwonthate.com) is too often overlooked, and people waste time and money building out nonsense APIs which don't help their customers. This usually means a v2 comes quickly after the v1, and maybe a v3 is required as a few more clients get involved and give more feedback. This usually means the API was too normalized, leading to the client needing to make 150 HTTP requests to solve their use case, or the resources are giant meaning there is good data hidden amongst 100 fields the user didn't need. [Use-case driven APIs](https://medium.com/@%5F%5Fxuorig%5F%5F/the-tension-between-data-use-case-driven-graphql-apis-8f982198653b?ref=apisyouwonthate.com) are usually way more useful than data-driven APIs, regardless of the API [paradigm you picked](https://apisyouwonthate.com/blog/picking-api-paradigm/) for the API build. Let your users share their feedback early, when it's still cheap and easy to change things - not when it's already in production and change gets more complex. ## Code-First, then Annotate ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/posts/api-design-first-vs-code-first/wf-2.png) This popular variation of the code-first approach to API descriptions the effort to speed up the "documentation later" part of the process, a lot of API developers decide to use annotations or code comments to litter their source code with bits of the API description in a special format. Multiple tools exist for this. In some strictly typed languages the annotation tooling contains very little information, mostly only things like human-readable descriptions. Information like basic types ("string" and "integer") can be inferred from the code, wether null is allowed, etc. can all be picked up. Sadly some people think that is all the information they need to put into a description document. They ignore things like example values, formats like "email" or "date-time" which can add validation benefits and make documentation more useful, and other more advanced features in OpenAPI or JSON Schema like allOf, oneOf, etc. Languages with annotations as a first class feature generally support this a bit better, like Java. They have a multitude of annotation systems which can give you syntax errors if you write rubbish in there. ```java class UserController { @OpenApi( path = "/users", method = HttpMethod.POST, // ... ) public static void createUser(Context ctx) { // ... } } ``` Other languages like PHP rely on doc block comments, and that's just writing nonsense into a text editor. ```php /** * @OA\Get(path="/2.0/users/{username}", * operationId="getUserByName", * @OA\Parameter(name="username", * in="path", * required=true, * description=Explaining all about the username parameter * @OA\Schema(type="string") * ), * @OA\Response(response="200", * description="The User", * @OA\JsonContent(ref="#/components/schemas/user"), * @OA\Link(link="userRepositories", ref="#/components/links/UserRepositories") * ) * ) */ public function getUserByName($username, $newparam) { } ``` This looks rough to me, but folks defend it with reasoning like: "having the annotations near the code means developers are more likely to keep it up to date". More likely is not definitely. ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/posts/api-design-first-vs-code-first/milk.jpg) Using annotations you still need to use one of the approaches to making sure code and descriptions are in sync, but you have to add a build step to export from source code and then run that generated OpenAPI file through Dredd or similar. Or you can just hope that all of your developers remember and "it'll be fine". The feedback loop here is still a bit long. It comes after you've written a whole bunch of code, or maybe you wrote all the routes to a bunch of empty controllers, and can export the OpenAPI to create a mock server, but that all still sounds like a lot of work. There are more improvements to be made. ## Design First, Ditch for Code First ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/posts/api-design-first-vs-code-first/wf-3.png) In general, "API Design First" is about closing the feedback loop substantially. You get mocks and docs before you write any code, so there is no more mucking about with code until a decent number of clients have confirmed the interfaces look good for their needs, and seeing as you already have what you need to generate docs you don't have to worry about doing it later. This specific flavour of design-first still has a lot of problems, yet recently a few big names in the API world have been advocating for this. Mainly I think they advocate for it because they are sick of writing API descriptions by hand: insert the usual complaints about "thousands of lines of YAML" here. Maybe they use a DSL to design things at first, then switch to annotations once things are done, again hoping "it will be more likely to stay up to date" that way. One of multiple falsehoods here is the idea that there is a design phase, then you stop designing things and it's time for the code to happen, and we don't need to do design new functionality after that. Regardless of whether devs write the API code by hand or generate it from API descriptions, there is no end to the design phase. Design is a circular life-cycle with a feedback loop which leads to new resources and endpoints, or new global versions, or just new properties. APIs evolve over time, and rolling out new functionality without gathering feedback from customers is always a bad idea, not just in the initial design phase. I have seen some success from folks at [Meetup](https://meetup.com/?ref=apisyouwonthate.com) using "immutable services", where they generate routes, controllers, data models, docker config, even all the Kubernetes setup, all from OpenAPI, then they just slap in a bit of business logic in the empty gaps and hit deploy. What happens when they need to make changes to the contract? That'll be a brand new service. No change allowed. Plan things well enough you don't need to tweak em for ages, then deprecate and replace them if change is required. Immutable services are not a common way to do things, and require a huge amount of discipline to get right. For everyone else, evolution is more common, because even folks using major global versions for their API will make backwards compatible changes as they go (new endpoints, etc.) Tooling which asks you to "Import" OpenAPI then go on from there without it is condemning you to a design-less future for new functionality, even if they offer an Export OpenAPI feature (which many don't). Worse than that, many of these tools keep their own version of your API description in the cloud, which can change independently of the API description you have in your Git repo, meaning you don't a sources of truth: *you have two* *sources of lies.* Let's look at a workflow which allows you to use API descriptions as a single source of truth, which evolves along with your code. ## Design-First, Evolve with Code ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/posts/api-design-first-vs-code-first/wf-4.png) This approach stops treating API description documents like an afterthought, or like a chore, because they aren't. DSL's might have been required to [make writing OpenAPI bearable](https://developer.squareup.com/blog/making-openapi-swagger-bearable-with-your-own-dsl/?ref=apisyouwonthate.com) in the past, but with stunning visual editors like [Stoplight Studio](https://stoplight.io/studio/?ref=apisyouwonthate.com), the days of using DSLs as a crutch to avoid hand-rolling YAML are behind us. Studio lets you work with your OpenAPI files on your local machine, for free, so anyone can easily build up powerful description documents, and even easily reuse models between multiple APIs so the whole "thousand lines of YAML" thing completely falls away. Whether you use Studio, DSL, or write it by hand, start off in your empty repo with just the description documents. Run a mock server early and often, get feedback from your customers, then commit the documents once things are agreed. Then you can start writing code. The amount of code you need to write can be drastically simplified with tooling that uses your description documents to power [server-side validation](https://apisyouwonthate.com/blog/server-side-validation-with-api-descriptions/), or even API Gateway validation. This is not code generation, but it is using your API descriptions to power production validation. The same description documents that you are using to render documentation are now powering the most complex aspect of your API, and things can never be "out of sync" because there is only one source of truth. When customers request new functionality, it is easy to add new endpoints, introduce new properties, etc., and get feedback on that new stuff before you start writing the code. At no point do you lose that ability, so you can benefit from design first, design again, and again. This does not help keep responses "in sync", but seeing as your description documents are sat right there in your repo, you can use them to [drastically simplify your unit/integration testing](https://apisyouwonthate.com/blog/writing-documentation-via-contract-testing) anyway, so the whole interface is covered. Don't half-ass your description documents. Use them to plan something amazing, and cut down the amount of recoding you need to do down the line. Create APIs which last longer, which are better documented, better tested, all whilst reducing the total amount of time spent developing the API overall. ### GraphQL & Caching: The Elephant in the Room URL: https://apisyouwonthate.com/blog/graphql-and-caching-elephant-in-the-room/ Last updated: 2023-03-24T12:24:14.000Z > This is a guest post by [Marc-Andre Giroux](https://twitter.com/%5F%5Fxuorig%5F%5F?ref=apisyouwonthate.com) and a preview of a book on GraphQL he's working on. If you're interested you can [check it out and subscribe here](https://book.graphqlschemadesign.com/?ref=apisyouwonthate.com) If you've followed the discussions around whether GraphQL is a good idea or not, you might have heard things like "GraphQL breaks caching", or "GraphQL is not cacheable". If not, I guarantee you'll hear similar things when you start displaying interest, or implementing GraphQL. This is something I see some companies starting to use GraphQL being scared of, and for which they never heard a clear answer on. Before we dive into the world of caching and GraphQL, it might be a good idea to address these common concerns, and where they originate from. Comments like "GraphQL breaks caching" lack the nuance required to actually have a proper discussion about caching and GraphQL. What kind of caching? Client side? Server side? HTTP caching? Application side caching? To have a proper discussion, and end with a better understanding of GraphQL's limitations in terms of caching, we must be more nuanced. ### GraphQL breaks server-side caching? This is a common thing to see thrown around when talking about GraphQL. The first thing to understand is that "server-side caching" is already vague. At this point, we know that GraphQL can actually be a thin layer over our existing servers, and that in no way GraphQL prevents us to cache on the server-side, specifically what we could call **Application Caching**. We will dive deeper into some concepts that can be applied at the application level later on in this chapter. If you're familiar with popular GraphQL clients, you know one of their major feature is a denormalized cache that allows client side application to avoid refetching data they already posess, using it to optimistically update an UI, and to keep a consistent version of the world across components. If we can actually cache things both at the server and client layers, why are we hearing so much about GraphQL "breaking", or making caching really hard? This is where it becomes more nuanced. ### HTTP Caching While certain API styles like REST make great use the powerful HTTP semantics, GraphQL does not really, at least, by default. Since GraphQL is transport agnostic, most server implementations out there use HTTP as a "dumb pipe", rather than using it to its full potential. This causes issues around certain things, like HTTP caching. There are multiple parts to HTTP caching that are important to understand before we go further. First, there are many different cache entities that can be involved in HTTP caching. **Client side caches**, such as browser caches, use HTTP caching to avoid refetching data that is still fresh. **Gateway caches** are usually deployed along with a server, to avoid requests from always hitting servers if the information is still up to date at the cache level. There are two concepts that are particularly important to understand when it comes HTTP caching: **freshness** and **validation**. Freshness lets the server transmit, through `Cache-Control` and `Expires` HTTP headers, the time for which a resource should be considered fresh. For example, a server returning this `Cache-Control` header is telling clients not to bother fetching this resource again until it has been at least one hour (3600 seconds). ``` Cache-Control: max-age=3600 ``` This is especially great for data that doesn't change often, such as browser assets. Whenever the age of the resource we fetched will be greater than this `max-age`, the client will emit a request instead of using the value in its cache. However, it doesn't mean that it actually changed on the server. This is where **validation** comes in. Validation is a way for clients to avoid refetching data when they're not sure if the data is still fresh or not. There are two common HTTP headers to achieve this. The first one is `Last-Modified`. When an HTTP cache on the server has a value for `Last-Modified`, a client can send a `If-Modified-Since` to avoid downloading the data if the data hasn't changed since last time it downloaded it. The other common way of validating caches is using `ETag`. `Etags` are server generated identifiers for reprensentations that change when the representation does. This lets the client track which "version" of the reprensentation it has and avoid re-downloading a representation for which the `Etag` is the same as the one the client has. Together, freshness and validation are a powerful way to control client and gateway caches. To get a deeper understanding of HTTP caching, I highly recommend [this article](https://www.mnot.net/cache%5Fdocs?ref=apisyouwonthate.com), by the great Mark Nottingham. ### GraphQL & HTTP Caching When we dig deeper into the issues with GraphQL & Caching, we discover some of these issues are purely related **HTTP Caching**. It is an important distinction to make since **server-side caching** could mean just as well an HTTP Gateway cache, or application side caching on the server. One of the first things that could influence how HTTP caching works with GraphQL is the HTTP verb used to send GraphQL queries. There is a lot of misinformation out there, that has lead to some people believing using `POST` on a GraphQL endpoint is the only way to make it work. HTTP caches will not caches `POST` requests, which means GraphQL is simply not cacheable at the HTTP level. However, `GET` is indeed a valid way to query a GraphQL server over HTTP. This means that caches could indeed cache GraphQL responses. The only issue with `GET` is with the size of the query string. For example, almost each browser has different limits for these. If this becomes an issue, [persisted queries](https://blog.apollographql.com/persisted-graphql-queries-with-apollo-client-119fd7e6bba5?ref=apisyouwonthate.com) become very useful. We'll cover those later on, but they let you store query strings on the server instead of the client, meaning a client could execute queries like this: ``` GET /graphql/my_query ``` There's one last blocker. Since most GraphQL implementations don't use much of HTTP semantics, most GraphQL servers will currently let you use `GET` along with a `mutation` operation. **This will not play well with caches**. One way to address this issue would be to design your server to reject mutations using `GET`, and require mutation operations to be run on `POST` only. At this point, we've got all the basic elements to have HTTP caching and GraphQL working together. In fact, as we talked about earlier in the book, if we see GraphQL queries as way to dynamically create a server side client specific representation, each query is in fact, something that could be cached. Can we apply HTTP concepts to GraphQL queries? Let's start with freshness. With freshness, what we would want is for a server to be able to tell a client how long the query can be considered as fresh, and when to request for this data again. The unfortunate thing here is that HTTP semantics operate on whole responses/representations, and doesn't care or understand GraphQL queries, meaning we don't have a way to do per-field freshness for example. Still, nothing could stop us from adding a freshness to a whole query: we could say that a GraphQL query's max-age is equal to the field in the query with the lowest max-age. Validation is similar. While we can't use HTTP to revalidate only parts of the query, we could set `Last-Modified` to the value of the field with oldest `Last-Modified` value, and we could also generate an `ETag` based on a combination of all data loaded within the query. While these are **possible**, they're not ideal. Since GraphQL queries possibly span multiple entities that could change, and that they need to be represented as one representation on the GraphQL side, the amount of invalidations would be quite high. A single field being invalidated would invalidate the entire query, even if the rest of it was still fresh. ### Customizability vs Optimizability, Again Remember the continum of customizability we covered earlier in this book? Well it turns out this also affect how "cacheable" GraphQL really is. The invalidation issue we discussed above is not something very specific to GraphQL. In fact, it is specific to highly customizable APIs. Take for example a typical HTTP endpoint for a web API: ``` GET /user/1 ``` This particular endpoint accepts no particular query parameters and simply returns the user associated to this URI. As a public API especially, this endpoint is highly cacheable across all API clients. Now imagine a more customizable version of this endpoint: ``` GET /user/1?partial=complete GET /user/1?partial=compact ``` This API uses a `partial` query parameter to change the level of detail of the response. An even more customizable API, just as we saw in the introduction could look like this: ``` GET /user/1?fields=name,friends ``` The more versions of an HTTP endpoint we have, the more we "dillute" the cache. Meaning someone requesting `fields=name` only can't actually use a cache, even though someone requested `fields=name,friends`. We've got the same issue happening with GraphQL, remove a field, change anything to a query in fact, and we lose the benefit of all queries that were cached with a superset or subset of the data. As you see however, this is not something specific to GraphQL at all, and can be found in any API over HTTP that decides to opt for a more customizable API. Hopefully, that tradeoff was deliberate and the cache invalidation issues were worth it on the long run. Instead of "GraphQL is not cacheable", how about "Highly customizable APIs benefit less from HTTP caching"? ### How Important is HTTP Caching to you? There's no doubt HTTP caching is a wonderful mechanism for data that doesn't change often, and can be shared across multiple users, especially when talking about gateway caches. For authenticated, web APIs, the eternal debate is on how useful HTTP caching really is. It is a debate which I won't solve here, but that is still interesting to discuss. An interesting fact is that shared caches actually **should not** cache any request with an `Authorization` header. If your API is authenticated, the "GraphQL breaks shared caches" argument simply does not apply. Private caches, such as browser caches and client side caches could still gain a lot from using HTTP caching. As we saw, it is **not out of question** with GraphQL, it is simply not as powerful as for highly optimized/one-size-fits-all APIs because of how often a query can be invalidated and how little can be shared. Another thing to keep in mind is that lot of web APIs actually can't have stale data for very long and freshness headers become less useful. Validators such as `ETag` and `Last-Modified` usually require the server to retrieve all necessary data and run business logic to be computed. This usually is the major part of the work, savings being mainly on serialization, and bandwith since no data needs to be transmited. If bandwidth or serialization is an issue, again, nothing could stop you to implement `Etag` or `Last-Modified` generation for a GraphQL query. GraphQL definitely made tradeoffs where it is much more suited to authenticated APIs and realtime data that changes often, versus serving long lived data as a public API. If your use case is the latter one, and it is the only thing your API does, considering using an API architecture that uses HTTP in a more meaningful way could be a better choice. ### Ways Forward HTTP Caching could benefit GraphQL in good ways. The lack of **GraphQL over HTTP** specification is something that makes things a bit harder. The mutations over `GET` is an example of something that could be solved by such specification. However, there are many other ways to cache GraphQL, be it at the client level, the whole response level, the individual resolver level, etc. In this chapter, we will mainly cover GraphQL specific approaches since these are the most used tools at the moment and can be more powerful in the long run since they understand GraphQL semantics. ### Testing API Client Applications URL: https://apisyouwonthate.com/blog/testing-api-interactions/ Last updated: 2023-03-24T12:24:28.000Z One problem with building applications that talk to external dependencies like APIs, is that the applications are talking to external dependencies. This opens up a whole can of worms when it comes to testing the application. You may well have heard developers saying: don’t let tests hit external dependencies! Some folks take that to various different extremes, and don’t even let their tests talk to a database they control. That might make sense in some situation, and not in others, but something pretty much everyone agrees on is that a test suite hitting an actual API over the wire is not ideal. If the test suite is hitting a production API, you could end up sending "funny" (offensive) test emails to a bunch of customers. If a special testing API exists, then multiple developers hitting that test server could cause state to bleed from one test to another, causing race conditions, false positives, false negatives, or all sorts of nonsense. Trying to reset an external API back to a specific state for each test is a fools errand. If you somehow manage it, your test suite now requires the internet, meaning anyone of your team is gonna be screwed next time they try working from a coffee shop, busy conference, plane, etc. Here are a bunch of solutions that not only help you cut the cord, but help you get the application into specific states, improving the quality of your tests. ## Mocking Code Dependencies with Unit Tests Hopefully your application is not littered with HTTP calls to this API or their SDK directly, because that would be some tight coupling and make it reeeeal hard to switch the API for another one if the company yank it for some reason. You probably have some thin layer wrapping their logic, giving you the chance to swap things out without changing too much of your own code. Maybe it looks a bit like this: ```ruby class Geocoder def address(str) google_sdk.geocode(str) end end ``` The application code has `VenueService` which is talking to `Geocoder` and using the `address` method, which pops off to the Google Maps API to do the thing. To avoid the test suite hitting the external API, the most likely move is to mock the `Geocoder` in the `VenueService` tests. ```ruby RSpec.describe VenueService do describe '.update' do it 'will geocode address to lat lon' do allow(Geocoder).to receive(:address).with('123 Main Street') do { lat: 23.534, lon: 45.432 } } subject.update(address: '123 Main Street') expect(subject.lat).to eql(23.534) expect(subject.lon).to eql(45.432) end end end ``` Basically what we have here is a test (using RSpec but whatever it’s all the same) which describes how the `VenueService` should work. The `update` method is being tested, and the `Geocoder` is being set up (monkey patched 🙈) to respond in a certain way. For the `VenueService` unit tests this is fine, because the intent is to make sure `VenueService` works with what we think `Geocoder` is going to return. Unit tests for `VenueService` only focus on that class, so what can we do to make sure Geocoder is working properly? Well, unit testing that class is one option, but it’s not really doing much other than talking to the Google Map SDK, and we really dont want to mock that. Why? Because we don’t own it, and mocking things you dont own is making guesses that might not be correct now, and might not be correct later. The Google Maps SDK might change, and if all we have are tests saying that the SDK works one way, but really it works another way, then you are in false positive world: a broken application with a lovely green test suite. *This will often be less of a problem for typed languages like Go,* *TypeScript, PHP 7, etc., but changes can happen which those type systems* *do not notice. For example, a `foo` property can still be a string, but* *suddenly have different contents within that string.* Integration tests are very important to make sure things work altogether. ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/posts/testing-api-interactions/door-handle.gif) ## Web Mocking in Integration Tests Integration tests will be a bit more realistic as they hit more real code, so the behaviour is closer to what is actually likely to happen in production. This does mean integration tests can be slower than unit tests. Some developers avoid integration tests for this reason, but that is reckless and daft premature optimization. Would you rather work on speeding up a slow but reliable test suite, or have a broken production with an untrustworthy test suite. As integration tests hit more code, some folks think hitting the external APIs is just going to happen, but not the case! One approach to avoid hitting the wire, yet still having realistic interactions, is to use something like [WebMock](https://github.com/bblimke/webmock?ref=apisyouwonthate.com) for Ruby, [Nock](https://github.com/nock/nock?ref=apisyouwonthate.com) for JavaScript, or the baked in [httptest](https://golang.org/pkg/net/http/httptest/?ref=apisyouwonthate.com) in Go. These tools are another type of mock, unlike the two other types of mocking discussed so far. Instead of mocking a class in your programming language, they mock a HTTP server. They are also very different from API specification based mocking tools like [Prism](http://github.com/stoplightio/prism/?ref=apisyouwonthate.com), which is a whole other article. Web mocking tools can be configured to respond in certain ways depending on what URL, HTTP method, or body params are sent to it, depending on how complex things want to get. Most of the time this is used for simple stuff. Here’s an example taken from the CLI-tool [Spectral](https://github.com/stoplightio/spectral?ref=apisyouwonthate.com). ```js const invalidOas3SpecPath = resolve(__dirname, '__fixtures__/openapi-3.0-no-contact.yaml'); describe('when loading specification files from web', () => { test.nock('http://foo.local', api => api.get('/openapi').replyWithFile(200, validOas3SpecPath, { 'Content-Type': 'application/yaml', }), ) .stdout() .command(['lint', 'http://foo.local/openapi']) .it('outputs no issues', ctx => { expect(ctx.stdout).toContain('No errors or warnings found!'); }); }); ``` This test is setting up a server on the arbitrary fake hostname `http://foo.local`, with a GET path `/openapi` that returns a YAML file with some specific content. Then other tests can confirm what Spectral will do if it tries to load an unsupported file type, the response contains a 404 status code, or any other number of edge cases. - **PHP:** [Guzzler](https://guzzler.dev/?ref=apisyouwonthate.com) - **Java:** [Wiremock](http://wiremock.org/docs/stubbing/?ref=apisyouwonthate.com) Web mocking is great for when you want to control the response, but once again you should only mock things you own. Using this approach for the Google Maps API example would only be confirming that the Geocoder works with an assumption of what the Google Maps API is going to do. When things change in the API there is no programmatic way to know about it. Even if the change is noticed, updating these mock setups can be time consuming. What we really want is something like Jest Snapshots, but for HTTP request... ## Record & Replay in Integration Tests There is a tactic called "record and replay", and it is available in pretty much every programming language in one form or another. Record & Replay has been around for years, but I did not discover it until I started using Ruby. They have a great tool called [VCR](http://relishapp.com/vcr/vcr?ref=apisyouwonthate.com) ("Video Cassette Recorder"). *For younger developers a VCR is like Blueray but terrible quality and* *the data is printed on a chunk of plastic you shove in a box under your* *TV. It was mostly used for recording telly you weren’t able to watch at* *the time, which is no longer a thing.* VCR explains the goals nicely, so I will use their words: > Record your test suite’s HTTP interactions and replay them during > future test runs for fast, deterministic, accurate tests. The basic approach is to put your test suite in "record mode", which will actually make real requests to the external services, but then it records the response. All the headers, body content, status code, the whole thing. Then when the test suite is run not in record mode, it will reuse the recorded responses instead of going over the wire, meaning it is quick, always going to give the same result, and the entire response is being used, so you know it is accurate. ```ruby require 'rubygems' require 'test/unit' require 'vcr' VCR.configure do |config| config.cassette_library_dir = "fixtures/vcr_cassettes" config.hook_into :webmock end class VCRTest < Test::Unit::TestCase def test_example_dot_com VCR.use_cassette("synopsis") do response = Net::HTTP.get_response(URI('http://www.iana.org/domains/reserved')) assert_match /Example domains/, response.body end end end ``` This is a rather verbose Ruby example for clarity. It includes the config which would normally be tucked away in a helper, and it is manually using a cassette block, but the idea is this: You can define multiple cassettes, and switch them out to see the code working differently. How exactly it works under the hood might be a bit too much of how the sausage is made, but it is very clever so I am going to nerd out a little. In Ruby once again there is some monkey patching going on. It knows to look out for common HTTP clients, and actually messes with their definitions a little (only in the test suite). This sounds a bit scary, but it means VCR can hijack the HTTP requests and use the recorded versions instead. Most of these record & replay tools can be configured to use the more static web mocking tools mentioned previously. Ruby VCR for example can use webmock, just think of VCR as a helper for creating these accurate web mocks. Another convenient thing about record & replay is the ability to have expiring cassettes. You can configure these recordings to automatically expire (vanish) after a certain amount of time, and then the test suite goes back into record mode. Or you can have them throw warnings, and hope some developers actually pay attention. This can be very annoying, but you would not believe how often I have seen client application developers use year old stubs with fields that did not exist anymore. When recorded responses expire, clients need to go over the wire and record new responses. This can be tricky if as the API might have different data now. Some amount of effort can go into getting good data on the API for recording, which might be a case of building a sort of seed script. This annoyance is worth it in the long run, but certainly takes some getting used to. Expiring recordings go hand in hand with [deprecations](https://apisyouwonthate.com/blog/surviving-deprecations-to-resources-and-properties-on-other-apis) and [evolution](https://apisyouwonthate.com/blog/api-evolution-for-rest-http-apis), especially `Sunset` and `Deprecated` headers. If your applications are using reasonably up-to-date recordings, then your test suite can start throwing deprecating warnings, and loudly report about the code hitting is URLs marked for removal with `Sunset`. The Ruby VCR was initially inspired by [Chris Young’s NetRecorder](https://github.com/chrisyoung/netrecorder?ref=apisyouwonthate.com) are the inspiration for a lot of other record and replay tools, and they maintain an impressive list of ports to other languages: - [Betamax](https://github.com/sigmavirus24/betamax?ref=apisyouwonthate.com) (Python) - [VCR.py](https://github.com/kevin1024/vcrpy?ref=apisyouwonthate.com) (Python) - [Betamax](https://github.com/thegreatape/betamax?ref=apisyouwonthate.com) (Go) - [DVR](https://github.com/orchestrate-io/dvr?ref=apisyouwonthate.com) (Go) - [Go VCR](https://github.com/dnaeon/go-vcr?ref=apisyouwonthate.com) (Go) - [Betamax](https://github.com/wjlroe/betamax?ref=apisyouwonthate.com) (Clojure) - [vcr-clj](https://github.com/ifesdjeen/vcr-clj?ref=apisyouwonthate.com) (Clojure) - [scotch](https://github.com/mleech/scotch?ref=apisyouwonthate.com) (C#/.NET) - [Betamax.NET](https://github.com/mfloryan/Betamax.Net?ref=apisyouwonthate.com) (C#/.NET) - [ExVCR](https://github.com/parroty/exvcr?ref=apisyouwonthate.com) (Elixir) - [HAVCR](https://github.com/cordawyn/havcr?ref=apisyouwonthate.com) (Haskell) - [Mimic](https://github.com/acoulton/mimic?ref=apisyouwonthate.com) (PHP/Kohana) - [PHP-VCR](https://github.com/php-vcr/php-vcr?ref=apisyouwonthate.com) (PHP) - [Polly.js](https://netflix.github.io/pollyjs/?ref=apisyouwonthate.com) (JavaScript/Node) - [Nock-VCR](https://github.com/carbonfive/nock-vcr?ref=apisyouwonthate.com) (JavaScript/Node) - [Sepia](https://github.com/linkedin/sepia?ref=apisyouwonthate.com) (JavaScript/Node) - [VCR.js](https://github.com/elcuervo/vcr.js?ref=apisyouwonthate.com) (JavaScript) - [yakbak](https://github.com/flickr/yakbak?ref=apisyouwonthate.com) (JavaScript/Node) - [NSURLConnectionVCR](https://bitbucket.org/martijnthe/nsurlconnectionvcr?ref=apisyouwonthate.com) (Objective-C) - [VCRURLConnection](https://github.com/dstnbrkr/VCRURLConnection?ref=apisyouwonthate.com) (Objective-C) - [DVR](https://github.com/venmo/DVR?ref=apisyouwonthate.com) (Swift) - [VHS](https://github.com/diegoeche/vhs?ref=apisyouwonthate.com) (Erlang) - [Betamax](https://github.com/betamaxteam/betamax?ref=apisyouwonthate.com) (Java) - [http\_replayer](https://github.com/ucarion/http%5Freplayer?ref=apisyouwonthate.com) (Rust) - [OkReplay](https://github.com/airbnb/okreplay?ref=apisyouwonthate.com) (Java/Android) - [vcr](https://github.com/ropensci/vcr?ref=apisyouwonthate.com) (R) If you are a JavaScript user then check out [Polly.js](https://netflix.github.io/pollyjs/?ref=apisyouwonthate.com), comically written by Netflix. It has some great config options. ```js polly.configure({ recordIfMissing: true, recordIfExpired: false, recordFailedRequests: false, expiresIn: null, timing: Timing.fixed(0), matchRequestsBy: { method: true, headers: true, body: true, order: true, } }) ``` The `recordIfMissing` is a good option, which means when folks add new tests it will try to record the request when it is run the first time. This can catch developers out if they are not expecting it, and can lead to a rubbish response being recorded so they have to delete and try again, but again it is worth getting used to. Another one I like is `recordFailedRequests: true`. This is yet another reminder that if the API is ignoring HTTP conventions like status codes, this will not work. Ask the API developers to stop ignoring conventions and build their APIs properly. Maybe send them a copy of *Build APIs You* *Won’t Hate*. if they need convincing. *All this and more in *[Surviving Other Peoples APIs](https://leanpub.com/surviving-other-peoples-web-apis/?ref=apisyouwonthate.com)*, currently available* *for pre-order, with roughly 80% of the book available for download.* ### Server-Side Validation with API Descriptions URL: https://apisyouwonthate.com/blog/server-side-validation-with-api-descriptions/ Last updated: 2025-02-27T16:54:03.000Z Validation can mean a lot of things, but in API land it generally means figuring out if the data being sent to the API is any good or not. Validation can happen in a lot of different places - it can happen on the server, and it can happen in the client. Traditionally client-side and server-side validation have both played a role, covering different use-cases. Client-side validation is generally used to very quickly provide feedback to a user, to do things like highlighting the input box that failed, with red outlines, tooltips explaining that the email address doesn't look valid, explaining that the "Amount to pay off your credit card" should be higher than 0, etc. These days browsers take care of a lot of the visual feedback so often client-side validation is not doing quite as much as it used to, but it is still required for "either field A or B should be set, but not both, and if B is set, then C should be too." sorts of thing. ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/posts/client-validation/strava-name-required.png) Server-side validation has always been required and for an API is the most important of the two. An API that relies entirely on the client is going to end up with problems. Data coming from the client can never be trusted because it's impossible for the server to know what happened on the client. Even if you're developing a private API for only two known clients, there are always chances that validation in those clients breaks down; or someone will hit those APIs with curl or Postman and send some invalid stuff. Even if the database catches invalid data, the [errors won't be useful](https://apisyouwonthate.com/blog/useful-api-errors-for-rest-graphql-and-grpc). Writing validation rules has always been a major source of pains in my... neck, for the last 15 years, so an approach to syncing the two has forever been on my mind. ## API Description Documents Server-side validation is usually doing the most mundane of tasks. - Is this property required - Is this property an email address - Is this property a credit card number - Is this property required if another property is present Some frameworks shove this logic in the controller, which is a pain when you need to validate the same payload in two different use cases. Others shove it in the data model, like Ruby on Rails: ```ruby class Person < ApplicationRecord validates :name, presence: true validates :bio, length: { maximum: 1000, too_long: "%{count} characters is the maximum allowed" } validates :games_played, numericality: { only_integer: true } end ``` Doesn't this all sound incredibly familiar? This is exactly what API description docs [(also known as specifications)](https://apisyouwonthate.com/blog/resolving-overloaded-terms-for-api-specifications-descriptions-contracts) are talking about, required, types, formats, etc... all of this is already handled for us entirely by the same API descriptions we used to [generate our mock servers](http://github.com/stoplightio/prism/?ref=apisyouwonthate.com) for trial integrations, that we wrote to get [beautiful reference docs](https://apisyouwonthate.com/blog/turning-contracts-into-beautiful-documentation), that we are using to manage our API Gateway, etc. Some of you may have read our article a while back about [using JSON Schema for client-side validation](https://apisyouwonthate.com/blog/json-schema-client-side-validation), and now we want to show you how to leverage your existing source of truth for drastically reducing the amount of validation code you need to write server-side too. ## Which Description Format OpenAPI and JSON Schema are the two biggest API description formats, and there are a lot of options for all the programming languages. OpenAPI tools are listed on [OpenAPI.Tools](https://openapi.tools/?ref=apisyouwonthate.com) and JSON Schema has a whole huge list on [JSON Schema: Implementations](https://json-schema.org/implementations.html?ref=apisyouwonthate.com). ### JSON Schema Example JSON Schema is not aware of metadata like URLs or HTTP methods as its designed to work with any JSON data instance. In API land the JSON data instance we're most likely to work with is the HTTP request body or the response body. For example purposes we are gonna make some JSON Schema validation happen in Node.js, but you can use any language that has a [JSON Schema Validator](https://json-schema.org/implementations.html?ref=apisyouwonthate.com#validators) (which is most of them). To use JSON Schema for server-side validation, you normally just grab one of the validators, shove it in your controller, or "routing" or whatever your language/framework of choice calls it. Using [Express.js](https://expressjs.com/?ref=apisyouwonthate.com), we can put this logic in our route. A fairly familiar `app.js` to Node users, we are just requiring a few bits of code, and loading the `userSchema`. ```js const Ajv = require('ajv'); const express = require('express'); const fs = require('fs'); const ajv = new Ajv(); const app = express(); const port = 3000; const userSchema = JSON.parse(fs.readFileSync('./schemas/user.json')); app.use(express.json()); app.put('/:id', async (req, res) => { const updateData = req.body; try { const valid = ajv.validate(userSchema, updateData); if (!valid) { res.status(400).json({ errors: ajv.errors }); return; } } catch (ex) { // ajv.validate may throw, you can handle exceptions here res.status(500); return; } // Updates the user with some ORM or whatever res.status(200).send('that was great'); }); app.listen(port, () => console.log(`Server on port ${port}!`)); ``` That `userSchema` file comes from `schemas/user.json` which you can make locally, and will have contents like this: ```json { "$id": "http://example.com/schemas/user.json", "type": "object", "$schema": "https://json-schema.org/draft-07/schema#", "properties": { "name": { "title": "Name", "type": "string", "description": "Users full name supporting unicode but no emojis.", "maxLength": 20 }, "email": { "title": "Email", "description": "Like a postal address but for computers.", "type": "string", "format": "email" }, "date_of_birth": { "title": "Date Of Birth", "type": "string", "description": "Date of users birth in the one and only date standard: ISO 8601.", "format": "date", "example": "1990–12–28" } }, "required": [ "name" ] } ``` When you inspect the structure, you can infer that our JSON object has the following list of properties: `name`, `email`, and `date_of_birth`. The first two are strings, and the third is a date. Also, `name` is marked as required. Now we can run this Node app with `node app.js` and fire HTTP requests at it: ```shell http -v PUT http://localhost:3000/123 name=Frank ``` ```http HTTP/1.1 200 OK Connection: keep-alive Content-Length: 14 Content-Type: text/html; charset=utf-8 Date: Sat, 25 May 2019 08:09:10 GMT ETag: W/"e-4o7E1rWH1O+7xJOCXIMFqIbMSxE" X-Powered-By: Express that was great ``` Ok it liked that because `name` was set but `email` and `date_of_birth` are optional. Let's try sending them, but bad. ```shell http -v PUT http://localhost:3000/123 name=Frank email=notanemail ``` ``` HTTP/1.1 400 Bad Request Connection: keep-alive Content-Length: 164 Content-Type: application/json; charset=utf-8 X-Powered-By: Express { "errors": [ { "dataPath": ".email", "keyword": "format", "message": "should match format \"email\"", "params": { "format": "email" }, "schemaPath": "#/properties/email/format" } ] } ``` Oh no! Some errors happened. Sorry! These errors are not the best format because we just dumped them out for demo purposes, but this can be tidied up with a simple helper. If you were already doing validation in the controller, then your controller should be a lot cleaner, and if you are doing extensive validation in your model then this will remove a lot of the cruft. If you did not have validation before, then using this approach means you don't need to start writing it. Win win win! Also, whilst this works in any language, Ruby folks using Rack (therefore anyone using Rails too) can use [committee](https://github.com/interagent/committee?ref=apisyouwonthate.com), a fantastic middleware for making this a bit easier. JSON Schema is pretty good at handling request body validation, but having to put this in every controller can be a bit annoying. OpenAPI can help us out here. ### OpenAPI Middleware Example OpenAPI can be a bit easier to implement here, due to it covering the service model too, not just the data model. ```yaml openapi: "3.0.0" # ... snip ... paths: /pets: post: description: Creates a new pet in the store operationId: addPet requestBody: description: Pet to add to the store required: true content: application/json: schema: $ref: '#/components/schemas/NewPet' responses: '200': description: pet response content: application/json: schema: $ref: '#/components/schemas/Pet' ``` Seeing as OpenAPI will say "this schema should be used for this combination of HTTP Method and Path" you do not need to provide the glue. Instead, many languages offer tools that let you just register a middleware, tell that middleware which OpenAPI file to use, then job done. Sticking with Node/Express for the examples, let's take a look at using OpenAPI and registering a middleware: ```js const express = require('express'); const oasValidator = require('express-ajv-swagger-validation'); const port = 3000; oasValidator.init('./openapi.yaml').then(function () { const app = express(); app.use(express.json()); app.get('/pets', oasValidator.validate, function (req, res, next) { return res.json({ result: 'OK' }); }); app.post('/pets', oasValidator.validate, function (req, res, next) { return res.json({ result: 'OK' }); }); app.get('/pets/:petId', oasValidator.validate, function (req, res, next) { return res.json({ result: 'OK' }); }); app.use(function (err, req, res) { if (err instanceof oasValidator.InputValidationError) { return res.status(400).json({ more_info: JSON.stringify(err.errors) }); } }); const server = app.listen(serverPort, function () {}); }); ``` Tadaaa! You don't have to put the validation checks in all the routes, because the middleware can handle that for you, and your route/controller code won't even bother getting invoked if the request coming in is invalid. The framework middleware is able to look at the request, compare it to the API descriptions, and reject it with an error format (hopefully something like [RFC 7807](https://tools.ietf.org/html/rfc7807?ref=apisyouwonthate.com)) before your code even needs to wake up. There are a decent number of options out there, but there should be more: For OpenAPI v3.0: - **PHP:** [openapi-psr7-validator](https://github.com/lezhnev74/openapi-psr7-validator?ref=apisyouwonthate.com) - **Node.js:** [fastify](https://github.com/fastify/fastify/blob/master/docs/Validation-and-Serialization.md?ref=apisyouwonthate.com) / [express-swagger-ajv-validator](https://github.com/Zooz/express-ajv-swagger-validation?ref=apisyouwonthate.com) / [express-openapi-validate](https://github.com/Hilzu/express-openapi-validate?ref=apisyouwonthate.com) - **Ruby/Rails:** [committee](https://github.com/interagent/committee?ref=apisyouwonthate.com) - **Python:** [connexion](https://github.com/zalando/connexion?ref=apisyouwonthate.com) - **Perl:** [Mojolicious::Plugin::OpenAPI](https://metacpan.org/pod/Mojolicious::Plugin::OpenAPI?ref=apisyouwonthate.com) For OpenAPI v2.0: - **Rails:** [swagger\_shield](https://github.com/amcaplan/swagger%5Fshield?ref=apisyouwonthate.com) The Rails tool swagger\_shield is great. It wins maximum "Wont Hate Points" for using RFC 7807 on failure: ```json { "errors": [ { "status": "422", "detail": "The property '#/widget/price' of type string did not match the following type: integer", "source": { "pointer": "#/widget/price" } } ] } ``` ## Some Validation Still Required This is only going to handle validation rules which do not require looking in a data store, or need some other sort of programming to run. You can do rather a lot with JSON Schema or OpenAPI, but it cannot tell you if the email address is valid, or if this resource is generally in the right state to be doing the thing you are trying to do. Not a problem. You can still perform your own checks after this validation is done, and because everything is all [using RFC 7807](https://tools.ietf.org/html/rfc7807?ref=apisyouwonthate.com) the whole way through then your code and the middleware and everything else will all match. Lovely! ## Testing Benefits There are two huge benefits we have not quite got to yet, beyond the time and money saved from not having to write out a bunch of validation code. The whole idea of trying to [keep docs in sync with code](https://apisyouwonthate.com/blog/keeping-documentation-honest) goes out the window when your description documents are literally code. Seeing as you are using the same description documents to handle request validation that you are using for your documentation, mock server, etc, there is no need for extra logic to ensure your requests are correctly described (or documented). You can just do your usual integration testing on requests, and you are all set. ```ruby it 'should conform to user schema' do get "/users/#{subject.id}" expect(response).to match_json_schema('user') end ``` Your test suite handling bog standard unit and integration tests are now proving your documented requests are correct, and if somebody changes how requests work without updating the description docs, they've been caught in the act and their pull requests will fail. If they update the description in the pull request to fix the tests, boom, we can now have a little chat about why they just tried to commit a breaking change...! 🧐 Using description docs for validations only covers requests, so [use the API description document to power contract testing](https://apisyouwonthate.com/blog/writing-documentation-via-contract-testing) to make sure responses are good too. ## Future of Server-Side Validation This is a very old concept which has recently picked up steam as more developers catch onto the API Design-first workflow. One of the benefits of writing HTTP APIs is that you usually are not locked into a single implementation, and do not have to try and force the "one size fits all" tooling that comes with it. Sadly that means some of the tools for some of the languages aren't as excellent as others, but as we are a community of open-source developers we can fix that. I've heard developers say "this is not performant" but there is no reason to believe that. A specific tool might not be written the most efficient way, but that can be fixed with PRs. So long as the tool is not parsing the entire document on the fly, and on startup constructs some sort of artifacts in memory, this could easily be more performant than running whatever behemoth of a "validation library" you've loaded up to do all this manually. Another approach is to skip out on doing it in the server-side, and move it up a level: to the API Gateway. We'll be writing more about that soon, but most API gateways are starting to get smarter about how they accept API descriptions as input, and how they use that input. One example is [Express Gateway](https://www.express-gateway.io/?ref=apisyouwonthate.com), who [added JSON Schema Validation](https://www.express-gateway.io/docs/policies/customization/conditions/?ref=apisyouwonthate.com#json-schema), a project maintained by my friend and [Stoplight.io](https://stoplight.io/?ref=apisyouwonthate.com) colleague [Vincenzo Chianese](https://twitter.com/D3DVincent?ref=apisyouwonthate.com). ## Summary The days of treating descriptions like some annoying thing you have to do later to get docs are long behind us. API descriptions now come first, are used for [contract testing](https://apisyouwonthate.com/blog/writing-documentation-via-contract-testing), getting feedback on implementation, and a whole [bunch of other stuff](https://www.toptal.com/api-developers/5-new-things-rest-specification?ref=apisyouwonthate.com). Use this as another carrot to convince your laggard teammates or boss that this is the way to go. Once again, using existing standards mean you are getting amazing benefits you have not even heard of yet, so implementing cool new stuff like this comes along almost for free once you've got descriptions. ### Does REST and Hypermedia Have a Place in the Modern API Landscape? URL: https://apisyouwonthate.com/blog/rest-and-richardson-maturity-model/ Last updated: 2024-12-12T12:52:34.000Z Nobody out there thinks you should always use REST for every single API being built, nor should you ever use any one paradigm for everything, but a growing number of people misunderstand REST so badly they think they never need it. As with most things, there is a whole bunch of nuance that the average developer doesn’t have time to consider when they’re trying to get their job done, but some of that busyness can come from solving problems that popped up because the wrong paradigm was being used in the first place. REST can be considered a bunch of layers of abstraction that help you talk to the client developers less, whilst enabling them to build more sturdy and consistent applications. This is achieved in a few ways, mostly by leaning on the existing semantics of whatever protocol the REST API exists in, and usually that is HTTP. Building a REST API in HTTP usually means deferring to existing HTTP semantics to achieve the concepts REST sets out to achieve instead of inventing your own conventions. Instead of having to know how Rachel built caching in their API, I should be able to just enable a [generic caching middleware](https://apisyouwonthate.com/blog/speeding-up-apisappssmart-toasters-with-http-response-caching) and have that work. Instead of writing a bunch of code to know every single application error an API might respond with and try to figure out if it’s one I can retry or not, clients can consult the `Retry-After` header. Folks often complain that REST is hard because they have to learn about HTTP methods and their meaning, and different status codes. Whilst learning HTTP might take some time, the conventions of HTTP are shared with a huge ecosystem of tooling, which you are supporting without even knowing about it by using these conventions. People have always tried to avoid having to learn this stuff, and a lot of the time RPC ends up being what people go with. For a lot of developers, having a collection of functions that operate over the Internet, fetching some data from a thing and triggering a thing is the pinnacle of an API. In doing that they are creating a lot of their own conventions and losing a lot of the benefits that REST sets out to provide. Going back to the comment “REST is a bunch of layers of abstraction on top of RPC”, let’s look at a diagram I’m sure most of you are familiar with: the Richardson Maturity Model. ![](https://miro.medium.com/v2/resize:fit:1000/0*CJ9DSMbbcig933JA.png) This is a visualization created by Martin Fowler in his article introducing [the Richardson Maturity Model](https://www.martinfowler.com/articles/richardsonMaturityModel.html?ref=apisyouwonthate.com). The model takes the name of [Leonard Richardson](https://twitter.com/leonardr?ref=apisyouwonthate.com), and his talk about [API maturity](https://www.crummy.com/writing/speaking/2008-QCon/act3.html?ref=apisyouwonthate.com). There are a few common concerns with this diagram, mainly with “Glory of REST” at the top, and the “Swamp” at the bottom. For the same reason I have concerns about the word “maturity” being used. This has the unfortunate effect of making it seem like REST APIs are amazeballs and everything else is stupid. That’s not what anyone was trying to say, but it’s the conclusion a lot of people draw. When talking about this diagram I usually explain that an RPC API doing a job that a REST API would be better suited at is gonna suck. I wouldn’t take my mountain bike on a 200mi ride and I wouldn’t take my carbon road racer on a downhill mountain bike course. A good thing used for the wrong task very quickly starts to look like a bad choice, without the thing being inherently bad. Without wanting to talk about glory: an API getting the full benefits of REST is going to be better protected against a lot of the awful bullshit I spend so much time trying to help companies avoid or solve, but again those benefits are all hardy awesome mountain biking components that will slow me down in a road race. Another concern is that Martin talks about Plain Old XML, and these days some folks talk about POJOs (Plain old JSON Objects). API specifications (metadata to describe your data model) are [available to all API developers](https://apisyouwonthate.com/blog/commit-to-api-contracts) regardless of their paradigm or implementation of choice, so that should be taken out of consideration. REST folks use JSON Schema, gRPC people use Protobuf, and GraphQL users have GraphQL Types. Some folks might be working with POX/POJO if the decision makers in charge of the API development team are committed to ignoring modern best practices, but it is nothing to do with paradigm. # Updated Richardson Maturity Model So with the holier-than-thou concern out the way, let’s take a look at my attempt to update the Richardson Maturity Model. I’m keeping the name because I am not changing anything conceptually from the original talk. Each layer briefly mentions some of the functionality it enabled. ![](https://miro.medium.com/v2/resize:fit:1400/0*6HPvBPr5yCy6mwpz.png) # 0: RPC What we really have as an issue is that RPC in its most basic form usually ignores a lot of HTTP concepts. Instead of leveraging the uniform interface of HTTP and its full semantics, and instead of using HTTP as a transfer protocol, it uses only the transportation aspect. A transfer protocol helps you know when or if you need to make a request, instead of just ferrying data up and down the wire. Most RPC implementations interact with a single endpoint, and most interaction is using a single HTTP method. Very few generic HTTP conventions will work for something that is just the most basic RPC. If the RPC is following a specific standard then tools built for that standard will work, but generic HTTP conventions do not apply. # 1: Resources Two common confusions here, firstly this is not about having `/bikes` and`/bikes/abc123` in that standard collections and plurals and resources CRUD pattern we are often used to. I have fallen for this in the past. Resources are technically the same thing as endpoints, but there is an intentional distinction. Endpoints are often thought of more like functions, and the intention is that you call a function whenever you want to do a thing, but that is again most transport that transfer, and is usually a sign of RPC thinking: call a thing and do a thing. Resources are more like identifiers, a unique thing which lives in a specific place, and can be identified by that thing. It is the ultimate unique identifier in HTTP world, because whilst two companies could have different products with the same alpha/numeric ID, and even UUID collisions are mathematically possible, we are never going to run into collisions with `https://cannondale.com/bikes/abc` and [https://surly.com/bikes/abc](https://surly.com/bikes/abc.?ref=apisyouwonthate.com)[.](https://surly.com/bikes/abc.?ref=apisyouwonthate.com) The URI (Uniform Resource Identifier) is not wildly exciting in itself, but having unique URIs for everything means you can start adding specific headers to different resources, which can be stored along with the responses as metadata. This lets resources declare their own cacheability, which is one of the big things REST talks about. > *Cache constraints require that the data within a response to a request be implicitly or explicitly labeled as cacheable or non-cacheable. If a response is cacheable, then a client cache is given the right to reuse that response data for later, equivalent requests.* > > *The advantage of adding cache constraints is that they have the potential to partially or completely eliminate some interactions, improving efficiency, scalability, and user-perceived performance by reducing the average latency of a series of interactions. The trade-off, however, is that a cache can decrease reliability if stale data within the cache differs significantly from the data that would have been obtained had the request been sent directly to the server.* > > *— Fielding, Roy Thomas.* [*Architectural Styles and the Design of Network-based Software Architectures*](https://www.ics.uci.edu/~fielding/pubs/dissertation/rest%5Farch%5Fstyle.htm?ref=apisyouwonthate.com) [RFC 7234](https://tools.ietf.org/html/rfc7234?ref=apisyouwonthate.com) handles this nicely. > *The goal of caching in HTTP/1.1 is to significantly improve performance by reusing a prior response message to satisfy a current request. A stored response is considered “fresh”, as defined in Section 4.2, if the response can be reused without “validation” (checking with the origin server to see if the cached response remains valid for this request). A fresh response can therefore reduce both latency and network overhead each time it is reused. When a cached response is not fresh, it might still be reusable if it can be freshened by validation or if the origin is unavailable.* > > *—* [*IETF: RFC 7234*](https://tools.ietf.org/html/rfc7234?ref=apisyouwonthate.com) Having unique URIs for things also means [HTTP/2 Server Push](https://www.smashingmagazine.com/2017/04/guide-http2-server-push/?ref=apisyouwonthate.com) can work as expected. This is a huge benefit of leveraging HTTP properly, using it as a transfer layer and not a dumb tunnel. # 2: HTTP Methods Methods add a lot of important semantics to the type of thing happening in the request. IF caching is used, the caching component will know it can cache a GET request, but if a POST or DELETE is made to that same resource, it knows it should get out of the way. Client-side logic like automatic retries are now possible. A retry can help when an API is taking a long time to respond, a client application might [bail on the request and try again](https://apisyouwonthate.com/blog/taking-a-timeout-from-poor-performance). With a GET there are barely any downsides here, because it is an idempotent request that should not have any destructive actions. You could GET a thing 3479 times and you would just have that data. Retrying a POST could be dangerous, as maybe before the timeout was reached, it had managed to change some records in the database, send some emails, charge a credit card, etc. PUT and PATCH would be fine, because PUT is idempotent and just obliterates the result, and PATCH usually has a “from” and “to” meaning if the request is made a second time the “from” would probably not match. People see POST vs PUT vs PATCH and get upset about having to [learn the difference](https://apisyouwonthate.com/blog/put-vs-patch-vs-json-patch), but again these semantics are baked into HTTP tooling instead of everyone being forced to `updatePartialThing` and `updateFullThing` and invent other conventions around idempotency... If you are a fan of gRPC you will be thinking that a lot of this stuff sounds possible, and you’re right! The gRPC “HTTP Bridge” adds these two layers of abstraction, to make it a bit more HTTPish. It’s not a REST bridge as some people call it, because it’s missing this next layer… # 3: Hypermedia Controls Hypermedia Controls is shorthand for “Hypermedia as the Engine of Application State” (HATEOAS), which is quite a simple concept. Instead of an API being just a datastore-over-HTTP, it becomes a state machine-over-HTTP. It’s still got data, but it can also offer “next available actions” in self describing ways. Think about an invoice saying it is payable, instead of you needing to figure out if it can be paid based on the lack of a `paid_date`, or maybe there is a `status: pending`, but maybe a new status gets added and pending doesn't mean you can pay it anymore... Client applications break, or need to be versioned, both of which wastes developer time and company money. Having a link show up called “pay” if the invoice is payable means the client application knows when to pay, and so long as a good hypermedia format is used the client application will know *how* to pay, as the controls can mention what data is required, offering the means to [validate that data client side](https://blog.apisyouwonthate.com/the-many-amazing-uses-of-json-schema-client-side-validation-c78a11fbde45?ref=apisyouwonthate.com) before you even send a HTTP request to the server… transfer > transportation, helped us out again! The most basic level of hypermedia is shoving links into the response body (thanks Resources!) but then the client has to do a lot of detective work to figure out what they can do next. In the past folks would just shrug and say “you have a URL and a link relation, that’s a good start”, but these days there are quite a few popular Hypermedia Formats around which make things a whole lot easier than that. We’ve talked in a lot more depth about [representing state in APIs](https://apisyouwonthate.com/blog/representing-state-in-rest-and-graphql) for more of a general overview. Most APIs that call themselves REST stop short of the last layer, which mean they are what many people all RESTish, or just a HTTP API. That’s not to be snotty, it’s because Hypermedia Controls make it a REST API, it is a huge chunk of the point. Sometimes it’s a lack of education on the topic, where people just literally have no idea what HATEOAS is about. Fair enough! Other times folks think they understand it, and think that HATEOAS is about forcing you to make loads of HTTP requests to get the same amount of data. That usually shows they’re thinking about transportation and not transfer, and these days with HTTP/2 even if you were needing to make “more calls” the performance impact is negligible. # Next Once you get to the REST part of the diagram that doesn’t mean your API is suddenly infallible and perfect in all ways forever. Shoddy resource design will make any API a pain to work with regardless of the paradigm being used, and [GraphQL developers are starting to notice that](https://medium.com/@%5F%5Fxuorig%5F%5F/the-tension-between-data-use-case-driven-graphql-apis-8f982198653b?ref=apisyouwonthate.com) now. A focus on model design that meets the needs of your clients is important, and APIs can evolve over time to trim away useless data, and create composite resources to minimize network chattiness. JSON Schema just got a [deprecated keyword](https://github.com/json-schema-org/json-schema-spec/pull/737?ref=apisyouwonthate.com) too which can make [API evolution](https://blog.apisyouwonthate.com/api-evolution-for-rest-http-apis-b4296519e564?ref=apisyouwonthate.com) a whole lot easier. # Hypermedia + gRPC / GraphQL When talking about Hypermedia Controls, people have said things like “That’s not just something REST can do, gRPC could do that if you used the HTTP Bridge and added links!” Comically they were saying this in a shouty, red faced, gRPC-defending way, and my answer was “Yes! Absolutely, if you add Hypermedia Controls to a RPC API along with all these other things then you have literally made it a REST API!” REST is a collection of ideas, and you can use those ideas anywhere you like. A few prominent GraphQL people have been trying to figure a way to get Hypermedia Controls into GraphQL for a while. If they figure it out, GraphQL would not be following this diagram exactly, but we can call “query” and “mutation” close enough to HTTP Methods to give them a pass, and the only thing missing is resources (URIs). Missing URIs is a larger problem for GraphQL because it pretty much destroys their chance of using HTTP/2 Server Push, meaning they’re left turning to vendor specific solutions like [Apollo Subscriptions](https://www.apollographql.com/docs/graphql-subscriptions/?ref=apisyouwonthate.com) and other non-standard [@defer](https://blog.apollographql.com/introducing-defer-in-apollo-server-f6797c4e9d6e?ref=apisyouwonthate.com) extensions things for that. # Summary Anyway, APIs don’t always need Hypermedia Controls, nor do they *need* any of this. For example, full-stack developers often think REST is a waste of time because they are just trying to query the database and get that information to the presentation layer. They do not need to bake cache controls into the message itself because they can just set the caching in the client application which is probably open in another window on their machine. They know when to use retries or not, because they wrote their application codes and know what they mean, so who cares about leaning on HTTP semantics for that. Those developers have absolutely nothing in common with developers trying to provide consistent functionality to a wide variety of client teams who might be on different floors or different continents, where communicating change or how to infer state might be a costly problem. Those teams might be using all sorts of network and client tooling like caching middlewares, monitoring services, inspection proxies, and you don’t want to restrict what tools they’re able to work with because that could lose you business. Then there are all the scenarios in between. Not all cars need to be bullet proof, not all conversations need a translator, not all underwear needs to be edible, and not all APIs need to be REST. 👍 Check out our article [*Picking the Right API Paradigm*](https://apisyouwonthate.com/blog/picking-the-right-api-paradigm) to see when you might want to consider using REST, and when you should use something else. ### PKCE vs Proxy URL: https://apisyouwonthate.com/blog/pkce-vs-proxy/ Last updated: 2023-03-24T12:25:10.000Z ## Background Logging in to a web service used to mean sending a username and password and getting a cookie in return. This cookie is then sent with every request so that the server knows who is making the request. However, when it became more common for services to interact with each other on a user's behalf, other forms of authorization systems evolved. One such system is OAuth, which uses token-based authentication instead of cookies. OAuth is an **Authorization** framework that allows three different parties to interact with a minimum level of trust. This way only one service needs to know your username and password. The service that doesn’t store this information can ask the service that does to verify who the user is and what they are allowed to do. In general, this provides a better, more secure, user experience. A user can maintain their profile and password information in a single place and have that referenced automatically when they connect to other services. No more typing in email addresses to every site and updating it in multiple places if it changes. This is how **Login with Facebook** or **Login with Google** buttons work. Unfortunately, there are a couple of inherent security risks with this approach. One service is not as trustworthy as another. You might want to allow that budgeting app to read your account balance, but you probably don’t want it to empty your checking account. Additionally, the first time services communicate they must go over a public network where anybody can be listening. In order to protect the credentials and limit what those credentials can do some initial trust needs to be bootstrapped ahead of time. This is done by combining two different components to ensure that the recipient of a temporary public credential is, in fact, the one it is intended for. If you have heard of two-factor authentication **(2FA)**, it's like that but for web services. There are a couple of methods that use this combination of factors to provide some extra confidence the service getting access is the right one. We’ll take a look at two of those; **Authorization Code Flow** and **Proof of Key Code Exchange**. ## Authorization Code Flow *Protection by Proxy* **Authorization code** flow is the classic way to do this when a client that has a server component. This server component protects an OAuth **client secret** and proxies the token request from the client to the **Authorization Server**. In this flow, a user is redirected from the service hosting the protected resources to an **Authorization Server**. The **Authorization Server’s** job is to authenticate the user and approve access. After successfully authenticating, usually with a username and password, the user is redirected back to the service with some extra URL parameters including an **authorization code**. The **authorization code** is a temporary code needed to retrieve the **access token** for permission to interact with the **relying service's** resources. The **relying** **service** then exchanges the **authorization code** combined with their **client id** and **client secret** for an **access token** and optionally a **refresh token**. The **relying** **service** then returns these to the user's client, usually a browser or mobile application. The client then uses this token to tell the **resource server** who the request is for and what permissions they have. ![Authorization Code Flow](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/posts/auth-code-flow.png) ## Proof of Key Code Exchange (PKCE) *Cryptographic Pixie Dust* With the rise and popularity of mobile apps and Single Page Applications (SPA) a new flow had to evolve to accomodate public clients. These clients cannot properly protect a **client secret** since they run in an environment not controlled by the application owner. These are often either a mobile device or JavaScript in the browser. To compensate for the lack of confidentiality the flow extends the **authorization** **code** flow using cryptography to generate a **code verifier** and a **code challenge** for every **authorization code** exchange. During the request to get an **authorization code** the **relying service** includes the **code challenge**. Then when the **relying service** receives the **authorization code** it exchanges that in addition to the **code verifier**. The **authorization server** then uses the **verifier** to confirm the original **code challenge**. ![PKCE Flow](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/posts/pkce-flow.png) ## Why not exchange the password for a token? Exchanging the password for a token is called the **Resource Owner Credential** **Flow**. This flow is not recommended unless there is a HIGH level of trust between the **Authorization Server** and the client. When the client is JavaScript or a mobile app the trust just isn’t there. The previous flows allow the for interaction outside of the initial client for exchanging the password with the **Authorization Server**. For example, on a mobile device, the application might open a browser to begin the exchange and finish at an app-specific URL to complete it. This way the mobile application never has to handle the username and password directly. ## What does this buy us? In both **Authorization Code** and **PKCE** flows, two factors must be exchanged for valid credentials. The **authorization code**, as presented as part of a redirect URL for consumption, along with some additional information posted in the body of the **access token** request. These two channels of communication mitigate a variety of attacks and misconfigurations where bearer tokens or authorization can be intercepted in flight. However, with **PKCE** there is one more consideration to take into account and that is where the **Authorization Server** redirects after authentication. The client tells the **Authorization Server** where it wants to receive the code in the URL. If this is not an exact match, it means there is an opening for a malicious client to trick the **Authorization Server** to send the code somewhere else. That same client can also generate the temporary secret code. While **PKCE** moves the goalposts in terms of difficulty, it still relies on this redirect as an anchor of trust. While having a fully qualified redirect URL is a best practice, **Authorization Code** flow mitigates an open redirect misconfiguration due to the fact that the server still holds a predetermined secret. Because the secret used in **PKCE** is generated at runtime, a malicious actor capitalizing on an open redirect can still follow to the protocol to get valid credentials. On mobile clients, this is an app-specific URL such as `app-foo://auth_code_handler#code=XXX`. For SPA’s this would be something like `https://app.foo.com#code=XXX`. While **PKCE** was originally intended for mobile applications the [OAuth Best Practices Working Group](https://tools.ietf.org/html/draft-ietf-oauth-security-topics-11?ref=apisyouwonthate.com#section-2.1.1) has recently started to recommend it for SPA’s as well. To add a **PKCE** flow to your application take a look at [https://appauth.io/](https://appauth.io/?ref=apisyouwonthate.com) for supported libraries and implementations. ### Creating Good API Errors in REST, GraphQL and gRPC URL: https://apisyouwonthate.com/blog/useful-api-errors-for-rest-graphql-and-grpc/ Last updated: 2023-03-24T12:25:22.000Z Dealing with the Happy Path™ in an API is pretty easy: When a client asks for a resource, show them the resource. When they trigger a procedure, let them know if it was triggered OK, and maybe if it completed without a problem. What to do when something doesn’t go according to plan? Well, that can be tricky. [HTTP status codes](http://http.cat/?ref=apisyouwonthate.com) are part of the picture, they can define a category of issue, but they are never going to explain the whole story. Two examples from a carpooling application which had a "simulated savings" endpoint, to let folks know how much they might save picking up a passenger on their daily commute: This error let the client know the coordinates were too close together, meaning it is not even worth driving let alone trying to pick anyone else up. ```http HTTP/1.1 400 Bad Request { "errors" : [{ "code" : 20002, "title" : "There are no savings for this user." }] } ``` This carpool driver is trying to create a trip from Colombia to Paris. ```http HTTP/1.1 400 Bad Request { "errors" : [{ "code" : 20010, "title" : "Invalid geopoints for possible trip." }] } ``` This is often touted as a failing of the HTTP status code concept, but it was never intended to cover every single possible application specific error message. Think of HTTP status codes like an exception. In Ruby you might get a ArgumentError or LoadError exception which gives you a pretty good hint as to what the issue is, but there is also data specific to the instance of that failure that helps with fixing the situation. Programming languages do not just give you the exception name, they give you instance information too. ```ruby > require "nonsense" LoadError (cannot load such file -- nonsense) ``` Errors in HTTP APIs are pretty similar to exceptions: they can tell the client what is going on, and combine a bunch of useful metadata to help both the client and the server solve problems. This is often in the response body, using JSON or whatever data format the API generally uses. ## Error Objects A well designed API error will have at the very least: - A 4xx or 5xx status code depending on the situation - A human readable short summary: Cannot checkout with an empty shopping cart - A human readable message: It looks like you have tried to check out but there is nothing in your…​ - An application-specific error code relating to the problem: `ERRCARTEMPTY` - Links to a documentation page or knowledge base where a client or user of the client can figure out what to do next This will help humans and machines to figure out what is happening. Missing out the error code means clients need to implement substring matching, which is awful for everyone, and turns contents of the error message into part of the agree contract. Imagine a text-change breaking integration with multiple unknown clients! 😳 This used to happen with Facebook and their rather bad Graph API, where any issue with an access token would return type: OAuthException, regardless of the type of issue. If it was an expired token which needed a refresh, or if it was just total nonsense, you would get the same type, and a different string. ```javascript { "error": { "type": "OAuthException", "message": "Session has expired at unix time 1385243766. The current unix time is 1385848532." } } ``` Without getting too much into Authentication at this point, there are times where the client would want to take different actions for different errors. For example, when an access token was previously good but expires, the client wants to suggest the user try logging in again, or reconnecting their Facebook account. When the token is just nonsense (a totally invalid token) then a different action needs to be taken. These days Facebook have a far more robust error object in their Graph API, with error codes and even "sub-codes", so the client developer has enough information to react programmatically to various errors. An improved version of that error message, with an error code and a link ```javascript { "error": { "message": "Message describing the error", "type": "OAuthException", "code": 190, "error_subcode": 460, "error_user_title": "A title", "error_user_msg": "A message", "fbtrace_id": "EJplcsCHuLu" } } ``` They explain the structure of the error object in their documentation. > - **message:** A human-readable description of the error. > - **code:** An error code. Common values are listed below, along with common recovery tactics. > - **error\_subcode:** Additional information about the error. Common values are listed below. > - **error\_user\_msg:** The message to display to the user. The language of the message is based on the locale of the API request. > - **error\_user\_title:** The title of the dialog, if shown. The language of the message is based on the locale of the API request. > - **fbtrace\_id:** Internal support identifier. When reporting a bug related to a Graph API call, include the fbtrace\_id to help us find log data for debugging. > > —- [Facebook GraphAPI Documentation](https://developers.facebook.com/docs/graph-api/using-graph-api/error-handling?ref=apisyouwonthate.com) ## Know Your Audience Making errors be useful for client users (not just client developers) can be a powerful thing. Clients can build their interface around the expectation that a link in an error will help their users out, without needing to know specifically what the actual error is. Whenever possible try to avoid creating an API error that you would not want to show to a user. Often a client will create a filter that checks for certain errors to do something, and anything left can be thrown up as a generic error box with the message in it. Clients doing this help future proof their application. For example, if a new validation rule pops up they might not have their UI code written to check for that, but an ugly alert box can pop up with instructions to the user and maybe that is better than the application just being completely unusable. Another useful thing to do is put a link for more information. Add a href/link/url property to your error object. ```javascript { "error": { ... "href": "http://example.org/docs/errors/#ERR-01234" } } ``` In some instances maybe this more information link points to a blog post or some documentation which explains that the user should update their application, or take some other action to resolve the situation, or email somebody, or reset their password. 👍 ## The Trouble with Custom Error Formats Everyone starts off building APIs with their own error format. It usually starts off as just a string. ```javascript { "error": "A thing went really wrong" } ``` Then somebody points out it would be nice to have application codes, and new versions of the API (or some different APIs built in the same architecture) start using a slightly modified format. ```javascript { "error": { "code": "100110", "message": "A thing went really wrong" } } ``` Guess what happens when a client is expecting the first example of a single string, but ends up getting that second example of an object? ![When an error is an object but the client shoves it into a string, you see object Object and it happens aaaall the time.](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/posts/useful-errors/object-Object.jpg) A wild \[object Object\] appears on Gelato — a discontinued API design and analytics platform acquimerged into Kong. These errors happened at my previous job all the time, because every one of the 50 APIs had a totally different error format, some had multiple different error formats in different API versions (v2 and v3 would have different error formats), and [you would be expected to hit both!](https://apisyouwonthate.com/blog/api-versioning-has-no-right-way) I remember writing a bunch of code that would check for various properties, if error is a string, if error is an object, if error is an object containing foo, if error is an object containing bar…​. ## Standard Error Formats There are two common standards out there for API errors which you should consider using for your next API, or maybe even consider adding to your existing APIs. ### Problem Details for HTTP APIs [Problem Details for HTTP APIs (RFC 7807)](https://tools.ietf.org/html/rfc7807?ref=apisyouwonthate.com) is a brilliant standard from Mark Nottingham, Erik Wilde, released through the IETF. > *This document defines a "problem detail" as a way to carry machine-readable details of errors in a HTTP response to avoid the need to define new error response formats for HTTP APIs.* > —[ Internet Engineering Task Force (IETF)](https://tools.ietf.org/html/rfc7807?ref=apisyouwonthate.com) The goal of this RFC is to give a standard structure for errors in HTTP APIs that use JSON (application/problem+json) or XML (application/problem+xml). ```http HTTP/1.1 403 Forbidden Content-Type: application/problem+json Content-Language: en { "type": "https://example.com/probs/out-of-credit", "title": "You do not have enough credit.", "detail": "Your current balance is 30, but that costs 50.", "instance": "/account/12345/msgs/abc", "balance": 30, "accounts": ["/account/12345", "/account/67890"] } ``` This example from the RFC shows the user was forbidden from taking that action, because the balance did not have enough credit. 403 would not have conveyed that (it could have meant the user was banned, or all sorts of things) but there is text, and there is a type, which is just an error code in the form of a URL. > *Note that this requires each of the sub-problems to be similar enough to use the same HTTP status code. If they do not, the 207 (Multi- Status) \[RFC4918\] code could be used to encapsulate multiple status messages.* > *A problem details object can have the following members:* > > - type (string) — A URI reference \[RFC3986\] that identifies the problem type. This specification encourages that, when dereferenced, it provide human-readable documentation for the problem type (e.g., using HTML \[W3C.REC-html5–20141028\]). When this member is not present, its value is assumed to be abou:blank". > - title (string) — A short, human-readable summary of the problem type. It SHOULD NOT change from occurrence to occurrence of the problem, except for purposes of localization (e.g., using proactive content negotiation; see \[RFC7231\], Section 3.4). > - status (number) — The HTTP status code (\[RFC7231\], Section 6) generated by the origin server for this occurrence of the problem. > - detail (string) — A human-readable explanation specific to this occurrence of the problem. > - instance (string) — A URI reference that identifies the specific occurrence of the problem. It may or may not yield further information if dereferenced. > > \-— [Internet Engineering Task Force (IETF)](https://tools.ietf.org/html/rfc7807?ref=apisyouwonthate.com) Remembering all of this might be a little tricky, and asking every API developer to go read and memorize an RFC might not be particularly successful. As with most things, there are implementations that can be slotted into place for languages and web application frameworks to make the whole thing easier. - **PHP:** [zendframework/zend-problem-details](https://github.com/zendframework/zend-problem-details?ref=apisyouwonthate.com) - **Java:** [problem](https://github.com/zalando/problem?ref=apisyouwonthate.com) & [problem-spring-web](https://github.com/zalando/problem-spring-web?ref=apisyouwonthate.com) - **Python:** [https://github.com/cbornet/python-httpproblem](https://github.com/cbornet/python-httpproblem?ref=apisyouwonthate.com) - **Node:** [https://www.npmjs.com/package/problem-json](https://www.npmjs.com/package/problem-json?ref=apisyouwonthate.com) ### JSON:API [JSON:API](https://jsonapi.org//format/?ref=apisyouwonthate.com#errors) is a standard for a lot more than just errors, it attempts to help with a lot of design choices for HTTP APIs, outlining the general format of requests and responses in JSON when working with HTTP APIs. In general it labels itself an anti-bikeshedding tool, and this is pretty accurate. HTTP API developers often feel like there are infinite possibilities, which can lead to a lot of discussions and arguments, so using implementations like JSON:API can get folks on the same page. The following is an excerpt from the JSON:API standard at time of writing. > *An error object MAY have the following members:* > > - **id** — A unique identifier for this particular occurrence of the problem. > - **href** — A URI that MAY yield further details about this particular occurrence of the problem. > - **status** — The HTTP status code applicable to this problem, expressed as a string value. > - **code** — An application-specific error code, expressed as a string value. > - **title** — A short, human-readable summary of the problem. It SHOULD NOT change from occurrence to occurrence of the problem, except for purposes of localization. > - **detail** — A human-readable explanation specific to this occurrence of the problem. > - **links** — Associated resources, which can be dereferenced from the request document. > - **path** — The relative path to the relevant attribute within the associated resource(s). Only appropriate for problems that apply to a single resource or type of resource. > > \-— [JSON:API](https://jsonapi.org/format/?ref=apisyouwonthate.com#errors) Pretty familiar stuff here! Whilst RFC 7807 has an interface that suggests one error object be returned with multiple problems provided using extra properties, JSON:API errors are an array of error objects. ```http HTTP/1.1 422 Unprocessable Entity Content-Type: application/vnd.api+json { "errors": [ { "source": { "pointer": "/data/attributes/firstName" }, "title": "Invalid Attribute", "detail": "First name must contain at least three characters." }, { "source": { "pointer": "/data/attributes/firstName" }, "title": "Invalid Attribute", "detail": "First name must contain an emoji." } ] } ``` That "pointer" is a [JSON Pointer (RFC 6901)](https://tools.ietf.org/html/rfc6901?ref=apisyouwonthate.com), and can be used to point to the specific location in the HTTP request body that failed. This is great for client developers who have a UI. They probably already have some logic which maps their form inputs to request data, so if they use that pointer they can trace the error back to a form input, and show custom validation errors even if they had not built that validation into their frontend. **Note:* Clients love copying validation rules into their applications and* *that leads to all sorts of problems. Provide them with an alternative using* *[JSON Schema client-side validation](https://apisyouwonthate.com/blog/json-schema-client-side-validation).* There are a lot of [implementations for JSON:API](https://jsonapi.org/implementations/?ref=apisyouwonthate.com). To be frank, some are better than others, by which I mean some are amazing and some are truly terrible. Check a few out. ## Should You Use a Standard? RFC 7807 was only released as a final RFC in 2016, and JSON:API is also fairly recent in the grand schema of the Internet. As such there are not many popular APIs using them. This is a common stalemate scenario where people do not implement standards until they see buy-in from a majority of the API community, or wait for a large company to champion it, and seeing as everyone is waiting for everyone else to go first nobody does anything. The end result of this stalemate is that most people roll their own solutions, making a standard less popular, and the vicious cycle continues. Many large companies are able to ignore these standards because they can create their own effective internal standards, and have enough people around with enough experience to avoid a lot of the common problems around. Smaller teams that are not in this privileged position, can benefit from differing to standards written by people who have more context on the task at hand. If you are Facebook then certainly roll your own error format, but if you are not then RFC 7807 will point you in the right direction, and implementations make it easy. ## 200 OK and Error Code HTTP 4XX or 5XX codes alert the client, monitoring systems, caching systems, and all sorts of other network components that something bad happened. ![The folks over at CommitStrip.com know what’s up.](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/posts/useful-api-errors-for-rest-graphql-and-grpc/comic.jpg) The folks over at [CommitStrip.com](https://commitstrip.com/?ref=apisyouwonthate.com) know what’s up. If you return an HTTP status code of 200 with an error code, then Chuck Norris will roundhouse your door in, destroy your computer, instantly 35-pass wipe your backups, cancel your Dropbox account, and block you from GitHub. It is the hidden error. HTTP-based monitoring systems do not know about your arbitrary `{ success : false }` conventions and so do not report errors. HTTP network caching will cache your errors because you are recording them as not errors. Do. Not. Use. 200\. For. Errors. Ok?! ## GraphQL GraphQL has an error object format defined, so in theory no choice should need to go into selecting one. It has a message and a location, the location being useful for GraphIQL and other visual query tools to help show which line the error was on. ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/posts/useful-errors/graphql-error.png) Errors seem to be tailored to helping the console show a pretty message, but make it tough for programmers to do anything useful. There is also a path property made available in some error responses: ```javascript "path": [ "name" ], ``` At first this may appear to be similar to the JSON:API pointer (JSON Pointer) approach, but is actually considerably more complex. > If an error can be associated to a particular field in the GraphQL result, it must contain an entry with the key path that details the path of the response field which experienced the error. This allows clients to identify whether a null result is intentional or caused by a runtime error. > > This field should be a list of path segments starting at the root of the response and ending with the field associated with the error. Path segments that represent fields should be strings, and path segments that represent list indices should be 0-indexed integers. If the error happens in an aliased field, the path to the error should use the aliased name, since it represents a path in the response, not in the query. > > For example, if fetching one of the friends’ names fails in the following query: > > ```javascript > { > hero(episode: $episode) { > name > heroFriends: friends { > id > name > } > } > } > > ``` > > The response might look like: > > ```javascript > { > "errors": [ > { > "message": "Name for character with ID 1002 could not be fetched.", > "locations": [ { "line": 6, "column": 7 } ], > "path": [ "hero", "heroFriends", 1, "name" ] > } > ], > "data": { > "hero": { > "name": "R2-D2", > "heroFriends": [ > { > "id": "1000", > "name": "Luke Skywalker" > }, > { > "id": "1002", > "name": null > }, > { > "id": "1003", > "name": "Leia Organa" > } > ] > } > } > } > > ``` > > \-— Lee Byron, [graphql-spec](https://github.com/graphql/graphql-spec?ref=apisyouwonthate.com) As you might have noticed here, GraphQL has an interesting spin on errors. With most HTTP APIs you are either trying to do something and succeed, or you fail, and it is usually rather binary. *\*\*Note: \*\*An exception to that rule might be trying to fetch a collection of* *things, searching, etc. and getting an empty result, but that is not an error,* *that is a fetch returning an empty result.* GraphQL has a different take, and it tries to provide as much data back even when a request contained incorrectness. It usually seems more like GraphQL considers errors to be merely warnings, which is why you can have data and also have errors, and that not be an issue. When trying to work out where people should put their own errors, there are a lot of disparate instructions. Some folks saying things like: > If the viewer should see the error, include the error as a field in the > response payload. For example, if someone uses an expired invitation token and > you want to tell them the token expired, your server shouldn’t throw an error > during resolution. It should return its normal payload that includes the error > field. It can be as simple as a string or as complicated as you desire: > > ```javascript > return { > error: { > id: '123', > type: 'expiredToken', > subType: 'expiredInvitationToken', > message: 'The invitation has expired, please request a new one', > title: 'Expired invitation', > helpText: 'https://yoursite.co/expired-invitation-token', > language: 'en-US' > } > } > > ``` > > — [Matt Krick](https://itnext.io/the-definitive-guide-to-handling-graphql-errors-e0c58b52b5e1?ref=apisyouwonthate.com) This is back to creating custom error formats, despite GraphQL having one bundled... Once again, GraphQL is so vague on a particular topic that it is not very helpful, and the vendors have to step in. Apollo has extension based [tooling for errors](https://www.apollographql.com/docs/apollo-server/features/errors?ref=apisyouwonthate.com) which can help you, but the usual concerns about vendor lockin, $$$ and having extension-riddled APIs apply. ## gRPC gRPC does not care about how you do errors, do what you want. The official documentation for gRPC Core has written down some [pre-defined error codes](https://grpc.io/grpc/core/md%5Fdoc%5Fstatuscodes.html?ref=apisyouwonthate.com), but you can invent your own too. The official documentation pushes readers towards [https://avi.im/grpc-errors/](https://avi.im/grpc-errors/?ref=apisyouwonthate.com), which is a convenient set of SDKs for most of the programming languages gRPC is implemented in. The code helps API developers use the status codes defined in gRPC Core, and add their own text too. *All this and more in *[Build APIs You Won't Hate: Second Edition](https://leanpub.com/build-apis-you-wont-hate-2/?ref=apisyouwonthate.com)*, currently available* *for pre-order with the early chapters available for download.* ### Resolving Overloaded Terms for API Specifications… Descriptions… Contract? URL: https://apisyouwonthate.com/blog/resolving-overloaded-terms-for-api-specifications-descriptions-contracts/ Last updated: 2023-09-19T10:50:36.000Z Recently a bunch of us API nerds got into a big old chat about what to call what you are writing when you write OpenAPI for *your* API. Some people call it the API specification, others think that "specification" is a word reserved only for [the markdown file in GitHub](https://github.com/OAI/OpenAPI-Specification/blob/master/versions/3.0.0.md?ref=apisyouwonthate.com) which stipulates the exact functionality of OpenAPI itself. Others call it a description document, others call it a contract, and I have definitely called it all of them from time to time. Sometimes it depends on the context. For example in the process of contract testing, the specification is definitely a contract. AGGGHH. The excellent [Matthew Reinbold](https://matthewreinbold.com/2019/02/27/APIsandPedantry/?ref=apisyouwonthate.com) took a stab at this problem by writing up his model: - **API Specification** — A technical standard, like OpenAPI 3.0, that defines how to describe an API’s interface in a general, broadly applicable way that is both a machine-parsable and human-readable. - **API Description** — A file articulating a single API’s interface that fulfills the requirements and expectations of an API specification. - **API Interface** — The obvious or described (and, according to Hyrum’s law, not so obvious) means to communicate with a set of functionality within a system. - **API Implementation/Code** — The gobbledygook created by software developers necessary to accomplish a business function to be exposed via an interface. - **API Executable/Deployable** — The version of code that is promoted to a live environment and responding to production requests. - **API Documentation** — The collection of things necessary to convey sufficient meaning so that clients can successfully use the deployed API through the interface. This includes, but is not limited to, the API Description. This certainly works, but lets just talk about specifications for a minute. If you want to get technical about it there are a lot of different specifications involved in an API. Say you have a HTTP/2 API that implements JSON:API, which happens to be written in Go and uses OAuth 2. - [JSON:API has a specification](https://jsonapi.org/?ref=apisyouwonthate.com) - [JSON has a specification](https://tools.ietf.org/html/rfc7159?ref=apisyouwonthate.com) - [HTTP/2 has a specification](https://tools.ietf.org/html/rfc7540?ref=apisyouwonthate.com) - [Golang has a specification](https://golang.org/ref/spec?ref=apisyouwonthate.com) - [OAuth 2 has a specification](https://tools.ietf.org/html/rfc6749?ref=apisyouwonthate.com) All of this can be described by OpenAPI (which has a spec) and is probably written in YAML (which guess what has a spec). Everything has a specification, so maybe that word is blown out, and we can use something else. Maybe API Description is fine. Originally I had concerns about description making it sound like it should come later, but you can absolutely describe what something should be before it exists, then check the implementation against the description to see if it matches up. Sure. At Stoplight, we had a few places where we said "the OpenAPI Specification" to mean OpenAPI itself, then a few other places where we used the word specification to mean your stuff. The former has mostly been switched to simply "OpenAPI" in the open-source docs, but and the latter in OSS docs is mostly "API description" following Matthew’s model. There are a few instances of "spec" left to disambiguate. ## "Resolving" OpenAPI files There are multiple distinct concepts surrounding resolution, and many of us seem to be using a few of them interchangeably. This might seem like pedantry, but there have been a few issues at work, and people using various API related tools, where they see the word "resolve" and expect it to do something other than what it does. ### Definition A > Looking for the value found at the end of a `$ref`, but no changes are made (to file or object) Aliases include: "lookup" Some folks say the resolving part is literally just following the `$ref`, and no lossy actions are performed. ### Definition B > External `$refs` were pulled in from various external sources Aliases include: "bundling" or "external inlining" Replacing `$ref: "http://"` and `$ref: "otherfile.yaml"` with `$ref: "#/components/..."`. This is a potentially lossy action and extra metadata needs to be tracked to figure out the origin file/line of a specific part of the API description. ### Definition C > All `$refs` and replaced with their values a’la copy & paste. Aliases include: "dereferencing", "internal inlining" or "transclusion" No more `$ref’s` exist in the file/object representation. If you have 10 operations referencing the same model 10 times, you now have 10 different models. ### So What Every utility on [OpenAPI.Tools](https://openapi.tools/?ref=apisyouwonthate.com) uses them slightly differently. [Speccy](https://github.com/wework/speccy?ref=apisyouwonthate.com) uses definition B for its optional (-r) resolve CLI switch. [Spectral](https://github.com/stoplightio/spectral?ref=apisyouwonthate.com) uses definition C in its CLI automatically for rules that need it. When folks switch from Speccy to Spectral they are confused to find out that only internal $ref’s work, and external $ref’s do not. It seems like a bug, but really it is just a different definition… The technical solution there is to allow multiple resolvers, or multiple… pre-processors, which allow the file to be mucked about with in different ways, so the source is maintained and the processed one is available for those that need the processed object. At some point we will probably need to a way to approach a consensus in the community. OpenAPI could help out here, maybe we could get some official guidance for tooling vendors. Right now there are a lot of different camps who all think their approach is the one true correct way, but tool vendors are starting to collaborate and it’s getting a bit weird to have a bunch of different words for the same thing, and also have the same word mean multiple things. Words are hard, so guidance from the OpenAPI overlords could improve things a smidge. For now, at Stoplight we are still using the term resolver, but we have different types of resolver, and are branding them as internal, external, etc. Spectral will get [an \--external switch](https://github.com/stoplightio/spectral/issues/134?ref=apisyouwonthate.com) and that’ll support refs that hit the filesystem and HTTP, etc. with support for others. In the future, we might try and re-imagine that a bit, but at least it should lessen some of the confusion. ### Caching is hard, draw me a picture URL: https://apisyouwonthate.com/blog/caching-is-hard-draw-me-a-picture/ Last updated: 2023-03-24T12:25:46.000Z *Another guest post from our friend [Darrel Miller](https://twitter.com/darrel%5Fmiller?ref=apisyouwonthate.com).* This is my attempt to make the HTTPbis caching rules more accessible and hopefully shine a light on how powerful HTTP caching can be. I’ve been working on a Pluralsight course that talks about how to use the Microsoft HttpClient library. One of the areas I cover is how to take advantage of HTTP caching. In the process I have been doing quite a bit of reading of the HTTPbis spec document on [caching](https://tools.ietf.org/html/draft-ietf-httpbis-p6-cache-26?ref=apisyouwonthate.com). It isn’t the easiest of specifications to read as there are many interdependencies between the directives and there are a many different scenarios that are supported. To help me get a grip, I decided I needed to draw some diagrams to help me get a clearer picture of the rules. The rules break down into two distinct steps: 1. Is a cache allowed to store a response that is returned from an Origin Server? 2. Can a response be served from the cache for a particular request? ![HTTP Cache diagram](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/posts/caching-is-hard-draw-me-a-picture/http-cache-diagram.png) ### Cacheable Methods GET and HEAD responses may be cacheable. POST responses may be cacheable, but will only be served to a subsequent GET request. A POST request will never receive a cached response. The response to PUT, DELETE, CONNECT, TRACE and OPTIONS are not cacheable. #### Is No-store present? In this test we much check both the response cache-control header that comes back from the server and the equivalent request header. If either contain the no-store directive then response should not be held onto for any longer than it takes to return it to the client. #### Is shared cache? HTTP Caches are classified into two distinct types, shared and private. A shared cache stores responses that are to be reused by more than one user. Shared caches are the ones you find sitting in front of web servers, or at the edge of corporate networks. Private caches are usually pieces of software that are either built into the client OS or the client application. Probably the most important difference in the behaviour of private caches is that they are allowed to store responses that contain authentication headers. This behaviour which keeps your authentication credentials out of shared caches is probably one of the best arguments for using authorization header instead of some custom header or URI query parameter. The presence of headers like must-revalidate, public and s-max-age override this limitation on shared caches not being able to store responses with an authorization header. I understand why using public and s-max-age might do that, but I’m puzzled as to why must-revalidate does. #### Contains Freshness Information? If a response contains freshness directives like max-age, Expires, or s-max-age, then we know that the server considers the response cacheable. #### Can be cached using heuristics? If no explicit freshness information is provided, then responses with the following status can still be cached using heuristic based caching: 200,203,204,206,300,301,404,405,410,414,501\. The details of the heuristics algorithm are specific to the particular client application. For example Internet Explorer uses a fraction of the difference between the last modified date and the current time as the max-age. This is the suggested algorithm in RFC2616\. If the last-modified header is not present, then it falls back to user defined settings for caching that response. Once a response has been stored, then future requests may reuse that stored response. ![An updated cache diagram](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/posts/caching-is-hard-draw-me-a-picture/http-cache-diagram-2.png) #### Effective URI matches stored response? The request URI is used as part of the primary cache lookup key. The other part is the request method which we will talk about next. The term "Effective URI” is used because on the server side [some processing](https://tools.ietf.org/html/draft-ietf-httpbis-p1-messaging-26?ref=apisyouwonthate.com#section-5.5) is needed to reconstruct the URI that the client used to make the request. #### Can HTTP request method return cached responses? The second part of the primary cache lookup key is the the HTTP method. As we mentioned earlier, only GET and HEAD requests can return cached responses. However, because the GET and HEAD request for the same resource return different representations they are treated as distinct cache entries. I wondered if it might be acceptable to generate the HEAD cached response directly from a stored GET response by simply stripping off the body. However, I’m not sure if that is allowed because you can use HEAD requests to freshen stored GET responses. [Here](https://tools.ietf.org/html/draft-ietf-httpbis-p6-cache-26?ref=apisyouwonthate.com#section-4.3.5) is the relevant part of the spec, maybe someone else can do a better job of decoding it than me! #### Do selecting header fields match? This innocuous little question deserves a blog post all of it’s own so please accept that I am only skimming the surface here. This test is used when the stored response contains a vary header. When the header fields identified in the vary header contain matching values in both the new request and the stored response then the stored response can be used to satisfy the request. #### Does request or response contain no-cache directive? A request that contains a no-cache directive in either the cache-control header or the pragma header will not allow a stored response to be used directly even if it is fresh. Before it can be used, the cache must make a conditional request back to the server to confirm that the stored response is still valid. Once that is confirmed, then the stored response can be returned. So, to re-iterate, just because you sent a request with no-cache, doesn’t mean that you won’t get a response served from a cache. However, you will be guaranteed that it is up to date. The same revalidation process occurs if the stored response contains a no-cache header. Most people are surprised when they find out that no-cache doesn’t mean "don’t cache”. It simply means "must-revalidate even if still fresh”. You might notice that there is no check for the no-store request header in the diagram. The no-store request header is only tested when determining if a response can be cached. If some other user of a shared cache issues a request to a resource that is cacheable and then you issue a request to the same resource with no-store, you could still return a cached response. #### So can we finally serve this response? If the stored response is still fresh. I.e. the expired date has not passed, or the date retrieved plus max-age has not passed, then the response can be served. If the response is stale and the client sends a max-stale directive then it may also be possible to serve the stale response. And finally, if we have just finished re-validating the response, then we can return it. That’s a high level overview of the process. There are lots of details I skipped, but that’s why the full [caching specification](https://tools.ietf.org/html/draft-ietf-httpbis-p6-cache-26?ref=apisyouwonthate.com) is 40 pages! Hopefully, this overview will make it easier when you want to dig into more details in the spec. #### Show me the code! If you are interested to see what this process might look like in code, I have started building a private cache implementation [here](https://github.com/tavis-software/Tavis.PrivateCache?ref=apisyouwonthate.com). Hopefully, I will get comments working again on this blog soon, but in the meanwhile, come find me on twitter with any questions you might have. [@darrel\_miller](https://twitter.com/darrel%5Fmiller?ref=apisyouwonthate.com) *Originally published at* [*bizcoder.com*](http://bizcoder.com/caching-is-hard-draw-me-a-picture?ref=apisyouwonthate.com)*.* ### Surviving Other People's APIs URL: https://apisyouwonthate.com/books/surviving-other-peoples-apis/ Last updated: 2024-05-16T20:16:41.000Z ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2023/03/surviving-other-peoples-apis--1--2.jpg) Designing the world's most beautiful API is only half the story, somebody needs to interact with it! API Developers focus so much on designing and building their APIs, yet often we seem to forget the folks on the other end of the line. You, the frontend and backend developers trying to integrate our data and functionality into your own work, often get left with junky docs, or are just assumed to know how things are going to work. [Preorder on Leanpub](https://leanpub.com/surviving-other-peoples-web-apis?ref=apisyouwonthate.com) --- Frontend engineers work magic. They deal with all sorts of awful browser stuff, and have to deal with all sorts of ginormously complex problems. Most of those problems seem to be browser compatibility and Webpack, but there's a lot of documentation around how most of these things work. One area where frontend engineers are often left entirely in the dark? Interacting with APIs. Following the success of [Build APIs You Won’t Hate](https://apisyouwonthate.com/books/build-apis-you-wont-hate), this book will take a non-academic, easy-to-read approach to some pretty complex topics around HTTP interactions, versioning, client-caching, state management, differences between how you interact with RPC, REST and GraphQL, using JSON Schema for local validation, and all sorts of other awesome stuff that nobody ever bothered to mention. This book is aimed at frontend engineers (web, mobile, whatever) and backend engineers that find themselves talking to other APIs a lot, which is probably most of us these days. ## Preorder **Surviving Other People's APIs* ### Get instant access to the book as we write it [Preorder on Leanpub](https://leanpub.com/surviving-other-peoples-web-apis?ref=apisyouwonthate.com) ### Organizing your AsyncAPI documents URL: https://apisyouwonthate.com/blog/organizing-asyncapi-documents/ Last updated: 2023-03-24T12:26:04.000Z *Written by [Fran Méndez](https://medium.com/@fmvilas?ref=apisyouwonthate.com) and re-posted here with permission.* A recurring question that I get very often is: "how do I organize my AsyncAPI documents?". Also, the related one: "I have two services, a publisher and a consumer, should I define both in the same AsyncAPI document?". Let’s break down some best practices and tips to avoid ending up in a hell of unmanageable documents. > I’m using the term microservices here because it’s the most common type of distributed architecture that you can find nowadays. The best practice for organizing AsyncAPI files in your microservices architecture is to have a file per microservice. This way, you end up with multiple independent files that define your application. ![PubSub diagram](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/posts/organizing-asyncapi-documents/yaml-rules.png) One publisher and one subscriber, both sharing the UserSignUp message. Microservices are meant to do a single thing and to do it well and, very importantly, they must be independently deployable. However, if you have a publisher and multiple consumers, you quickly end up having something like the following: ![One publisher and various subscribers. All of them sharing the UserSignUp message.](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/posts/organizing-asyncapi-documents/pub-sub.png) One publisher and various subscribers. All of them sharing the UserSignUp message. It’s clear there’s a dependency between all of them: the *UserSignedUp* message. If at some point to want to change it, you’ll have to go through all of the files and change it. It’s a tedious task we want to avoid, so we can make use of the *$ref* capability of AsyncAPI to simplify things: ![A diagram of dependencies in the pubsub relationship](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/posts/organizing-asyncapi-documents/pub-sub2.png) The value of $ref should be "common/messages.json#UserSignedUp". Now, if you have to add something to the *UserSignedUp* message, it’s just a matter of changing one file. Depending on your setup, you may have to restart your services to get the new definition. However, as simple and straightforward as it may seem, you must take care not to introduce breaking changes in the message definition. Otherwise, you’ll have inconsistent states while some services got the new message definition and some didn’t yet. Here comes the importance of versioning your messages, but that makes for another blog post alone. Since microservices tend to be small in scope, most probably their AsyncAPI document will not be very extensive too. And I found out this is one of the reasons people tend to re-use the same file for many services. They think the file is very small and because the publisher and the subscriber share the same message, why not putting everything there? It’s tempting at first, but the reason why you should avoid doing this is that you lose context and semantics, and it causes problems: 1. If a single document contains publish and subscribe for the same channel (topic), how do you know which one is defining what your application does? 2. Since your document may contain many channels, how do you know if your application is publishing or subscribing to each channel or just a subset of them? That itself causes more problems, for instance, when you want to generate the documentation for a single service but you have a single file defining your whole architecture and thus a single documentation page describing all the services without any clue which one is doing what. **An AsyncAPI file is meant to define the behavior of a single application**. You can obviously break the rules and use it to define the whole architecture but expect all sorts of problems to appear because you’re using a hammer to saw a piece of wood. The same way you don’t use a single OpenAPI (Swagger) file to define many of your REST APIs, you shouldn’t use a single AsyncAPI document to define many of your message-driven APIs. Say, for instance, you have a WebSockets API and front-end application using it. The paradigm is very similar to the one we’re used to with HTTP APIs, with the subtle difference that the communication is [full duplex](https://en.wikipedia.org/wiki/Duplex%5F%28telecommunications%29?ref=apisyouwonthate.com#Full%5Fduplex), i.e., the client can send and receive messages over the same channel, at any time. This case is not very different from the microservices one. If you think about it, we can look at it as a small distributed architecture, where you only have two services: the client and the server. So the recommended best practice is to follow the same approach and have one document for each of the applications — one for the server and another for the client or front-end. I want to reinforce the point that an AsyncAPI file is meant to define **the behavior of a single application**. Keep this always in mind, and everything will make sense to you. Happy coding! ✌️ *AsyncAPI is an open source project running on donations so please, [*consider donating*](https://opencollective.com/asyncapi?ref=apisyouwonthate.com).* 🙌 ### Optimizing for the Speed of Light URL: https://apisyouwonthate.com/blog/optimizing-for-the-speed-of-light/ Last updated: 2023-03-24T12:26:14.000Z *Written by Darrel Miller, on his blog [Bizcoder.com](http://www.bizcoder.com/optimizing-for-the-speed-of-light?ref=apisyouwonthate.com)*. This article makes a case for trying to acoid batch requests (or compound documents, included resources, etc) in situations that make interactions slower. HTTP/2 is here, and we need tos foip some of the ways we think on our head! – Phil\_ It is true three women can’t make a baby in 3 months, but it also doesn’t take 27 months for them to have 3 babies! You might be wondering what on earth I am talking about but this is something I see API developers getting confused about regularly. There is **oft-repeated guidance that if an API needs to make multiple calls, it is more efficient to bundle them into a single call**. The reasoning is based on one universal truth and one piece of legacy dogma. ### Damn Physics I regularly tell folks that until we find a way of making network calls faster than the speed of light there are **fundamental rules of network based computing that just won’t change**. Network latency between user and server is usually in the 10s of milliseconds and making many round-trips sequentially is going to impact user experience. **Minimizing round-trips is generally good guidance**, except for when it is not. ### Abandoned Runways ![An empty runway](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/posts/optimizing-for-the-speed-of-light/runway.jpg) If I make 1000 HTTP requests simultaneously I only pay the speed of light tax once. So, why is it that we don’t just make every request really small and make many of them at once. The primary reason is based on piece of history. **HTTP/1.1 implementations aren’t very good at sending more than one request simultaneously over a single connection** and generally clients are limited to only opening a certain number of connections with a server. Historically, building servers that could maintain open connections with tens of thousands of clients was challenging. In order to achieve scalable systems it was necessary to limit the state that servers needed to hold onto for each client, and therefore limiting open connections was one way to do this. Over the years, massively increased hardware capacity and clever engineering optimizations have made maintaining large numbers of server connections far less of an issue. More recently the introduction of **HTTP/2 has made it possible to easily tunnel many requests simultaneously over a single connection**. We no longer need to be constrained by connection limits. ### Solutions to unnatural problems In order to work around the connection limit, it has become a best practice in the web world to "bundle” resources together to allow retrieving a set of resources in a single request. Clever tooling has evolved to make the bundling process almost automatic. **Few people question why it is being done**. The same mantra of "minimizing round trips” has become a best practice of the HTTP API world with a slight twist. Instead of the server creating bundles of content for the client to consume, **the client aggregates a set of requests into a single request** and the server splits that request, collects the data and creates a custom aggregated response. The client can then tear apart the composed result into the component parts. Some APIs present this as a kind of "batch request”, others in a form of query language against an aggregated server side data model. ### Knobs and switches This model of a single batch or query request has some advantages to the client developer. For client developers to make simultaneous HTTP calls it is necessary to deal with multiple asynchronous requests that don’t block on IO. This experience is not the easiest to manage. In Java Script you end up with multiple callbacks. In C#/typescript you can no longer use the friendly \`await\` keyword that makes async calls almost as easy synchronous calls. **Ironically, the \`await\` keyword makes it really easy to force HTTP requests to execute sequentially that could easily execute in parallel.** This is what the pit of fail looks like! ![An empty quarry](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/posts/optimizing-for-the-speed-of-light/quarry.jpg) ### Consequences So, what’s the problem? The speed of light doesn’t let multiple smaller requests return any faster than the single large result. The composite payload size is not any bigger than the sum of the smaller results. The server has to process less HTTP requests which will probably create less overhead work. To answer this question we need to point to a failing of my initial analogy. **Human babies tend to take approximately the same time to produce. Individual HTTP requests have no such constraint.** Executing a query that returns a user’s profile information combined with the calculated statistics of how much time they spent writing emails this week, and their online presence indicator will cause varying degrees of work to determine those values. Using a batch or query usually means that we wait for all the results to be calculated before get any answers. Maybe we want to render the results to a user interface. Should we need to wait for the statistics to be calculated before we can render their profile information and presence? ### Do you really want to minimize round trips? There is another industry mantra that is very useful here: "The fastest round-trip is no round trip”. ![No U turns here](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/posts/optimizing-for-the-speed-of-light/no-u-turn.png) This sounds a whole lot like what lead us to batching, but in fact this is referring to local caching. **If I have previously retrieved the information and it is still good, there is no need to retrieve it again.** With a batch request it is the responsibility of a client application to choose which requests to batch together to ensure data stays fresh. When the client wants to update the user presence indicator, it probably doesn’t need to update the user profile information. At least not nearly as often. If the server calculates the email statistics only once a day, refreshing that data more often is a waste of time. It is a non-trivial task for a client to choose when to refresh those pieces of data and aggregate the requests into batches to minimize roundtrips. However, relying on standard out-of-box caching intermediaries allows the server to communicate when the data will become stale. The client can continue making independent requests to the distinct resources and if a fresh copy exists in the local private cache, then no round-trip will happen. **The client programming model becomes massively simpler.** It is possible for generic client infrastructure code to maintain a cache of individual responses and transparently return those responses if they are not stale. Web Browsers do this all the time. Web sites are heavily optimized to take advantage of caching. **Developers of HTTPs API tend to towards claiming that their API data is not cacheable.** That may be the case if you have bundled some highly volatile data along with rarely changing data in order to reduce round trips. Data volatility is a critical factor to consider when designing API resources. ### You have options, until you don’t There is no doubt that high latency HTTP requests can make an application unusable. Just try your favorite app from airplane WIFI sometime. However, **chunkier round trips are not always the most effective way to reduce latency costs**. Local HTTP caching is a low cost, low impact option that can drastically improve performance in some cases. Be careful not to close that door because you are busy bundling your requests. ### Build APIs You Won't Hate URL: https://apisyouwonthate.com/books/build-apis-you-wont-hate/ Last updated: 2023-03-25T13:35:22.000Z ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/2023/03/build-apis-you-wont-hate-1.jpg) Everyone and their dog wants an API, so you should probably learn how to build them. Tasked with building an API for your company but don't have a clue where to start? Taken over an existing API and hate it? Built your own API and still hate it? This book is for you. This book has been up on the LeanPub top 10 for most of its lifetime, and was the #5 Best Selling Book of 2015, so it can't be too bad. [Buy eBook with Shopify](https://apisyouwonthate.myshopify.com/cart/46864794762:1?channel=buy%5Fbutton&ref=apisyouwonthate.com) Also available on these sites *(but they take a bigger cut of sales):* [Amazon US](https://www.amazon.com/Build-APIs-You-Wont-Hate/dp/0692232699/?ref=apisyouwonthate.com) [Amazon UK](https://www.amazon.co.uk/dp/0692232699/?ref=apisyouwonthate.com) [Amazon FR](https://www.amazon.fr/Build-APIs-You-Wont-Hate/dp/0692232699/?ref=apisyouwonthate.com) [Amazon DE](https://www.amazon.de/Build-APIs-You-Wont-Hate/dp/0692232699/?ref=apisyouwonthate.com) --- API development is increasingly common for server-side developers thanks to the rise of front-end JavaScript frameworks, iPhone applications, and API-centric architectures. It might seem like grabbing stuff from a data source and shoving it out as JSON would be easy, but surviving changes in business logic, database schema updates, new features, or deprecated endpoints can be a nightmare. After finding many of the existing resources for API development to be lacking, Phil learned a lot of things the hard way through years of trial and error. This book aims to condense that experience, taking examples and explanations further than the trivial apples and pears nonsense tutorials often provide. Phil worked primarily as an API developer for the last three years. One horror was managing an API built in FuelPHP by a freelancer at the million dollar startup he joined. It was utilizing a then deprecated ORM which had been hacked to death by the previous developer, so took the time to delete that mess and build the next version in Laravel, leveraging it's simple routing, database migrations, schema, seeding, etc. When the following major version of the API was built no rewrite was required, and both managed to live side-by-side on the same "API" servers. By passing on some best practices and general good advice you can hit the ground running with API development, combined with some horror stories and how they were overcome/avoided/averted. This book will discuss the theory of designing and building APIs in any language or framework, with this theory applied in PHP-based examples. Some of the more advanced topics covered here are endpoint testing, embedding data objects in a consistent and scalable manner, paginating responses (including embedded objects) and hypermedia "HATEOAS" controls. ## Get *Build APIs You Won't Hate* ### You'll be building better APIs in no time [Buy the book!](https://apisyouwonthate.myshopify.com/cart/46864794762:1?channel=buy%5Fbutton&ref=apisyouwonthate.com) ### Taking a Timeout from Poor Performance URL: https://apisyouwonthate.com/blog/taking-a-timeout-from-poor-performance/ Last updated: 2023-03-24T12:26:28.000Z In a system-oriented architecture, it is crucial to communicate with other systems. In an ideal world each service knows enough information to satisfy its clients, but often there are unfortunate requirements for data to be fetched on the fly. Broker patterns, proxies, etc., or even just a remote procedure being triggered synchronously, like confirming an email has been sent successfully. **All of these things take time**. Frontend applications (desktop, web, iOS, Android, etc.) talk to services, and services talk to other services. This chain of calls can stack up, as service A calls service B, unaware that system is calling service C and D… So long as A, B, C and D are functioning normally, the frontend application can hope to get a response from service A within a "reasonable time”, but if B, C or D are having a bad time, it can cause a domino effect that takes out a large chunk of your architecture, and the ripple effects result in a slow experience for the end users. Slow applications can cost you a lot of money. A [Kissmetrics survey](https://blog.kissmetrics.com/loading-time/?wide=1&ref=apisyouwonthate.com) suggests that **every** **1s slower a page loads, 7% fewer conversions will occur**. This article explains how you can make your applications remain performant when upstream dependencies are not, using timeouts and retries. ### Other People’s Problems You own service A, and are making calls to service B. What happens when service B has a bad day, and instead of responding within the usual \~350ms, it starts to take 10 seconds? Do you want to wait 10 seconds for that response? ![Response time chart showing a spike](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/posts/taking-a-timeout-from-poor-performance/chart1.png) What about if service B is fine, but C is taking 20s and D is taking 25s? Are you happy to wait 45 seconds for the response from B? What about **two minutes**?! 😱 ![2 minute response times](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/posts/taking-a-timeout-from-poor-performance/2-minutes.png) When a server is under load, it can do some pretty wild stuff, and not all servers know to give up. Even fewer applications know when to give up, and those that do will take a while to do it. For example, if service B is on Heroku, we can be confident the request is not going to last for more than 30 seconds. Heroku’s router has a policy: applications get 30 seconds to send the first byte, and if that doesn’t happen then the request gets dropped. Quite often in monitoring systems like NewRelic or CA APM, you will see things like this: ![Showing an average response time of 2.9s](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/posts/taking-a-timeout-from-poor-performance/transactions.png) This controller has an average response time of 2.9s, but the slow ones are floating right around that 30s mark. The Heroku Chop saves the caller from being stuck there indefinitely, but this behavior is not widespread. Other web servers with different policies will hang forever. For this reason, **never presume any service is going to respond as fast as it** **usually does**. Even if that team’s developers are super confident. Even if it autoscales. Even if it’s built in Scala. If you don’t set timeouts, other people’s problems become your problems. So, how can we prevent this from happening? ### Set Timeouts in the HTTP Client HTTP clients are usually generic, and out of the box will wait *indefinitely* for a response. Luckily any HTTP client worth its salt will allow you to configure timeouts. For Ruby users, the HTTP client [Faraday](https://github.com/lostisland/faraday/?ref=apisyouwonthate.com) might look like this: ```ruby conn = Faraday.new('http://example.com'); conn.get do |req| req.url '/search' req.options.timeout = 5 # open/read timeout in seconds req.options.open_timeout = 2 # connection open timeout in seconds end ``` For PHP, [Guzzle](http://docs.guzzlephp.org/en/stable/request-options.html?highlight=timeout&ref=apisyouwonthate.com#timeout) does this: ```php $client->request('GET', '/delay/5', ['timeout' => 5]); ``` There are two types of timeout that a lot of HTTP clients use: 1. Open (Connection) Timeout 2. Read Timeout An **open timeout** asks: how long do you want to wait around to see if this server is actually accepting requests. That can mean many things but often means a server is too busy to take a request (there are no available workers processing the traffic). It also depends in part on expected network latency. If you are making an HTTP call to another service in the same data center, the latency is going to be a few milliseconds, but going to another continent takes time. The **read timeout** is how long you want to spend reading data from the server once the connection is open. It’s common for this to be set higher, as waiting for a server to generate an answer (run queries, fetch data, serialize it, etc.) should take longer than opening the connection. When you see the term "timeout” on its own (not an open timeout or a read timeout) that usually means the total timeout. [Faraday](https://github.com/lostisland/faraday/?ref=apisyouwonthate.com) takes `timeout = 5` and `open_timeout = 2` to mean "I demand the server marks the connection as open within 2 seconds, then regardless of how long that took, it only has 5 seconds to finish responding.” ### Some must die, so that others may live Any time spent waiting for a request that may never come is time that could be spent doing something useful. When the HTTP call is coming from a background worker, that’s a worker blocked from processing other jobs. Depending on how you have your background workers configured, the same threads might be shared for multiple jobs. If Job X is stuck for 30s waiting for this server that’s failing, Job Y and Job Z will not be processed or will be processed incredibly slowly. That same principle applies when the HTTP call is coming from a web thread. That’s a web thread that could have been handling other requests! For example, `POST /payment_attempts` is making an HTTP call in the thread which is usually super quick, but unfortunately, some service it talks to is now blocking it for 30s. Other endpoints usually respond in 100ms, and they will continue to respond so long as there are threads available in the various workers… but if the performance issues for the dependency continue, every time a user hits `POST /payment_attempts`, another thread becomes unavailable for that 30s. Let’s do a bit of math. For each thread that gets stuck, given that thread is stuck for 30s, and most requests go through in 100ms, **that’s 3000 potential** **requests not being handled**. 3000 requests not being handled because of a single endpoint. There will continue to be fewer and fewer available workers, and given enough traffic to that payment endpoint, there might be zero available workers left to work on any the traffic to any other endpoints. **Setting that timeout to 10s would result in the processing of 2000 more** **successful requests.** **As a general advice**: please do try and avoid making requests from the web thread, put them in background jobs whenever possible. Sometimes it is unavoidable, but please go to painstaking efforts to try and avoid doing it. Making timeouts happen early is much more important than getting a fast failure. The most important benefit of failing fast is to give other workers the chance to work. If the server is a third party company, you might have a service-level agreement stating: "Our API will always respond in 150ms”. Great, set it to 150ms (and retry on failure if the thing is important.) If the service is in-house, then try to get access to NewRelic, CA APM or whatever monitoring tool is being used. Looking at the response times, you can get an idea of what should be acceptable. Be careful though, **do not look only** **at the average**. ![Web transactions time chart](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/posts/taking-a-timeout-from-poor-performance/web-transactions.png) Looking at this graph may lead you to think 300ms is an appropriate timeout. Seems fair right? The biggest spike there is 250ms and so round it up a bit and let’s go for 300ms? Nope! These are averages, and averages are going to be far far lower than the slowest transactions. Click the drop-down and find "Web transaction percentiles.” ![Chart showing percentiles for transactions](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/posts/taking-a-timeout-from-poor-performance/percentiles.png) That is a more honest representation. Most of the responses are 30–50ms, and the average is usually sub 100ms. That said, under **high load** this service starts to stutter a bit, and these peaks can **lead to responses coming in around** **850ms**! Clicking around to show the slowest traces will show a handful of requests over the last few weeks coming in at 2s, 3.4s, and another at 5s! Those are ridiculous, and looking at the error rate we can see that those requests didn’t even succeed. Whatever happens, setting the timeout low enough to cut those off is something we want to do, so far I’m thinking about 1s. If the transactions are failing anyway, there is no point waiting. Next: if the call is being made from a background worker, that 99 percentile of 850ms may well be acceptable. Background workers are usually in less of a rush, so go with 1s and off you go. Keep an eye on things and trim that down if your jobs continue to back up, but that’s probably good enough. ### Backup plan: retry If it’s a web process… well, 2s+ is certainly no good, especially seeing as it might fail anyway. Waiting around for this unstable transaction to complete is as much of a good plan as skydiving with just the one chute. Let’s create a backup plan using retries. So we have this special web application that absolutely has to have this web request to Service B in the thread. We know this endpoint generally responds in 35–100ms and on a bad day it can take anywhere from 300–850\. We do not want to wait around for anything over 1s as its unlikely to even respond, but we don’t want this endpoint to take more than 1s… Here’s a plan: set the timeout to 400ms, add a retry after 50ms, then if the first attempt is taking a while *boom*, it’ll give up and try again! ```ruby conn = Faraday.new('http://example.com'); conn.post('/payment_attempts', { ... }) do |req| conn.options.timeout = 0.4 conn.request :retry, max: 1, interval: 0.05 end ``` There is potential for trouble here, as the second and first attempts might end up in a race condition. The interval there will hopefully give the database long enough to notice the first response was successful, meaning the 2nd request will fail and say "already paid” or something intelligent, which can be inspected and potentially treated as a success by the client. Anyway, (400 \* 2) + 50 = 950, with another 50ms for whatever other random gumf is happening in the application, should mean that we come in at under 1 second! This is a good place to be in. You have 2x the chance of success, and you’re setting tight controls to avoid service B messing your own application up. **An important note for Ruby users**: you are [already using retries](https://engineering.wework.com/ruby-users-be-wary-of-net-http-f284747288b2?ref=apisyouwonthate.com) on idempotent requests, and you probably had no idea. It’s wild that NetHTTP does this by default. ### Next Step: Circuit Breakers Timeouts are a great way to avoid unexpected hangs from slowing a service down too much, and retries are a great solution to having another when that unexpected problem happens. These two concepts are both reactive, and as such can be improved with the addition of a third proactive concept: **circuit** **breakers**. Circuit breakers are just a few lines of code, maybe using something like Redis to maintain counters of failures and their timestamps. With each failure to a service (or a particular endpoint on that service), the client increments a failure counter and compares it to a certain threshold. Maybe that threshold is 10 failures in 1 minute, or for higher volume systems maybe 5 failures in a second. So in our example, Service A might notice that service B is down after the 10th error in 1 second, and at that point it opens the circuit breaker, meaning it completely stops making calls to that system. This will decrease the load on downstream services (B, C, and D), giving them a chance to recover. This also avoids the "running out of threads” issue we discussed previously. Even with service A giving up after 1s, that’s still 1s that thread could have spent handling other requests. What to do when a circuit breaker is open? It depends on the feature the circuit breaker is wrapping. - Immediately respond with an error, letting the user know the required system is down, and to try again later - Have a secondary system kick in that handles things in a different way - Divert traffic to a cluster of servers elsewhere - Record information about the attempt and have customer services reach out That’s only a quick intro to circuit breakers, so head over to see [Martin Fowler explain circuit breakers in depth](https://martinfowler.com/bliki/CircuitBreaker.html?ref=apisyouwonthate.com) if you want more information on the topic. [Envoy](https://www.envoyproxy.io/?ref=apisyouwonthate.com) is also a great example of a tool that can handle a lot of this for you at a network level, instead of asking you to code it up yourself and the application level. Hopefully, these tips will help keep your applications ticking along smoothly and avoid a domino effect taking out a giant chunk of your architecture. ### Health Checks for REST/gRPC APIs: Kubernetes and Beyond URL: https://apisyouwonthate.com/blog/health-checks-with-kubernetes/ Last updated: 2023-03-24T12:26:37.000Z Is your API working right now? It can be a tough question to answer. A lot of monitoring tools are out there, like [Prometheus](https://prometheus.io/?ref=apisyouwonthate.com) which can track all sorts of metrics, but that assumes metrics are being sent to it which let you know there’s a problem. Maybe [NewRelic](https://newrelic.com/?ref=apisyouwonthate.com) can track response times, and [Rollbar](https://rollbar.com/?ref=apisyouwonthate.com) can track exceptions, but there could be a lot of things relating to "application health” which do not fall into those categories. Application health is an important difference to test suite success. Test suites confirm that the code you’ve pointed some tests at work with the environment variables in your test suite, but that doesn’t mean that same code will work with different environment variables. Some people confuse application health with "server health”, but that is rather different. Server health is knowing things like the amount of disk space or memory that your server has left, and reporting that the database is running perfectly. This is easily solved with tools like [collectd](https://collectd.org/?ref=apisyouwonthate.com), which can shuffle these metrics back to collectors like Prometheus. Unfortunately that doesn’t mean your application knows how to talk to that perfectly functioning database. To get a current snapshot of the health of a specific application (even if its only the specific instance you happen to get routed to when you make that HTTP request), there are a lot of tools which help you expose some sort of "health check endpoint”. Every programming language seems to have a prominent tool for doing this, at a totally different endpoint, with a totally different payload. - Ruby: [OkComputer](https://github.com/sportngin/okcomputer?ref=apisyouwonthate.com) runs on `/okcomputer` with a mega-response - Python: [health-check](https://pypi.org/project/health-check/?ref=apisyouwonthate.com) has a bunch of various `/api/health/database` endpoints - Java/Spring Boot: [Actuator](https://www.baeldung.com/spring-boot-actuators?ref=apisyouwonthate.com) adds `/health`, `/metrics`, and `/info` Whatever language/framework you use, one of these tools is probably a good idea. As somebody working at a company with a whole bunch of languages being used, I still somehow have delusions of having consistency across those various systems. Luckily there is a [draft RFC HTTP response format for health checks](https://inadarei.github.io/rfc-healthcheck/?ref=apisyouwonthate.com), which anyone can conform to right now. Sure draft RFCs are not final RFCs, but if people do not implement the drafts, they never become final RFCs. Check out [Sunset](https://apisyouwonthate.com/blog/surviving-deprecations-to-resources-and-properties-on-other-apis) for another important draft RFC. Each of those tools need to start supporting "formats”, and there should be a "legacy” format (whatever they currently have), and a "RFC HealthCheck” format which supports this draft RFC. Using these tools has allowed departments at WeWork to build internal "Status Dashboards” that let support staff have great insight into the status of various applications. Especially if the health checks go beyond the standard "database” and "cache” checks, and start reporting circuit breaker statuses for other services. ## Kubernetes Cares about Application Health Above we talked about making a health endpoint that we could hit up for a status page or whatever, but how about if we knew that health status *before* we deployed? If you happen to be using Kubernetes, you can use HTTP-based [readiness probes](https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/?ref=apisyouwonthate.com), which make sure that application doesn’t make it into rotation in the load balancer until it is ready. After it becomes ready, if at any point in the future it seems to be suffering, Kubernetes will remove it from the load balancer automatically! Kubernetes pods have a whole [lifecycle](https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/?ref=apisyouwonthate.com), but basically pods are created, and remain as "pending” until they are "ready”. Once a pod the pod is considered "ready” it will be placed in the load balancer for that service, and it will start accepting traffic. When a new pod is placed into the load balancer, a corresponding pod for that service from a previous release will be removed. If a service has multiple pods it will do this one at a time, and as each new pod becomes ready a corresponding old one will be removed until the new release is complete. Exactly how Kubernetes determines a pod to be ready depends on how the readiness probe was configured the [your Helm chart](https://docs.google.com/document/d/1CELGSyDLOQ0D9TFgbVA19eKKzlUbU%5Fynowd40ilomKQ/edit?ref=apisyouwonthate.com), specifically in `-chart/templates/-deployment.yaml` ```yaml spec: containers: - name: yourapp-web ports: - containerPort: {{ .Values.service.targetPort }} name: {{ lower .Values.service.targetProtocol }} protocol: TCP readinessProbe: httpGet: scheme: {{ upper .Values.service.targetProtocol }} path: /health port: {{ .Values.service.port }} initialDelaySeconds: 5 timeoutSeconds: 2 ``` This is telling Kubernetes to wait 5 seconds, then try and send a HTTP GET request to `/health`. If that endpoint returns a HTTP response code is between 200 and 399 then the pod is considered ready. If it returns any other status, then it will stay as "pending”, and another probe will be send later on. Another type of probe is "liveness”. Once a pod has been defined as ready, it is then live, until a liveness check decides it is not live. This could mean the server crashed, the pod ran out of memory, or any one of many other things that can cause code to fail. A liveness probe works just the same as a readiness probe, and accepts the same configuration. ### Different Types of Probe There are three different types of probe that can be used by Kubernetes: #### tcpSocket A TCP probe just checks that a process is listening on that port and accepting TCP connections, it does not confirm that whatever is listening (and its related dependencies) are fully booted. If a pod is considered ready prematurely it will replace a fully functioning pod from the previous release with a pod that is not yet ready for traffic. For this reason, a TCP probe should only be used for processes that do not start listening until initialization is complete. #### httpGet A HTTP probe is preferable for any HTTP-based applications, like a REST API, web UI, etc. If the application has a fully functioning web server, and the health endpoint is written to check the important dependencies are available to it, then we should have strong confidence the pod is ready. The HTTP response status code should be between 200 and 399, and bonus points for HTTP applications that return a body that conforms to [RFC Draft: Health Check](https://inadarei.github.io/rfc-healthcheck/?ref=apisyouwonthate.com). #### exec Exec is a fantastic "other” option, which does not rely on TCP or HTTP at all. For example, Redis could be implemented as a TCP check, but it is so much better handled with an `exec` probe: ```yaml livenessProbe: exec: command: ["redis-cli", "ping"] initialDelaySeconds: 15 periodSeconds: 5 readinessProbe: exec: command: ["redis-cli", "ping"] initialDelaySeconds: 15 periodSeconds: 5 ``` If the `redis-cli ping` command fails, it will not be ready, or stop it from being live, and that has a lot more intelligence built in than a simple TCP check. This is also useful for gRPC APIs. Seeing as they are not pure HTTP, they need special handling. For a long time there were three fairly awful choices: ![gRPC Handling choices](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/posts/health-checks-with-kubernetes/grpc-diagram.png) 1. httpGet means you have to set up a HTTP/1.1 bridge for an otherwise HTTP/2-only app 2. tcpSocket means you might have traffic directed to a pod which is not yet truly ready, meaning you blue/green deployments are looking a bit brown 3. You built some special snowflake exec based checking, which involves special documentation and monitoring Since gRPC v1.15 there is a [standardized approach to exec health checks!](https://kubernetes.io/blog/2018/10/01/health-checking-grpc-servers-on-kubernetes/?ref=apisyouwonthate.com) This serves as a warning that using non-standard API implementations means you need non-standard solutions. gRPC only got a standard for doing this recently, and implementing it needs awkward treatment in Kubernetes. If you use GraphQL there still isn’t really a plan. Googling "graphql health checks” somebody is just tweeting about maybe automating a simple query. Standards are great folks! When I write about [the differences between REST and Other Implementations™](https://apisyouwonthate.com/blog/picking-api-paradigm) and say things like "standard HTTP things do not work for GraphQL/gRPC”, this is the sort of thing being discussed. Over time implementation specific solutions are created and there are various workarounds floating about, but ignoring the conventions of HTTP can lead to complications. If you understand the pros and cons, and are willing to get involved with all that, then fair enough. But don’t forget about this non-standardness when deciding between various paradigms or implementations! ### JSON API, OpenAPI and JSON Schema Working in Harmony URL: https://apisyouwonthate.com/blog/json-api-openapi-and-json-schema/ Last updated: 2024-11-12T13:48:40.000Z A regular question on Twitter, at WeWork, and in the [APIs You Won’t Hate Slack community,](https://slack.apisyouwonthate.com/?ref=apisyouwonthate.com) is: "What standard should my API follow? Should it be JSON API or something else?” At which point a bunch of people usually start discussing things like OpenAPI and/or JSON Schema. In this article, I’d like to talk about how they can all work together in harmony, to do different things. ## JSON API [JSON API](https://jsonapi.org/?ref=apisyouwonthate.com) is a specification written by a group of folks, with the goal of being an anti-[bikeshedding](https://bikeshed.org/?ref=apisyouwonthate.com) tool for writing JSON APIs. The name often causes a bit of confusion, but JSON API is one of many data formats that is often applied to REST (or RESTish) APIs, as an alternative to [Siren, HAL, Uber, etc](https://sookocheff.com/post/api/on-choosing-a-hypermedia-format/?ref=apisyouwonthate.com). The goal of JSON API is to standardize some of the specifics of API design that the REST paradigm leaves to the implementer. REST has no opinions on how you implement resources vs collections, or where meta data should go, how to include related resources, how pagination should work, or anything else, so JSON API tries to fill in a lot of those gaps for you. Many people try to figure those things out their selves, and there are a lot of things that can go wrong. [*Build APIs You Won’t Hate*](https://apisyouwonthate.com/books/build-apis-you-wont-hate.html) would have been a lot slimmer if everyone had just used JSON API back then. Firstly, JSON API explains what shape the body of the HTTP request and response should take. Specifically, where primary data goes, where meta data goes, where should links to other resources be placed, and how exactly should related data be included. ```json { "links": { "self": "http://example.com/articles", "next": "http://example.com/articles?page[offset]=2", "last": "http://example.com/articles?page[offset]=10" }, "data": [ { "type": "articles", "id": "1", "attributes": { "title": "JSON API paints my bikeshed!", "stuff": "and nonsense" }, "relationships": { "author": { "links": { "self": "http://example.com/articles/1/relationships/author", "related": "http://example.com/articles/1/author" } }, "comments": { "links": { "self": "http://example.com/articles/1/relationships/comments", "related": "http://example.com/articles/1/comments" } } }, "links": { "self": "http://example.com/articles/1" } } ] } ``` This structure initially seems like a lot of noise to somebody who is expecting an API to do one thing: transfer a few fields from the server to their client, but when an API is more of a [*state machine over HTTP*](https://apisyouwonthate.com/blog/representing-state-in-rest-and-graphql), a lot of this starts to make sense. Secondly, beyond just the shape of the request/response body, JSON API helps with a few other things, like [Sparse Fieldsets](https://jsonapi.org//format/?ref=apisyouwonthate.com#fetching-sparse-fieldsets). You can let clients pass `/articles?fields[articles]=title,body` to get just the title and body fields, a feature much loved by GraphQL advocates. They’re a controversial topic in the REST world because they can slim down responses, but also [screw up cache ratios](https://philsturgeon.uk/api/2017/08/13/a-happy-compromise-between-customization-and-cacheability/?ref=apisyouwonthate.com). Maybe use them maybe don’t, but JSON API is there to give you the option for this. It helps out with a lot of other things, like `?includes=` for [compound documents](https://jsonapi.org//format/?ref=apisyouwonthate.com#document-compound-documents), it takes a stab at [pagination](https://jsonapi.org//format/?ref=apisyouwonthate.com#fetching-pagination), makes [a vague hand-wave at filtering](https://jsonapi.org//format/?ref=apisyouwonthate.com#fetching-filtering), and covers things like [error objects](https://jsonapi.org//examples/?ref=apisyouwonthate.com#error-objects-basics) for those not using [RFC 7807](https://tools.ietf.org/html/rfc7807?ref=apisyouwonthate.com). All of this is basically very structural, but none of it tells you anything about *what* the data is, or anything about the data. There is no "schema” functionality (or types, as some folks call them). For that, you need to look at something like OpenAPI, or JSON Schema. ## OpenAPI and JSON Schema [OpenAPI](http://openapis.org/?ref=apisyouwonthate.com) and [JSON Schema](https://json-schema.org/?ref=apisyouwonthate.com) are two different specifications that have inspired each other a lot, but are subtly different. OpenAPI aims to describe both the service model (the API in general, endpoints, request metadata like headers, authentication strategies, response metadata, etc.), and it also covers the HTTP request/response body using a bunch of keywords based on JSON Schema, that have [diverged over time](https://apisyouwonthate.com/blog/openapi-json-schema-divergence). The [divergence is slowly being solved](https://apisyouwonthate.com/blog/openapi-v3-1-and-json-schema). JSON Schema aims to describe an instance of JSON data, like the ones found in a HTTP request or response, but is in no way limited to a HTTP API. In describing the data, you can say which fields are required, mention common formats like email, UUID, etc., add more complex validation rules to those fields like maximum length, or regex patterns, and all sorts of other functionality. The data could be in any shape, but whatever data is there can be described with one of these two tools. At WeWork, we use JSON Schema to describe the data models, OpenAPI to describe everything else, then the message being described is usually JSON API. OpenAPI has a lot of design-time and build-time [tooling](https://openapi.tools/?ref=apisyouwonthate.com), so it’s popularly used for mocking services and generating SDKs. It is not commonly used for run-time functionality. For that, most folks use JSON Schema, which can do many amazing things like [client-side validation](https://apisyouwonthate.com/blog/json-schema-client-side-validation). Seeing as JSON Schema does not touch the service model, it’s quite tough to build things like SDK generators as they do not know anything about the endpoints. There could be a future where JSON HyperSchema can take over this role from OpenAPI, as a fully RESTful API with JSON HyperSchema would be able to build a HATEOAS compatible SDK, which would focus on navigating from root, to resource, to resource, instead of worrying about memorizing endpoints in the first place. JSON API and JSON HyperSchema can actually complement each other quite well. Aaron Hedges recently wrote about [how JSON HyperSchema can be used to describe JSON API payloads](https://apisyouwonthate.com/blog/common-hypermedia-patterns-with-json-hyper-schema), taking the plain old "here is a HTTP link do with it what you will” approach to HATEOAS, and peppering it with all sorts of useful information. Some folks refer to this as upgrading an APIs "Hypermedia Maturity Model”, taking JSON API from HMM 1 to HMM 2\. More on the [Hypermedia Maturity Model here](https://8thlight.com/blog/jason-desrosiers/2018/05/30/the-hypermedia-maturity-model.html?ref=apisyouwonthate.com). ## Summary The point here is that these three things all have rather different jobs, and can be used together. You don’t want multiple teams cranking out handfuls of REST APIs with completely different payload shapes and completely different approaches to things, and whatever shape the API data is in, you need to know more about what that data actually is. Think of one as a "Data Format” and another as a "Data Contract” and things make a bunch more sense. I’ve used JSON API extensively for years, flipping from being a big fan to an outspoken critic, and back again multiple times. JSON API is a great way to solve a lot of problems in a HTTP/1 world, but [*HTTP/2 should handle a lot of the things that JSON API focuses on*](https://apisyouwonthate.com/blog/lets-stop-building-apis-around-a-network-hack). That said, a lot of tooling (especially in the Ruby world) is still completely ignoring HTTP/2\. I recommend JSON API as an alternative to anarchy, but it is full of potential foot-guns, like any powerful concept used carelessly. Please read [*Making the Most of JSON API*](https://apisyouwonthate.com/blog/making-the-most-of-json-api) if you are considering using it, and do not take my mentioning of it here as tacit approval for every idea in the specification. *The cover photo is an awesome [Zelda Triforce Lamp](https://www.etsy.com/listing/121737141/zelda-triforce-lamp?ref=apisyouwonthate.com) which I would absolutely buy if I lived somewhere.* ### WeWork’s API Specification Workflow URL: https://apisyouwonthate.com/blog/weworks-api-specification-workflow/ Last updated: 2023-03-24T12:26:58.000Z **Update: I spent a lot of time duct-taping and glueing crap together to get a** **coherent workflow together because tooling at the time was rather lacking for** **API Design-First. Most of this experience has gone into making [Stoplight Studio](https://stoplight.io/studio/?ref=apisyouwonthate.com) and [Stoplight Platform](https://stoplight.io/welcome/create?ref=apisyouwonthate.com) awesome, covering all the [docs](https://stoplight.io/docs?ref=apisyouwonthate.com), [mocks](https://meta.stoplight.io/docs/platform/3.-design/d.setting-up-a-mock-server.md?ref=apisyouwonthate.com), [linting](https://stoplight.io/open-source/spectral?ref=apisyouwonthate.com),** **[style guides](https://apisyouwonthate.com/blog/automated-style-guides-for-rest-graphql-and-grpc), [visual editing](https://stoplight.io/studio/?ref=apisyouwonthate.com), and [contract validation](https://meta.stoplight.io/docs/prism/docs/guides/03-validation-proxy.md?ref=apisyouwonthate.com) you could ever need.** A year ago we started trying to figure out the best way to not just document HTTP APIs, but to leverage API specifications to avoid duplicating efforts on loads of similar-but-different tasks; maintaining Postman Collections, creating mocks, contract testing, payload validation, etc. To us, it felt like API developers were wasting a lot of time writing up the same logic over and over again. Listing endpoints, defining what fields should have what data, figuring out validation logic, writing up examples, doing this all over and over again in loads of different formats, then — the few folks that have enough time and interest — would write loads of tests to make sure all these different formats were saying the same thing. The initial goals were: - One source of truth where developers need to update stuff - API Design / Prototyping — Design first, code later when you’ve agreed on contracts - Beautiful documentation for humans - Specifications and Documentation should be kept in sync - Validate payloads before you send them on the client side - RSpec/PHPUnit/etc. assertions providing contract testing API responses in existing test suites - SDK Generators with customisable templates The first low hanging fruit was to get some [human readable documentation](https://apisyouwonthate.com/blog/turning-contracts-into-beautiful-documentation). We had *nothing* for 99% of our APIs, and engineers were being forced to [guess at contracts](https://apisyouwonthate.com/blog/commit-to-api-contracts). We had two of our APIs documented with API Blueprint, but nobody was particularly interested in writing documentation. We decided we’d need a few more carrots to dangle in order to get folks writing specs, and to do that we’d need a API specification language that could do more. We evaluated [RAML](https://raml.org/?ref=apisyouwonthate.com) and [OpenAPI](https://www.openapis.org/?ref=apisyouwonthate.com) as alternatives, and in the end [OpenAPI was chosen](https://phil.tech/api/2017/07/20/my-vision-for-a-perfect-world-in-api-specification/?ref=apisyouwonthate.com). Switching OpenAPI v2.0 (hereby referred to as OASv2) to power our internal documentation portal got us from 2 documented APIs to 6 documented APIs, but a few people were hitting troubles due to hitting limitations in the language. Luckily [OASv3 solved the problems we were facing](https://phil.tech/api/2018/03/01/api-specification-workflow-matures/?ref=apisyouwonthate.com). Tools like [ReDoc](https://github.com/Rebilly/ReDoc?ref=apisyouwonthate.com) supported v2.0 and v3.0 (using swagger2openapi internally), so we could easily produce documentation written in either version. There was still some confusion about how [JSON Schema](https://json-schema.org/?ref=apisyouwonthate.com) and OpenAPI fit together. JSON Schema is incredibly powerful, and is the basis for how OpenAPI handles the schema portion of its specifications, but [the two have some subtle discrepancies.](https://apisyouwonthate.com/blog/openapi-json-schema-divergence) We knew we wanted to leverage JSON Schema, but we also had to make it fit with OpenAPI, so we built tooling to [convert from JSON Schema-proper, to OpenAPI-flavoured JSON Schema](https://apisyouwonthate.com/blog/solving-openapi-and-json-schema-divergence). With documentation nailed, and folks starting to ask questions about more advanced functionality, it was time to think about how to offer the full suite of functionality API specifications can provide. Here is the workflow we created. ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/posts/wework-workflow/workfow.jpeg) *The legend there shows which parts we have implemented at WeWork, and which* *parts are still in progress.* Most of our APIs already exist, and we’re trying to get them caught up with the benefits of API specifications. There are a lot of benefits to writing your specs first, but if you learned the power of API specifications after you’d already built the API, there’s nothing you can do short of invent a time machine. That said, we can still use a spec-first approach for new functionality — new endpoints, or whole new global versions in the same codebase, etc. If you’re adding a new endpoint, we recommend you add just the specs first. This lets the mocks and SDK steps do their job: allow people to play with it, and get feedback before you bother writing the application code. If you want to jump right in there and write all the code and specs all at once — knowing that you might have to change it later — then that’s something you can do too. This works for us as our APIs are private, but having specs for endpoints that do not exist in `master` might be a problem for some. Perhaps a `develop` branch helps out there, or we need to make mocks and SDKs support feature branches… Let us know how it goes for you. **Write OpenAPI:** Writing YAML files by hand is daunting and too open-ended. Many beginners (including my past self) just don’t want to do it. Thankfully several IDE/editors like Atom, VS Code, Eclipse, and the JetBrains suite, [have plugins](https://openapi.tools/?ref=apisyouwonthate.com#editors) that help you autocomplete keywords, provide validation, linting, and more. Many want a more GUI approach, and currently that’s not quite there. Some such as [Rápido](https://rapidodesigner.com/?ref=apisyouwonthate.com) help during the sketching phase, but lacks editing so once you’ve created your initial specs you’re back to editing by hand anyway. There are online editors which work great, but either require you to copy and paste file contents up and down (a recipe for losing changes), or they fail to support multiple files [using the](https://swagger.io/docs/specification/using-ref/?ref=apisyouwonthate.com) `[$ref](https://swagger.io/docs/specification/using-ref/)` [keyword](https://swagger.io/docs/specification/using-ref/?ref=apisyouwonthate.com). This means if you want to continue using an editor you need to maintain one giant megafile. Some require paying per user, some offer limited import/export functionality, or lock you into using special git repos if you’re lucky. Integrating these online editors into a larger workflow can be hard, so we stick to using open-source tools and write the glue ourselves. One company is working on a GUI editor for Windows, OS X and Linux, which just works with local files. This will be amazing for anyone’s workflow because you can use Git, Hg, Bazaar, FTP Sync… whatever you want. This is sadly not yet available, so the "Write OpenAPI" step remains "In Progress". For now at WeWork we recommend folks use their IDE/editor of choice, which hopefully has a plugin available. If their editor of choice does not have a plugin, we suggest they consider switching to VS Code, because the editor and its [OpenAPI plugin](https://marketplace.visualstudio.com/items?itemName=mermade.openapi-lint&ref=apisyouwonthate.com) are both fantastic. If not, they can rely entirely on our CLI linter to get feedback on the validity of their files. **Lint:** There are plenty of OASv3 CLI validators around which anyone can use to confirm OpenAPI files are written correctly, but "correct" is not quite enough when you have a landslide of developers coming into OpenAPI for potentially the first time, writing specs for tricky APIs. We wrote up a huge Style Guide on a wiki which nobody was really looking at, and would change, so even when folks read it they’d miss out on updates. To improve the quality of our API specs (like reminding people to use Tags, and add descriptions to parameters so humans know what they’re for), [Speccy](https://speccy.io/?ref=apisyouwonthate.com) was created. Now recommendations can be made programatically, by [adding new rules to the existing rules files](https://apisyouwonthate.com/blog/govern-your-apis-with-speccy). **Update (2019-10-04):* When I left Speccy I left it with a few colleagues, who* *have continued to keep Speccy maintained. No major work has been done on it in ages,* *so if you like the idea of Speccy but want more power, use [Spectral](https://stoplight.io/spectral/?ref=apisyouwonthate.com) by Stoplight. It* *has a bunch of developers adding amazing features on the regular, and is used to power* *style guides in our amazing visual OpenAPI editor [Stoplight Studio](https://stoplight.io/studio/?ref=apisyouwonthate.com).* **Pull Request:** Committing to master is generally frowned upon, and by making a pull request we get a chance to ask our team for review. It also allows us to utilize Continuous Integration for our API specification workflow. **Team Review:** Currently there are a lot of teams at WeWork, and +1 is very much required for most of them. Team Review of specs is helpful, because if somebody is designing a new endpoint you should probably have another pair of eyes on it. Business requirements need to be met, and it’s better to catch mistakes early on before people start using those specs. **Lint:** Speccy again! Just like code linters and unit tests, it’s a good idea to run this stuff locally then on CI too. We all forget things when we’re in a rush. If you’re using CircleCI, add the following to your `.circle/config.yml`: ```yaml version: 2 jobs: # other jobs speccy: docker: - image: 'circleci/node:8' steps: - checkout - run: 'npx speccy lint ./api/openapi.yml --rules=strict' workflows: version: 2 commit: jobs: - speccy: context: org-global requires: - build ``` This will fail the job not only when somebody creates something like an OpenAPI syntax error, but also when they do something Speccy doesn’t like, like adding a parameter without a description. As soon as Speccy is enabled, the quality of specs can only go up. **Contract Testing:** This step is implemented at the same time as the code, because a JSON Schema file is a great way to check your code is working according to expectations. I’ve referred to this as [Tricking Colleagues into Writing Documentation via Contract Testing](https://apisyouwonthate.com/blog/writing-documentation-via-contract-testing), because developers often hate writing docs. Many developers want contract testing, so getting them to write contract tests can actually create documentation at the same time… Write contract tests, and get awesome documentation that you know is up-to-date "for free". JSON Schema is used here because many many more languages have JSON Schema validators, than have OpenAPI validators. JSON Schema can also cover more dynamic cases thanks to `if`/`then`/`else` keywords. Our specification files live in the same repositories as the source code for each API, as we want the usual PR process, want code changes to match specification changes, and generally it’s less faffing around for develpers this way. The teams experimenting with gRPC are making one central repository and everyone sends their schema (protobuf) updates there, but we wanted to avoid this extra step for developers. They can control their own schemas, and we have a central process that aggregates API specifications, doing a lot of handy stuff in the process. **Clone All Repos:** This started off as a really simple Rakefile (Ruby-flavoured make) which looks through a `data/apis.yml` file which lists repositories, and adds metadata about that API. ```yaml service-name: name: Service Name description: Short blurb about the thing repo: "git@github.com:wework/service-name.git" spec_dir: api/ entry_file: openapi.yml ``` Simple bit of ruby glues YAML and the shell together: ```ruby apis = YAML.load(File.open('./data/apis.yml')) task :clone do apis.each do |nickname, api| sh "rm -rf ./tmp/#{nickname} 2>&1" # Shallow clone and only get the branch we want sh "git clone #{api['repo']} ./tmp/#{nickname} -b #{api['branch'] || 'master'} — single-branch — depth=1 2>&1" end end ``` Done! So long as the user running this has SSH access to all the Git repos we’re talking to, we’ll have a `./tmp/` directory full of cloned repos. The next step is where things get extra interesting! **Normalizing and Resolving Specs:** Whilst we strongly recommend OASv3 be used as an input, reality had other ideas. Some teams are still on OpenAPI v2.0, and one or two folks are on an older "Postman First" workflow. Whilst that’s been deprecated (more on that later), we wanted their APIs to at least have some level of inclusion in our workflow. We pass OpenAPI v2.0 through swagger2openapi, and if the `data/apis.yml` indicates the input file is `format: postman`, we chuck it through the [Apimatic Transformer](https://apimatic.io/transformer?ref=apisyouwonthate.com) API and have them convert it OASv3. Even the OASv3 specifications need a little work done. Most contain `$ref`, and some weaker OpenAPI tooling does not support that feature. On top of that, some of the referenced files are written in JSON Schema-proper instead of OpenAPI-flavoured JSON Schema. Speccy’s `resolve` command solves these problems: ``` $ speccy resolve api/openapi.yml --json-schema ``` This turns multiple files into one megafile, and downgrades JSON Schema-proper to OpenAPI-flavoured JSON Schema in the process. Now the aggregator service has one format to work with, and it’s all in one file, so we don’t have to worry about how good the OpenAPI tooling is that’s being used for the task at hand. **API Server:** Our API Aggregator has a web API (meta!) and exposes a lot of this information for other aspects of the workflow. ```json [ { "name": "Service Name", "slug": "service-name", "contact": { "team": "Some Team", "email": "some-team@example.com" }, "openapi_url": "http://localhost:3000/specs/service-name/openapi", "postman_url": "http://localhost:3000/specs/service-name/postman-collection", "mock_server_url": "http://localhost:3000/mocks/service-name/" }, { ... } ] ``` The root of our API server contains links to all the other useful stuff it has to offer, so it can be navigated around without documentation (HATEOAS!). Needing documentation to find the documentation would just be… oww. **Mock Server:** There are two different types of "mock server". What you usually see when talking about "mocks" is a HTTP server running static endpoints, which will respond with the same sort of example you see in your documentation when you send it a request that matches the path and method. There will usually be no validation, no way to trigger errors, no way to get dynamic values back, etc. Then there’s a sandbox, which is often a fairly realistic API instance which may well even have a database in the background, that saves values you send it, can trigger errors if your payload is invalid, and various other configurable things. These are really powerful, but we’re ignoring them for now as they’re quite advanced, and a lot of them require user interfaces, user sign in, which is another walled garden to think about paying for and integrating with. Sticking to traditional "mocking", we’re still deciding between [Prism](https://github.com/stoplightio/prism?ref=apisyouwonthate.com) and [APISprout](https://github.com/danielgtaylor/apisprout?ref=apisyouwonthate.com). Prism has been around for a long time and is very powerful, but it’s stuck waiting for one of its [Go dependencies to get OAIv3 support](https://github.com/go-openapi/spec/issues/21?ref=apisyouwonthate.com). APISprout is very new and looks very promising, but we’ve had a few issues getting it to work. We’re focusing on other bits for now and will loop back in another month to finish up the mock server. **SDK Generator:** Currently it’s very common for every application contacting another API/service to build a "Connection" class. This class will usually be a HTTP wrapper, and it’s essentially an SDK which the caller builds. These are hand written, in every application, by different developers. The connection classes vary across applications, and they vary within the same application (one connection class will work differently to another in the same application). They often miss common error cases, ignore concepts like "Retry-After", do not help with validation errors, force clients to figure out appropriate timeouts, etc. It means we’re all using the very bare minimums of HTTP and ignoring the most useful features. ![Pie chart showing only 12% of customers had a client library for any given API or service in their language (and we were all using ruby)](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/posts/wework-workflow/sdk.png) A handful of teams have written a module/library to act as an SDK, but these take a lot of effort, are manually built and tested, and usually only available in one language (usually Ruby). Due to the very low-level nature of working with the HTTP APIs, integrating another API into a codebase takes several days, when it should be much much quicker than that. Using [OpenAPI Generator](https://github.com/OpenAPITools/openapi-generator?ref=apisyouwonthate.com) (and our own custom templates) we’ve just finished a proof of concept gem, which is going into production as I write. ```ruby sdk = We::Sdk.new user = sdk.apis.locations.get_building('abc-uuid') puts user.data # { pegasus: 123, name: "Adam Neumann", ... } user = sdk.apis.memberships.get_user_memberships(user_uuid: 123, status_scope: 'active') puts user.data # [ ... ] ``` The `.data` property is an abstraction on `.body` which could contain a string of [JSON:API](https://jsonapi.org//?ref=apisyouwonthate.com) data or our own propietary data format. ```ruby def data if headers['content-type'] =~ /^application\/vnd.api+json/ check_json_api_response else check_we_response end end ``` Seeing as we’ve only recently made the JSON:API recommendation, it’s really handy that we can roll out code which dynamically supports both. It’s also baked in [faraday-sunset](https://github.com/wework/faraday-sunset?ref=apisyouwonthate.com), so more teams will start to notice deprecation warnings showing up in their logs (and since faraday-sunset v0.2.0 things can also get reported to [Rollbar](https://rollbar.com/?ref=apisyouwonthate.com)). Over time we can programmatically add other things like [HTTP caching](https://apisyouwonthate.com/blog/http-client-response-caching) and [circuit breakers](https://blog.runscope.com/posts/phil-sturgeon-taking-a-timeout-from-poor-performance?ref=apisyouwonthate.com) and enable them one API at a time if we so want, and thanks to [Dependabot](https://dependabot.com/?ref=apisyouwonthate.com) we can have reasonable expectations of updates happening within a decent timeframe. We also have an async interface being added later which will allow for HTTP/1.1 (and later HTTP/2) calls to be executed in a batch. We’ll see what parts of this we can open-source too of course. **Postman Mirroring:** We used to have teams that would use Postman for documentation, but Postman documentation is very limited. It’s basically a bookmark collection in HTML hosted on their website, and it doesn’t mention anything about the actual body of the request/response unless you specifically copy and paste some JSON into their GUI. These examples may be entirely inaccurate and stagnate quickly. Postman is an amazing HTTP client, and having shared collections is very useful, so we still use it! We convert our normalized OASv3 files to shared Postman Collections via Apimatic Transformer. Once it’s given us a Postman Collection v2, we upload that file to the [Postman API](https://docs.api.getpostman.com/?ref=apisyouwonthate.com), and within seconds everyone's Postman GUI will have updated to contain any new endpoints, improved examples, etc. Our documentation hub initially had to do all the normalization and resolution itself, but thanks to the API Aggregator it now just has a single Rakefile which looks at the API Server, and passes each `openapi_url` through [ReDoc](https://github.com/Rebilly/ReDoc?ref=apisyouwonthate.com). The static HTML is put in the `public/` folder along with all our other technical documentation, and job done. We have a fantastic cross-API testing suite, which is similar in functionality to [Stoplight: Scenarios](https://stoplight.io/platform/scenarios/?ref=apisyouwonthate.com). API’s don’t exist in isolation, clients talk to them and APIs talk to other APIs, so you need a test suite that exists outside of the one APIs repo. This end-to-end test suite runs periodically and on build, leveraging docker compose to spin up and dependencies it requires. Built using RSpec and Faraday, it’s easy for us to add middlewares into these interactions. One middleware we cooked up in a hackathon is faraday-openapi (we’ll open-source this as soon as we’ve got it more fleshed out!), and it basically just looks at the provided OpenAPI spec, checks the request and response that comes through it, then raises an exception on failure. You’d probably never want to enable this in production, but in a test suite this is fantastic! Now instead of relying on each and every team to enable contract testing (to confirm that both their code is doing what the specs say, and their specs are correct to the code), we can enable this automatically for any API in the E2E suite. OpenAPI and JSON Schema are fantastic tools and the tooling is only getting stronger. Every week it feels like a new tool pops up that makes me improve another section of the workflow, and we’ve not even got into server generation yet. \~The API Aggregator is being wrapped up in Docker to make it something we can open-source, so you too can have mocks, docs, SDKs and Postman mirroring, all spawned from your own Git repos with very little work. We figure we’ve spent a lot of time working all this out, no reason why you should too.\~ I took all this experience to [Stoplight](https://stoplight.io/?ref=apisyouwonthate.com) and we've solved "DocOps" for you, so don't waste your time duct-taping all this nonsense together. ### Common Hypermedia Patterns with JSON Hyper-Schema URL: https://apisyouwonthate.com/blog/common-hypermedia-patterns-with-json-hyper-schema/ Last updated: 2023-03-24T12:27:18.000Z *Guest post from [Aaron Hedges](https://twitter.com/Dashron?ref=apisyouwonthate.com).* In the last two JSON-Hyper Schema articles (Getting Started — [Part One](https://apisyouwonthate.com/blog/getting-started-with-json-hyper-schema) and [Part Two](https://apisyouwonthate.com/blog/getting-started-with-json-hyper-schema-part-2)), we covered the basics: - Basic Links - Request and response bodies - Request and response headers - HTTP methods But while working with JSON Hyper-Schema I have discovered a couple of common API patterns that could use a little more explanation. In this article I’m going to show you common hypermedia patterns and how to describe them with JSON Hyper-Schema. ## **Hypermedia for Collections** When you are handling a collection of items, such as a list of users, there are two forms of hypermedia you want to think about: the hypermedia associated with the collection and the hypermedia associated with each item in that collection. ### Collection Representation Hypermedia Let’s start with hypermedia associated with the collection representation. Here’s an example of the representation returned from GET /users ```js { "data": [{ "name": "User 1", "id": 5 }, { "name": "User 2", "id": 20 }], "pagination": { "next_cursor": "afe412aew3813jwa" } } ``` This example includes pagination metadata. Pagination limits the items on the current "page” of data, and gives you a way to request the next page. This resource handles pagination via cursors, which you use with the following url `/users?cursor={next_cursor}` Instead of exposing cursors and instructing your client how to build that url, I highly recommend you automate the process. Automating the URL generation with JSON Hyper-Schema requires use of the links keyword: ```js { "type": "object", "properties": { "data": { "type": "array", "items": { "type": "object", "properties": { "id": { "type": "number" }, "name": { "type": "string" } } }, "maxItems": 10 }, "pagination": { "type": "object", "properties": { "next_page_cursor": { "type": "string" } } } }, "links": [ { "rel": "next", "href": "/users{?cursor}", "templateRequired": ["cursor"], "templatePointers": { "cursor": "/pagination/next_page_cursor" }, "targetSchema": {"$ref": "#"} } ] } ``` Here, our next Link Description Object (or LDO, as described in a [previous article](https://apisyouwonthate.com/blog/getting-started-with-json-hyper-schema)) has an optional query parameter called cursor (as defined in the URI template href). The templatePointers object states that the cursor parameter should come from the next\_page\_cursor value in your JSON instance. This gives your client all the necessary to automate constructing the "next page” url. ### Collection Item Hypermedia The other type of collection hypermedia is the hypermedia associated with each item. Each item will have it’s own links, the most common of which is described by the rel item. This relation states that the link describes a single item in the collection. The hyper-schema to describe this relation would look like this… ```js { "type": "object", "properties": { "data": { "type": "array", "items": { "type": "object", "properties": { "id": { "type": "number" }, "name": { "type": "string" } }, "links": [{ "rel": "item", "href": "/users/{user_id}", "templateRequired": ["user_id"], "templatePointers": { "user_id": "/data/items/id" }, "targetSchema": { "/properties/data/items" } }] }, }, "maxItems": 10 }, "pagination": { "type": "object", "properties": { "next_page_cursor": { "type": "string" } } } } ``` Notice the links keyword nested deeper in the schema. The links keyword can be included at any depth of the JSON Hyper-Schema document. This is powerful because it directly associates links with portions of your schema. If the subschema associated with the links is not valid (such as a missing next\_page\_cursor property on the pagination object) the LDO is also invalid and can be ignored. There are a lot of creative ways to associate LDOs with schema fields, but that’s well out of the scope of this article. Leave a comment if you discover any cool real world examples! ## Describing conditional LDOs One of the best parts of hypermedia is how easily and intuitively it can define the capabilities of a client. In JSON Hyper-Schema, we have two options for describing whether or not a hypermedia link is applicable to your client. We just described one option, an invalid subschema invalidates all associated links. But what about a more complicated example. Let’s say you can only look at a users blog posts (`GET /users/{user_id}/posts`) if that user made them public. Here’s an example API response for a user with private blog posts. ```js { "id": 234, "post_availability": "private" } ``` Here is another with public blog posts. ```js { "id": 345, "post_availability": "public" } ``` To tell your client that access to `GET /users/234/posts` is only available for public collections we will use one of the more powerful features of JSON Schema, if statements. An if statement comes in 3 parts: if, then and else. And here’s an example of how you would use an if statement to control access to a users blog posts. ```js { "type": "object", "properties": { "id": { "type": "number" }, "post_availability": { "type": "enum", "values": ["public", "private"], "if": { "const": "public" }, "then": { "links": [{ "rel": "posts", "href": "/users/{user_id}/posts", "templatePointers": { "user_id": "/id" } }] } } } } ``` ### Let’s break it down. The value of the if block is a schema. If the schema is valid, the contents of then will be used along with the rest of the schema. When the if schema is invalid the then schema will be ignored and the else schema will be used instead. So in the earlier example, if post\_availability is public the client will use the links. If post\_availability is anything else the links are ignored (and nothing else happens, because we do not have an else statement). ## Building on top of existing hypermedia What if your API already uses a hypermedia format such as [JSON:API](https://jsonapi.org/?ref=apisyouwonthate.com), [HAL](https://stateless.group/hal%5Fspecification.html?ref=apisyouwonthate.com) or [Siren](https://github.com/kevinswiber/siren?ref=apisyouwonthate.com)? Well, you can easily reference your hypermedia from JSON Hyper-Schema! Here’s a very basic JSON:API document. ```js { "links": { "self": "/users/14", }, "data": { "type": "user", "id": 14, "attributes": { "name": "Aaron" } } } ``` JSON:API has a very specific way of organizing your data. Top level fields are all reserved and you define the object properties within /data/attributes. /data/type and /data/id are used to describe the specific resource you are accessing, and /links holds the connections between this data and other resources. Now our goal is to reuse the exact hypermedia data included in the JSON:API link for your hyper-schema links. In the previous examples you combined a URI template with JSON pointers to tell a client how to construct a URL. In this case we pull the entire URL from the JSON instance, as demonstrated in the self LDO of the example below. ```js { "type": "object", "properties": { "links": { "type": "object", "properties": { "self": { "type": "string" } } }, "data": { "type": "object", "properties": { "type": { "const": "user" }, "id": { "type": "number" }, "attributes": { "type": "object", "properties": { "name": { "type": "string" } } } } } }. "links": [{ "rel": "self", "href": "{+user_href}", "templateRequired": ['user_href'], "templatePointers": { "user_href": "/links/self" } }] } ``` But let’s take this a step further and expand this common use case with the full power of JSON Hyper-Schema’s LDOs. Usually, services let you change your name. In JSON:API, you would change your name like this: ```http PATCH /users/14 Content-Type: application/vnd.api+json { "data": { "type": "user", "id": 14, "attributes": { "name": "My New Name" } } } ``` JSON:API doesn’t support describing JSON payloads, it only describes the links between resources. If we make some minor tweaks to the earlier LDO we can augment the JSON:API data with JSON Hyper-Schema to fully describe the capabilities of this user endpoint ```js { "type": "object", "properties": { "links": { "type": "object", "properties": { "self": { "type": "string" } } }, "data": { "type": "object", "properties": { "type": { "const": "user" }, "id": { "type": "number" }, "attributes": { "type": "object", "properties": { "name": { "type": "string" } } } } } }. "links": [{ "rel": "self", "href": "{+user_href}", "templateRequired": ['user_href'], "templatePointers": { "user_href": "/links/self" }, "targetHints": { "allow": ["PATCH"], "accept-patch": ["application/vnd.api+json"] }, "targetSchema": { "$ref": "#/properties/data" } }] } ``` I’m very excited to see what people create with this pattern. Many hypermedia formats included links, but stopped there. With JSON Hyper-Schema we can describe the request and response bodies of any hypermedia link, in any hypermedia format. ### That covers it! Now it’s time to find or build a JSON Hyper-Schema library. You can find most of the existing libraries [here](hhttps://json-schema.org/implementations.html#hyper-schema). If you’d like to learn more about the concept of hypermedia, check out this article on [representing state](https://apisyouwonthate.com/blog/representing-state-in-rest-and-graphql). If you have any questions or comments, leave a reply. It just might lead to another article! ### Govern your APIs with Speccy URL: https://apisyouwonthate.com/blog/govern-your-apis-with-speccy/ Last updated: 2023-03-24T12:27:30.000Z **Update (2019-10-04):* When I left WeWork I left Speccy with a few* *colleagues, who have thankfully done a lot of maintainance work. If you like the* *idea of Speccy but want more power, use* *[Spectral](https://stoplight.io/spectral/?ref=apisyouwonthate.com) by Stoplight. It has a bunch of* *developers and community contributors regularly adding amazing features, and* *does everything Speccy used to do and loads more. See how you can use this, and* *other tools, to make [automated enforceable style guides](https://apisyouwonthate.com/blog/automated-style-guides-for-rest-graphql-grpc)!* The last year has involved a lot of teaching engineers how to use OpenAPI, with the day job growing from a startup to a sizeable corporation. Our engineering department has exploded, with new engineers starting every day. On one hand OpenAPI has been embraced as our approach to designing and documenting APIs, but on the other hand very few engineers have had any experience with it. As such, getting them up to speed with how to efficiently write descriptive, readable, and concise OpenAPI definitions has been a sizeable challenge. To meet that challenge, [Speccy](https://github.com/wework/speccy?ref=apisyouwonthate.com) was created, with a huge amount of help from [oas-kit](https://github.com/Mermade/oas-kit/?ref=apisyouwonthate.com) author [Mike Ralphson](https://github.com/MikeRalphson?ref=apisyouwonthate.com). It has many functions as an OpenAPI local development workflow assistant, but it primarily exists as a linter. As a linter, it can provide opinions, advice, and suggestions more than just "is this valid or not". The advice moves into "You should add descriptions to your parameters so people know what they’re for" and "Add contact info so other folks know who to talk to when they need help". There are a whole bunch of rules for Speccy already, and they’re split across "default" and "strict" meaning you don’t need to use all of them. You can provide arguments via `--skip="contact-properties,license-url"` if any specific rules are rubbing you up the wrong way. What Speccy really needs, is a lot more rules. Here are some I thought up at work recently: - No need to provide both example and default, when they contain the same value - No need to provide `enum: [true, false]` for a `type: boolean` I want to introduce a more file-level set of validation rules too, maybe under `speccy doctor openapi.yaml` to keep it away from the more object-level linter: - This file is 12824 lines long and does not contain a single [$ref](https://swagger.io/docs/specification/using-ref/?ref=apisyouwonthate.com) - No [links](https://swagger.io/specification/?ref=apisyouwonthate.com#linkObject) are used anywhere in this file, how will people know that a response is somehow connected to another response? Kin Lane wrote about the future of Speccy a while back, in [*An OpenAPI-Driven,* *API Governance Rules* *Engine*](https://apievangelist.com/2018/03/05/an-openapi-rules-engine/?ref=apisyouwonthate.com). > My only feedback right now, is that we need lots of people using it, and > helping contribute rules. Oh, and wrap it in an API, and make it available as an > easy to use, and deploy containerized microservice. Then lets get to work on the > Github Gist-driven marketplace of rules, where I can publish the rules I develop > across the projects I’m working on, and of the clients I consult with. Let’s get > to work making sure there are a wealth of rules, broken down into different > categories for API providers to choose from. Then let’s get API tooling and > service providers to begin baking a Speccy rules engine into their solutions, > and allow for the import and management of open source rules. All of that is on the to-do list. An API will exist soon, and was one of the multiple products of a recent hackathon. I’ll get it cleaned up and out on speccy.io at some point. As far as a rules marketplace goes, that’s a while off. For now, how about you folks get in the comments, [Twitter](https://twitter.com/apisyouwonthate/?ref=apisyouwonthate.com), or [Slack,](https://slack.apisyouwonthate.com/?ref=apisyouwonthate.com) with suggestions for rules that you’d like to see. ### Surviving Deprecations to Resources and Properties on Other APIs URL: https://apisyouwonthate.com/blog/surviving-deprecations-to-resources-and-properties-on-other-apis/ Last updated: 2024-06-29T14:08:26.000Z Deprecation, is the art of telling people to stop using that thing, because that thing is probably going away at some point. Even if there are no solid plans to delete the thing entirely, the thing might be deprecated just because it sucks to work with, but usually it's because it has been replaced with a better idea. Regardless of the [type of API](https://apisyouwonthate.com/blog/picking-api-paradigm) you are interacting with, and the [approach they take to versioning](https://apisyouwonthate.com/blog/api-versioning-has-no-right-way), there are no absolute promises the team in charge will deprecate things carefully. Some API providers [act like jerks](https://groups.google.com/forum/?ref=apisyouwonthate.com#!topic/strava-api/%5FHW3UN4Mxt4) and just arbitrarily change things, remove things, and break stuff. Closing APIs down with no warning (or poor warning) is utterly disgraceful behavior, but it does happen. Wrapping dependencies and having backup plans for this sort of situation is another article. Luckily, more often than not a more reasonable approach is taken, and API providers will communicate change in a variety of ways. This chapter is going to outline a few ways in which you might find out about changing happening to an API you're interacting with, so you don't lose business from not noticing a breaking change. ## Low-Tech General Solutions As ludicrous as it might sound, the entire extent of some API providers attempts to communicate upcoming change is to write a blog about it and hope you notice. These blog articles explain things like "Hey we're closing off the free access to this API”, or "We are replacing this Foo API with a Bar API, and you have 3 months to stop using the Foo API before we shut it down”. Fingers crossed you see it! This low-tech approach has no real solutions I can think of, other than… subscribe to the api providers engineering blog on RSS, and — so long as the RSS reader you're using doesn't get shut down — you might notice the article. If they have a newsletter then subscribe to that, maybe with a "devs@" email address or team-specific email address, to improve the bus factor on this. You don't want the announcement going to a member of the team who's quit, then the API breaks just because the rest of the team didn't know about the change. APIs requiring authentication will usually ask for an email address in return for the authentication credentials (API keys, access tokens, etc.) This gives API providers one more way to get in touch about change, and again, make sure it's not the email of a manager or team lead, but a distribution email address for the team. ## Endpoint-based APIs If an API is using global versioning, then they might deprecate all the endpoints under `/v1/` at the same time, and suggest you use all the `/v2/` endpoints. If they're using evolution they might suggest that `/companies` is going away, and you should start to work with `/accounts` instead. The suggestions here may come in the form of low-tech announcements, but they can also be communicated in-band. ### Deprecating Endpoints There is a a proposed standard: [Sunset Header](https://tools.ietf.org/html/draft-wilde-sunset-header-05?ref=apisyouwonthate.com), which at the time of writing is at draft 05\. APIs can add a simple header to the HTTP response, to let clients know the endpoint is nearing the end of its life. **Update 20th May, 2019:** Sunset HTTP Header was released as [RFC 8594](https://tools.ietf.org/html/rfc8594?ref=apisyouwonthate.com). Supporting Sunset is as simple as sniffing for the `sunset` header, and it contains a [HTTP date](https://tools.ietf.org/html/rfc7231?ref=apisyouwonthate.com#section-7.1.1.1) which looks a little like this: ```http Sunset: Sat, 31 Dec 2018 23:59:59 GMT ``` The proposed standard also allows responses to contain a HTTP link header, with `rel=sunset`. The link can be a link to anything, the endpoint replacing it, a link to some human readable documentation, a blog post, whatever. Keep an eye out for sunset headers and their accompanying links coming back in responses your code is getting, and log, or alert, or pass to a dashboard accordingly. If you are using Ruby or PHP there are already easy to use middleware for the most popular HTTP clients: - **PHP:** [hskrasek/guzzle-sunset](https://github.com/hskrasek/guzzle-sunset?ref=apisyouwonthate.com) - **Ruby:** [wework/faraday-sunset](https://github.com/wework/faraday-sunset?ref=apisyouwonthate.com) Some API gateways like [Tyk](https://tyk.io/?ref=apisyouwonthate.com) are [building in support](https://github.com/TykTechnologies/tyk/issues/1626?ref=apisyouwonthate.com), and it's going to become far more wide-spread over time. If you make an implementation for your favorite HTTP client please get in touch in the comments. Another approach commonly used by API providers is to provide SDKs, and if they are kept up to date they'll fling out deprecation warnings about resources that are going away. When you get these notifications, check the errors for what to do next, or go to their documentation to figure it out, or shout at their customer support for not making it clear what you're meant to be doing next and they'll do a better job next time. ### Deprecating Properties Other than the entire endpoint going away, specific properties may over time be deprecated. This is less common in global versioning as they would just remove those in the next global version. It is not a regular occurrence in evolution as they too would just wait until the creation of a new resource for the concept, but it can happen if a property absolutely has to go away. JSON is the main content type in use these days, which does not have a type system built in. There is no way to mark a JSON field as deprecated in just JSON, ~~and no~~ ~~standards exist to help, even JSON Schema still has this on the todo~~ ~~list~~ but I [added a deprecated keyword to JSON Schema](https://github.com/json-schema-org/json-schema-spec/issues/74?ref=apisyouwonthate.com) as part of draft 08. Again SDKs can mark things as deprecated over time, especially those built with OpenAPI v3.0 or later. OpenAPI added the `deprecated: true` keyword, so SDKs can now look for this and fire off deprecation warnings for clients using that property in their code. Keeping up to date is important, so make sure something like [Dependabot](https://https//dependabot.com/?ref=apisyouwonthate.com) is implemented to keep tabs on dependencies. ## GraphQL GraphQL pushes hard for evolution in its marketing and most advice in the ecosystem, which is awesome. They do not have a huge amount of documentation covering deprecations, and much of it comes from third-parties like Apollo, but it's certainly possible. ### Deprecating Types Instead of endpoints, GraphQL has types. There is no first class support for deprecating types in GraphQL at the time of writing. When an API is unable to evolve a concept through adding new properties alone, a new type will pop up somewhere. A low-tech solution like a blog or email announcement may be used to communicate this new type, and the deprecation of the old type. Another possible solution is to deprecate all the properties in the type, and mark in the deprecation reason that the whole type is going away. ### Deprecating Properties The API provider will add the `@deprecated` keyword to the type: ```graphql type User { name: String @deprecated(reason: "Property 'name' was split into 'firstname' and lastname'") firstname: String lastname: String } ``` When looking at a GraphQL API through GraphiQL, the documentation that it autogenerates will show deprecated fields, visually separated from the other fields (smart!) ![Screenshot of API docs showing a deprecated field](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/posts/surviving-deprecations-to-resources-and-properties-on-other-apis/fields.png) Image is from [Kevin Simper's](https://medium.com/@kevinsimper/?ref=apisyouwonthate.com) awesome article: [How to deprecate fields in GraphQL.](https://medium.com/@kevinsimper/how-to-deprecate-fields-in-graphql-52fbd03fb9d7?ref=apisyouwonthate.com) ## GRPC (Protobuf) Protobuf has an option for marking a "field” as deprecated. ```protobuf optional int32 old_field = 6 [deprecated=true]; ``` Full snipped from their documentation: > `deprecated` (field option): If set to `true`, indicates that the field is deprecated and should not be used by new code. In most languages this has no actual effect. In Java, this becomes a `@Deprecated` annotation. In the future, other language-specific code generators may generate deprecation annotations on the field's accessors, which will in turn cause a warning to be emitted when compiling code which attempts to use the field. If the field is not used by anyone and you want to prevent new users from using it, consider replacing the field declaration with a reserved statement. Seems like they don't have an option for deprecating the whole message, but theoretically you can just mark anything as deprecated. The implementations won't actually do anything about it, but you can probably hack something into something or send some PRs to get that done. ## Summary Whatever system you're using, things feel pretty similar. Keep an eye out for deprecations at various levels if you're talking to an evolving API, and keep an eye out for new versions if the API is globally versioned. If you work for a company that considers itself an API provider, what is your approach to letting folks know something is going away? Do you have a support team who reach out? Is there another approach I missed? Answer in the comments. ### Picking the right API Paradigm URL: https://apisyouwonthate.com/blog/picking-api-paradigm/ Last updated: 2023-03-24T12:32:26.000Z This article provides a decision flow diagram for selecting the "approach” to use for your next API. To oversimplify things a bit, it’s reasonably fair to say that all APIs conform to a paradigm: "RPC”, "REST”, or "query language”. These are general approaches to building APIs, but not a specific tool or specification. They are merely a concept, not a tangible thing. Implementations are something you can actually download, install, and use to build an API, that conforms to whatever rules the implementors picked, from whichever paradigms they wanted to use at the time. Specifications (standard, recommendation, etc.) are often drafted up by various working groups, to help implementations share functionality in the same way. An API and a client in different languages can work together perfectly if they’re all following the specification correctly. For example: - [SOAP](https://www.w3.org/TR/soap/?ref=apisyouwonthate.com) is a W3C recommendation, following the RPC paradigm, with implementations like [gSOAP](https://www.genivia.com/dev.html?ref=apisyouwonthate.com) - [gRPC](https://grpc.io/?ref=apisyouwonthate.com) is a implementation, following the RPC paradigm, which has no standard or specification by any working group, but authors Google Inc. did [document the protocol](https://github.com/grpc/grpc/blob/master/doc/PROTOCOL-HTTP2.md?ref=apisyouwonthate.com) - [REST](https://www.ics.uci.edu/~fielding/pubs/dissertation/top.htm?ref=apisyouwonthate.com) is a paradigm, which has never been turned into a specification, and has no official implementations, but building a REST API is usually just a case of picking [appropriate standards](http://standards.rest/?ref=apisyouwonthate.com) and tooling As you can see a direct comparison between any of these things is difficult, but at work I needed to find a way to help people make decisions about how to build an API. We’ve covered the [technical differences between RPC, GraphQL and REST](https://apisyouwonthate.com/blog/understanding-rpc-rest-and-graphql) before, so all that was left was to highlight *when* one should be used over another. I initially wanted to make a diagram to point folks to the appropriate paradigm, but that gets really open-ended. For example, you might want to ask if having a type-system is important for the messages, but some implementations and standards from all three mentioned paradigms use types, and some do not. Making the decision between paradigms alone was so vague it was useless, and trying to consider all implementations just sounded awful (XML-RPC, JSON-RPC, SOAP, SPARQL, FIQL, Micro, … 😴). Instead we went with gRPC, REST, and GraphQL. ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/posts/picking-paradigms/dfd.png) A quick guide to picking an approach for your next API, in the form of a decision flow diagram. Wait, what is that "context boundary” thing all about?! Basically, it’s the idea that whenever a the line is crossed between any imaginary boundary, a few more layers of abstraction should be used to help with the longevity of the system. REST provides those layers of abstraction, and GraphQL provides a few too. That boundary could be as simple as another team/department/company, or a group of systems that just shouldn’t know about each other. Things within the context can treat their own APIs like "private classes” in programming languages, they can change whenever they want, spin up and down, delete, evolve, change, who cares. When going to another context… probably use things like REST (with Hypermedia and JSON Schema) to help those clients last longer without needing developer involvement for most change. ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/posts/picking-paradigms/context-boundary.png) This bounded context bit is really the crux of a lot of the deciding between when to use gRPC, and when to use something else. Internally you can do whatever you want, but when there’s a chance that the developers involved in clients and servers not in close communication (when they have other priorities in the sprint, are on a work retreat, or literally don’t know each other or have any way to communicate), these layers of abstraction become a lot more useful. Pushing [client-side validation to JSON Schema](https://apisyouwonthate.com/blog/json-schema-client-side-validation), for instance, is a layer of abstraction that REST allows (and you could totally do in your own RPC APIs if not using gRPC). Another example would be [pushing workflows and resource state to the API](https://apisyouwonthate.com/blog/representing-state-in-rest-and-graphql) instead of having your RPC clients have to try and figure it out by looking at random properties. The when here is important, because should every API be REST or RESTish? Hell no! But REST is very important for more use-cases than folks seem to think these days. GraphQL fits in here when the more important parts of REST are not relevant, and the shape of clients is super different from each other. We’ve not been recommending it actively at WeWork, and one of the two teams using it has ditched the thing, but I do expect to see it pop up after making this diagram part of our API design guide. ### Implementations gRPC and GraphQL have officially approved implementations for a wide array of languages, so use those as a starting point. - [gRPC](https://grpc.io/?ref=apisyouwonthate.com) - [GraphQL](https://graphql.org/?ref=apisyouwonthate.com) There is a bunch of REST tooling floating around that’s not awful. - [API Platform](https://api-platform.com/?ref=apisyouwonthate.com) - [Apigility](https://www.apigility.org/?ref=apisyouwonthate.com) - [Flask-Potion](https://github.com/biosustain/potion?ref=apisyouwonthate.com) - [Fusio](http://fusio-project.org/?ref=apisyouwonthate.com) - [Go has a bunch of frameworks](https://nordicapis.com/7-frameworks-to-build-a-rest-api-in-go/?ref=apisyouwonthate.com) - [Silkapp](https://github.com/silkapp/rest?ref=apisyouwonthate.com) - [VertX](https://vertx.io/docs/?ref=apisyouwonthate.com#web) - [jsonapi-rb](https://github.com/jsonapi-rb/jsonapi-rb?ref=apisyouwonthate.com) It certainly would be lovely if there was a go-to REST implementation, like gRPC + HTTP URLs + with JSON Schema for [client+server-side validation](https://apisyouwonthate.com/blog/json-schema-client-side-validation) and [discovery through HATEOAS](https://apisyouwonthate.com/blog/getting-started-with-json-hyper-schema-part-2)… that’d be dope. ### What is API Rate Limiting All About? URL: https://apisyouwonthate.com/blog/what-is-api-rate-limiting-about/ Last updated: 2023-03-24T12:32:16.000Z Something API clients (applications talking to APIs) are not often aware of, but run into often, is rate limiting; the API telling you to calm down a bit, and slow down how many requests are being made in a certain timeframe. The most basic rate limiting strategy is often "clients can only send X requests per second." Many APIs implement rate limiting to ensure relative stability when unexpected things happen. If for some reason one client causes a spike in traffic, the API has to continue running smoothly for other users instead of crashing. A misbehaving (or malicious script) could be hogging resources, or the API systems could be struggling and they need to cut down the rate limit for "lower priority" traffic. Sometimes it is just because the company providing the API has grown beyond their wildest dreams, and want to charge money for increasing the rate limit for high capacity users. Often the rate limit will be associated to an "API key" or "access token", and our friends over at [Nordic APIs](https://nordicapis.com/?ref=apisyouwonthate.com) very nicely explain some other rate limiting strategies: > Server rate limits are a good choice as well. By setting rates on specific > servers, developers can make sure that common use servers, such as those used to > log in, can handle a lot more requests than specialized or seldom used servers, > such as data conversion devices. > Finally, the API developer can implement regional data limits, which limit > calls by region. This is especially useful when implementing behavior-based > limiting; for instance, a developer would expect the number of requests during > midnight in North America to be lower than the baseline daytime rate, and any > behavior contrary to this without a good reason would suggest questionable > activity. By limiting the region for a period of time, this can be > prevented. — [Nordic APIs](https://nordicapis.com/stemming-the-flood-how-to-rate-limit-an-api/?ref=apisyouwonthate.com) All fair reasons, but for the client it can be a little pesky. #### Throttling Your API Calls There are a lot of ways to go about throttling your API calls, and it very much depends on where the calls are being made from. One of the hardest things to limit are API calls to a third party being made directly to the client. For example, if your iOS/web/etc clients are making Google Map API calls directly from the application, there is very little you can do to throttle that. You’re just gonna have to pay for the appropriate usage tier for how many users you have. Other setups can be a little easier. If the rate limited API is being spoken to via some sort of backend process, and you control how many of those processes there are, you can limit often that function is called in the backend code. For example, if you are hitting an API that allows only 20 requests per second, you could have 1 process that allows 20 requests per second to pass through. If this process is handling things synchronously that might not quite work out, and you might need to have something like 4 processes handling 5 requests per second each, but you get the idea. If this process was being implemented in NodeJS, you could use [Bottleneck](https://www.npmjs.com/package/bottleneck?ref=apisyouwonthate.com). ```js const Bottleneck = require("bottleneck"); // Never more than 5 requests running at a time. // Wait at least 1000ms between each request. const limiter = new Bottleneck({ maxConcurrent: 5, minTime: 1000 }); const fetchPokemon = id => { return pokedex.getPokemon(id); }; limiter.schedule(fetchPokemon, id).then(result => { /* ... */ }); ``` Ruby users who are already using tools like Sidekiq can add plugins like [Sidekiq::Throttled](https://github.com/sensortower/sidekiq-throttled?ref=apisyouwonthate.com), or pay for [Sidekiq Enterprise](https://github.com/mperham/sidekiq/wiki/Ent-Rate-Limiting?ref=apisyouwonthate.com), to get rate limiting functionality. Worth every penny in my books. Every language will have some sort of throttling, job queue limiting, etc. tooling, but you will need to go a step further. Doing your best to avoid hitting rate limits is a good start, but nothing is perfect, and the API might lower its limits for some reason. #### Am I Being Rate Limited? The appropriate HTTP status code for rate limiting has been argued over about as much as tabs vs spaces, but there is a clear winner now; RFC 6585 defines it as 429, so APIs should be using 429. ![http cat 429 error](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/posts/what-is-api-rate-limiting-about/429.jpg) [http cat](https://http.cat/?ref=apisyouwonthate.com) is an amazing resource. Twitter’s API existed for a few years before this standard, and they chose "420 — Enhance Your Calm". They’ve dropped this and moved over to 429, but some others copied them at the time, and might not have updated since. You cannot rule out bumping into a copycat API, still using that outdated unofficial status. ![http cat 420 error](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/posts/what-is-api-rate-limiting-about/420.jpg) [http cat](https://http.cat/?ref=apisyouwonthate.com) shows us how a cat can enhance their calm. Google also got a little "creative" with their status code utilization. For a long time were using 403 for their rate limiting, but I have no idea if they are still doing that. GitHub v3 (a RESTful API that was replaced with a GraphQL, but is still floating around) is still using 403: ```http HTTP/1.1 403 Forbidden X-RateLimit-Limit: 60 X-RateLimit-Remaining: 0 X-RateLimit-Reset: 1377013266 { "message": "API rate limit exceeded for xxx.xxx.xxx.xxx. (But here’s the good news: Authenticated requests get a higher rate limit. Check out the documentation for more details.)", "documentation_url": "https://developer.github.com/v3/#rate-limiting" } ``` Getting a 429 (or a 420) is a clear indication that a rate limit has been hit, and a 403 combined with an error code, or maybe some HTTP headers can also be a thing to check for. Either way, when you’re sure it’s a rate limit error, you can move onto the next step: figuring out how long to wait before trying again. #### Proprietary Headers Github here are using some proprietary headers, all beginning with `X-RateLimit-`. These are not at all standard (you can tell by the X-), and could be very different from whatever API you are working with. Successful requests with Github here will show how many requests are remaining, so maybe keep an eye on those and try to avoid making requests if the remaining amount on the last response was 0. ```bash $ curl -i https://api.github.com/users/octocat HTTP/1.1 200 OK X-RateLimit-Limit: 60 X-RateLimit-Remaining: 56 X-RateLimit-Reset: 1372700873 ``` You can use a shared key (maybe in Redis or similar) to track that, and have it expire on the reset provided in UTC time in X-RateLimit-Reset. #### Retry-After According to the RFCs for HTTP/1.1 (the obsoleted and irrelevant RFC 2616, and the replacement RFC 7230–7235), the header `Retry-After` is only for 503 server errors, and maybe redirects. Luckily [RFC 6584](https://tools.ietf.org/html/rfc6584?ref=apisyouwonthate.com) (the same one which added HTTP status code 429) says it’s totally cool for APIs to use `Retry-After` there. So, instead of potentially infinite proprietary alternatives, you should start to see something like this: ```http HTTP/1.1 429 Too Many Requests Retry-After: 3600 Content-Type: application/json { "message": "API rate limit exceeded for xxx.xxx.xxx.xxx.", "documentation_url": "https://developer.example.com/#rate-limiting" } ``` An alternative value for `Retry-After` is an HTTP date: ```http Retry-After: Wed, 21 Oct 2015 07:28:00 GMT ``` Same idea, it just tells the client to wait until then before bothering the API further. By checking for these errors, you can catch then retry (or re-queue) requests that have failed, or if thats not an option try sleeping for a bit to calm workers down. ***Warning:*** *Make sure your sleep does not block your background processes* *from processing other jobs. This can happen in languages where sleep sleeps the* *whole process, and that process is running multiple types job on the same* *thread. Don’t back up your whole system with an overzealous sleep!* Faraday, a Ruby gem I work with often, is [now aware of Retry-After](https://github.com/lostisland/faraday/pull/773?ref=apisyouwonthate.com). It uses the value to help calculate the interval between retry requests. This can be useful for anyone considering implementing rate limiting detection code, even if you aren’t a Ruby fan. To learn *how* to implement rate limiting, Google it! [Nginx can help you](https://www.nginx.com/blog/rate-limiting-nginx/?ref=apisyouwonthate.com), API Management gateways like [Kong](https://getkong.org/plugins/rate-limiting/?ref=apisyouwonthate.com) and [Tyk](https://tyk.io/docs/control-limit-traffic/rate-limiting/?ref=apisyouwonthate.com) do a great job, or you can try to implement it at the application level for aaaaallllll of your APIs, which is a bit of a pain in the butt. Some frameworks like [Laravel (PHP)](https://laravel.com/?ref=apisyouwonthate.com) support [basic rate limiting out of the box](https://laravel-news.com/laravel-5-6-dynamic-rate-limiting?ref=apisyouwonthate.com), which is pretty darn cool. *All this and more in *[Surviving Other Peoples APIs](https://leanpub.com/surviving-other-peoples-web-apis/?ref=apisyouwonthate.com)*, currently available* *for pre-order, with roughly 80% of the book available for download.* ### API Evolution for REST/HTTP APIs URL: https://apisyouwonthate.com/blog/api-evolution-for-rest-http-apis/ Last updated: 2023-04-13T11:55:43.000Z There are a lot of pros and cons to various approaches to API versioning, but that has been covered in depth before in [API Versioning Has No "Right" Way](https://apisyouwonthate.com/blog/api-versioning-has-no-right-way). API evolution is making a comeback these days with GraphQL and gRPC advocates shouting about it. Whatever [API paradigm or implementation](https://apisyouwonthate.com/blog/understanding-rpc-rest-and-graphql) you subscribe to, evolution is available to you. REST advocates have been recommending API evolution for decades, but in the past I failed to understand how exactly to handle evolution. Luckily, as always, tooling and standards for HTTP have been improving, and these days API evolution is a lot easier to wrap your head around. ## What is API Evolution API evolution is the concept of striving to maintain the "I" in API, the request/response body, query parameters, general functionality, etc., only breaking them when you absolutely, *absolutely*, have to. It's the idea that API developers bending over backwards to maintain a contract, no matter how annoying that might be, is often more financially and logistically viable than dumping the workload onto a wide array of clients. At some point change cannot be prevented, so at that time evolution suggests you provide sensible warnings to clients, letting them know if a feature they're using is going away, and not bothering them otherwise. ## Examples > The property `name` exists, and that needs to be split into `first_name` and `last_name`. Easy enough. However the data is handled internally (splitting on first space or last space or some other [falsehood defying assumption](https://www.kalzumeus.com/2010/06/17/falsehoods-programmers-believe-about-names/?ref=apisyouwonthate.com)) you now have two new properties. The serializer can change from outputting just their name, to outputting all three properties: ```ruby class UserSerializer include FastJsonapi::ObjectSerializer attributes :name, :first_name, :last_name attribute :name do |object| "#{object.first_name} #{object.last_name}" end end ``` When folks `POST` or `PATCH` to your API, if they send a `name` you can convert it, or if they send `first_name` and `last_name` it'll get picked up fine on the serializer. Job done. > The property `price` needs to stop being dollars/pounds/whatever as we're > starting to support currencies that don't fit into "unit" and "subunit". Switching to an integer to place your cents, pence, etc. would be just as much of a [Fallacies Programmers Think About Currencies](https://gist.github.com/rgs/6509585?ref=apisyouwonthate.com) as using float dollars/pounds, etc. To support the widest array of currencies, some folks like to use "micros", a concept explained well here by [Sift Science](https://support.siftscience.com/hc/en-us/articles/203869406-The-amount-field?ref=apisyouwonthate.com). In this case, the new property could easily be called `price_micros`. If somebody grumps about that and you want a more concise name, just call it `amount` and point folks towards that property instead. A thesaurus is handy. Why don't we just outright change this value from dollars to micros? Because then we'd start charging $1,000,000 for stuff that should only cost $1, and folks probably wouldn't like that. Now clients can either send the `price` property, and it'll convert, or send the new `price_micros` property. If `currency` is a property in the resource (or something nearby) then it's easy enough to support `price` for whatever initial currencies you had (dollar/pound/euro) and throw an error if somebody tries using price for these newer currencies, pointing them instead to the micro property. Nothing broke for existing use cases, and new functionality was added seamlessly. > We have too many old properties kicking around, we need to get rid of them. Deprecations can be communicated in a few ways for API's. For those using OpenAPI v3, you can mark it as `deprecated: true` in the documentation. That's not ideal, of course, as OpenAPI is usually human-readable documentation, sat out of band on a developer portal somewhere. Rarely are OpenAPI schemas shoved into the response like JSON Schema is, so programmatically clients have no real way to access this. JSON Schema is ~~considering~~ [adding a deprecated keyword](https://github.com/json-schema-org/json-schema-spec/issues/74?ref=apisyouwonthate.com), ~~and oops I think I'm in charge of making that happen. I'll get back to doing that after this blog post.~~ as part of draft 08, which matches the OpenAPI keyword. The idea here would be to pair the schema with a smart SDK (client code) which detects which properties are being used. If the schema marks the `foo` field as deprecated, and the client code then calls `$response->foo`, the SDK can raise a deprecation warning. This is achieved by inspecting the schema file at runtime if you offer your schemas in the `Link` header, or at compile time if you're distributing schema files with the SDK. ![GraphQL allows you to evolve an api over time without versioning](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/posts/api-evolution-for-rest-http-apis/api-evolution.gif) GraphQL has the advantage when it comes to property deprecation for sure, as their type system demands clients to specify the properties they want. By knowing which clients are requesting a deprecated property, you can either reach out to that client (manually or automatically), or shove some warnings into the response somewhere to let them know they're asking for a thing which is going away. This is the sort of advantage you get when your type system, clients, etc. are all part of the same package, but HTTP in general can achieve this same functionality through standards. All of that said, removing old properties is usually not all that much of a rush or an issue. Over time new developers will be writing new integrations, looking at your new documentation that tells them to use the new property, and your developer newsletters or changelogs just let them know to move away from it over time. > A carpooling company that has "matches" as a relationship between "drivers" and "passengers", suggesting folks who could ride together, containing properties like `passenger_id` and `driver_id`. Now we need to support carpools that can have multiple drivers (i.e. Frank and Sally both take it in turns to drive), so this whole matches concept is garbage. At a lot of startups, this sort of conceptual change is common. No number of new properties is going to help out here, as the whole "one record = one match = one driver + one passenger" concept was junk. We'd need to make it so folks could accept a carpool based on the group, and any one of those folks could drive on a given day. Luckily, business names often change fairly regularly in the sort of companies that have fundamental changes like this. There is often a better word that folks have been itching to switch to, and evolution gives you a chance to leverage that change to your benefit. Deprecating the whole concept of "matches", a new concept of "riders" can be created. This resource would track folks far beyond just being "matched", through the whole life-cycle of the carpool, thanks to a status property containing pending, active, inactive, blocked, etc. By creating the `/riders` endpoint, this resource can have a brand new representation. As always, the same database fields can be used internally, the same internal alerting tools are used for letting folks know about matches (v1 app) or new pending riders (v2 app). The API can create and update "matches" through this new "riders" interface. Clients can then use either one, and the code just figures itself out in the background. Over time the refactoring can be done to move the internal logic more towards riders, and your [integration tests / contract tests](https://apisyouwonthate.com/blog/writing-documentation-via-contract-testing) will confirm that things aren't changing on the outside. > [@cebe](https://github.com/philsturgeon/philsturgeon.github.io/pull/30?ref=apisyouwonthate.com#discussion%5Fr185912780) asks: How would the matches endpoint return data where there is more than one driver? If the data does not fit the endpoint anymore, it must be broken or fail for such data? Ex-coworker and API mastermind [Nicolás Hock-Isaza](https://github.com/philsturgeon/philsturgeon.github.io/pull/30?ref=apisyouwonthate.com#discussion%5Fr185948655) says: *We only exposed the first driver match to the older apps. If the user accepted it, great. The other driver riders would be denied. If the user rejected the first one, we would show the next one, and the next one, and the next one.* All it takes is a little ingenuity, and API evolution isn't so scary. > We have all these old endpoints hanging around, can we get rid of these > slightly more intelligently than just sending some emails? Yes! API Endpoints can be marked with a `Sunset` header to signal deprecation (and eventual removal) of an endpoint. The [Sunset header](https://tools.ietf.org/html/rfc8594?ref=apisyouwonthate.com) is a HTTP response header that is aiming to standardize how URLs are marked for deprecation. tl;dr it looks a bit like this: ``` Sunset: Sat, 31 Dec 2018 23:59:59 GMT ``` The date is a [HTTP date](https://tools.ietf.org/html/rfc7231?ref=apisyouwonthate.com#section-7.1.1.1), and can be combined with a `Link: rel="sunset"` which can be anything that might help a developer know what is going on. Maybe link to your API documentation for the new resource, the OpenAPI/JSON documents, or even a blog post explaining the change. Ruby on Rails has [rails-sunset](https://github.com/wework/rails-sunset/?ref=apisyouwonthate.com), and hopefully other frameworks will start adding this functionality. Open-source API Gateway system [Tyk](https://tyk.io/?ref=apisyouwonthate.com) is [adding support to an upcoming version](https://github.com/TykTechnologies/tyk/issues/1626?ref=apisyouwonthate.com). Clients then add a middleware to their HTTP calls, checking for Sunset headers. We do this with [faraday-sunset](https://github.com/wework/faraday-sunset?ref=apisyouwonthate.com) (Ruby), [Hunter Skrasek](https://twitter.com/HSkrasek/?ref=apisyouwonthate.com) made [guzzle-sunset](https://github.com/hskrasek/guzzle-sunset?ref=apisyouwonthate.com) (PHP), and anyone can write a thing that looks for a header and logs it to whatever logging thing they're using. > We need to change some validation rules, but the clients have rules baked in. How do we let them know change is coming? Certain validation rules are very clearly breaking. For example, lowering the maximum length of a string property would break clients who are expecting to be able to send longer names. Folks would have to shorten the property on certain devices which would be really weird, especially as the client may well be showing it as valid, only to then surface an error from the API. Other rules may *seem* like they're backwards compatible, but can still break clients in all sorts of ways. For example, making a string property accept a longer value can lead to problems where an out-of-date client is expecting a length of 20, but an up-to-date client has already been used to get that property up to 40\. Again they user would find that data is valid on one device, but be stuck unable to submit the form on another device. Baking validation rules into client applications based on whatever the documentation says is brittle, so moving [client-side validation logic to server-defined JSON Schema](https://apisyouwonthate.com/blog/json-schema-client-side-validation) can solve these problems, and a bunch more. It also makes evolution a whole bunch easier, because this is just another category of change you are automatically communicating to client applications, without any developers needing to get involved. > *Deprecating a specific type of authentication from an endpoint, it's time to say goodbye to HTTP Basic Auth.* If the client is making a request with an authorization header, they have some sort of account. If during the signup for that account you've asked them for an email, you can contact them. If you've not got any way to contact them… tough. Monitor how many folks are using HTTP basic, blog about it, shove some videos up, and eventually you're just going to have to turn it off. The only other approach to helping out here is an SDK. If you slide some deprecation notices into the code months ahead of the cutoff date, you can throw some warnings saying the code is no longer going to work. This gives you a fighting chance for anyone that keeps a bit up to date. For those that don't, you don't have much choice. Shoving a clear error into your HTTP response (here using the amazing [RFC 7807: Problems for HTTP APIs](https://tools.ietf.org/html/rfc7807?ref=apisyouwonthate.com)): ```json HTTP/1.1 403 Forbidden Content-Type: application/problem+json { "type": "https://example.org/docs/errors#http-basic-removed", "title": "Basic authentication is no longer supported", "detail": "HTTP Basic has been deprecated since January, 1st 2018, and was removed May, 1st 2018. Applications should switch to OAuth to resume service." } ``` Google Maps are using this approach to [remove keyless interactions from Google Maps API](https://cloud.google.com/maps-platform/user-guide/?ref=apisyouwonthate.com). It can be handy in other situations too, like if you're dropping `application/xml` from your API and want people to know it won't be there forever. ## More Power The above solutions are a little ad-hoc, and can lead to branched code paths with a bunch of if statements. You can feature flag some of this stuff to help keep things a little tidy, and another approach is to write up change as libraries, something [Stripe refer to as "declarative changes"](https://stripe.com/blog/api-versioning?ref=apisyouwonthate.com). This approach can be a little heavy handed, but it's something to keep in mind. ## Summary Evolution involves thinking a little differently on how you approach change. Often there are simple things you can do to keep clients ticking along, and whilst clients will have to change at *some point*, the whole goal here is to allow them a decent amount of time to make that switch, with the minimal change possible during that change, and **no lock-step deploys** required. And yes, whilst making a new endpoint to switch `/matches` and `/riders` is essentially the same as `/v1/matches` and `/v2/matches`, you've skipped the [quagmire of tradeoffs](https://apisyouwonthate.com/blog/api-versioning-has-no-right-way/) between global versioning, resource versioning, or **gulp** method versioning. Global versioning has its place, but so does evolution. Think about it this way. If implementing some change takes twice as long for API developers compared to other versioning approaches, but save 6 or 7 client developer teams from having to do a whole bunch of work, testing, etc. to chase new versions, this has been worthwhile to the company in terms of engineering hours spent. If you've only got a small number of clients (maybe an iOS and Android version) of an API that changes drastically every year or two, then global versioning is clearly the way to go. ### Writing Documentation via Contract Testing URL: https://apisyouwonthate.com/blog/writing-documentation-via-contract-testing/ Last updated: 2023-03-24T12:32:50.000Z Part of my day job is trying to get folks to write API documentation, so the hordes of new developers joining the company on a weekly basis are not stuck trying [guess at contracts](https://apisyouwonthate.com/blog/commit-to-api-contracts) by poking around years old code. Convincing API developers to take the time to write specs has involved dangling a myriad of carrots. One was a continuous integration pipeline to convert their in-repo OpenAPI specs to read-only Postman Mirrors (because who wants to maintain Postman collections manually?!!), and some folks are excited to [use JSON Schema to offer client-side validation](https://apisyouwonthate.com/blog/json-schema-client-side-validation) (now that we’ve solved the JSON Schema >< OpenAPI issue.) The carrot we’ll be discussing here is contract testing: the art of validating the shape of data provided by the API is what you think it is. Folks are regularly asking "how can we keep our documentation up to date”, and we have [covered a few solutions to that,](https://apisyouwonthate.com/blog/keeping-documentation-honest) one of which is using JSON Schema. Unfortunately thinking of it this way leads to responses like "ok greeeeeat, when I have finished spending a week writing up documentation, I get to write a bunch more stuff to ensure its up to date. Yaaaay! 🙄” Well that’s a fairly backwards way to think about the whole thing. I tell people: API specifications are a *really* powerful way to get contract testing, and they just so happen to render as documentation too. Once you’ve confirmed your code conforms to the contract, you know your documentation is correct, so it’s pretty much been done for free. ## What is Contract Testing A lot of applications just check that the endpoint doesn’t entirely crap out, which is the most basic sort of unit test: ```bash it 'should return HTTP OK (200)' do get "/users/#{subject.id}" expect(response).to have_http_status(:ok) end ``` Sure ok, that’s better than literally nothing, but we need to know what fields are there, that those fields have a specific type, and we need to make sure that doesn’t randomly change! Some folks will add some specific checks to their unit tests, maybe something like [include\_json](https://relishapp.com/waterlink/rspec-json-expectations/docs/json-expectations/include-json-matcher?ref=apisyouwonthate.com): ``` it "has basic info about user" do expect(subject).to include_json( id: 25, email: "john.smith@example.com", name: "John" ) end ``` Others will bump this off to BDD-style testing with Cucumber or similar tools, which is essentially doing the exact same thing: ``` Feature: User API Scenario: Show action When I visit "/users/1" Then the JSON response at "first_name" should be "Steve" And the JSON response at "last_name" should be "Richert" And the JSON response should have "username" And the JSON response at "username" should be a string And the JSON response should have "created_at" And the JSON response at "created_at" should be a string And the JSON response should have "updated_at" And the JSON response at "updated_at" should be a string ... ugh it goes on and on ... ``` At this point it is essentially contract testing, but its very laborious, and is a whitelist. If a new field shows up and we’ve not written a bunch of assertions for its type, we have no way of knowing if its correct. Clients might start using it, later it could change, and our test suite would never know. Something else to consider, is that writing these assertions for each field sucks. You want your unit tests to confirm the shape of the response under all sorts of scenarios, but you do not want to copy these lines over and over again. This leads to folks shoving some assertions into the first unit test, then just crossing their fingers that other scenarios output the correct shape. 🤷‍♂️ ## JSON Schema for "DRY” Contract Testing Don’t Repeat Yourself, and define reusable `schemas/components/user.json` files, populated by JSON Schema: ```json { "$schema": "https://json-schema.org/draft-06/schema#", "title": "Foo", "type": "object", "properties": { "id": { "readOnly": true, "type": "string", "example": "123" }, "uuid": { "type": "string", "format": "uuid", "example": "50f50f52-0d41-4a08-85ea-56423b2803c8" }, "email": { "type": "string", "example": "john@example.com" }, "name": { "type": "string", "example": "john.smith" }, "status": { "type": "string", "example": "inactive" }, "created_at": { "type": ["string", "null"], "format": "date-time", "example": "2018-04-09T15:45:44.358Z" }, }, "required": [ "email", "name", "uuid" ] } ``` *Learn more about the basic JSON Schema keywords over here:* [**The basics - Understanding JSON Schema 1.0 documentation** \_When learning any new language, it's often helpful to start with the simplest thing possible. In JSON Schema, an empty…\_spacetelescope.github.io](https://spacetelescope.github.io/understanding-json-schema/basics.html?ref=apisyouwonthate.com "https://spacetelescope.github.io/understanding-json-schema/basics.html")[](https://spacetelescope.github.io/understanding-json-schema/basics.html?ref=apisyouwonthate.com) Armed with a bunch of reusable contracts, the only thing to do is check if those contracts match the code. Ruby users can do this with [json\_matchers](https://github.com/thoughtbot/json%5Fmatchers?ref=apisyouwonthate.com) by Thoughtbot. Put this config wherever you handle your RSpec setup (or check out their [alternative approaches](https://github.com/thoughtbot/json%5Fmatchers?ref=apisyouwonthate.com#usage)): ```ruby require "json_matchers/rspec" JsonMatchers.schema_root = "docs/components/schemas" ``` Then it’s simply a case of adding new tests for that context, or adding the expect line to your existing tests: ```ruby it 'should return HTTP OK (200)' do get "/users/#{subject.id}" expect(response).to have_http_status(:ok) end it 'should conform to user schema' do get "/users/#{subject.id}" expect(response).to match_json_schema('user') end ``` That’s it as far as contract testing goes. Run your tests, tweak the responses and JSON Schema, try and get it to fail. Remove a required field and it’ll moan. Typecast an int to a string and it’ll get sad. If you need help creating these JSON Schema files, there’s [a few hacks to speed you up](http://localhost:3000/blog/creating-api-specifications-from-bullst?ref=apisyouwonthate.com). #### What about Docs? Often early stage OpenAPI documentation is a lot of paths and maybe some properties, but folks often leave the schemas "until later” (never). ```yaml responses: 200: description: OK ``` With the schema missing you lose all benefit of API specifications: you have no example values, attribute tables, enums for status fields, types, etc. Luckily the new JSON Schema fill exactly that gap! ```yaml responses: 200: description: OK content: application/json: schema: $ref: ./components/schemas/user.json ``` *You’ll need to watch out for the [divergence issue](https://apisyouwonthate.com/blog/openapi-json-schema-divergence),* *or use* *[json-schema-to-openapi-schema](https://github.com/openapi-contrib/json-schema-to-openapi-schema?ref=apisyouwonthate.com)* *to convert this JSON Schema & OpenAPI combo to "pure OpenAPI" in some build/CI* *step, but this works juuuust fine.* Then it’s just a case of [turning that OpenAPI into beautiful documentation](https://apisyouwonthate.com/blog/turning-contracts-into-beautiful-documentation). Give it a try and let me know what you think. I really like this simple, powerful, approach to keeping code up-to-date with your API specs. **JSON Schema** - JavaScript - [jest-json-schema](https://github.com/americanexpress/jest-json-schema?ref=apisyouwonthate.com) (by American Express 😲) - [chai-json-schema](https://www.chaijs.com/plugins/chai-json-schema/?ref=apisyouwonthate.com) - PHP - [estahn/phpunit-json-assertions](https://github.com/estahn/phpunit-json-assertions?ref=apisyouwonthate.com) - Ruby - [thougtbot/json\_matchers](https://github.com/thoughtbot/json%5Fmatchers?ref=apisyouwonthate.com) *Tell us about your API description-based testing and what languages and* *frameworks you're using on* *[@apisyouwonthate](https://twitter.com/apisyouwonthate?ref=apisyouwonthate.com) or [our Slack](https://slack.apisyouwonthate.com/?ref=apisyouwonthate.com).* ### Getting started with JSON Hyper-Schema: Part 2 URL: https://apisyouwonthate.com/blog/getting-started-with-json-hyper-schema-part-2/ Last updated: 2023-03-24T12:33:02.000Z *Guest post from [Aaron Hedges](https://twitter.com/Dashron?ref=apisyouwonthate.com).* In my last [article](https://apisyouwonthate.com/blog/getting-started-with-json-hyper-schema) I described how JSON Hyper-Schema can describe basic links. This included… - The basics of JSON Schema - Describing links with Link Description Objects (LDOs) - Automating URI construction with URI templates, templatePointers, templateRequired and hrefSchema - Reducing schema duplication with schema references In this article I’m going to build on that foundation with resource representations, arbitrary request bodies, HTTP headers, and HTTP methods. ## Describing resource representations Interacting with a resource via it’s URI often involves a resource representation. The most common time you see a resource representation is as the response of an HTTP GET request. In JSON Hyper-Schema a resource representation is defined by two fields, targetSchema and targetMediaType. targetMediaType tells the client what format to expect such as application/json, application/hal+json, etc. targetSchema provides a validation schema for that representation. So if you used the following JSON Hyper-Schema… ```json { "type": "object", "properties": { "id": { "type": "number" }, "name": { "type": "string" } }, "links": [{ "rel": "self", "href": "https://api.dashron.com/users/12345", "targetMediaType": "application/json", "targetSchema": { "$ref": "#"} }] } ``` You would expect the following request… ```http GET https://api.dashron.com/users/12345 ``` To return something that looks like the following response… ```json { "id": 12345, "name": "Zena" } ``` Let’s dig into why. First, the LDO has an href of [https://api.dashron.com/users/12345](https://api.dashron.com/users/12345.?ref=apisyouwonthate.com). The scheme is https, so we know that URI can receive secure HTTP requests. Next, targetMediaType suggests that we will receive a representation that matches the application/json media type. Nothing fancy here, but we now know that we need to pull out our trusty JSON parser. Finally, targetSchema uses a schema reference (`{"$ref”: "#”}`) to point to the root document. This pattern is particularly common for LDOs with a rel of self because these LDOs describe how to access the current context. Other LDOs may reference other JSON Hyper-Schema documents or embed an entire JSON schema. The following example does not use a schema reference. It has an LDO that describes the author of the current resource, and states that the author’s representation will contain an id and name . ```json { "type": "object", "properties": { "id": { "type": "number" }, "title": { "type": "string" } }, "links": [{ "rel": "author", "href": "https://api.dashron.com/users/12345", "targetMediaType": "application/json", "targetSchema": { "id": { "type": "number" }, "name": { "type": "string" } } }] } ``` ## Describing request bodies Most HTTP requests can include a request body. Two of the most common uses of request bodies are replacing an existing resource or sending arbitrary data. ### Replacing resource representations Sometimes you send a request body to replace the representation of an existing resource. To explain this further, let’s bring back a previous JSON Hyper-Schema example. ```json { "type": "object", "properties": { "id": { "type": "number" }, "name": { "type": "string" } }, "links": [{ "rel": "self", "href": "https://api.dashron.com/users/12345", "targetMediaType": "application/json", "targetSchema": { "$ref": "#"} }] } ``` Earlier we described how this LDO can accept HTTP GET requests. Those GET requests will return JSON that validates against this schema. If we want to replace that representation we can make an HTTP PUT request with a new JSON instance that also validates against this schema. ```http PUT /users/12345 { "id": 12345, "name": "Not Zena" } ``` And that’s it. The very same schema can describe GET and PUT requests. If you would like your Hyper-Schema to be more specific about what is and is not supported, check out the HTTP Methods section below. ### Sending arbitrary data What if your request body doesn’t match the target representation? You commonly see this with collection resources. For example, the following resource will return a collection of all videos owned by user 12345 . ``` GET https://api.dashron.com/users/12345/videos ``` That request might return the following resource representation, which contains multiple videos ```json { "items": [{ "id": 12345, "title": "My first video" }, { "id": 23456, "title": "My second video" }], "total": 2 } ``` If you want to add a new video to that collection, you might make the following request… ```http POST /users/12345/videos { "title": "My third video" } ``` In this case, we can’t use targetSchema because the request body isn’t identical to the target representation. When the body doesn’t match the representation we should use submissionSchema and submissionMediaType. This is the main difference between the target and submission keywords. The values are both JSON schemas, but the intent differs. Use submission keywords for arbitrary request bodies, and target keywords to describe the representation that you send to, or receive from, the resource. Now let’s describe those HTTP requests with JSON Hyper-Schema. ```json { "type": "object", "properties": { "items": { "type": "array", "item": { "id": {"type": "number"}, "title": {"type": "string"} } } }, "links": [{ "rel": "self", "href": "https://api.dashron.com/users/12345/videos", "submissionMediaType": "application/json", "submissionSchema": { "type": "object", "properties": { "title": {"type": "string"} } } }] } ``` As mentioned earlier the submissionSchema value describes the HTTP POST request body and the submissionMediaType indicates the the request body should be JSON. There’s one more type of request that I haven’t covered here; partial edits. Partial edits in HTTP are covered by the HTTP PATCH method and a specific HTTP header. Because of this I need to first explain HTTP headers, and then I’ll jump back into the HTTP PATCH method. ## Describing HTTP Headers There are two different types of HTTP headers, one for HTTP requests and one for HTTP responses. ### Request Headers To describe how clients can provide request headers, use headerSchema. headerSchema is a lot like submissionSchema and targetSchema, except it describes the key value pairs of a clients request headers. ```json { "type": "object", "properties": { "id": {"type": "number"}, "name": {"type": "string"} }, "links": [{ "rel": "self", "href": "https://api.dashron.com/users/12345", "headerSchema": { "if-modified-since": { "type": "string" } } }] } ``` This LDO can accept the following header ``` if-modified-since: Thu, 05 Apr 2018 20:08:44 GMT ``` Each header value is a JSON schema, which means it supports any of the [validation keywords](https://json-schema.org/latest/json-schema-validation.html?ref=apisyouwonthate.com). ### Response Headers For response headers, we’ve got something a little different. We’re not describing a response pattern or response schema, we’re actually going to be providing the exact response header values. ```json { "type": "object", "properties": { "id": {"type": "number"}, "name": {"type": "string"} }, "links": [{ "rel": "self", "href": "https://api.dashron.com/users/12345", "targetHints": { "transfer-encoding": ["chunked"] } }] } ``` The value of targetHints should be an object, with each key being a header, and each value being an array of header values. Header values are always arrays because HTTP allows for duplicate headers (as is commonly seen with set-cookie). The example above tells you to expect the resource to return the following response header. ``` transfer-encoding: chunked ``` ## Describing HTTP Methods You might be wondering why I left HTTP methods for last. HTTP methods are a crucial part of a REST API, but are not first class citizens in JSON Hyper-Schema. This is because, as mentioned in the first article, JSON Hyper-Schema is protocol agnostic. To simplify the process of describing HTTP methods I have listed out each HTTP method below and the associated Hyper-Schema keywords. You can usually assume that if these fields are present you can use the associated HTTP method. ### GET All an HTTP GET request needs is a URI, so you might be able to make an HTTP GET request on any LDO with an href property. ### POST The POST method is best for when you want to send arbitrary data to an endpoint. In JSON Hyper-Schema, arbitrary data is described by submissionSchema. Any LDO with a submissionSchema will likely support HTTP POST requests. ### DELETE The DELETE method only needs a URI, just like GET. You might be able to make an HTTP DELETE request to any LDO with an href property. If you want to be a little more careful about what LDOs support DELETE actions, check out the "Want to be more specific?” section below. ### PUT The PUT method is similar to POST, except it expects the request body to replace the target resource in its entirety. targetSchema describes the target resource representation, so if you see targetSchema, it might accept a PUT. ### PATCH PATCH requests are a little unique. It’s kinda like a PUT, but because you aren’t replacing the entire representation, your request body uses a special format (such as [JSON merge patch](https://tools.ietf.org/html/rfc7386?ref=apisyouwonthate.com)). To tell the client which formats are acceptable, use the accept-patch header in the targetHints object. ```json { "type": "object", "properties": { "id": {"type": "number"}, "name": {"type": "string"} }, "links": [{ "rel": "self", "href": "https://api.dashron.com/users/12345", "targetHints": { "allow": ["PATCH"], "accept-patch": ["application/merge-patch+json"] }, "targetSchema": { "name": {"type": "string"} } }] } ``` This JSON Hyper-Schema states that you can make PATCH requests with the merge-patch format, and the body can contain an id or name. Don’t worry about the allow hint just yet, I’ll explain that in more detail in the following section. ### Want to be more specific? If you don’t want to rely on assumptions, you can explicitly state which HTTP methods are supported by including the [allow header](https://tools.ietf.org/html/rfc7231?ref=apisyouwonthate.com#section-7.4.1) in the targetHints object. The allow header tells the client exactly which HTTP methods can be used with a resource. Here’s how it looks in a JSON Hyper-Schema document ```json { "type": "object", "properties": { "id": { "type": "number"}, "name": { "type": "string"} }, "links": [{ "rel": "self", "href": "https://api.dashron.com/users/12345", "targetMediaType": "application/json", "targetSchema": {"$ref": "#"}, "targetHints": { "allow": ["GET", "DELETE"] } }] } ``` ## Wrapping Up And with that last piece of the puzzle in place, you now have the details necessary to describe any basic API endpoint. - URL construction from [Part 1](https://apisyouwonthate.com/blog/getting-started-with-json-hyper-schema) of this article - Request and Response bodies with targetSchema and submissionSchema - Request and response headers with headerSchema and targetHints - HTTP methods via implied methods, or the allow header inside of targetHints But what about something more complex? Does JSON Hyper-Schema hold up to a real world API? Over the last couple of months I have found some edge cases that require a little more explanation. In my next [article](https://apisyouwonthate.com/blog/common-hypermedia-patterns-with-json-hyper-schema) I share some of those examples and explain how best to describe them with JSON Hyper-Schema. ### The OpenAPI and JSON Schema Divergence Problem URL: https://apisyouwonthate.com/blog/openapi-json-schema-divergence/ Last updated: 2024-07-03T17:53:40.000Z > **Update 2020-02-03:** [OpenAPI v3.1 will be using JSON Schema Draft 2019-09](https://apisyouwonthate.com/blog/openapi-v3-1-and-json-schema/), so tooling vendors should get to work on upgrading support for JSON Schema 2019-09 and the other OpenAPI v3.1 changes. This article is going to explain OpenAPI and JSON Schema divergence, which I’ve been calling the subset/superset/sideset problem. It’ll finish up explaining how we’re going to solve it. Whenever talking about API descriptions it is impossible to avoid mentioning OpenAPI and JSON Schema. They’re the two main solutions for any sort of API that doesn’t have a type system forcibly jammed into it by default. Often you’ll need OpenAPI for one thing, and JSON Schema for another. OpenAPI has [amazing API documentation tools](https://apisyouwonthate.com/blog/turning-contracts-into-beautiful-documentation), fancy SDK generators, and handles loads of API-specific functionality that JSON Schema doesn’t even go near. It also has a focus on keeping this static, for strictly typed languages, where properties should be 1 type and 1 type only. JSON Schema focuses on very flexible data modeling with the same sort of validation vocabulary as OpenAPI, but for more flexible data sets. Whilst it doesn’t focus just on APIs, by using more advanced vocabularies like [JSON Hyper-Schema](https://apisyouwonthate.com/blog/getting-started-with-json-hyper-schema) it can model a fully RESTful API and its hypermedia controls (HATEOAS). JSON Schema can [offer server-defined client-side validation](https://apisyouwonthate.com/blog/json-schema-client-side-validation), and a bunch of other fantastic stuff that OpenAPI doesn’t really aim to do. Over the last year I’ve been [chasing the perfect workflow](https://philsturgeon.com/my-vision-for-a-perfect-world-in-api-specification/?ref=apisyouwonthate.com), and one of my main requirements when evaluating the common API specs was JSON Schema support. The situation overall was pretty bleak, not just in supporting JSON Schema, but a lot of tooling was just... not great. Eight months after that article things are *better*, and the API design-first workflow [has been maturing](https://philsturgeon.com/api-specification-workflow-matures/?ref=apisyouwonthate.com) around me, to a point where I’m really happy about most stuff! OpenAPI is often described as an extension of JSON Schema, but both specs have changed over time and grown independently. OpenAPI v2 based based on JSON Schema draft 4 with a long list of deviations, but OpenAPI v3 shrank that list, upping their support to draft v4 and making the list of discrepancies shorter. Despite OpenAPI v3 closing the gap, the issue of JSON Schema divergence has not been resolved fully, and with newer drafts of JSON Schema coming out, the divergence is actually getting worse over time. Currently OpenAPI is still on draft 5, and JSON Schema is on draft 7. ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/posts/openapi-json-schema-divergence/json-schema-oas-differences.png) *A list of caveats to the JSON Schema support in OpenAPI v3.0.* I’ve been punting this issue for a while in my articles and recommendations at work. The hope was that by the time folks at work had upgraded to v3, there might be a v3.1 out solving the situation, but that has not come to pass. Now I find myself suggesting folks find some way to convert one to the other, or try to write JSON Schema that *is* compatible with OpenAPI. ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/posts/openapi-divergence.png) *Carefully writing JSON Schema for your data model kiiiinda works* The latter can be done, but eventually you’ll get bit by something. At work we’ve been writing JSON Schema files, using them for contract testing and a bunch of other stuff, then rendering them as part of our OpenAPI Docs with ReDoc. ReDoc will let you use `type: [string, null]`, but now we've got [Speccy](https://github.com/wework/speccy?ref=apisyouwonthate.com) linting our packages, it's reporting that as invalid OpenAPI... ``` $ speccy lint docs/openapi.ymlSpecification schema is invalid. #/paths/~1foo/post/requestBody/content/application~1json/properties/user_uuid expected Array [ 'string', 'null' ] to be a string expected Array [ 'string', 'null' ] to have type string expected 'object' to be 'string' ``` If I change that to valid OpenAPI and use `type: string` with `nullable: true` instead, validators like Speccy will be happy, but my JSON Schema contract tests will break as they no longer know that `null` is an acceptable value for that field. This error was the final straw for me. At work I have been recommending everyone enable Speccy on CircleCI to (amongst other things) make sure we’re writing valid OpenAPI, and I am failing to write valid OpenAPI in an API I manage. I’m also a little tired of explaining this awkward difference to people who would like to use some JSON Schema-based tools. After grumping at Darrel Miller (a contributor to OpenAPI) and others on the [APIs You Won’t Hate slack](https://slack.apisyouwonthate.com/?ref=apisyouwonthate.com), some good ideas started to pop up. Darrel is going to try and draft up an extension to OpenAPI that could theoretically end up in a future version — like 3.1 or 4.0: ```yaml x-oas-draft-alternate-schema: schemaType: json-schema schemaRef: ./real-json-schema.json ``` This would allow support for various JSON Schema drafts, and any other data model you can think of; including protobuf. The decisions of which data model formats to support would be in the hands of tool vendors. Of course this would increase work for these vendors, and decrease portability for a while as you can only use tools that support your alternate schema, but ultimately solve a *lot* of problems. There’s some stuff to flesh out, and obvious limitations around which schemas can `$ref` which other schemas, but there is definitely a solution here. It’ll take some time to get it done, and in that time we all need a solution. One approach would be trying to use OpenAPI for all the things, write validators and RSpec tools that do this, but we’d have to be 100% OpenAPI for everything, and we’d never get to play with client validation, Hyper-Schema, etc. Another approach would be converting our OpenAPI models to JSON Schema, but that seems a bit lossy. OpenAPI v3 is based on JSON Schema draft v5, and at time of writing JSON Schema is up to draft v8... This also adds a build step that gets in the way. If you are using JSON Schema for your contract testing, with something like [thoughtbot/json\_matchers](https://github.com/thoughtbot/json%5Fmatchers?ref=apisyouwonthate.com), you would need to edit your OpenAPI model, run the conversion, then run the tests. Or crowbar a conversion into your test suite, meaning the tool handling the conversion needs to be written in that specific language... or pipe a shell command to the CLI... AGH RUN AWAY. No, I think making JSON Schema (latest possible draft) the one and only source of truth for the data model, then "downgrading" to a flavour of JSON Schema that OpenAPI likes, is going to be the way to go. Anyway, gonna keep at this until I come up with some solutions. **Part Two:** [Problem Solved!](https://apisyouwonthate.com/blog/openapi-v3-1-and-json-schema/) ### Creating API Specifications from Bulls**t URL: https://apisyouwonthate.com/blog/creating-api-specifications-from-bullst/ Last updated: 2023-03-24T12:33:23.000Z So you want to get into design-first API development. Getting started can seem a little tricky, especially when you have a bunch of old APIs floating around and feature work keeps piling up. Writing out full specs for code that already exists can seem like a challenge, and a lot of people get tempted down the dark nonsense path of annotation-based specifications. Don’t do that to yourself, just use one of these handy tricks to catch your legacy APIs up with a bunch of specs, then your old APIs and new APIs can be on the [same workflow](https://apisyouwonthate.com/blog/weworks-api-specification-workflow), maintained with awesome [OpenAPI editors](https://openapi.tools/?ref=apisyouwonthate.com), [used for contract testing](https://apisyouwonthate.com/blog/writing-documentation-via-contract-testing), etc. #### Convert from Postman Most teams will at the very least half-assedly update a Postman Collection for their more important endpoints, and sometimes these are vaguely up-to-date. Postman Collections are essentially just another API specification format, so taking this can be a good start. Accepting that it is not going to be a complete list of endpoints, aware that it is missing all sorts of metadata like field descriptions, formats, enum values, etc., you can use this as a head start. Step 1 is to Export your Postman Collection. ![Open the Postman application, find your API and click Export.](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/posts/creating-api-specifications-from-bullst/export.png) Open the Postman application, find your API and click Export. There are a lot of "Postman to Swagger” or "Postman to OpenAPI” tools out there, but if you are luck enough to find one that works, it is definitely outdated. Many of them only work wiht Postman Collection v1, and they all only convert to OpenAPI v2.0… The best closest CLI tool out there is API Flow, but even after [sending 7 pull requests](https://github.com/luckymarmot/API-Flow/pulls?ref=apisyouwonthate.com) that tool is still OpenAPI v2.0 only, and their version is completely broken. Give [my fork](https://github.com/philsturgeon/API-Flow?ref=apisyouwonthate.com) a try if you are interested. Luckily the [APImatic Transformer](https://apimatic.io/transformer?ref=apisyouwonthate.com) is amazing. You can use their website to upload the Postman Collection, and download the OpenAPI v3.0 in JSON or YAML. Fantastic! Their output is not perfect, but it is a start. Run the [Speccy](https://github.com/wework/speccy?ref=apisyouwonthate.com) lint command on the output to get advice on things to add, and use the serve command to get a preview in your browser. #### Generate from Real Traffic From a mere chunk of JSON, a "schema generator” can attempt to make a very basic schema for you. If something looks like a string, and maybe contains a date, it can rather easily list the field as`type: string` and `format: date-time`. That might not be mind blowing, and it’s certainly not the most advanced usage of schema, but it is rather useful when you have a whole API with a few thousand fields spread over various resources. There are a few tools around for OpenAPI and JSON Schema. ***Update 2018–05–16:*** *Smartbear (creators of the Swagger tooling for OpenAPI) have just released* [*Swagger Inspector*](https://inspector.swagger.io/builder?ref=apisyouwonthate.com)*, which is explained in* [*this blog post*](https://swagger.io/blog/convert-oas-3-swagger-inspector/?ref=apisyouwonthate.com)*. You basically use it like a HTTP client, import it into SwaggerHub, then export as OpenAPI v2 or v3 and use it however you like.* If you want to create JSON Schema there are a whole bunch of [JSON Schema Generators](https://json-schema.org/implementations.html?ref=apisyouwonthate.com#schema-generation), including [one written in Ruby](https://github.com/maxlinc/json-schema-generator?ref=apisyouwonthate.com), and [one written for NodeJS](https://github.com/stoplightio/json-schema-generator?ref=apisyouwonthate.com). There is also [jsonschema.net](https://jsonschema.net/?ref=apisyouwonthate.com), which is a really handy online tool that supports loads of different drafts of JSON Schema. ![Throw JSON in on the left, get a bunch of JSON Schema out on the right.](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/posts/creating-api-specifications-from-bullst/json-schema.png) Throw JSON in on the left, get a bunch of JSON Schema out on the right. *Caveat, if you are referencing these JSON Schema files in an OpenAPI v3.0 specification, make sure your JSON Schema Generator is outputting the JSON Schema Draft 04, or it will break OpenAPI.* #### Generate Specs from Code Annotation based systems are terrifying and not something I can understand people wanting to do. That said, if you have a strictly typed language like Go, you probably have enough information in your code-base to generate some OpenAPI specs. To create a bunch of specs rather quickly, [goswagger](https://goswagger.io/generate/spec.html?ref=apisyouwonthate.com) can help you out. ``` swagger generate spec -o ./swagger.json ``` It will only return OpenAPI v2 right now, but slap the v2 spec through [APImatic Transformer](https://apimatic.io/transformer?ref=apisyouwonthate.com) and you’ll have v3 ready to go. #### Onwards When you have both your data model defined in OpenAPI or JSON Schema, and your service model defined in OpenAPI or [JSON HyperSchema](https://apisyouwonthate.com/blog/getting-started-with-json-hyper-schema), you’re off to the races! ![What is data model vs what is service model.](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/posts/creating-api-specifications-from-bullst/service-model.png) What is data model vs what is service model. Now all you need to do is save those specification files in your API’s GitHub repository, create some [beautiful documentation](https://apisyouwonthate.com/blog/turning-contracts-into-beautiful-documentation), serve them up for [client-side validation,](https://apisyouwonthate.com/blog/json-schema-client-side-validation) and all sorts of other handy stuff. ### The Many Amazing Uses of JSON Schema: Client-side Validation URL: https://apisyouwonthate.com/blog/json-schema-client-side-validation/ Last updated: 2023-03-24T12:33:35.000Z These days type systems are all the rage in APIs, and with gRPC and GraphQL fans touting their baked in systems. REST fans have a few options for type systems, but JSON Schema seems to be powering forwards as the primary candidate. JSON Schema is metadata for JSON, which can be used for a whole bunch of things. We've written about how [JSON Schema can really simplify contract testing](https://apisyouwonthate.com/blog/keeping-documentation-honest), how JSON Schema can add non-invasive [hypermedia controls to existing APIs](https://apisyouwonthate.com/blog/getting-started-with-json-hyper-schema), and - seeing as JSON Schema is [mostly compatible with OpenAPI](https://github.com/OAI/OpenAPI-Specification/blob/master/versions/3.0.1.md?ref=apisyouwonthate.com#schema-object) \- it can be used to [generate meaningful and beautiful human-readable documentation](https://apisyouwonthate.com/blog/turning-contracts-into-beautiful-documentation) too! Now let's talk about validating API requests. Again, this is another chunk taken from the book. Download the [latest version from Leanpub](https://leanpub.com/surviving-other-peoples-web-apis?ref=apisyouwonthate.com) to get this chapter and more (existing customers should have had an email about it.) Whenever an API client attempts an operation (creating a REST resource, triggering an RPC procedure, etc.) there are usually some validation rules to consider. For example, the `name` field is required and and cannot be longer than 20 characters long, `email` must be a valid email address, the `date` field should be a valid [ISO 8601](https://en.wikipedia.org/wiki/ISO%5F8601?ref=apisyouwonthate.com) date, and either the home phone or mobile phone field must be entered to send a text message, etc. There are two locations in which these rules can live, the server and the client. Client-side validation is incredibly important, as it provides immediate visual feedback to the user. Depending on the UI/UX of the client application, this might come in the form of making invalid boxes red, scrolling the user to the problem, showing basic alert boxes, or disabling the submit button until the local data looks good. ![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/posts/client-validation/strava-name-required.png) To create this functionality, a common approach is to reproduce server-side validation in the client-side. **This should be avoided at all costs.** It is awfully common for client developers to read the API documentation, take note of prominent validation rules, then write a bunch of code to handle those rules on their own end. If their code considers the data valid, it will pass it all onto the server on some event (probably on submit) and hopefully the server will agree. This seems to make sense at first, as frontend applications want immediate feedback on input without having to ask the server about validity. Unfortunately this approach does not scale particularly well, and cannot handle even the smallest evolution of functionality. Rules can be added, removed or changed for the API as the business requirements change, and clients struggle to keep up to date. Arguably the API development team should not be randomly changing things, but change always happens in some form. The [varying versioning strategies for APIs](https://apisyouwonthate.com/blog/api-versioning-has-no-right-way) aside, even extremely cautious API developers can introduce unexpectedly breaking change when it comes to validation. The most simple example would be the above mentioned `name` field, with a max length of 20 characters. Months later, requirements come in from the business to increase the max length to 40, and the API rolls out that change. (Still [not how names work](https://www.kalzumeus.com/2010/06/17/falsehoods-programmers-believe-about-names/?ref=apisyouwonthate.com), but Twitter just did this.) The API developers have a chat, and decide that making validation rules more lenient cannot be considered breaking, as all previous requests will still function. So they deploy the change with a max length of 40 characters, and one client-app deploys an update to match, increasing the validation to 40\. Despite various client applications still having the hardcoded max length at 20 characters, everyone feels pretty happy about this. One user jumps on this chance for a long name, going well over the 20 character limit. Later on the iOS application, they try to edit another field on the same form, but notice their name has been truncated to 20 characters as the input field will not take any more than that. Confused, they grab their friends phone and try it out there. The Android application does something a little different: the full 40 characters are populated into the form, but client-side validation is showing an error when the form is submitted. The user profile cannot be updated at all on this application without the user truncating their name… Well built APIs generally offer a copy of their contracts in a programmatically accessible format. The idea is that validation rules should be written down somewhere, and not just inside the backend code. If the validation rules can be seen by API clients, clients are going to be far more robust, and not break on tiny validation changes. ## JSON Schema to the Rescue JSON Schema is very simple; point out which fields might exist, which are required or optional, what data format they use. Other validation rules can be added on top of that basic premise, along with human-readable information. The metadata lives in `.json` files, which might look a bit like this: ```json { "$id": "http://example.com/schemas/user.json", "type": "object", "definitions": {}, "$schema": "https://json-schema.org/draft-07/schema#", "properties": { "name": { "title": "Name", "type": "string", "description": "Users full name supporting unicode but no emojis.", "maxLength": 20 }, "email": { "title": "Email", "description": "Like a postal address but for computers.", "type": "string", "format": "email" }, "date_of_birth": { "title": "Date Of Birth", "type": "string", "description": "Date of uses birth in the one and only date standard: ISO 8601.", "format": "date", "example": "1990–12–28" } }, "required": ["name"] } ``` There is quite a lot of stuff here, but most of it should make sense to the human eye without too much guesswork. We are listing properties by name, giving them a data type, setting up `maxLength` for the name according to the example earlier, and putting human-readable descriptions in there too. Also some examples have been thrown in for giggles. The one bit that probably needs more explaining is the $schema key, which is pointing to the draft version of JSON Schema in use. Knowing which draft you are validating against is important, as a JSON Schema file written for Draft 07 cannot be validated with a Draft 06 validator, or lower. Luckily most JSON Schema tools keep up fairly well, and JSON Schema is not madly changing crap at random. Anyway, back to it: An example of a valid instance for that `.json` schema file might look like this. ```json { "name": "Lucrezia Nethersole", "email": "l.nethersole@hotmail.com", "date_of_birth": "2007–01–23" } ``` To try playing around with this, head over to [jsonschemavalidator.net](https://www.jsonschemavalidator.net/?ref=apisyouwonthate.com) and paste those in. Removing the name field triggers an error as we have `"required": ["name"]` in there. ![screenshot showing object:name is missing](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/posts/json-schema-client-side-validation/missing-name.png) Another validation rule could be triggered if you enter date of birth in an incorrect format. ![screenshot showing '2007-01' does not valiate against format 'date'](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/posts/json-schema-client-side-validation/missing-date.png) Conceptually that probably makes enough sense, but how to actually programmatically get this done? The JSON Schema `.json` files are usually made available somewhere in a HTTP Link header with a rel of "describedby". `Link: ; rel="describedby"`. This might look a bit off to those not used to Link headers, but this is how a lot of links are handled these days. The one difficulty here is parsing the value, which can be done with some extremely awful regex, or with a [RFC5988](https://tools.ietf.org/html/rfc5988?ref=apisyouwonthate.com) compliant link parser — like [parse-link-header](https://github.com/thlorenz/parse-link-header?ref=apisyouwonthate.com) for JavaScript. Once a API client has the schema URL they can download the file. This involves simply making a GET request to the URL provided. Fear not about performance, these are usually stored on CDNs, like S3 with CloudFlare in front of it. They are also likely to have cache headers set, so make sure your HTTP client knows how to handle that. *Learn about [HTTP caching for clients](https://apisyouwonthate.com/blog/http-client-response-caching).* Triggering validation rules on a random website is one thing, but learning how to do that with code is going to be far more useful. For JavaScript a module called [ajv](https://www.npmjs.com/package/ajv?ref=apisyouwonthate.com) is fairly popular, so install that with a simple `yarn add ajv@6`, then shove it in a JavaScript file. This code is available on [the new books GitHub Repo](https://github.com/apisyouwonthate/surviving-other-peoples-apis-code?ref=apisyouwonthate.com). ```javascript const Ajv = require('ajv'); const ajv = new Ajv(); // Fetch the JSON content, pretending it was downloaded from a URL const userSchema = require('./cached-schema.json') // Make a little helper for validating function validate(schema, data) { return ajv.validate(schema, data) ? true : ajv.errors; } // Pretend we've submitted a form const input = { name: "Lucrezia Nethersole", email: "l.nethersole@hotmail.com", registered\_date: "2007-01-23T23:01:32Z" } // Is the whole input valid? console.log('valid', validate(userSchema, input)) // true // Ok screw up validation... input['email'] = 123 console.log('fail', validate(userSchema, input)) // [ { keyword: 'type', dataPath: '.email', ... ``` For the sake of keeping the example short, the actual JSON Schema has been "downloaded" from `http://example.com/schemas/user.json` and put into a local file. This is not quite how you would normally do things, and it will become clear why in a moment. A `validation()` function is created to wrap the validation logic in a simple helper, then we move on to pretending we have some input. The input would realistically probably be pulled from a form or another dynamic source, so use your imagination there. Finally onto the meat, calling the validation, and triggering errors. Calling this script should show the first validation to succeed, and the second should fail with an array of errors. ```bash $ node ./1-simple.js true [ { keyword: 'type', dataPath: '.email', schemaPath: '#/properties/email/type', params: { type: 'string' }, message: 'should be string' } ] ``` At first this may seem like a pile of unusable gibberish, but it is actually incredibly useful. How? The `dataPath` by default uses JavaScript property access notation, so you can easily write a bit of code that figures out the input.email was the problem. That said, [JSON Pointers](https://tools.ietf.org/html/rfc6901?ref=apisyouwonthate.com) might be a better idea. A much larger example, again available on Github, will show how JSON Pointers can be used to create dynamic errors. Sadly a lot of this example is going to be specific to AJV, but the concepts should translate to any JSON Schema validator out there. ```javascript const Ajv = require('ajv'); const ajv = new Ajv({ jsonPointers: true }); const pointer = require('json-pointer'); const userSchema = require('./cached-schema.json') function validate(schema, data) { return ajv.validate(schema, data) ? [] : ajv.errors; } function buildHumanErrors(errors) { return errors.map(function(error) { if (error.params.missingProperty) { const property = pointer.get(userSchema, '/properties/' + error.params.missingProperty); return property.title + ' is a required field'; } const property = pointer.get(userSchema, '/properties' + error.dataPath); if (error.keyword == 'format' && property.example) { return property.title + ' is in an invalid format, e.g: ' + property.example; } return property.title + ' ' + error.message; }); } ``` The important things to note in this example are the `new Ajv({ jsonPointers: true });` property, which makes `dataPath` return a JSON Path instead of dot notation stuff. Then we use that pointer to look into the schema objects (using the [json-pointer](https://www.npmjs.com/package/json-pointer?ref=apisyouwonthate.com) npm package), and find the relevant property object. From there we now have access to the human readable title, and we can build out some human readable errors based off of the various properties returned. This code might be a little odd looking, but we support a few types of error quite nicely. Consider the following inputs. ```javascript [ { }, { name: "Lucrezia Nethersole", email: "not-an-email" }, { name: "Lucrezia Nethersole", date_of_birth: 'n/a' }, { name: "Lucrezia Nethersole Has Many Many Names" } ].forEach(function(input) { console.log( buildHumanErrors(validate(userSchema, input)) ); }); ``` These inputs give us a whole bunch of useful human errors back, that can be placed into our UI to explain to users that stuff is no good. ```bash $ node 2-useful-errors.js [ 'Name is a required field' ] [ 'Email should match format "email"' ] [ 'Date Of Birth is in an invalid format, e.g: 1990–12–28' ] [ 'Name should NOT be longer than 20 characters' ] ``` The errors we built from the JSON Schema using the properties that exist can get really intelligent depending on how good the schema files are, and how many edge cases you cover. Putting the examples in is a really nice little touch, and makes a lot more sense to folks reading the messages than just saying the rather vague statement "it should be a date". If you were to instead find a way to tie these back to the DOM, you could update your forms with visual updates as discussed earlier: making invalid boxes red, scroll the user to the problem, show basic alert boxes, or disable the submit button until the local data looks good! ## What about Validation Hell? Earlier validation hell was mentioned, and JSON Schema is supposed to avoid it. But how? The API client now has this JSON Schema file locally, and if the server changes… how does it know? This sample code storing the schema in the repo along with the source code, which — generally speaking — is a pretty bad idea, only done for simplicity of the example. Put very simply, if the API developers change the schema file to have a `maxLength` of 40, any client should then get that change the next time they request the schema file, meaning things are not out of sync. ## Almost That is a fluffy simplicity which has a few real-world caveats to explain… Consider schema can be found on the HTTP response like this: ```http Link: ; rel="describedby" ``` This URL is not versioned, which suggests that it might change. This is… possibly ok, as long as they have not set a long cache expiry. If a client application is respecting cache headers, and the schema file has cache headers, then your application could suffer from validation hell for the duration of that expiry. If the cache is only set to something short like 5 minutes, and the change is only a minor one, that honestly might not be too bad. The whole "multiple devices being used to try to make profile changes and getting clobbered by a `maxLength` change" scenario we have been discussing actually is not ideal when you need developers to rush in and fix it, but not so bad if it'll fix itself after a few minutes. Some APIs version their schema files, and as such new versions should be published as a new URL. ```http Link: ; rel="describedby" ``` When a minor change is released like the `maxLength` one, API developers may well release another version. ```http Link: ; rel="describedby" ``` So long as URLs are not hardcoded in your application, and the URL is being read from the response (taken from wherever the API provides the link: body or link header), then the change of URL will automatically cause your application to fetch the new schema, allowing your application to notice the new validation essentially immediately. This is yet another amazing thing JSON Schema can do for your HTTP APIs. Keep an eye out for more articles showing amazing uses of JSON Schema, such as dynamic form generation with tools like [React JSON Schema Form](https://mozilla-services.github.io/react-jsonschema-form/?ref=apisyouwonthate.com). ### Understanding RPC, REST and GraphQL URL: https://apisyouwonthate.com/blog/understanding-rpc-rest-and-graphql/ Last updated: 2023-03-24T12:33:45.000Z Every API in the world is following some sort of paradigm, whether it knows it or not. They will fall under RPC, REST, or a "query language." Even if you are confident you understand the difference, do yourself a favor and read them anyway. About 99% of people get this wrong, so you can be in the top 1% with a quick read. ## Remote Procedure Call (RPC) RPC is the earliest, simplest form of API interaction. It is about executing a block of code on another server, and when implemented in HTTP or AMQP it can become a Web API. There is a method and some arguments, and that is pretty much it. Think of it like calling a function in JavaScript, taking a method name and arguments. For example: ```http POST /sayHello HTTP/1.1 HOST: api.example.com Content-Type: application/json {"name": "Racey McRacerson"} ``` In JavaScript, we would do the same by defining a function, and later we’d call it elsewhere: ```js /* Signature */ function sayHello(name) { // ... } /* Usage */ sayHello("Racey McRacerson"); ``` The idea is the same. An API is built by defining public methods; then, the methods are called with arguments. RPC is just a bunch of functions, but in the context of an HTTP API, that entails putting the method in the URL and the arguments in the query string or body. When used for CRUD, RPC is just a case of sending up and down data fields, which is fine, but one downside is that the client is entirely in charge of pretty much everything. The client must know which methods (endpoints) to hit at what time, in order to construct its own workflow out of otherwise naive and non-descriptive endpoints. RPC is merely a concept, but that concept has a lot of specifications, all of which have concrete implementations: - [XML-RPC](https://en.wikipedia.org/wiki/XML-RPC?ref=apisyouwonthate.com) - [JSON-RPC](https://en.wikipedia.org/wiki/JSON-RPC?ref=apisyouwonthate.com) - [Simple Object Access Protocol (SOAP)](https://en.wikipedia.org/wiki/SOAP?ref=apisyouwonthate.com) XML-RPC and JSON-RPC are not used all that much other than by a minority of entrenched fanatics, but SOAP is still kicking around for a lot of financial services and corporate systems like [Salesforce](https://developer.salesforce.com/docs/atlas.en-us.api.meta/api/sforce%5Fapi%5Fquickstart%5Fintro.htm?ref=apisyouwonthate.com). XML-RPC was problematic, because ensuring data types of XML payloads is tough. In XML, a lot of things are just strings, which JSON does improve, but has trouble differentiating different data formats like integers and decimals. You need to layer metadata on top in order to describe things such as which fields correspond to which data types. This became part of the basis for SOAP, which used [XML Schema](https://wikipedia.org/wiki/XML%5Fschema?ref=apisyouwonthate.com) and a [Web Services Description Language (WSDL)](https://wikipedia.org/wiki/Web%5FServices%5FDescription%5FLanguage?ref=apisyouwonthate.com) to explain what went where and what it contained. This metadata is essentially what most science teachers drill into you from a young age: "label your units!" The sort of thing that stops people paying $100 for something that should have been $1 but was just marked as "price: 100" which was meant to be cents… It is also worth pointing out if your "distance" field is metric or imperial, to avoid bad math crashing your billion dollar satellite into Mars. A modern RPC implementation is [gRPC](https://grpc.io/?ref=apisyouwonthate.com), which can easily be considered modern (and drastically better) SOAP. It uses a data format called [ProtoBuff](https://developers.google.com/protocol-buffers/?ref=apisyouwonthate.com), which requires a schema as well as the data instance, much like the WSDL in SOAP. GRPC focuses on making single interactions as quick as possible, thanks to HTTP/2, and the fact that Protobuff packs down smaller than JSON, but JSON can also be used easily enough. ## Representational State Transfer (REST) REST is a network paradigm described by Roy Fielding in a [dissertation](https://www.ics.uci.edu/~fielding/pubs/dissertation/top.htm?ref=apisyouwonthate.com) in 2000\. REST is all about a client-server relationship, where server-side data are made available through representations of data in simple formats. This format is usually JSON or XML but could be anything. These representations portray data from various sources as simple "resources", or "collections" of resources, which are then potentially modifiable with actions and relationships being made discoverable via a concept known as hypermedia controls (HATEOAS). Hypermedia is fundamental to REST, and is essentially just the concept of providing "next available actions", which could be related data, or in the example of an "Invoice" resource, it might be a link to a "Payment Attempts" collection so that the client can attempt paying the invoice. These actions are just links, but the idea is the client knows that an invoice is payable by the presence of a "pay" link, and if that link is not there it should not show that option to the end user. ```json { "data"": { "type": "invoice", "id": "093b941d", "attributes": { "created_at": "2017–06–15 12:31:01Z", "sent_at": "2017–06–15 12:34:29Z", "paid_at": "2017–06–16 09:05:00Z", "status": "published" } }, "links": { "pay": "https://api.acme.com/invoices/093b941d/payment_attempts" } } ``` This is quite different to RPC. Imagine the two approaches were humans answering the phones for a doctors office: > **Client:** Hi, I would like to speak to Dr Watson, is he there? > > **RPC:** No. \*click\* > > \*Client calls back\* > > **Client:** I checked his calendar, and it looks like he is off for the day. I would like to visit another doctor, and it looks like Dr Jones is available at 3pm, can I see her then? > > **RPC:** Yes The burden of knowing what to do is entirely on the client. It needs to know all the data, come to the appropriate conclusion itself, then has to figure out what to do next. REST however presents you with the next available options: > **Client:** Hi, I would like to speak to Dr Watson, is he there? > > **REST:** Doctor Watson is not currently in the office, he’ll be back tomorrow, but you have a few options. If it’s not urgent you could leave a message and I’ll get it to him tomorrow, or I can book you with another doctor, would you like to hear who is available today? > > **Client:** Yes, please let me know who is there! > > REST: Doctors Smith and Jones, here are links to their profiles. > > **Client:** Ok, Doctor Jones looks like my sort of Doctor, I would like to see them, let’s make that appointment. > > **REST:** Appointment created, here’s a link to the appointment details. REST provided all of the relevant information with the response, and the client was able to pick through the options to resolve the situation. Of course REST would needed to know to follow the `"alternative_doctors": "https://api.example.com/available_doctors?available_at=2017-01-01 03:00:00 GMT"` link, but that is far less of a burden on the client than forcing it to check the calendar itself, seek for availability, etc. This centralization of state into the server has benefits for systems with multiple different clients who offer similar workflows. Instead of distributing all the logic, checking data fields, showing lists of "Actions", etc. around various clients — who might come to different conclusions — REST keeps it all in one place. Other than hypermedia (the [most powerful yet most ignored aspect of REST](https://apisyouwonthate.com/blog/representing-state-in-rest-and-graphql)) there are a few other requirements for a system to be a REST API: - REST must be stateless: not persisting sessions between requests - Responses should declare cacheablility: helps your API scale if clients respect the rules - REST focuses on uniformity: if you’re using HTTP you should utilize HTTP features whenever possible, instead of inventing conventions The goal of these constraints is to make the REST architecture help APIs last for decades, which is almost impossible to do without these concepts. REST also does not require the use of schema metadata, which many API developers hated in SOAP. For a long time nobody was building REST APIs with schema, but these days it is far more common thanks to [JSON Schema](https://json-schema.org/?ref=apisyouwonthate.com). JSON Schema is inspired by XML Schema — but not functionally identical — and is one of the most important things to happen to HTTP APIs in years, and will be mentioned a lot in further articles. Unfortunately, REST become a marketing buzzword for most of 2006–2014\. It became a metric of quality that developers would aspire to, fail to understand, then label as REST anyway, so most systems saying they’re REST are little more than RPC with HTTP verbs and pretty URLs. As such, you might not get cacheability provided, it might have a bunch of wacky conventions, and there might not be any links for you to use to discover next available actions. These APIs are jokingly called REST\_ish\_ by people aware of the difference. On the flip side, a REST API can be used in an RPC fashion if you as the client developer chose to ignore the links. It is not advisable of course, but it is possible. A huge source of confusion for people with REST is that they do not understand "all the extra faffing about", such as hypermedia controls and HTTP caching. They do not see the point, and many consider RPC to be the almighty. To them, it is all about executing the remote code as fast possible, but REST (which can still absolutely be performant) focuses far more on longevity and reduced client-coupling. REST can theoretically work in any transportation protocol that provides it the ability to fulfill the constraints, but no transportation protocol other than HTTP has the functionality. To fit REST into AMQP you would need to define hypermedia controls somehow (potentially an array of messages you could call next), a standard for declaring cacheability of the AMQP messages, etc., and create a lot of tooling that does not exist. Basically REST is too powerful for other existing transportation protocols, so it is generally implemented in HTTP. REST has no specification which is what leads to some of this confusion, nor does it have concrete implementations. That said, there are two large popular specifications which provide a whole lot of standardization for REST APIs that chose to use them: - [OData](https://www.odata.org//?ref=apisyouwonthate.com) - [JSON-API](https://jsonapi.org//?ref=apisyouwonthate.com) If the API advertises itself as using these, there is a chance it is a good one. Find a OData client or a JSON-API client in your programming language to save yourself some work. Otherwise go at it yourself with a plain-old HTTP client and you should be ok with a little bit of elbow grease. ## GraphQL Listing [GraphQL](https://graphql.org/?ref=apisyouwonthate.com) as a direct comparison to these other two concepts is a little odd, as GraphQL is essentially RPC, with a lot of good ideas from the REST/HTTP community tacked in. Still, it is one of the fastest growing API ecosystems out there, mostly due to some of the confusion outlined above. GraphQL is basically RPC with a default procedure providing a query language, a little like SQL — if that is something you are familiar with. You ask for specific resources and specific fields, and it will return that data in the response. ![Screenshot of a graphql request and its response](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/posts/understanding-rpc-rest-and-graphql/graphql.png) GraphQL Request (left) and the corresponding response (right) It has [Mutations](https://graphql.org/learn/queries/?ref=apisyouwonthate.com#mutations) for creates, updates, deletes, etc. and again they are exactly RPC. ![Definition of a mutation (top left), the mutation request (bottom left), and the response (right).](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/posts/understanding-rpc-rest-and-graphql/graphql2.png) Definition of a mutation (top left), the mutation request (bottom left), and the response (right). GraphQL has many fantastic features and benefits, which are all bundled in one package, with a nice marketing site. If you are trying to learn how to make calls to a GraphQL API, the [Learn GraphQL](https://graphql.org/learn/?ref=apisyouwonthate.com) documentation will help, and their site has a bunch of other resources. Seeing as GraphQL was built by Facebook, who had previously built a REST\_ish\_ API, they’re familiar with various REST/HTTP API concepts. Many of those existing concepts were used as inspiration for GraphQL functionality, or carbon copied straight into GraphQL. Sadly a few of the most powerful REST concepts were completely ignored. The backstory to GraphQL, is an interesting one. Facebook has experimented with various different approaches to sharing all their data between apps over the years; remember FQL? Executing SQL-like syntax over a GET endpoint was a bit odd. ``` GET /fql?q=SELECT%2Buid2%2BFROM%2Bfriend%2BWHERE%2Buid1%3Dme()&access\_token=… ``` Facebook got a bit fed up with having a REST\_ish\_ approach to get data, and then having the FQL approach for more targeted queries as well, as they both require different code. As such, GraphQL was created as a middle-ground between endpoint-based APIs and FQL, the latter being an approach most API developers would never consider — or want. In the end, they developed this RPC-style query language system, to ignore most of the transportation layer, meaning they had full control over the concepts. Endpoints are gone, resources declaring their own cacheability is gone, the concept of the uniform interface (as REST defines it) is obliterated, which has the supposed benefit of making GraphQL so incredibly simple it could fit into AMQP or any other transportation protocol. The main selling point of GraphQL is that it defaults to providing the very smallest response from an API, as you are requesting only the specific bits of data that you want, which minimizes the Content Download portion of the HTTP request. It also reduces the number of HTTP requests necessary to retrieve data for multiple resources, known as the "HTTP N+1 Problem" that has been a problem for API developers through the lifetime of HTTP/1.1, but thankfully was [solved quite nicely in HTTP/2](https://apisyouwonthate.com/blog/lets-stop-building-apis-around-a-network-hack). The rest of the chapter, and the rest of the book, will be talking about various pros and cons of these approaches, such as how [HTTP caching really suffers](https://apisyouwonthate.com/blog/graphql-vs-rest-caching) in GraphQL compared to other REST/RESTish approaches, how [Representing State](https://apisyouwonthate.com/blog/representing-state-in-rest-and-graphql) can be tough, and plenty of other things. All this in [*Build APIs You Won’t Hate: Second Edition*](https://leanpub.com/build-apis-you-wont-hate-2/?ref=apisyouwonthate.com), currently available for pre-order with the early chapters available for download. ***Update 2018–06–01:*** *With this article only covering the technical differences, some folks thought it was about trying to establish which is "the best", instead of trying to be a helpful resource in getting folks thinking about how these different paradigms work. For help picking the right paradigm for any give task, check out* [***Picking the right Paradigm***](https://apisyouwonthate.com/blog/picking-api-paradigm)*.* ### Getting started with JSON Hyper-Schema URL: https://apisyouwonthate.com/blog/getting-started-with-json-hyper-schema/ Last updated: 2023-03-24T12:33:57.000Z *Guest post from [Aaron Hedges](https://twitter.com/Dashron?ref=apisyouwonthate.com).* JSON Hyper-Schema is a format for describing an API. You can automate a lot with a document that describes your API. - API SDKs - Documentation - Tools such as API sandboxes or Postman collections to simplify the development process - Contract tests to ensure your API doesn’t change unexpectedly - Mock API servers to test your design before building the API OpenAPI is another API description format, but whereas OpenAPI usually remains content with server-based uses, clients can use JSON Hyper-Schema directly. A client can use this schema to build their API requests including URLs, query parameters, and the request body. When an API changes, they simply have to update the schema and all clients will adapt accordingly. The client doesn’t have to write a single line of code. In this article I’m going to show some JSON Hyper-Schema examples and walk you through what makes them tick. This information should give you a foundation to continue learning about JSON Hyper-Schema. In future articles I will expand on that foundation and help you describe your entire API. But before we can discuss JSON Hyper-Schema, you need a basic understanding of JSON schema. ## **JSON Schema** JSON Schema describes JSON data. It’s like a database schema for JSON and can be used to validate a JSON instance before it’s sent to an API. Here is an example schema for a blog post ``` { "type": "object", "properties": { "id": { "type": "number" }, "title": { "type": "string" }, "urlSlug": { "type": "string" }, "body": { "type": "string" } }, "required": ["id"] } ``` And here is JSON data that matches that schema ``` { "id": 5, "title": "JSON Hyper-Schema", "urlSlug": "json-hyper-schema", "body": "My long post about JSON Hyper-Schema..." } ``` Let’s go through each property in the schema ### **type** type describes the data type such as string, object or number. You can find the whole list [here](https://json-schema.org/latest/json-schema-validation.html?ref=apisyouwonthate.com#rfc.section.6.1). Each type has its own set of [validation keywords](https://json-schema.org/latest/json-schema-validation.html?ref=apisyouwonthate.com#rfc.section.6) that make up the rest of the schema. We’ll describe two validation keywords below. ### **properties** properties is validation keyword for JSON objects. This properties object defines each valid property along with an additional level of validation schema. If one of the properties is an object, you can continue to nest that representation as deep as necessary. ### **required** The required property is another validation keyword specific to objects. The value of required should be an array of strings, where each string is a key in the properties object. JSON data is not valid against this schema if any of the properties in the required array are missing. So using the example above, the following JSON data is valid because the title and body are optional ``` { "id": 5 } ``` And this JSON data is not valid because id is required ``` { "title": "JSON Hyper-Schema" } ``` Once your API consumers have a JSON schema, they can use one of the many [JSON Schema libraries](https://json-schema.org/implementations.html?ref=apisyouwonthate.com) to validate their JSON data. ## **JSON Hyper-Schema** Now that we have a foundation of JSON Schema, let’s add the "Hyper” part. JSON Schema was built with expansion in mind, via a system called vocabularies. The spec defines this well, so here’s a quote. > A JSON Schema vocabulary is a set of keywords defined for a particular purpose. [JSON schema \[Section\] 4.3.2](https://tools.ietf.org/html/draft-handrews-json-schema-00?ref=apisyouwonthate.com#section-4.3.2) [JSON Schema core](https://json-schema.org/latest/json-schema-core.html?ref=apisyouwonthate.com) defines the general vocabulary. In the JSON Schema example above I described part of the validation vocabulary. In the following section I will describe part of the hypermedia vocabulary, which include everything needed to describe an API request. Let’s start with an example. Here’s a JSON Hyper-Schema document for a blog post. ```json { "type": "object", "properties": { "id": { "type": "number" }, "title": { "type": "string" }, "urlSlug": { "type": "string" }, "post": { "type": "string" } }, "required": ["id"], "base": "[http://api.dashron.com/](http://api.dashron.com/)", "links": [{ "rel": "self", "href": "posts/{id}", "templateRequired": ["id"] }] } ``` This hyper-schema starts with the blog post schema from earlier and adds two new new properties. These new properties are base and links. ### **base** base is similar to the HTML [](https://developer.mozilla.org/en-US/docs/Web/HTML/Element/base?ref=apisyouwonthate.com) tag. The value should be a URI, which becomes the root of any relative URI in the JSON instance. For example, a base of [https://dashron.com/api](https://dashron.com/api?ref=apisyouwonthate.com) would be combined with the relative URI of users and become [http://dashron.com/api/users](http://dashron.com/api/users?ref=apisyouwonthate.com). ### **links** links is an array of Link Description Objects (LDOs) as described below. ## **Link Description Object (LDO)** An LDO contains all the information necessary to describe the actions available to your client. Here’s the LDO from the previous example. ```js { "rel": "self", "href": "posts/{id}", "templateRequired": ["id"] } ``` You might be expecting to see things like method or requestBody for HTTP requests, but you won’t find them. JSON Hyper-Schema doesn’t limit itself to any one protocol. JSON fans can use JSON Hyper-Schema to describe a file system as easily as an HTTP API. Let’s dig into each property of the LDO above. ### **rel** rel is a string that describes how the LDO relates to the JSON. The LDO in this example uses the relation self. Because this schema describes a blog post, the self relation describes how you access that blog post. There are many different predefined relations you can use to describe your LDOs. Start by checking if your relation is defined in [this list](https://www.iana.org/assignments/link-relations/link-relations.xhtml?ref=apisyouwonthate.com). If you can’t find what you need, JSON Hyper-Schema [recommends](https://tools.ietf.org/html/rfc8288?ref=apisyouwonthate.com#section-2.1.2) you use a URI as your relation. If you need your relation to be human readable, check out the [tag URI scheme](https://tools.ietf.org/html/rfc4151?ref=apisyouwonthate.com) for more flexibility. Before we move on it’s important to point out that no two LDOs should share a relation. The relation is the defining information of an LDO and should be unique across every LDO. ### **href** href is a [URI template](https://tools.ietf.org/html/rfc6570?ref=apisyouwonthate.com). URI templates let you describe the structure of your URLs. ```js "href": "posts/{id}" ``` JSON Hyper-Schema builds on top of URI templates by explaining how to populate the URI template variables. ### **Populating URI templates with JSON data** In the href example earlier, JSON Hyper-Schema assumes the `{id}` variable lines up with the id property in your JSON schema. So our blog post has an id of 5 and our final href would be /posts/5. When all the variables in the template are populated, you check to see if the URI is a relative or absolute URI. Absolute URIs start with a URI scheme, such as http:// and should be considered complete URIs. Relative URIs do not start with a scheme, and should be resolved according to [RFC 3986](https://tools.ietf.org/html/rfc3986?ref=apisyouwonthate.com#section-5). In this case because our base URI ends with a / and our relative URI does not start with a /, we can simply append the relative URI onto the base, giving us [https://api.dashron.com/posts/5](https://api.dashron.com/posts/5?ref=apisyouwonthate.com). If your URI variables don’t line up with your JSON properties you can use templatePointers. This object allows you to map URI template variables to a part of your JSON instance. In the following example the pointer indicates that the id variable should use the urlSlug property. ```js "href": "/posts/{id}", "templatePointers": { "id": "/urlSlug" } ``` A template pointer is a [absolute](https://tools.ietf.org/html/rfc6901?ref=apisyouwonthate.com) or [relative](https://json-schema.org/latest/relative-json-pointer.html?ref=apisyouwonthate.com) JSON pointer. JSON Pointers allow you to refer directly to sections of a JSON document, but are out of the scope of this article. I recommend learning more when you have the time! But what if urlSlug is not in your JSON? We won’t have this problem with id because it is a required field, but urlSlug is optional. When a property is missing JSON schema assumes the value is an empty string. In the above example our fully resolved url would become [https://api.dashron.com/posts/](https://api.dashron.com/posts/?ref=apisyouwonthate.com), which isn’t the URL we are describing. To avoid this problem use templateRequired. ``` "templateRequired": ["id"] ``` templateRequired is an array of strings, where each string is a mandatory URI template variable. If the JSON Hyper-Schema library can not fill the required URI template variables, the LDO is considered invalid and must be ignored by the client. ### **Populating URI templates with outside input** Instead of populating the URI from the JSON you can ask for outside input such as user submitted data. To do this, add the hrefSchema property to define a validation schema for any outside variables. Using the blog post example again, let’s say we want the mobile app to prompt the user for a blog id. Your hrefSchema would look like this. ```js "href": "/posts/{id}", "hrefSchema": { "id": { "type": "number" } } ``` In this example I copied the id schema from earlier in the document. It’s hard to maintain duplicate data, so I’m going to introduce the final concept of this article. To ensure we have a single source of truth for our id schema we can use schema references. ### Schema References Schema References are JSON objects that allow you to point to a schema. The schema could be this schema, an entirely different schema, or a portion of a schema. These objects will always contain a single property, $ref with a URI value. Let’s start with the simplest example, a reference to a different schema. The following schema reference points outside of our schema to another document. That document is identified by the URI [https://schemas.dashron.com/users](https://schemas.dashron.com/users?ref=apisyouwonthate.com). ```js { "$ref": "https://schemas.dashron.com/users" } Let’s assume that the current schema is identified by https://schemas.dashron.com/posts. We can reference the id property of the current schema by including a JSON pointer in the fragment portion of the URI. ```js { "$ref": "https://schemas.dashron.com/posts#/properties/id" } ``` If you plan on referring to the current schema, you can leave out everything but the fragment portion of the URI. The above schema can be simplified to the following example. ```js { "$ref": "#/properties/id" } ``` And if you want to refer directly to the entire current schema, you can leave off the JSON pointer. ```js { "$ref": "#" } ``` This self-referencing schema reference is the most common schema reference. Now that I’ve covered schema references, let’s clean up that hrefSchema example. The example below points to the id property of the current schema instead of duplicating it. ```js "href": "/posts/{id}", "hrefSchema": {"$ref": "#/properties/id"} ``` ### What’s next? With this information, you can start writing schemas for all the JSON in your API, along with their hypermedia links. In the [next article](https://apisyouwonthate.com/blog/getting-started-with-json-hyper-schema-part-2) I cover request and response bodies. In the meanwhile try building a JSON Hyper-Schema for one of your API endpoints, and validating your JSON with one of the many [JSON Schema libraries](https://json-schema.org/implementations.html?ref=apisyouwonthate.com). *This is part one of a three-parter. Check out [part two](https://apisyouwonthate.com/blog/getting-started-with-json-hyper-schema-part-2).* ### Ruby Users: Be Wary of Net::HTTP URL: https://apisyouwonthate.com/blog/ruby-users-be-wary-of-net-http/ Last updated: 2023-03-24T12:34:09.000Z A few months ago we released [We::Call](https://github.com/wework/we-call-gem?ref=apisyouwonthate.com), a Ruby gem that wraps popular HTTP-client [Faraday](https://github.com/lostisland/faraday/?ref=apisyouwonthate.com). We::Call aims to configure Faraday with as many best practices for HTTP communications by default as it reasonably can. It’s come a long way over the last few development versions, most of which revolves around removing the deprecation logic, putting that functionality in a more "Rails-like” gem [rails-sunset](https://github.com/wework/rails-sunset?ref=apisyouwonthate.com), and removing Rails as a dependency altogether. Something else we’ve done, is force the [use of timeouts](https://github.com/wework/we-call-gem?ref=apisyouwonthate.com#timeouts). If you don’t define a timeout upon creating the connection, We::Call will raise an exception. This had the knock-on effect of demanding the entire dev team to consider timeouts, which has in turn improved performance throughout our architecture. Timeouts are another article entirely, but fail fast, fail often. This leads us to the matter at hand: we noticed that timeouts were taking twice as long as they should. If we set `We::Call::Connection.new(…, timeout: 5)` then it would take 10 seconds. This was having awful affects downstream as you can imagine, and the thread being blocked for that long was causing awful backups in the request queues! Somebody suggested maybe We::Call was failing to pass the timeout to Faraday correctly, and 10s was the server just timing out… This seemed reasonable, so I immediately started debugging. After some debugging, I noticed a timeout of 1 would take 2s, 2s would take 4s, 4s would take 8s… WHAT?! Exasperated, I think I dumped some ridiculous search query into Google like: > why are Faraday timeouts doubled Amazingly, I found an answer. [**timeout is always doubled · Issue #612 · lostisland/faraday**](https://github.com/lostisland/faraday/issues/612?ref=apisyouwonthate.com) > \_I guess this happens because it sets open and read timeout to the same value ... but it's kind of irritating : Not a fault of `We::Call` or Faraday at all, but the default adapter it uses as the guts for its operations: [Net::HTTP](http://ruby-doc.org/stdlib-2.4.2/libdoc/net/http/rdoc/Net/HTTP.html?ref=apisyouwonthate.com). This is clearly used by Faraday as the default because it comes baked into Ruby. Back in 2012 [a commit was made](https://github.com/ruby/ruby/commit/bee7ccddd254c7b6e781f5ea9bbf651b50b590ee?ref=apisyouwonthate.com) that ended up in v2.0.0, to automatically retry any request made that was idempotent: ``` IDEMPOTENT_METHODS_ = %w/GET HEAD PUT DELETE OPTIONS TRACE/ ``` This is theoretically a great idea, but it ends up creating a huge mystery for anyone who is not expecting it. Some of the most experienced developers at this company had no idea it was happening, so how can we expect anyone else to know?! It’s not like we’re just an unfortunate collection of daft people. This advice is everywhere, like in [Common Patterns > Retry on Failure](https://github.com/jordansissel/software-patterns/tree/master/retry-on-failure/ruby?ref=apisyouwonthate.com), a gist that appears rather high on certain search terms: response = Net::HTTP.get\_response("google.com", "/") rescue retry This works, but it’s going to be making two attempts before that `rescue retry` is even hit. 😅 NewRelic does not report this as 2x requests either, just shows one NetHTTP call that measures twice as long. Maybe the NewRelic reporting could be improved, or maybe its opaque to their Ruby agent, but having this act as an uncontrollable default is terrifying. The pull request to make it configurable [has been merged](https://github.com/mrkn/ruby/commit/439550a0692518bb2827ee178f2207d0687dc3c8?ref=apisyouwonthate.com) and will be landing in Ruby v2.5.0 when it’s released. It keeps the default behavior unchanged, which will continue to confuse people, but at least it will allow Faraday to implement control over it in their adapter. They’ll probably default to 0 retries then using their `faraday.request :retry, max: 1` you could increase that optionally. Waiting for every application to get to Ruby v2.5.0 didn’t seem like a solution to this problem, so I started looking into alternative adapters that Faraday can use (so we can continue using all of our awesome middlewares), and typheous popped up as a strong contender. #### Typhoeus Is Fantastic [Typhoeus](https://github.com/typhoeus/typhoeus?ref=apisyouwonthate.com) is a popular choice for those wanting to make async calls, but are not able to upgrade their architecture to HTTP/2 yet for various reasons. I’m still pushing for more HTTP/2 to help [improve and simplify our API designs](https://apisyouwonthate.com/blog/lets-stop-building-apis-around-a-network-hack), but in the meantime letting folks use typheous async stuff seems like a great move. Not only does typheous’ async functionality mean multiple things can be done faster (making a lot of calls async is [a lot faster](https://philsturgeon.uk/php/2013/11/12/benchmarking-codswallop-nodejs-v-php/?ref=apisyouwonthate.com) than making a lot of calls in serial), but making bog-standard calls with typheous is a lot faster too. Typhoeus wraps [ethon](https://github.com/typhoeus/ethon?ref=apisyouwonthate.com), which is a low-level wrapper around libcurl. Having libcurl as a base gets us HTTP/1.1 KeepAlive support, which [drastically improves the time](https://blog.insightdatascience.com/learning-about-the-http-connection-keep-alive-header-7ebe0efa209d?ref=apisyouwonthate.com) it takes for a HTTP connection to be created. [Benchmarking this was pretty simple](https://gist.github.com/philsturgeon/f4003f5ae14ca76665d8e002824a50aa?ref=apisyouwonthate.com). Let’s create 100 realistic POST requests to ensure no caching is happening, and don’t use any features like built-in JSON serialization or async functionality. ``` user system total real net\_http: 0.910000 0.190000 1.100000 ( 9.382847) net\_http\_persistent: 0.980000 0.170000 1.150000 ( 9.245193) patron: 0.150000 0.110000 0.260000 ( 2.233322) httpclient: 0.150000 0.100000 0.250000 ( 2.142556) typhoeus: 0.120000 0.070000 0.190000 ( 2.138841) ``` Patron, httpclient and typhoeus all wrap libcurl, and they switched being fastest by `.1` a few times. In the end it was a clear and easy choice to go with typhoeus due to the similarly amazing performance boost, plus the async stuff. We trialed Typhoeus on a system that gets plenty of hits, and operates as a evaluator for remote data sources. It was the perfect system to trial it on, as making HTTP requests is pretty much all it does, and whatever can be done to speed that up speeds the whole system up. ![Slight improvement in times for connections to some otherwise rather slow services, after the Nov 4ish deployment.](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/posts/ruby-users-be-wary-of-net-http/chart.png) Slight improvement in times for connections to some otherwise rather slow services, after the Nov 4ish deployment. We deployed a We::Call v0.7.0-pre1 and lost instrumentation for the rest of the day, leading to that exaggerated dip in the graph. Typheous does support NewRelic, so we [deployed a fix](https://github.com/wework/we-call-gem/commit/f8e0e0da41269d07dc631d8046e02ada6319c6fb?ref=apisyouwonthate.com), and you can easily see the improvement. #### Typhoeus Is Not Perfect Things cant always be 100% unicorns. Typheous right now does not seem to distinguish between "open timeout” and "read timeout”, which [for our needs](https://github.com/lostisland/faraday/issues/718?ref=apisyouwonthate.com#issuecomment-344360416) (and for debugging in general) is very important. This sucks, but it sucks a whole lot less than having everything mysteriously double up without anyone knowing why. The Typheous adapter in Faraday ≤ v0.13 is also horrendously out of date, and Typheous bundles their own now. This has been [removed in master](https://github.com/lostisland/faraday/pull/715?ref=apisyouwonthate.com) (and will be gone in v0.14). The Faraday test suite [runs against the external typheous adapter,](https://github.com/lostisland/faraday/pull/745?ref=apisyouwonthate.com) thanks to [some help](https://github.com/lostisland/faraday/pull/748?ref=apisyouwonthate.com) from the Faraday maintainer "iMacTia”. All in all the situation is better now than it was, but there are more PRs to send, and more open-sourcing to do! #### Summary Whatever you do, make sure you’re conscious about Net::HTTP doing this thing, and if it scares you switch to one of those libcurl based adapters! Also keep this speed difference in mind when considering switching from "traditional” REST or RESTish/RPC HTTP calls. Using a HTTP client that does not handle keep-alive will be much slower than switching to some fancy new tool that does, but that’s a false improvement. Try to avoid throwing the baby out with the bathwater, and just use a HTTP client that turns keep-alive on for you. Or find one that allows you to turn it on yourself. 👍🏼 ### Keeping Documentation Honest URL: https://apisyouwonthate.com/blog/keeping-documentation-honest/ Last updated: 2023-03-24T12:34:22.000Z We've been talking a lot about documentation and API descriptions recently. About how it’s important to [write down your contract](https://apisyouwonthate.com/blog/commit-to-api-contracts) using API descriptions, and how to turn these descriptions into [beautiful human-readable documentation](https://apisyouwonthate.com/blog/turning-contracts-into-beautiful-documentation). Now let’s look at how we can ensure that documentation is actually telling the truth! API description documents come in a few forms, and if you’re writing JSON Schema you can use things like [json\_matchers](https://github.com/thoughtbot/json%5Fmatchers?ref=apisyouwonthate.com) (Ruby/Rspec) to simplify your integration tests, and confirm your response matches a certain schema. ```yaml context 'with a valid payload' do it 'has a valid contract' do result = JSON.parse(subject.body)['result'] expect(result).to match_response_schema('foo') end end context 'with an invalid payload' do it 'has a valid error' do result = JSON.parse(subject.body)['result'] expect(result).to match_response_schema('shared/error') end end ``` That `foo` lines up with `schemas/foo.json` and the error matches up with `schemas/shared/error.json`. This is really handy if your documentation is based entirely off of JSON Schema, or if you’re managing to walk that creepy line of [writing JSON Schema and having it generate OpenAPI](https://philsturgeon.uk/api/2017/07/20/my-vision-for-a-perfect-world-in-api-specification/?ref=apisyouwonthate.com) despite their discrepancies. When your code is guaranteed to match the schema, then when you generate documentation from the schema you *know* the documented responses are going to be honest. For example, if docs show the `foo` field is going to be there, but your code doesn’t have it, your tests should fail. If you say `bar` is going to be a string, but it somehow is output as an integer, you should know about that too. Using JSON Schema combined with a schema matcher in your integration tests, you have contract testing *and* documentation testing all in one. The only downside there, is that this approach *only confirms responses*. Request bodies, query strings and their values, possible enum values, etc. are all kinda ignored, and you’re left hoping that whatever you wrote in your specs is accurate… There are two tools which set out to help ensure more than just the responses are validated. #### Dredd Dredd supports [API Blueprint](https://apiblueprint.org/?ref=apisyouwonthate.com) and OpenAPI v2.0\. The idea with Dredd is that you want to test your documentation works, and seeing as your documentation is full of URLs, query string parameters, enums, and example values, it can throw those at a locally running instance of your API and see how it responds. Dredd provides documentation testing, and essentially you end up with generated integration and contract testing as a side benefit. It’s not intended to replace integration tests or contract testing, but seeing as it’s making requests and testing the response is the right shape, you could consider it basic contract testing. Dredd can be pretty complex, and I’ve made videos in the past showing how to get it going. You’ll need to create a database seed to generate test data for your tests to play with. You’ll need to use the `--sorted` switch or corresponding YAML config to ensure `GET` runs before your `DELETE`, otherwise you get a bunch of 404s as there is no rollback ability. There are plenty of other gotchas to figure out. As complex as Dredd can be, it’s an absolute lifesaver, which is why I’ve been recommending it for the last few years, but I’ve been curious if an alternative tool could live inside the test suite a little more… Transactions and rollbacks are important, and with Dredd just being a node cli tool that hits your API from the outside, you can’t play with that sort of thing. I’ve never known anything like this to exist from time in PHP, but working in Ruby land these days meant a tool was recommended... #### Apivore [Apivore](https://github.com/westfieldlabs/apivore?ref=apisyouwonthate.com) initially looked to be the answer to my hopes and dreams. I read the article [*Automating Empathy: Test Your Documentation With Swagger and Apivore*](hhttps://amcaplan.ninja/blog/2016/12/27/automating-empathy-test-your-documentation-with-swagger-and-apivore/), which gives a bunch of insight into how it works. The idea is that you make an RSpec test, pass your OpenAPI file, and Apivore will do two things. First it will validate the file (which is handy), but what is fantastic is that it’ll then let you hit each of your API endpoints to make sure they’re all valid against the responses you’ve defined. The promise here immediately seemed ideal, but as soon as I started implementing it I was hitting problems. Apivore expects your OpenAPI file to be available on URL instead of a filepath and [the PR for that has been abandoned](https://github.com/westfieldlabs/apivore/pull/88?ref=apisyouwonthate.com) since July 2016… I also noticed its failure to load YAML files, as it just runs `JSON.parse()` on any file you give it regardless of the extension… so [I added YAML support](https://github.com/westfieldlabs/apivore/pull/118?ref=apisyouwonthate.com). With YAML being loaded I hit a fresh problem: `$ref` is not respected to the extent that the OpenAPI spec allows it. Another stale conflict-ridden PR exists for supporting `$ref` inside `responses`, but I want it inside `paths`. ```yaml paths: /foos: $ref: paths/foos.yml /foos/{id}: $ref: paths/foos-id.yml ``` To avoid spending another half day on a PR, I temporarily used [swagger-cli](https://github.com/BigstickCarpet/swagger-cli?ref=apisyouwonthate.com) to bundle up a JSON file with no `$ref` usage: ``` swagger-cli bundle -r docs/api.yml > docs/api.json ``` This temporary solution got me far enough to notice that the API for sending query string, headers, body data, etc. is rather convoluted. I found myself building a params hash from smaller lets as the "Autiomating Empathy” article suggested: ```ruby require 'rails_helper' RSpec.describe 'Valid OpenAPI', type: :apivore, order: :defined do subject { Apivore::SwaggerChecker.instance_for('docs/api.json') } let(:api_key) { create(:api_key) } let(:url_params) {{}} let(:query_string_params) {{}} let(:data_params) {{}} let(:headers) do { 'Authorization' => "Token token=#{api_key.access_token}", 'Content-Type' => 'application/json' } end let(:params) { url_params.merge( '_headers' => headers, '_query_string' => query_string_params.to_query, '_data' => data_params.to_json ) } describe '/foos' do context 'get' do before { create(:foo) } it { is_expected.to validate(:get, '/foos', 200, params) } end context 'post' do let(:data_params) do { user_uuid: SecureRandom.uuid, account_uuid: SecureRandom.uuid, } end it { is_expected.to validate(:post, '/foos', 201, params) } end end ``` This starts to seem fairly cool, and tests started passing… but I have already noticed myself copying code from my integration tests to make this work. This file is going to get huge, especially as I have the `validate_all_paths` in there. ``` Failure/Error: expect(subject).to validate_all_paths post /foos is untested for response code 400 ``` If I have to test all success and failure scenarios in this special type of RSpec test, I’m really wasting my time, as my integration tests are already doing that. Now I need to copy all of the business logic from all of the existing integration tests, stub things out, make sure VCR requests are happening, etc. just to make Apivore happy… I commented out that `validate_all_paths` test to make this error go away, and my tests pass, but it’s left me a bit confused about the goals of this thing. Building this special `type: :apivore` test file, repeating the URLs, copying items from my integration tests to make it work, and doing this all manually… it seems like a lot of extra work. I would prefer an RSpec helper much like json\_matcher like… openapi\_matcher which just helps me confirm the response is correct. Setting everything up myself seems rough, as Dredd would automatically test all paths for the default response and let you know which didn’t work. I don’t need to write the test, Dredd generates that test from example values. Conveniently Dredd will not try to cover every response status, which means if you list your success first and failures after, it’ll skip those. That is fine as I’m using `expect(result).to match_response_schema('shared/error')` in the integration tests failure cases. Once again JSON Schema has saved the day. Apivore seems especially useless as it turns out, [Apivore does not help with query string parameters](https://github.com/westfieldlabs/apivore/issues/91?ref=apisyouwonthate.com)*.* > Tests your rails API against its Swagger description of end-points, models, and query parameters. — [https://github.com/westfieldlabs/apivore](https://github.com/westfieldlabs/apivore?ref=apisyouwonthate.com) It lies... That means all it does is check the response, which I am already doing with [json\_matchers](https://github.com/thoughtbot/json%5Fmatchers?ref=apisyouwonthate.com)… so… #### Summary For me I’ll keep using [json\_matchers](https://github.com/thoughtbot/json%5Fmatchers?ref=apisyouwonthate.com) in integration tests to ensure the contract of each response, and use Dredd to check everything else is working. I’ll suggest my PHP coworkers use [JsonGuard](http://json-guard.thephpleague.com/?ref=apisyouwonthate.com) in a similar fashion, and take Apivore off the recommended tool list here at the day job for now. I’ll be writing more about Dredd in the future, so subscribe if you want to get that! You could also buy our book [Build APIs You Won’t Hate](https://apisyouwonthate.com/books/build-apis-you-wont-hate.html)! ### Making the Most of JSON:API URL: https://apisyouwonthate.com/blog/making-the-most-of-json-api/ Last updated: 2023-03-24T12:34:33.000Z With [JSON:API](https://jsonapi.org//?ref=apisyouwonthate.com), HAL, and other compound document formats [becoming less necessary thanks to HTTP/2](https://apisyouwonthate.com/blog/lets-stop-building-apis-around-a-network-hack), using these formats is no longer a given. That said, there are plenty of reasons for some teams to continue using JSON:API for the near future: 1. Existing APIs with an underlying codebase tightly coupled to JSON:API, that cannot serialize multiple formats 2. Teams that are all very used to JSON:API and are on tight deadlines to deliver a working solution 3. APIs that are consumed in the browser, especially those which need support for mobile browsers, or operating systems older than Windows 10, or OS X 10.11 ([source](https://caniuse.com/?ref=apisyouwonthate.com#feat=http2)) Let's look at a few of the problems I've hit using JSON:API in production for two years. ## Scope for Compound Documents At a carpooling company, with trips which could contain many riders. We exposed `/trips?include=riders` and everything was great, until the web and iOS clients let us know months later they'd been filtering on the client side to map only active riders. Client-side filtering is a concern as it overworks the application server and database server. It also forces clients to wait on unnecessary data to transferred over the network, and makes the browser use more juice than it needs to. Some developers thought it was obvious to restrict the "riders” relationship to only "active riders”, and pushed code to make that happen. It turns out that broke the Android client, which actually had a Previous Riders screen and was expecting the "riders” relationship to have all statuses not just active. My suggestion was to deprecate the `riders` include, and create `active_riders` and `inactive_riders` includes, a trend which we continued throughout the life of that API. Avoiding default scopes is important, as the "default” for one application might be different to another. Don't try to assume what default scopes should be for ever client, just make it very clear what relationships have scope by putting it in the name. ```js "relationships": { "active_riders": { "data": [ { "type": "people", "id": "1" } ] }, "inactive_riders": { "data": [ { "type": "people", "id": "2" }, { "type": "people", "id": "3" }, { "type": "people", "id": "4" } ] }, "pending_riders": { "data": [ { "type": "people", "id": "10" } ] } } ``` This avoids the trouble of expectation not matching reality, avoids accidental changes to reality, and makes it very clear what you're getting. So long as you're careful to handle this code on the backend, and you're able to fetch your data in a way that isn't now three times slower. If you're querying a database for example, you need to avoid temptation to make three separate extra queries: 1. `... WHERE status = active` 2. `... WHERE status = inactive` 3. `... WHERE status = pending` If you can avoid doing that, performance should not suffer too badly from multiple relationships. Basically you need to fetch everything then split it yourself, which… is what we were trying to avoid all the clients doing. At least adding [HTTP headers will help clients hopefully cache that result](https://apisyouwonthate.com/blog/http-client-response-caching). Some people try solve this in another way: filtering relationships. ## Filtering Relationships If by default the whole relationship is everything, most SQL-minded API developers will then think: I need a `WHERE` clause. JSON:API doesn't really provide anything for this, but does suggest using a query string parameter called `?filter=`. The spec is [extremely vague about filtering](https://jsonapi.org//format/?ref=apisyouwonthate.com#fetching-filtering): > The `filter` query parameter is reserved for filtering data. Servers and clients **SHOULD** use this key for filtering operations. > > **Note:** JSON:API is agnostic about the strategies supported by a server. The `filter` query parameter can be used as the basis for any number of filtering strategies. That's all the spec says, but various conventions exist. The most popular suggest is `filter[riders][status]=active`, which takes the relationship as the first array key, then the resource field as the next key, before finally taking the value. Filtering like this is not so bad when it's a tightly controlled whitelist of possible fields, but many folks implementing JSON:API get a bit carried away and allow anyone to filter by anything. This is rather common with SQL-minded developers, but an API should not be treated like an SQL database, especially if you're handing out API keys to developers who could perform query-like requests that you don't know about. Allowing people to filter by any field on any relationship on a API collection that's pulling from a database would lead to slow queries. It's impossible to know which indexes to add ahead of time. The first rule of API development is "Clients will surprise you." Trying to index everything is [damaging to performance](https://redmondmag.com/articles/2014/07/16/too-many-sql-server-indexes.aspx?ref=apisyouwonthate.com) in a bunch of ways. Restricting the ways in which clients can filter is a good idea, and optimize the ways in which they do. As clients request further filters, weigh it up, and maybe add it if you think it's a good idea. It's a little tough to code and document these nested filter options, but you'll need to figure it out, or suffer from awful performance. ## Limitations on Relationships Another tricky choice, is figuring out how many items you put into a relationship. Do you put 10? 100? Literally all of them? At first you'll say "Oh there wont ever be more than X”, and again, that's almost certainly going to change since that initial assumption. The carpooling company said "Oh there's never going to be more than 5 riders, because a car wont fit more than 5!" Then we added vans with 20 riders, and im sure at some point there would have been actual buses with 100\. After a few months there could be several hundreds of riders with various status, and that's getting to be some chunky JSON. JSON:API offers fantastic advice in the [Pagination section](https://jsonapi.org//format/?ref=apisyouwonthate.com#fetching-pagination): > A server **MAY** provide links to traverse a paginated data set ("pagination links”). > > Pagination links **MUST** appear in the links object that corresponds to a collection. To paginate the primary data, supply pagination links in the top-level `links` object. To paginate an included collection returned in a [compound document](https://jsonapi.org//format/?ref=apisyouwonthate.com#document-compound-documents), supply pagination links in the corresponding links object. Basically they recommend shoving some `"next"` links in there. ```ruby "relationships": { "active_riders": { "links": { "related": "https://api.example.com/riders?trip_id=23124" "next": "https://api.example.com/riders?trip_id=23124&cursor=learn-about-cursors-in-the-book" } "data": [ { "type": "people", "id": "11" }, { "type": "people", "id": "2819" }, { "type": "people", "id": "3223" }, { "type": "people", "id": "36" }, { "type": "people", "id": "643" }, { "type": "people", "id": "3434" }, { "type": "people", "id": "232" }, { "type": "people", "id": "1123" }, { "type": "people", "id": "1563" }, { "type": "people", "id": "1357" } ] } } ``` Ooo, links. Otherwise known as HATEOAS! Hypermedia as the Engine of Application State which in its most simple form often is "Hey you can go and get more stuff over here if you want it." Putting in just a few items for the relationship (say… 10!), then having a URL for clients to fetch more stuff if they want it is awesome. They can go grab that either when the end user wants it, or preemptively. This is a lovely combination of two approaches. On one side its exactly the sort of thing the REST is asking you to do, whilst still providing a bit of an optimization for HTTP/1.1-stuck clients scared to make multiple requests. You do of course need to make sure you have that `/riders?trip_id=23124` endpoint, and you'll probably need to make sure you have all the same query string filter options as you do with `filter[riders][status]`. By the time you have all these things done twice, you start to wonder if it would be better to have just the one way of doing things… and that's where I'm at these days. I advice against using includes, and if you do use them you need to really restrict how and where you use them, and also put `links` in for all of your relationships, pagination, related actions, and anywhere else that makes sense. ## Summary For those really set on using JSON:API, watch out for these troubles I've bumped into, and check out other articles by [other smart folks](https://jeremiahlee.com/blog/2017/10/10/pragmatic-design-with-json:api/?ref=apisyouwonthate.com) who are getting into it. Avoid building a slow SQL-over-HTTP API, that appears to works for a few months, and fails horribly once your database tables are a little better stocked than your test data. Sure it'll fulfill any query that anyone could possibly want, but it'll do it poorly. 😅 One cool thing about using JSON:API for now, is that you can offer includes *and* links. Then you can slowly move over to a more hypermedia based approach as HTTP/2 usage becomes more prolific; deprecating and removing includes as clients upgrade to just making web requests to nice, HTTP-level cacheable collections, with top level query strings and no messy compound document stuff getting in the way. Personally I want to [ditch compound documents](https://apisyouwonthate.com/blog/lets-stop-building-apis-around-a-network-hack), pass most of the work off to HTTP/2, [RFC 7807](https://tools.ietf.org/html/rfc7807?ref=apisyouwonthate.com), [RFC 7386](https://tools.ietf.org/html/rfc7386?ref=apisyouwonthate.com), [JSON HyperSchema](https://twitter.com/philsturgeon/status/923209204567101440?ref=apisyouwonthate.com), and maybe knock up a baaaasic little micro-standard that just covers items, collections, and a handful of little bits of metadata like pagination, sorting, filters, etc., but that's a topic for another day. ### Let's Stop Building APIs Around a Network Hack URL: https://apisyouwonthate.com/blog/lets-stop-building-apis-around-a-network-hack/ Last updated: 2024-12-17T11:35:22.000Z [JSON-API](http://jsonapi.org/?ref=apisyouwonthate.com) has been one of the most popular standards for API development for a while now. It was conceived in 2013, battled through some rather different RC versions (changing drastically as it went), finally stabilizing with v1.0 back in mid-2015. My interest in JSON-API has changed substantially over the years. Initially I avoided it like the plague, later I recommending it, and now I generally suggest folks skip it if at all possible. Besides trying to standardize how you represent items, collections, meta data and errors (an excellent goal!), it sets out to solve a lot of problems such as [Compound Documents](http://jsonapi.org/format/?ref=apisyouwonthate.com#document-compound-documents). The concept of a “compound document” — in the API world at least — is the art of squashing related data into the main requested resources. This is done to reduce the number of HTTP calls a client has to make. Clients have for a long time been very unhappy making multiple HTTP/1.1 calls, especially over mobile networks. That concern has had a noticeable impact on how we design APIs. Multiple calls is considered a point of bad design, to the point where fetching a user will include the companies they work for, and that includes the locations a company might have, and that includes something else… That was a bit of a mess, so conventions and standards were invented to make those includes optional, and JSON-API was one of the main contenders. Somewhere in 2013 I also released a tool called [Fractal](http://fractal.thephpleague.com/?ref=apisyouwonthate.com), which was based off of code I’d been running in production since early 2013\. The goal there (other than serialization) was the same thing, to facilitate embedding and nesting includes optionally. In 2013 this all made sense. Multiple calls were bad, so we had to reduce those, without assuming too much and making our requests slower than they should be. This continued to be valid for a long time, even in 2015 when JSON-API was finalized. Guess what else was released in May 2015? RFC 7540, otherwise known as [HTTP/2](https://tools.ietf.org/html/rfc7540?ref=apisyouwonthate.com). In retrospect this seems highly poetic, as HTTP/2 kinda makes the compound document aspect of JSON-API a little bit pointless, and compound documents to me go hand in hand with what JSON-API is as a standard. Not to just pick on JSON-API, it also makes Fractal’s nesting/embedding feel silly, like it should be ripped out in a future version. ## What’s the Difference? There are plenty of resources out there explaining the differences between HTTP/1.1 and HTTP/2, so I’ll let them cover it. [](https://http2.github.io/faq/?source=post%5Fpage-----9a68f7e83dd2--------------------------------#what-are-the-key-differences-to-http1x) The main point here is that HTTP/1.1 forces you to handle DNS, HTTP handshakes, SSL negotiation, etc., on every single call, and if your homepage wants 10 things from the API then it’s doing all of that junk 10 times… HTTP/2 does all of that once, then you just make those calls back and forth. You can make those calls asynchronously, and fetch more data for those items as the responses come back. You requests all the A’s, then the callback for that async request asks for some B’s as and when the A’s respond. And C and D. If you can use [Server Push](https://http2.github.io/faq/?ref=apisyouwonthate.com#whats-the-benefit-of-server-push) then you don’t have to wait for the A’s to respond before you can request the B’s. The server will just start sending you the B’s! Seriously, [this shit is fast](https://gist.github.com/philsturgeon/ea4c3966731c8e2e92e70b11fe10d1bc?ref=apisyouwonthate.com). ## Bla Bla Compound Documents. What else? To me JSON-API has always mainly been about reducing the size of a payload response via network hacks like compoound documents, but there is another strategy used. [Sparse Fieldsets](http://jsonapi.org/format/?ref=apisyouwonthate.com#fetching-sparse-fieldsets) — an idea fairly fundamental to GraphQL also — lets clients specify the fields they’re interested in, so the API can send only those fields. ``` GET /articles?fields[articles]=title,body ``` Whilst sparse fieldsets can save a few kbs, there are plenty of other ways of trimming down the size of the response. The entire mindset of “more handshakes = bad” has forced us to shove fields into the same resource to avoid making new resources, or sub-resources. Now that we don’t need to be scared of it, we can stop using hacks like [the partials rade-off](https://blog.apisyouwonthate.com/a-happy-compromise-between-customization-and-cacheability-e48dc083ed10?ref=apisyouwonthate.com) I recently proposed, and move towards having more resources that are a bit more targeted in scope. Invoices don’t need payment status, or an array of order items, so we don’t need to worry about trimming down the response at the field level, they can just be their own related resources, discoverable with hypermedia controls. If splitting up resources sounds like a big change and you want to just squash that actual response, HTTP/2 has your back there too. Headers are compressed with HPACK, which is similar to GZIP but a little better, and its something [Cloudflare are pretty excited about.](https://blog.cloudflare.com/hpack-the-silent-killer-feature-of-http-2/?ref=apisyouwonthate.com) That’s not to mention that the whole protocol is binary instead of plain-text, which gets you another nice boost. If you’re still sad about JSON being slow, you can use content-negotiation to add BSON or Protobuff to your REST API, which is a large part of whats getting people so excited about gRPC. In general, once again, the potential speed benefit of skipping a bit of that JSON coming back using sparse fieldsets, seems to be lost in the idea of making your API be less variable, having more targeted resources, which are drastically more cacheable at the network level. ## HTTP/2 Support in the Real World [NGINX already supports HTTP/2](https://www.nginx.com/blog/nginx-1-9-5/?ref=apisyouwonthate.com). Some programming languages like [Go have built-in HTTP/2 support](https://http2.golang.org/?ref=apisyouwonthate.com), so you can enjoy the full benefits of HTTP/2 like Server Push. [NodeJS too](https://nodejs.org/api/http2.html?ref=apisyouwonthate.com). Everyone else can at least leverage [multiplexing](https://http2.github.io/faq/?ref=apisyouwonthate.com#why-is-http2-multiplexed) by slapping Cloudflare, Fastly, etc., in front of their HTTP/1.1 setup. # But the Browsers… Currently not 100% of browsers support HTTP/2 in it’s entirety, but the situation is pretty good. ![](https://miro.medium.com/v2/resize:fit:1400/1*76stN3E_Nxb8fRq3BqTxJw.png) [caniuse.com](http://caniuse.com/?ref=apisyouwonthate.com#feat=http2) shows support for HTTP/2\. Green is good, snot is partial, red is nope. If your API is predominantly web-based, and older browser support is important, or you have *no idea* which browsers are going to be hitting it, then maybe you need to continue worrying about network hacks like compound documentation. ## The New World of API Design Combining HTTP/2 with targeted, network cached endpoints (via Fastly, Varnish, Squid, etc), makes Hypermedia Controls (HATEOAS) seem waaaaaaaay less silly. > I’m not waiting for a response before I see what actions I can take next, that’s slow! **— Fictional Narrative Device** No longer! HTTP clients [net-http2](https://github.com/ostinelli/net-http2?ref=apisyouwonthate.com) in Ruby make asyncing up your calls nice and easy, so you can run actions in callbacks as soon as the responses come in. Hypermedia in general seems a lot more viable when you’re not terrified of the transportation layer being slow. It also means you’re not going to be scared of making a sub resource. At work right now we have `GET /users` and they want to shove `"locations”` in there (every WeWork location that a user has access to). That would be how you’d want to do it in 2014, but in 2017 that should be `GET /users/123/locations` or `GET /locations?user_uuid=123`, or *who cares.* It just doesn’t need to be jammed up in there, slowing everyone down, just *in case* somebody needs it. Includes, [partials](https://blog.apisyouwonthate.com/a-happy-compromise-between-customization-and-cacheability-e48dc083ed10?ref=apisyouwonthate.com), etc., can all graduate to being actual brand new requests, instead of being some weird network hack because we were scared of handshake time. ## But Adding This to REST Sounds Hard! Folks often say “Why would I want to waste time making a REST API do this when I could just use X.” In the same way that JSON-API feels a little silly to me now, it makes GraphQL feel substantially less useful… Anything designed around this network hack no longer feels as useful as it once was. Well, GraphQL is the king of all glorified network hacks, and gRPC is literally just HTTP/2 + Protobuff, I’m left realizing these little niche API setups are not quite as useful as their marketing sites are making out. If you already have a REST(ish) API, you can just start using HTTP/2 right now and get half the benefits of gRPC, and no longer care about many of the “benefits” of GraphQL. [You Might Not Need GraphQLDo you like the look of GraphQL, but have an existing REST/RPC API that you don’t want to ditch? GraphQL definitely has some cool features and benefits. Those are all bundled in one package, with a nice marketing site, documenting how to do all the cool stuff, which![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/icon/apis-logo-square-1.png)APIs You Won't HatePhil Sturgeon![](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/thumbnail/cover.png)](https://apisyouwonthate.com/blog/you-might-not-need-graphql/) Maybe you can remove the (ish) from your REST(ish) now that you’re not quite as scared of making more HTTP calls. [Summary](http://apibusters.com/003-why-hypermedia?source=post%5Fpage-----9a68f7e83dd2--------------------------------) These previous solution were far from perfect, and I’m going to write about real-world issues I’ve struggled through with compound documents in the next article ([done!](https://blog.apisyouwonthate.com/making-the-most-of-json-api-7fb51f4407aa?ref=apisyouwonthate.com)) I don’t think anyone should rush out and change everything right now, but recognize the changing environment around the web. As the web evolves the standards evolve to meet changing requirements, and as they evolve we no longer need as many hacks to get the job done. Frontend [developers aren’t make CSS image sprites anymore](https://developer.mozilla.org/en-US/docs/Web/CSS/CSS%5FImages/Implementing%5Fimage%5Fsprites%5Fin%5FCSS?ref=apisyouwonthate.com), and JS/CSS combination is going away too. ![](https://miro.medium.com/v2/resize:fit:1400/1*6JQWMKWR8GQklbSVbfVwUw.png) “Implementing image sprites in CSS” ****Source:** [developer.mozilla.org](https://developer.mozilla.org/en-US/docs/Web/CSS/CSS%5FImages/Implementing%5Fimage%5Fsprites%5Fin%5FCSS?ref=apisyouwonthate.com) HTTP/2 is a representation of what developers needed, and now we can use HTTP in its entirety, without needing so many network hacks to make it performant enough to get the job done. *Btw if you’re wondering what standard/specification I would recommend instead of JSON-API, I’m wondering the same thing. It’s between* [*Siren*](https://github.com/kevinswiber/siren?ref=apisyouwonthate.com) *and just naked-JSON +* [*HTTP Problems RFC*](https://tools.ietf.org/html/rfc7807?ref=apisyouwonthate.com) *\+ JSON Schema. Picking one format to rule them all is impossible, but Siren certainly looks excellent.* ### Speeding Up APIs/Apps/Smart Toasters with HTTP Response Caching URL: https://apisyouwonthate.com/blog/http-client-response-caching/ Last updated: 2023-03-24T12:32:06.000Z Caching is a huge topic, and there’s a lot of different types of caching. No one type of cache is going to suite all needs and cover everything needed to make a performant application, but one type that’s often overlooked is client caching at the HTTP level. Building a permissions API at work, a user and company UUID are provided. The permissions API fetches the user and company data from another API, and may make a few other calls. The evaluator decides if they can do the thing or not, based on their data, and some rules. [ABAC is fun y’all](https://en.wikipedia.org/wiki/Attribute-based%5Faccess%5Fcontrol?ref=apisyouwonthate.com). Checking one policy could internally generate 3–5 HTTP calls to a handful of dependencies. The client calling permissions (client A) would be waiting for those responses, because permissions (client B) is making a whole bunch of calls. Client A could be asking to check 10 policies, all of which might request different remote data sources, and the user could belong to multiple companies, all of which need to be looked up too. This could mean 20 HTTP requests are need be made, which is a lot. We implemented a little Ruby memoization to avoid duplicate requests, but that will not help with multiple Client A calls making similar requests, or the same requests shortly after. At this point people usually recommend adding some cache logic to the application (application caching), which [in Rails](https://guides.rubyonrails.org/caching%5Fwith%5Frails.html?ref=apisyouwonthate.com#low-level-caching) looks a bit like this: ```ruby Rails.cache.fetch("users/#{uuid}") do UserAPI.find_user(uuid) end ``` If there is nothing in the cache matching that user then it’ll run the block and fetch the thing. That’s handy and all, but how long does that cache entry last? Forever! Infinity is a long time, so we have to provide a reasonable date. ```ruby Rails.cache.fetch("users/#{uuid}", expires_in: 12.hours.from_now) do UserAPI.find_user(uuid) end ``` That’s great and all, but 12 is an arbitrary number plucked out of thin air. Client B is now making up its own rules about resources it doesn’t own… At the most basic this leads to "My email address is showing up differently in two systems”, but beyond that there may be all sorts of business logic potentially involved with how long data should be cached. Ignoring those concerns, I still have to litter my codebase with all of this code, wrapping every HTTP call in caching logic that I had to guess at. Or... The server can tell the client how long to store things, just as browsers do! Whenever you go to pretty much any website, the server defines various cache-related headers. These headers are outlined in [RFC 7234: HTTP/1.1 Caching](https://tools.ietf.org/html/rfc7234?ref=apisyouwonthate.com) and Darrel Miller broke it down in *[Caching is Hard, Draw Me a Picture](https://apisyouwonthate.com/blog/caching-is-hard-draw-me-a-picture/).* ``` Expires: Sat, 06 Oct 2018 12:00:00 GMT ``` This is an example of the most basic cache header, but there are [many more](https://developers.google.com/web/fundamentals/performance/optimizing-content-efficiency/http-caching?ref=apisyouwonthate.com). Your browser will respect these HTTP headers unless it’s told not to (e.g: hard refresh) and that’s how CSS/JS/HTML is cached. Basically the browser will skip the request that response is already in the cache. When we build systems that call other systems, we often skip out this step, and performance can suffer. #### Implementing HTTP Response Caching Using a single gem, I calmed Client B down substantially. Benchmarking with [siege](https://www.joedog.org/siege-home/?ref=apisyouwonthate.com): ``` siege -c 5 --time=5m --content-type "application/json" -H "Authorization: Token token=snip" https://permission-api.example/endpoint POST { ...not relevant... } ``` All of a sudden the Permissions API (client B) went from this: ``` Transactions: 443 hits Response time: 3.35 secs Transaction rate: 1.48 trans/sec Throughput: 0.00 MB/sec Successful transactions: 443 Failed transactions: 0 Longest transaction: 5.95 Shortest transaction: 0.80 ``` ... to this: ``` Transactions: 5904 hits Response time: 0.25 secs Transaction rate: 19.75 trans/sec Throughput: 0.00 MB/sec Successful transactions: 5904 Failed transactions: 0 Longest transaction: 1.75 Shortest transaction: 0.12 ``` Boom! This benchmark is of course fairly artificial due to requesting the same user and membership data thousands of times, but the initial requests are \~800ms (down from \~3.5s) and repeat requests are down to \~200ms (from also 3.5s). This is substantial however you spin it. This is all done with standard expire-based caching, and not [conditional caching](https://www.joedog.org/siege-home/?ref=apisyouwonthate.com). That’s a whole other barrel of fish, which we’re not going to get into here. ### I WANT THIS! Writing all the code to do this would be a big job. Luckily, there are solutions built in pretty much every single language. #### Ruby ```ruby client = Faraday.new do |builder| builder.use :http_cache, store: Rails.cache ... end ``` **[plataformatec/faraday-http-cache](https://github.com/plataformatec/faraday-http-cache?ref=apisyouwonthate.com)** \- a faraday middleware that respects HTTP cache. #### PHP ```php use GuzzleHttp\Client; use GuzzleHttp\HandlerStack; use Kevinrob\GuzzleCache\CacheMiddleware; // Create default HandlerStack $stack = HandlerStack::create(); // Add this middleware to the top with \`push\` $stack->push(new CacheMiddleware(), 'cache'); // Initialize the client with the handler option $client = new Client(['handler' => $stack]); ``` **[Kevinrob/guzzle-cache-middleware](https://github.com/Kevinrob/guzzle-cache-middleware?ref=apisyouwonthate.com)** \- A HTTP Cache for Guzzle 6\. It's a simple Middleware to be added in the HandlerStack. #### Python ```python import requests import requests_cache requests_cache.install_cache('demo_cache') ``` **[requests-cache](https://pypi.python.org/pypi/requests-cache?ref=apisyouwonthate.com)** \- Persistent cache for requests library. #### JavaScript (node) ```JavaScript const http = require('http'); const CacheableRequest = require('cacheable-request'); const cacheableRequest = new CacheableRequest(http.request); const cacheReq = cacheableRequest('http://example.com', cb); cacheReq.on('request', req => req.end()); ``` **[cacheable-request](https://www.npmjs.com/package/cacheable-request?ref=apisyouwonthate.com)** \- Wrap native HTTP requests with RFC compliant cache support. #### Go ```go proxy := &httputil.ReverseProxy{ Director: func(r *http.Request) { }, } handler := httpcache.NewHandler(httpcache.NewMemoryCache(), proxy) handler.Shared = true log.Printf("proxy listening on http://%s", listen) log.Fatal(http.ListenAndServe(listen, handler)) ``` **[lox/httpcache](https://github.com/lox/httpcache?ref=apisyouwonthate.com)** \- An RFC7234 compliant golang http.Handler for caching HTTP. #### On The Other Hand Not every HTTP GET request is one you want to cache. The middleware will generally do the correct thing so long as the server has declared their intentions well. Regardless of how well the server declares its cacheability, you may way to store things for longer. Disrespecting the use-by date can have similar effects to ignoring the date on a carton of milk, but if you’re aware of what you’re doing then sometimes ignoring the intentions of the server to persist longer makes sense. #### Sometimes its Inefficient If you are making multiple calls to APIs with large responses to create one composite resource (one local thing made out of multiple remote things) you might not want to cache the calls. If the client is only using a few fields from each response, caching all of the responses is going to swamp the cache server. File-based cache stores might be slower than making the HTTP call, and Redis or Memcache caches may well run out of space. Besides, restitching the data from those multiple requests to make the composite resource locally may be too costly on the CPU. In that case absolutely stick to application-level caching the composite resource instead of using the low level HTTP cache. You can use your own rules and logic on expiry, etc. because the composite item is yours. One final example: if you have data that changes based on the authenticated user, you’ll need to use `Vary: Authentication`, which basically segments the caches by Authentication header. Two requests that are identical in all ways other than the Authentication header will result in two different cache results. This can lower cache hit ratios so much it might not be worth worrying about. Depends. Give it a try. #### Conventions = Prizes The cool thing about HTTP caching is that once a server has declared its cacheability, not only can clients leverage that metadata to seamlessly know when to skip requests, but cache proxies can then offer cached responses to requests clients do actually make. Properly explaining cache proxies is another article in itself, but the topic is outlined in the previous article [GraphQL vs REST: Caching](https://apisyouwonthate.com/blog/graphql-vs-rest-caching). Caching is a great way to avoid doing slow stuff multiple times. Of course keep working on making things not be slow in the first place, but being able to have servers define their rules and clients automatically follow those rules is a fantastic way to add some stability to your architecture. The fewer HTTP calls we make, the better our smart toasters will run, and with all that saved energy we’ll stop the ice caps melting! 👍🏼 ### API Versioning Has No "Right Way" URL: https://apisyouwonthate.com/blog/api-versioning-has-no-right-way/ Last updated: 2023-03-24T12:31:56.000Z API versioning is a really difficult topic, and sometimes seen as a merely religious debate. It’s done differently at different companies, and different teams within different companies often vary. Some folks move from approach A to approach B, and when approach B solves their specific issues they act like approach B is "best" and nobody should ever use approach A. That’s all well and good until they realize approach C might be a good idea. There are so many approaches to versioning that it can be hard to talk about — let alone understand — them all, but I’m going to attempt to create a definitive guide. At work I am tasked with recommending one approach to API versioning, because right now we have a mixture of approaches, and we’re using a few subtle variants of the worst possible approach. My recommendations and research are being converted to a guide for all of y’all. ## Global URI Versioning Versioning the entire API. Usually in the path (e.g., `/api/v1/companies`), this could be a subdomain (e.g., `api-v2.foo.com/companies`). If functionality was required that forced the API to change the representation of companies drastically, a new version of the entire API would be created, even if users did not change at all. Usually these are planned timed releases, where — for example — a v2 of the iOS app and v2 of the API were timed deployments, and v1 probably has a sunset date. The downside here is that rarely are changes communicated clearly. In our example, when the users had not changed, client developers have a rough time confirming that. Also, it’s very common for smaller breaking changes to sneak in, because "It’s a major release, that’s time to change stuff". With that mindset, it’s very easy for API developers to shove in breaking changes that may not be communicated or documented particularly well. Some companies avoid this by being honest about the situation, and just make entirely new API applications, on new servers, with new domains. They build [http://new-api.example.com](http://new-api.example.com,/?ref=apisyouwonthate.com), or use codenames, then ditch the old. Eventually they’ll blackhole the old DNS once the old version has hit acceptable low or zero usage. This has the benefit of eradicating legacy as you go, but regardless of using names, version numbers, or anything else, building out multiple APIs and forcing the clients to consistently develop against new version after new version is *the* most time consuming approach to versioning, for not just the API developers but the client developers too. Everyone has to test *everything* on each new upgrade, and this all takes forever. ## Resource Versioning (URI-based) Versioning specific resources by theirselves. In the same example of `/api/v1/companies` and `/api/v1/users`, if companies changed substantially then maybe a `/api/v2/companies` is created, but `/api/v1/users` remains untouched. This makes upgrades easier for clients as they know where to focus their attention. If there is no v2 for users, they just keep on using the v1. One problem with this approach is it looks nearly identical to global versioning on the outside, and isn’t. I’ve seen apps built with `API::V1::BaseController`, `API::V2::BaseController`, and `API::V3::BaseController`, each of which have their own error format. That would be entirely fine in global versioning, as the assumption is a client will use all of v1, all of v2, or all of v3\. In resource versioning a client might hit a selection of v1, v2 and v3 endpoints at any time, and that means they need to support *three different error formats*. One end-user application was producing errors like `Error: [object Object]` to actual end users, because the JavaScript code was expecting the v2 format `{ "error" : "some message" }`, but a v3 error of `{ "error" : { "message" :"something", "code" :"err-123" } }` has come back. We could counter this specific issue with the use of a standard error response across all systems (shout-out to [*RFC 7807 Problem Details for HTTP* *APIs*](https://tools.ietf.org/html/rfc7807?ref=apisyouwonthate.com)*)*, but that is just fixing one symptom of a larger problem. ## Resource Versioning (Mime-Based) This approach is very similar, but instead of versioning in the URL, versions go in the `Accept` and `Content-Type` headers. ```http GET /api/companies Accept: application/vnd.acme.companies.v2+json ``` The keeps the URL issue out of the way, and avoids API developers mistakenly sharing conventions in version specific base controllers. This is what many considered for a long time to be the height of API versioning (myself included), and was the approach used by GitHub for a long time. It keeps the same endpoint working for a long time, but if the header is optional it can cause confusion. If clients don’t request a specific version, should they get the earliest supported version, or the latest? If v3 is added and v2 is eventually deprecated, clients would rather suddenly start getting a whole new version which might break their apps. Resource versioning does help, but can still be voodoo magic to many. ## Method-based Versioning (URI-based) Versioning by combination of URI and HTTP method. I have only ever seen this one approach used at WeWork, but I’m sure it’s been done elsewhere. It’s a variant of "Resource Versioning (URI-based)", which basically does not version an entire resource, only the method for that resource. That’s hard to wrap your head around for a good reason. Let’s see how it might work: The latest "fetch all" for companies might be`GET /api/v2/companies`, but to create a company you might `POST /api/v3/companies`, even though `GET/api/v3/companies` does not exist. The resource is not even guaranteed to match on the same method, as method + URI is what makes it unique, so `GET/api/v1/companies/{id}` might be the best way to grab a single resource, and who *knows* if those serializers are the same. This has all of the cons of the previously mentioned solutions, with a few more thrown in. Primarily, that API developers get used to disposable endpoints. Every time Client B needs to update a resource, they create their own new method, ignore the existing one, and this is tolerated as normal due to a lack of API versioning strategy. Disposable endpoint thinking has lead to situations where new update methods have been created that only contain subset of earlier update method functionality. The API developer adding functionality was unaware of the functionality in previous versions, and didn’t worry about looking because "nobody ever upgrades anyway." Due to this divergence, *even if a client wanted to upgrade, they could not!* They are trapped on older versions, which then have to be supported forever, for fear of breaking the clients using the newer endpoint. This increased functionality is essentially asking all API teams to support more functionality than they should, and that is time and money that could be spent on more useful things. ## API Evolution API Evolution is the concept of never breaking your contracts until you absolutely absolutely have to, then when you do you manage that change with sensible warnings to clients. It is **not** about making arbitrary changes and breaking stuff. Generally you add new fields, or add a new resource, and if you absolutely must you can deprecate and eventually remove the old bits when it’s no longer being used. This is an approach currently being popularized by GraphQL, but one that REST advocates have been talking about for decades. A great writeup on [HTTP Evolution](https://www.mnot.net/blog/2012/12/04/api-evolution.html?ref=apisyouwonthate.com) was done back in 2012, and more recently an excellent post on [API Change Management](https://blog.goodapi.co/api-change-management-2fe5bba32e9b?ref=apisyouwonthate.com). ![API evolution without versioning in graphql](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/posts/api-versioning-has-no-right-way/api-versioning.gif) *Evolution is not a GraphQL feature, but a concept that works rather well in most types of API.* At a [previous company offering crowdsourced carpooling](https://www.businessinsider.com/uber-cofounder-oscar-salazar-launches-ride-2015-4?ref=apisyouwonthate.com), we only added fields, never removed them. This was fine for the majority of changes, but when backwards compatible issues absolutely could not at all be avoided, we took advantage of the fact that a business name for a concept had changed, and took the chance to make our API match the business name. Matches were a relationship between a driver and a passenger, and multiple matches would make up the carpool. The BC change was to add multiple drivers, which completely screwed the concept of matches. When we switched from `/matches` to `/riders`, we changed the JSON representation entirely, but internally those two representations shared the same code. After a few months, the matches concept was deprecated, and clients started using the new riders concept. The internals changed multiple times as we worked on recoding towards the new goal, and we converted matches to riders, then eventually matches was just a different serializer sat on top of riders. During this process *the contract never changed.* We went through a few backflips to make that happen, but our clients didn’t need to do a thing, other than switch to "riders" in their own time. Android and iOS apps launched independently, and that didn’t matter for us at all. The approach of evolution pushes some extra work onto the API development team, but avoids a similar amount of work being pushed onto the client teams. If one API has 5 clients, then we’re saving 5x the development, testing, etc. ## Deprecations Are Tricky Whether you use Global URI Versioning, or want to use Evolution, you are going to need to consider how you handle deprecations. If your team is small enough, you can just email the iOS developer and suggest they use the newer endpoint, or the newer whole damn API, but if you work at a company with *hundreds* of developers (and 30+ services!) this might not be a successful approach. There are a few approaches to handling deprecations which you can use right now. ### Deprecating Endpoints with Sunset At WeWork we built a Faraday wrapper (popular Ruby HTTP client) called [we-call](https://engineering.wework.com/introducing-we-call-a-ruby-gem-for-making-http-calls-awesome-4ce3f7e54f46?ref=apisyouwonthate.com) that forces some good conventions onto client and server alike, using the power of middleware. One of those conventions is syntactic sugar around using the still-in-draft [Sunset header](https://tools.ietf.org/html/rfc8594?ref=apisyouwonthate.com), using annotations to make it super easy. You don’t need to use the whole we-call gem, you can just use [faraday-sunset](https://github.com/philsturgeon/faraday-sunset?ref=apisyouwonthate.com), or if you’re using PHP you can use [guzzle-sunset](https://github.com/hskrasek/guzzle-sunset?ref=apisyouwonthate.com). ### Deprecating Fields with OpenAPI v3.0 You can advertise to humans that fields are going away with OpenAPI and it’s ["deprecated" keyword](https://swagger.io/specification/?ref=apisyouwonthate.com#operationDeprecated), added in v3.0\. This is far from perfect as there is not really a computery way to detect these deprecations without shoving a link to your OpenAPI schemas into the JSON response. [JSON Schema is working on adding this](https://github.com/json-schema-org/json-schema-spec/issues/74?ref=apisyouwonthate.com), and it’s far more common for JSON Schema links to be placed into the JSON response. For now, don’t remove any fields, just add new fields, and use a new representation if you really just cannot handle the old contract. **Update 19th May, 2019:* JSON Schema got a deprecated keyword to match OpenAPI v3.0.* [PayPal do this](https://github.com/paypal/api-standards/blob/master/api-style-guide.md?ref=apisyouwonthate.com#deprecation) for their APIs, but they’re using vendor extensions that pre-date OpenAPI’s deprecated functionality. ### Using Migrations a’la Stripe Stripe have outlined an approach to [making their public API evolvable](https://stripe.com/blog/api-versioning?ref=apisyouwonthate.com), which is way above and beyond. Instead of using HTTP / in-band metadata to advertise the fact that it’s going away, they build out migrations which translate older requests into newer ones internally. They keep things backwards compatible for a certain amount of time before destroying the migration, emailing the client developers a whole bunch about the change before that point. This was done by Facebook for years, before they threw everything in and went with global versioning in the path… ## Be Honest About Building RPC APIs Most URI-based versioning is thinly veiled RPC, that just *look* a bit RESTish. If companies doing URI-based versioning using RPC naming conventions, the APIs would look a bit more like this: - `GET /api/v2/users` would be `GET /GetAllUsers` (without pagination for some client that nobody remembers) - `GET /api/v3/users` would be `GET /GetAllUsersPaginated` (if you want pagination just use this one!) - `GET /api/v2/users/123` would be `GET /GetUserAppA` - `GET /api/v2/users/123/simple` would be `GET /GetUsersAppB` - `PATCH /api/v2/companies` would be `POST /UpdateCompanyFromAppY` - `PATCH /api/v3/companies` would be `POST /UpdateCompanyFromAppZ` If people are going to design their APIs as RPC with a RESTish facade, they should just commit to being an RPC API and build endpoint for specific clients like they’re literally already doing. Just be honest about it. Hide the false intention, RPC the lot, document as such, and maybe just use [gRPC](https://grpc.io/?ref=apisyouwonthate.com). ## Summary API Evolution is incredibly powerful if you put in the work to make it possible. A [commitment to contracts](https://apisyouwonthate.com/blog/commit-to-api-contracts) will make things easier, especially with JSON Schema bringing deprecated in the future. Global URI Versioning is the least awful alternative. I would only recommend this to teams who cannot commit to contracts, and build incredibly different APIs to match incredibly different iterations of applications (e.g., v1, v2, v3 of a mobile app). Maybe just use codenames for each version and delete the old APIs, instead of trying to jam it all into one codebase and potentially taint one version with changes to some shared code in another version. Either way, global versioning is literally not at all RESTful, and hints towards RPC, and that’s coming straight from Roy. Resource-based content-negotiation is ok if you’ve got some very HTTP-savvy clients and document your intentions incredibly well, but it’s essentially a shortcut to kinda evolution, which doesn’t really add any benefit. And for the love of The Flying Spaghetti Monster, please don’t ever do any form of method-based versioning. It never makes any sense. You’re making things terrible for everyone, splitting expectations, creating overhead, and focusing on short-term gains and long-term confusion. However you decide to version, so long as you’re using one of the few more sensible approaches, remember that it’s not about deciding which is "best". It’s about sharing and understanding the pros and cons of various approaches, and the priority is educated decisions based on that information. ### PUT vs PATCH vs JSON-PATCH URL: https://apisyouwonthate.com/blog/put-vs-patch-vs-json-patch/ Last updated: 2023-03-24T12:30:40.000Z A question that is asked with increasing regularity in the [APIs You Won’t Hate Slack Group](https://slack.apisyouwonthate.com/?ref=apisyouwonthate.com) is one which has been asked for years, but does not always have a good answer. The question is: > What is the different between PUT and PATCH, and when do I use them? And WTF is JSON-PATCH? To start off, `PUT` and `PATCH` are two different HTTP methods, which are both commonly used in REST APIs. For people who think of REST APIs as only being CRUD (Create, Read, Update, Delete) there can be confusion over trying to work out which one is "best." People have preferences, people argue, and really the conversation is rarely had in a reasonable way. You can totally have both `PUT` and `PATCH` in your API, and no, they should not be an alias of each other (looking at you Rails). Quite simply, they do different things. The RFC for `PATCH` ([RFC 5789](https://tools.ietf.org/html/rfc5789?ref=apisyouwonthate.com)) actually explains the difference rather elegantly in its abstract: > *The existing HTTP PUT method only allows a complete replacement of a document. This proposal adds a new HTTP method, PATCH, to modify an existing HTTP resource.* One is for when you know *all* the answers, and the other is for updating little bits at a time. Some consider this a performance benefit (sending less stuff is quicker than sending lots of stuff), but there are some more racy benefits than that. ### Conflicts Think about a resource that has two fields, `field1` and `field2`. Two different requests (Request A and (Request B) try to update one of these field values as a `PUT` after getting the initial value of the resource with a `GET` request. Both `field1` and `field2` are `false` in response of the `GET` request. **Request A** Updating `field1` to be `true`. ``` PUT /foos/123 ``` ```json { "field1": true, "field2": false } ``` **Request B** Updating `field2` to be `true`. ``` PUT /foos/123 ``` ```json { "field1": false, "field2": true } ``` If both fields start `false`, and each request only intends to update one field, little do they know they are clobbering the results and essentially reverting them each time. Instead of ending up with both values being `true`, you'll simply have whatever the last request was, which is going to be `"field1": false` and `"field2": true`. To some this is a feature, but others consider it a bug because if they only want to update one field, why do they need to send everything? These people decide to just send the relevant fields they want to change, which is a flagrant misuse of how `PUT` is supposed to work and leads to a lot of problems. ### Expectations When building an API, you and your coworkers are not the only people that need to have fair expectations of how things are going to work. Other systems, such as EmberJS for example, are going to have some expectations of how a `PUT` request is going to work, and if you start going against the grain and making `PUT` send partial updates, you're going to have a bad time. We had a case at work, where an EmberJS application only had some models representing existing resources in the API, that were populated locally from partial data. They wanted to change the value of one field in this model and save the resource back to the API. When they saved, EmberData would notice it was a `PUT` and try to send as much data as it could. As it only had some of the field values, it would end up sending a body with every unknown field as `null`, which in turn was emptying values out of the database, and/or triggering validation errors for fields that it didn't want emptied. ### PATCHing the Problem Something that helped a lot at work, was implementing `PATCH`. We can now simply send the fields we intend to update, and anything else is left alone. Let’s just assume we’re using [JSON-API](https://jsonapi.org//?ref=apisyouwonthate.com) and building something for a carpooling company, like [Ride](https://ride.com/?ref=apisyouwonthate.com). We want to be able to "start” a trip, by changing the status from `"pending"` to `"in_progress"`. In v1.0, we used to use `PUT`. It worked something like this: ``` PUT /trips/123 ``` ```json { "data": [ { "type": "trips", "id": "123", "attributes": { "status": "in_progress", "started_at": null, "finished_at": null } } ], "relationships": { "driver": { "data": { "type": "users", "id": "999" } } } } ``` Here we’ve changed the `status` from whatever it was to `in_progress`. **Sidenote:* Am I meant to update the `started_at` myself or let the API do it? Who knows!* The main problem here is back to the conflict example above. If somebody else was changing another value like who the driver is, and accidentally changed the `status`, back to `pending` then they'd get an error message saying "The trip has already started, it cannot go back to pending" and they'd be thinking "I didn't even know it had started, I just wanted Gary to drive today." and everyone just gets confused. The same request as a `PATCH` could look something like this: ```http PATCH /trips/123 ``` ```json { "data":[ { "type": "trips", "id": "123", "attributes": { "status":"in_progress" } } } ``` This has solved the problem of not accidentally clobbering other values, as we are no longer sending things along for the sake of it. Only the attributes we send should be validated, and anything missing should be ignored entirely. ### WellActually In RFC 5789 (the RFC for the `PATCH` method), the example shows things working like so: ```http PATCH /file.txt HTTP/1.1 Host: www.example.com Content-Type: application/example If-Match: "e0023aa4e" Content-Length: 100 [description of changes] ``` This `[description of changes]` is considered by some to be a sequence of operations to be done to the resource in question, manifesting itself in a list of JSON objects like this: ```http PATCH /my/data HTTP/1.1 Host: example.org Content-Length: 326 Content-Type: application/json-patch+json If-Match: "abc123" [ { "op":"test", "path":"/a/b/c", "value":"foo" }, { "op":"remove", "path":"/a/b/c" }, { "op":"add", "path":"/a/b/c", "value":[ "foo", "bar" ] }, { "op":"replace", "path":"/a/b/c", "value":42 }, { "op":"move", "from":"/a/b/c", "path":"/a/b/d" }, { "op":"copy", "from":"/a/b/d", "path":"/a/b/e" } ] ``` This example is taken from [RFC 6902](https://tools.ietf.org/html/rfc6902?ref=apisyouwonthate.com), which builds on top of the `PATCH` method itself, to provide a standardised approach to giving multiple atomic changes. This approach is known as [JSON PATCH](http://jsonpatch.com/?ref=apisyouwonthate.com), and it has it’s own content type to make it clear when it’s in use. Finding information about it is rather fruitless, but there are a few articles around, Unfortunately the #WellActually starts to get pretty heavy. One of the most prolific articles out there on using PATCH is from [William Durand](https://williamdurand.fr/2014/02/14/please-do-not-patch-like-an-idiot/?ref=apisyouwonthate.com). It’s a great technical article, but the assertions of "do this really complicated thing which you might not need or you’re Doing It Wrong” do rub me up the wrong way a bit. Read this article, consider implementing it, and if you don’t want it, don’t do it. You’re fine. Yes, [JSON PATCH is lovely](https://www.mnot.net/blog/2012/09/05/patch?ref=apisyouwonthate.com), and you might need it for your API, but it also might be a complication you don’t need to worry about, depending on the complexity of your actions [JSON-API](https://jsonapi.org//?ref=apisyouwonthate.com) previously recommend using JSON PATCH, but since settled on the "just send what you need” approach instead. Something very similar to the "just send what you need" approach has been standardised as [RFC 7396: JSON Merge Patch](https://tools.ietf.org/html/rfc7396?ref=apisyouwonthate.com). One of the best things about HTTP-based APIs is the ability to respond to `Content-Type` headers. You can work with plain-old JSON in your `PATCH` requests for now, and in the future add JSON PATCH support if you find that you need it. You can use `Accept-Patch` ([more here](https://tools.ietf.org/html/rfc7396?ref=apisyouwonthate.com)) to advertise the availability of these two different modes. ``` OPTIONS /turtles/123 HTTP/1.1 Host: www.amazingpets.com ``` ``` HTTP/1.1 200 OK Allow: GET, PUT, POST, OPTIONS, HEAD, DELETE, PATCH Accept-Patch: application/json, application/json-patch+json ``` You can keep `PUT` in there too, if you like the idea of having idempotent saves. I’ve not used `PUT` for JSON in years as I almost never want an idempotent endpoint for this stuff when building CRUDish APIs, but they're pretty great for [file uploads](https://philsturgeon.uk/2016/01/04/http-rest-api-file-uploads/?ref=apisyouwonthate.com) which could fail and need idempotent retries. ### Turning Contracts into Beautiful Documentation URL: https://apisyouwonthate.com/blog/turning-contracts-into-beautiful-documentation/ Last updated: 2023-03-24T12:30:25.000Z Continuing on the topic of contracts — metadata that describes your data — it makes sense to see what easy early wins you can make from this stuff. Having a bunch of files written in JSON Schema, API Blueprint, RAML, OpenAPI, etc is not particularly exciting for developers who understand the format, and it’s completely useless for anyone else. Taking those contracts and turning them into beautiful documentation is one of the most rewarding results of contracting your API, although there are many more. Sadly, some of these formats suffer from a lack of documentation tools. For example, API Blueprint has Aglio, which has themes and looks quite nice, but does not document Data Structures (the most important part). OpenAPI (formerly known as Swagger) has some amazing tools, but people often only know of the one: Swagger UI. There are two others you should know about, and here’s a comparison. #### [**Swagger UI**](https://swagger.io/swagger-ui/?ref=apisyouwonthate.com) *This is not a tool I recommend. If you don’t like the look of it, please scroll down to two amazing tools.* ![Right from swagger.io, this RPC-style API shows that Swagger UI is RPC first](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/posts/turning-contracts-into-beautiful-documentation/api-docs.png) Right from swagger.io, this RPC-style API shows that Swagger UI is RPC first Looking at Swagger UI, it’s clear that it was designed by a developer, and also clear that developer was a big fan of RPC. Documenting an API as purely HTTP requests with the actual objects being hidden away as an afterthought is entirely backwards for a REST API. These tools promote RPC thinking in APIs that have Hypermedia and it’s just propagating the problem. ![Trying to find the object is hidden under "model” and it looks really bad.](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/posts/turning-contracts-into-beautiful-documentation/200-code.png) Trying to find the object is hidden under "model” and it looks really bad. Whilst having docs is better than not having docs (people will just [guess the contracts](https://apisyouwonthate.com/blog/commit-to-api-contracts) and RPC your REST API anyway), you don’t want to use a doc tool that really seems to suggest you completely ignore Hypermedia entirely. Trying to install Swagger UI is a weird one too. The docs send you to [their GitHub repo](https://github.com/swagger-api/swagger-ui?ref=apisyouwonthate.com), and the README is banging on about Docker. As wonderful as it is to work with Docker on a daily basis, it is not something somebody should need to learn to turn their OpenAPI files into HTML. I think (hope!) this is going away, as Swagger CodeGen has a HTML mode. CodeGen is designed primarily to generate code like SDKs, but it provides other templates too. Two of those templates output HTML, even though it’s not ready to be used as docs, and seems to be missing a bunch of stuff. ### Community to the Rescue! #### [Spectacle](https://sourcey.com/spectacle/?ref=apisyouwonthate.com) With an obvious install (npm/yarn install) you get a simple CLI tool that converts OpenAPI into great looking HTML. ![A cheese store API](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/posts/turning-contracts-into-beautiful-documentation/cheese-store.png) Demo API talks about Cheese **Source:** [sourcey.com/spectacle](https://sourcey.com/spectacle/?ref=apisyouwonthate.com) Spectacle splits the logic into two different sections, Operations and Schema Definitions. Operations give it the usual RPC-ish explanation you’re probably used to, and Schema Definitions is dedicated to outlining objects and their properties. Whilst this isn’t as 100% HATEOAStastic as some would like, this is a big step towards highlighting the importance of the objects instead of "lets just fire these fields at this endpoint” RPC thinking. In general I’m a fan, but there are a few small downsides noticed from my experiments. The enum output is bit weird, with lines looking like: **status:** *string* , `x ∈ { available , pending , sold }` I’m no designer so I really don’t mean to belittle anyone, but… I’d have asked a friend to help me instead of doing that. Also despite having the objects linked in there (under request and response will be a "Cheese” link), it’s not particularly clear, and if Object A embeds Object B, that’s not made clear in the example. Some systems will annotate the JSON Example with more details, such as… #### [ReDoc](https://github.com/Rebilly/ReDoc/?ref=apisyouwonthate.com) By far my favourite of the lot, ReDoc just looks absolutely stunning. You can add a logo with the `x-logo` vendor extension, tweak colors, and you get that awesome three-column style popularized by docs like the Stripe. ![Fancy/modern looking three-col API documentation from OpenAPI](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/posts/turning-contracts-into-beautiful-documentation/pet-store.png) Fancy/modern looking three-col API documentation from OpenAPI The "data structures” unfortunately don’t live in their own navigation section like Spectacle, but they are fantastic to look at. ![Animation of category docs from a CLI tool](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/posts/turning-contracts-into-beautiful-documentation/category.gif) It works a little differently to many of the CLI-based tools though, as it’s a node module that you embed into a HTML document and load through the browser. To make the CLI stuff a little easier, I made a shell script that would fire off a web server: # !/bin/sh ```bash set -e echo "Launching Documentation on [http://localhost:8088/](http://localhost:8088/)" echo "Press CTRL+C to terminate server" ruby -run -ehttpd ./docs/ -p8088 ``` Then I have a `docs/index.html` that looks like this: ```html Permissions API Documentation ``` This is just the development version so I can build out my specs as I’m adding functionality, and the actual HTML is collated for our many many APIs with a fancy Ruby called [Redocaholic](https://github.com/philsturgeon/redocaholic?ref=apisyouwonthate.com). This workflow lets your specs live in your repo, and Jenkins just rebuilds a Redocaholic-based static site every push/nightly so folks can see the wonderful API docs in their browsers. ReDoc is still only on OpenAPI v2.0, but they’re working on a [ReactJS rewrite](https://github.com/Rebilly/ReDoc/issues/312?ref=apisyouwonthate.com) which will then make the [OpenAPI v3.0 support](https://github.com/Rebilly/ReDoc/issues/312?ref=apisyouwonthate.com) easier. *I’ll certainly be pitching in to help with the OpenAPI v3.0 efforts once it’s in React, because I’m like 60% good at that, vs 20% at AngularJS.* #### Is OpenAPI The Right Tool? Breaking away from this RPC-style approach to documenting your REST APIs is hard. Some folks are coming up with [drastic new approaches](http://mfadhel.com/2017/API%5FTables/?ref=apisyouwonthate.com) which may or may not float down into the common documentation tools, but the way we’ve been doing it aint right. OpenAPI has some hypermedia support in v3.0 which hasn’t made it to any of the tools, but that functionality isn’t the best anyway… More on that another time. #### Summary Whilst there are plenty of people who think OpenAPI is a poor tool for documenting your REST APIs, I think its mostly the design of the HTML output that creates this concern. Hopefully we’ll see tools like ReDoc elevate the importance of objects over operations, but in the mean time I’m really glad that RPC APIs (and REST APIs that are used in an RPC-style) can eliminate guesswork for their consumers. *Documentation is only one benefit to having contracts for your API. I’ll be writing more about using your in-repo contracts to sync to Postman collections that stay up-to-date, integration testing, client/server-side validation, and much much more soon.* ### Commit to API Contracts URL: https://apisyouwonthate.com/blog/commit-to-api-contracts/ Last updated: 2023-03-24T12:30:01.000Z When building and maintaining a Web API, it's surprisingly common for the "I" (interface) part to be overlooked. Often new functionality is built in close contact with a specific client, with the backend and frontend developers sitting together to hash out the functionality. The frontend developer will write their side of the code, and the backend developer will write theirs. As they go, the fields and types are explained verbally, DMed over slack, dumped into a Google Doc somewhere, shoved in a wiki, or written up in HTML. These methods are only a snapshot of the "contract" at a certain point in time, and due to their useless format they will not kept up to date. > Fred: Hey Sarah, there's a new "fudge" field and it can be "blah" or "whatever" > Sarah: Great! Thanks I'll chuck that in now. After-all, why bother keeping them up to date? The code works, the iOS app runs fine, we're making money, and we're onto the next thing! Well, when another client or dependant service comes along, how do they know how to use this endpoint? #### The Hunt for a Contract When no contract is written, we have to deduce it ourselves, which is hard even when you have access to the source code. If you don't it is almost impossible, and results in guesswork. #### **Guessing from a HTTP Response** Using the request to guess what the contract could be is black magic guesswork, and is about as fun as interacting with an LED panel that has no driver or docs. One single example is not enough to extrapolate all potential variations. #### **Guessing from Source Code** When you're trying to integrate with an internal API, it can be very tempting for people to suggest that you RTFC (Read the F\*\*king Code). Even if you're familiar enough with the language and framework used, that's not always easy. Different serializers, ORMs, controllers, observers, and whatever other layers of abstraction can have sneaky effects on the data coming in and out of an API, especially when it comes to validation. Don't make people learn your entire application just to use an endpoint. #### **Guessing from Tests** Regardless of how you handle your testing, most frameworks will have a thing to unit/integration/something on controllers and check responses to give you some JSON to play with. Maybe you work with Rails and have a "request test" like: ```ruby get '/something/123' expect(response[:foo]).to be_a(String) expect(response[:bar]).to be true ``` Ok, that's great, but this test doesn't show that a `baz` field was added. The test still passes so the API team is happy about it, but the new client won't know about it. And this is when there's actually a test for this, often there isn't. #### Gaaaah! Commit to your contract by writing it down, in a standard way, and keep it up to date. It doesn't matter which of the tools you use, but if you don't use any of these tools you're doing it incredibly wrong. 1. [JSON Schema](https://json-schema.org/?ref=apisyouwonthate.com) 2. [API Blueprint](https://apiblueprint.org/?ref=apisyouwonthate.com) 3. [OpenAPI](https://www.openapis.org/?ref=apisyouwonthate.com) (formerly Swagger) 4. [RAML](https://raml.org/?ref=apisyouwonthate.com) 5. [GraphQL Types](https://graphql.org/?ref=apisyouwonthate.com) 6. [Protobuff](https://developers.google.com/protocol-buffers/?ref=apisyouwonthate.com) 7. [JSON-LD](https://www.w3.org/TR/json-ld/?ref=apisyouwonthate.com) / [Hydra](https://www.markus-lanthaler.com/hydra/?ref=apisyouwonthate.com) 8. [XML Schema](https://www.w3.org/standards/techs/xmlschema?ref=apisyouwonthate.com) 9. [WSDL](https://www.w3.org/TR/wsdl?ref=apisyouwonthate.com) 10. [OData](https://www.odata.org/?ref=apisyouwonthate.com) There's a lot of tools out there that aren't the same sort of thing. JSON Schema and OpenAPI are where I'm putting my efforts, but API Blueprint is a great, simple, starting point. You can use OpenAPI or API Blueprint with [Dredd](https://github.com/apiaryio/dredd?ref=apisyouwonthate.com) to ensure your specifications are still valid over time. These tools can create docs, mocks, contract tests, client validation, server validation, and help do amazing things. Articles coming on all of these topics, so follow for updates! Unspecified JSON (or any unexplained data in general) [makes people mad](https://apisyouwonthate.com/blog/why-some-people-dislike-json), and it's really easy to avoid pissing people off. #### Summary In my experience, fear and uncertainty lead to whole new endpoints being developed for this client, when it easily could have used the existing endpoint. This double the surface area for bugs, security issues, performance problems, and just doubles, triples, and quadruples work for no reason. Don't create a boatload of extra work for everyone who ever goes near this API. This pain is passed onto the frontend and the backend, and your whole team suffer, just because you're "too busy" to write down the contract and maintain it. ### Why Do People Dislike JSON? URL: https://apisyouwonthate.com/blog/why-some-people-dislike-json/ Last updated: 2023-03-24T12:28:51.000Z I’ve seen a few people promoting GraphQL and gRPC over REST, almost purely on the merits of being "not JSON”. They moan about JSON and instil the virtues of these architectures for their use of stricter type systems. As well as blaming JSON for various imagined faults, they continue on to blame REST for an entirely non existent reliance on JSON. REST is a architectural concept, and doesn’t care what data format you use. A REST API can receive and transmit data in any combination of JSON, XML, CSV, images, videos, binary or lolcat. It really doesn’t matter. As explained in [You Might Not Need GraphQL](https://apisyouwonthate.com/blog/you-might-not-need-graphql), if you’re mainly interested in this specific feature, your REST API can absolutely implement [Protobuf](https://developers.google.com/protocol-buffers/?ref=apisyouwonthate.com), [Cap’n Proto](https://capnproto.org/?ref=apisyouwonthate.com), or any other similar stricter type system. I posted a tweet along these lines, and got a great question back. #### Dislike of JSON When you hear somebody stating their dislike for JSON, it usually stems from the same thing: weak types. A field in JSON could be an integer in one instance, string in another, object on another response. It could change at any moment, even accidentally, and it could go unnoticed by either client or server for a while. Some weakly typed languages like PHP don’t really worry about it, as a `123` and a `"123"` are the same thing, but other languages certainly do fall over. To imagine the difference, think about data structures in programming languages. When you type hint on "object” or "array” you have no idea what shape that data will be, but if you hint against a specific class or interface name then you’ve got some assurances. JSON is like an unnamed object/array, and a lot of people don’t like things being that vague. There’s also a lack of explanation about what a field could contain. If you’re looking at a `status` field, you do not know what other values it could contain. None of this has anything to do with JSON. This would be the same if you built an API in XML or CSV, yet still people blame JSON. Data by itself is vague. Data needs metadata. This concept isn’t new, it’s just been ignored for a while, and it’s time to remember how things used to work. #### Everything Has Happened Before In the late 90's, people were building [XML-RPC](http://xml-rpc.net/?ref=apisyouwonthate.com) APIs with arbitrary fields being sent up and down. There was not much explanation about what anything was, and a string could change to an integer rather easily. This was much worse in XML as everything is a string anyway, so [XML Schema (XSD)](https://www.w3.org/standards/xml/schema?ref=apisyouwonthate.com) was developed to describe the contracts. Humans and computers then knew which types which fields were expected to be, and loads of expectations could be built around that. XML Schema was one of the core concepts of SOAP, which people then started to hate due to being overly complicated. They rushed to REST, with some people trying to "fit the entire request in the URL” as a testiment to how simple REST was. If your idea of a good API interaction is `json_decode(file_get_contents("http://api.example.org/v2/something.csv"))` then you might need to [read this book](https://apisyouwonthate.com/books/build-apis-you-wont-hate.html). Confusing REST for this oversimplified style of interaction, then blaming it for tripping you up, is exceedingly daft. ![devs be like](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/posts/rest-baton.jpeg) #### Everything Will Happen Again [JSON Schema](https://json-schema.org/?ref=apisyouwonthate.com) is a similar tool to XML Schema. You can design JSON Schema with [UML-style tools](https://www.liquid-technologies.com/json-schema-editor?ref=apisyouwonthate.com), use that JSON Schema for documentation, mocking, response validation, and all sorts of other things. In REST this stuff is optional. You can use it if you want, or you can ignore it. Even if you implement JSON Schema, nobody has to interact with it. ```js { "schema": "http://api.turtles.com/schemas/turtle.json", "id": "213-1398-fdsf", "foo": "bar", ... } ``` That’s about it. Or you can use Protobuf. Or, you can still use JSON Schema and add a step in your build chain to [generate Protobuf contracts from JSON Schema](https://www.npmjs.com/package/jsonschema-protobuf?ref=apisyouwonthate.com); Offering both depending on the `Content-Type` header. The world is your oyster once you write specifications. The ecosystem for [JSON Schema, specifications, documentation, etc](https://philsturgeon.uk/api/2017/07/20/my-vision-for-a-perfect-world-in-api-specification/?ref=apisyouwonthate.com)., is still ongoing and improving, but a lot of people are throwing the baby out with the bath water and jumping into whole API rewrites for a benefit they could easily have in their existing APIs. Most importantly, let’s not confuse people by blaming JSON or REST for problems that are nothing to do with either of them. #### Further Reading If you want an easy start with JSON Schema, there are [online editors](http://jsoneditoronline.org/?ref=apisyouwonthate.com) you can play with. Alternatively there’s [OData](https://www.odata.org/?ref=apisyouwonthate.com), [JSON-LD](https://json-ld.org/?ref=apisyouwonthate.com) and a few others kicking around. ### Partials: Happy Compromise Between Customization and Cacheability URL: https://apisyouwonthate.com/blog/partials-compromise-between-customization-and-cacheability/ Last updated: 2023-03-24T12:28:40.000Z With endpoint-based APIs (REST, RESTish, SOAP, RPC, AJAX-ish junk, etc.) you get to choose if you want increased likelihood of network cache hits, or the ability to slim down the response. The two goals are mutually exclusive. Slimming responses can be achieved with concepts like [sparse fieldsets](https://philsturgeon.uk/api/2017/08/13/a-happy-compromise-between-customization-and-cacheability/jsonapi.org/format/?ref=apisyouwonthate.com#fetching-sparse-fieldsets) (tl:dr; send `?fields=foo,bar` to just get foo and bar). This is common and popular for two reasons: 1. It takes fractionally less time to download a smaller response over the network 2. Some of those fields might be computed on request, and as such are slow Specifications like [OData](https://www.odata.org//?ref=apisyouwonthate.com) and [JSON:API](https://jsonapi.org//format/?ref=apisyouwonthate.com#fetching-sparse-fieldsets) support this, and most of their tooling also support it. The trouble is, the more you allow a response to be customized, the lower the chance of a cache hit. If two different clients (eg: the web app and the iOS app) request slightly different fields, then they’re not shared, and the network cache is keeping one set of caches for web and one for ios. That’s half as useful, and the more different clients you have, the less useful network caching becomes (if you’re using sparse fieldsets). This is a massive shame! [Varnish](https://www.varnish-software.com/?ref=apisyouwonthate.com), [Squid](http://www.squid-cache.org/?ref=apisyouwonthate.com), [Fastly](https://www.fastly.com/?ref=apisyouwonthate.com), etc., can all be a huge boost to the speed and stability of your architecture, and can save a bunch of money by reducing your application server hosting resources. For those using network caching, there are two common scenarios. ### High Customization, Low Cache Hits The Turtles.com web app requests `GET /turtles?fields=name,lifespan` which takes 200ms. The Turtles.com iOS app then requests `GET /turtles?fields=name` which takes 192ms, because the network cache did not consider this request as a match for the first request (and rightly so). Skipping the `lifespan` field made the JSON serializer do a little less work, and the time in transit was reduced too, so we saved a few ms. ### No Customization, High Cache Hits The Turtles.com web app `GET /turtles` which takes 220ms. That's a little slower as we've got all the fields (maybe 20 of them) coming back. The Turtles.com iOS app then requests `GET /turtles` the network cache considers that a hit, and short circuits the application server entirely, responding in 118ms. Even though the iOS app doesn’t need all the fields, we’ve hit the network cache, and subsequent requests are matching. This scenario is what most REST APIs do. It’s usually not an issue if fields are all precomputed and simply being fetched, but this scenario really falls apart when computed fields are part of the resource. ### Computed Fields Suck I’ve seen a simple `"is_enterprise": true` field snuck into a serializer, which then makes 3-4 SQL queries to establish if the company should be considered enterprise (if it has more than 500 members, going via a pivot table, etc). It also had `"locations"` field, which is an array of data fetched from another system. This API is a fairly slow responder on the best of days, and sometimes it has bad days. Bad days maybe this endpoint takes 1-2s instead of the 118-220ms we've been discussing. No amount of clever eager-loading or caching trickery can fully solve the fact that multiple computed fields are going to slow down all/some responses, especially when fetching a collection. Clients having their requests slowed down by being handed expensive information (that they didn’t even want) is a completely understandable source of frustration. The obvious advice is "don’t have expensive computed fields in the resource", and just pre-compute them, but sometimes these things are either unavoidable, or somebody else did it and you don’t have time to fix everything all in one go. Let’s accept them as an unfortunate reality and move on. One solution is see often a new endpoint: `GET /turtles/lean` which doesn't really help, and is occasionally a slippery slope to client and use-case specific endpoints (`GET /turtles/mobile`, `GET /turtles/ios`, `GET /turtles/ios-random-usecase-tuesday5`. That's no ideal (cache missing all over the shop and forces the team maintaining that API to know too much about specific use cases). It also does not help us with collections. Others do `?full=true` and have a "Basic" and "Full" serializer for the same response, which is a bit too binary to be considered customization. If you want to call the collection and only need one or two fields from that full version, you're stuck with *all* the fields, including whatever computed values are there which you still don't want. ### Reasonable Cache Hits, Partial Customization One solution to make your endpoint-based API responses partially customizable is… partials! Google have been recommending this solution for years as a common best practice, and YouTube was doing before that. > GET /turtles?partial=dimensions Instead of specifying fields, we specify a partial, which is like a nickname for a group of fields. Let’s stop talking about turtles for a minute, and go back to that company example. Here’s a simplified version of some code I wrote for WeWork: ```ruby class Api::V3::CompanyPresenter attr_accessor :company, :partials def initialize(company, partials: []) [@company](https://twitter.com/company "Twitter profile for @company") = company [@partials](https://twitter.com/partials "Twitter profile for @partials") = Array(partials) end def as_json { uuid: company.uuid, name: company.name, status: company.status, }.tap do |hash| hash[:locations] = locations if (partials & ['locations', 'full']).any? hash[:enterprise] = company.enterprise? if (partials & ['enterprise', 'full']).any? if (partials & ['contact_info', 'full']).any? hash[:contact_info] = { name: company.contact_name, email: company.contact_email, phone: company.contact_phone, } end if partials.include? 'full' # A bunch of other fields end end end def locations ThirdPartyFooClient.fetch_slow_ass_data end end ``` Then in the collection controller I implement: ```ruby def index companies = fetch_filtered_paginated_companies(params).map do |company| Api::V3::CompanyPresenter.new(company, partials: whitelist_partials).as_json end render json: { companies: companies } end def show company = find_company(params) company = Api::V3::CompanyPresenter.new(company, partials: whitelist_partials).as_json render json: { company: company } end def whitelist_partials if params[:partials] partials = Array(params[:include].split(',')) ['contact_info', 'locations', 'enterprise', 'full'] & partials end end ``` Now calling `GET /api/v3/companies` will result in only `uuid`, `name` and `status` being added, and if a client wants locations or any of the other allowed fields they can call them. The potential values will be advertised in OpenAPI documentation so people can find them, as this API does not even come close to having any HATEOAS descriptors or anything else. Yet. This code and this exact approach is not meant to be an example of wonderfully perfect API design. - I don’t like versions in the URL and prefer [evolution](https://www.mnot.net/blog/2012/12/04/api-evolution.html?ref=apisyouwonthate.com) - Locations should almost certainly be a sub-collection - Using [existing serializers](https://philsturgeon.uk/api/2015/05/30/serializing-api-output/?ref=apisyouwonthate.com) instead of hacking my own would be better - Using JSON:API, HAL or Siren would be more useful instead of this naked JSON Personal preferences aside, I was able to implement this partial functionality in one endpoint very quickly and easily without breaking anything, and without rewriting the entire API. Sometimes you need partial solutions, and partial customization with `?partials=` is partial enough for me. ### Further Steps Maybe make two different whitelists, one for resources and another for collections of resources. If your API has a high pagination limit (or no pagination 😱) and you’re letting `?partial=full` then you could be in for a really bad time. ### What About GraphQL So, anyone reading this who’s been hearing a lot about GraphQL would think: REST is dumb, GraphQL fixes all of this! Well no. REST provides these trade-offs for you to consider, and GraphQL forces your hand into the one option that it allows: high customization and no network cachability. As discussed in [GraphQL vs REST: Caching](https://philsturgeon.uk/api/2017/01/26/graphql-vs-rest-caching/?ref=apisyouwonthate.com), GraphQL does not work with existing HTTP endpoint-based network caching tools, so the only client options fall on the client attempting to handle it with very little insight, and cache invalidation/expires potentially being very different in two different clients, or your data all has to be restructured to support potentially any call, with any huge amount of nesting, and all of the [serialization](https://philsturgeon.uk/api/2015/05/30/serializing-api-output/?ref=apisyouwonthate.com) that comes with it. REST suggests you to make targeted endpoints with a very specific goal in mind, that handle the specific set of data in a performant way and offer a bunch of caching metadata for network tools to leverage. This should not be seen as a negative point, especially with [HTTP/2 removing the issue of "multiple handshakes = slow"](https://apisyouwonthate.com/blog/lets-stop-building-apis-around-a-network-hack). ### You Might Not Need GraphQL URL: https://apisyouwonthate.com/blog/you-might-not-need-graphql/ Last updated: 2023-03-24T12:28:32.000Z Do you like the look of GraphQL, but have an existing REST/RPC API that you don’t want to ditch? GraphQL definitely has some cool features and benefits. Those are all bundled in one package, with a nice marketing site, documenting how to do all the cool stuff, which makes GraphQL seem more attractive to many. Obviously seeing as GraphQL was built by Facebook, makers of the RESTish [Graph API](https://developers.facebook.com/docs/graph-api/?ref=apisyouwonthate.com), they’re familiar with various endpoint-based API concepts. Many of those existing concepts were used as inspiration for GraphQL functionality. Other concepts were carbon copied straight into GraphQL. Facebook has experimented with various different approaches to sharing all their data between apps; remember [FQL](https://blog.runscope.com/posts/you-might-not-need-graphql?ref=apisyouwonthate.com)? Executing SQL-like syntax over a GET endpoint was a bit odd. ``` GET /fql?q=SELECT%2Buid2%2BFROM%2Bfriend%2BWHERE%2Buid1%3Dme()&access_token=... ``` Facebook got a bit fed up with having a one-endpoint-based approach to get data and this other totally different thing, as they both require different code. As such, GraphQL was created as a middle-ground between endpoint-based APIs and FQL, the latter being an approach most teams would never consider — or want. That said, Facebook (and others like [GitHub](https://www.youtube.com/watch?v=hT-4pVmkGt0&ref=apisyouwonthate.com)) switching from RESTish to GraphQL makes folks consider GraphQL as a replacement for REST. **It is not**. It is an alternative. Whilst the use-cases for the sort of API you’d build in GraphQL are quite different from those you’d build with REST ([more on this](https://philsturgeon.uk/api/2017/01/24/graphql-vs-rest-overview/?ref=apisyouwonthate.com)) or RPC, that does not stop some folks jumping on the hot new thing. If you can withhold the urge to jump on the shiny new thing but are interested in some of the functionality GraphQL has to offer, you can brush up your endpoint-based APIs with these excellent existing concepts. **Note**: This article uses the term [REST](https://www.ics.uci.edu/~fielding/pubs/dissertation/top.htm?ref=apisyouwonthate.com) as defined in Roy Fielding’s dissertation, RESTish to mean it’s somewhere on the [Richardson Maturity Model](https://apisyouwonthate.com/blog/rest-and-richardson-maturity-model) but didn’t make it to the top, and endpoint-based APIs to mean any REST/RESTish/RPC/etc. API that uses endpoints instead of POSTing to a single `/graphql` endpoint. ### Sparse Fieldsets / Partials GraphQL allows you to specify the fields you would like to be returned, allowing you to skip all data that is not relevant to your response. This makes the request a little bit faster to download over the network, as the tubes do not get quite so full. REST certainly does not talk about this out of the box, because REST does not concern itself with such implementation specifics. The practice is however very common in the endpoint-based API world. A common standard for REST APIs to implement is [JSON-API](https://jsonapi.org//?ref=apisyouwonthate.com), which talks about [sparse fieldsets](https://jsonapi.org//format/?ref=apisyouwonthate.com#fetching-sparse-fieldsets). The basic idea is that you can specify the fields in the request: ``` GET /articles?fields[articles]=title,body ``` In the past YouTube had some [really whacky partial syntax](https://developers.google.com/youtube/2.0/developers%5Fguide%5Fprotocol%5Fpartial?ref=apisyouwonthate.com) for this sort of thing: ``` GET /feeds/api/users/default/uploads?fields=entry(title,gd:comments,yt:statistics) ``` Facebook also has this in their Graph API: ``` GET /frankcarter?fields=id,name,picture ``` Read more on building your own implementation of this if you’re interested, but I would stick to using existing solutions like [ActiveModel::Serializer](https://github.com/rails-api/active%5Fmodel%5Fserializers/blob/master/docs/jsonapi/schema.md?ref=apisyouwonthate.com) (Rails), [tobscure/json-api](https://github.com/tobscure/json-api?ref=apisyouwonthate.com#sparse-fieldsets) (PHP), etc. which handle standard implementations for you. ### Types / Schemas Folks talking about GraphQL get really excited about the concept of Types. JSON can be a little vague when it comes to types. Requests require a lot of validation, and responses require clarification. Some weakly typed languages might send a numeric string when they really meant to send an integer. A numeric JSON response field could look like an integer one minute, but then a wild decimal place appears. > GraphQL APIs are organized in terms of types and fields, not endpoints. Access the full capabilities of your data from a single endpoint. GraphQL uses types to ensure Apps only ask for what’s possible and provide clear and helpful errors. > **Source:** [graphql.org](https://graphql.org/?ref=apisyouwonthate.com) Many endpoint-based API developers solve this with [HTTP documentation](https://philsturgeon.uk/api/2015/10/08/http-documentation-with-api-blueprint/?ref=apisyouwonthate.com) using tools like [MSON](https://apiblueprint.org/documentation/mson/tutorial.html?ref=apisyouwonthate.com) to describe their data, but another approach is to use [JSON Schema](https://json-schema.org/?ref=apisyouwonthate.com). JSON Schema is very cool and lets you **describe your JSON** using a **JSON metadata file**. This metadata file can be linked into your main JSON outputs, letting JSON Schema-aware clients discover the metadata over the wire. One of the many benefits of this is allowing client apps (like a web frontend) to **validate their form data using the exact same rules as the server-side**, without needing to go over the wire. 😲 Taken from [JSON Schema examples](https://json-schema.org/examples.html?ref=apisyouwonthate.com): ``` { "title": "Person", "type": "object", "properties": { "firstName": { "type": "string" }, "lastName": { "type": "string" }, "age": { "description": "Age in years", "type": "integer", "minimum": 0 } }, "required": [ "firstName", "lastName" ] } ``` Comically, all of these concepts (type systems, schemas, metadata in general, etc.) is exactly what most people hated about SOAP. Instead of just the payload, you had to mess about with a WSDL, and that lead folks to enjoy REST. Well, in REST, using something like JSON Schema, you have the choice of using them, as do your clients. Aren’t interested in JSON? Look into Protocol Buffers like [ProtoBuff](https://developers.google.com/protocol-buffers/?ref=apisyouwonthate.com) or [Cap’n Proto](https://capnproto.org/?ref=apisyouwonthate.com) . They’re an identical concept to the GraphQL type system, and have been implemented in a bajillion languages too. ### Evolution / Versioning Versioning is a really big, muddy, awful topic in the world of APIs, and most of us agree that [every approach is a minefield](https://www.troyhunt.com/your-api-versioning-is-wrong-which-is/?ref=apisyouwonthate.com) . Some folks will version in the URI, making `/v1/foos` and `/v1/bars`, then make `/v2/foos` and `/v2/bars`. If nothing changed between v1 and v2 for bars, anyone using the API doesn't know that, and has to go read some documentation to find out before they can upgrade. That leads to slow uptake, and now you have two endpoints for ages. Some folks will start the same, with `/v1/foos` and `/v1/bars`, but then only create `/v2/foos`, leaving `/v1/bars` in place. That can be slightly better, but if (for example) the `Api::v1::AppController` and `Api::v2::AppController` have ever-so-slightly different error formats and your client is not aware of that, then a v1 or v2 error might pop up when the client is only coded to support v2 or v1\. I've seen this cause a JavaScript error in production that broke the app. That whole mess of nonsense can be avoided by not versioning your API. As daft as this might initially sound, if you can **avoid changing a contract**, that means **less work for clients**. The server can handle converting data from one format to another, or a new representation can eventually be created to replace the old representation. Replacing representations and fields carefully over time as things change is called [evolution](https://www.mnot.net/blog/2012/12/04/api-evolution?ref=apisyouwonthate.com) . At a previous company offering crowd-sourced carpooling, we switched from `/matches` to `/riders`, and internally those two representations shared a lot of code. "Matches" was deprecated, and clients started using the new "riders" concept. Over a few months, the internals changed to a much cleaner solution for riders. We eventually dropped the matches endpoint/serializers/logic entirely, without the riders contract changing. That helped us move fast on the code, but keep our contracts relevant for our clients until they didn't need them anymore. That is not always possible in a rapid-application environment where things change drastically all the time. Startups love their "agile philosophies”, pivots, etc., which can throw evolution out the window. As somebody who has worked for those companies, we would simply make a new API (copy, paste, tweak) and throw the old one out once a new iPhone app was launched and usage dropped below acceptable levels. There was no interest in "keeping the service alive for decades”, which is a core tenant of REST, but that’s ok, we were RESTish. As pointed out in my [GraphQL vs REST: Overview](https://philsturgeon.uk/api/2017/01/24/graphql-vs-rest-overview/?ref=apisyouwonthate.com) , easily being able to **track field usage by clients** is something that gives GraphQL an edge on most endpoint-based APIs. If a client wants to use field "foo” and you want to remove it, you know that their app will break. Earlier in the article, we looked at sparse fieldsets, which can help here. If the endpoint-based API offers sparse fieldsets as an option, and clients use them, there will be trackable insight into which clients are using specific fields just like GraphQL. One downside here is that only clients requesting `?fields=` will be detectable, unless the API goes a step further and requires the use of `?fields=`! I don’t know if I would do it, but it’s an option. ### Query Language GraphQL is primarily a query language, but if you’d like an endpoint-based API to have a query language, you can give it one. [OData](https://www.odata.org/?ref=apisyouwonthate.com) has a strong slogan: the best way to REST. That’s a big claim, but OData provides a lot of stuff that people seem to like about GraphQL. Not only does it offer [machine-readable metadata](https://docs.oasis-open.org/odata/odata/v4.0/odata-v4.0-part3-csdl.html?ref=apisyouwonthate.com) similar to JSON Schema, and a powerful explorer similar to [GraphiQL](https://pragmatiqa.com/xodata/?ref=apisyouwonthate.com) , but it [provides a syntax](https://www.odata.org/getting-started/basic-tutorial/?ref=apisyouwonthate.com) and tooling to allow you to use a query language: ``` GET /Airports?$filter=contains(Location/Address, 'San Francisco') ``` Or even: ``` GET serviceRoot/People?$filter=Emails/any(s:endswith(s, 'contoso.com')) ``` OData has some other boring grown-up benefits, like allowing your endpoint-based API to be the source of truth for Salesforce, utilizing External Objects. Basically, instead of making janky sync logic between your applications and theirs, a custom object can be created to let the data live only in the one OData API, but still look like it’s sitting in Salesforce. Guess what I’m working on at the moment. ### Data Inclusion / Compound Documents JSON-API suggests an ability to [include related data](https://jsonapi.org/format/?ref=apisyouwonthate.com#fetching-includes) from multiple resources in a single HTTP request. That is a big pro for some, as it reduces the number of HTTP requests, which under the right conditions can often speed things up for the client. It could also slow it down a bunch, but usually, it’s a helper. These includes are very similar to the nested field queries possible in GraphQL. When using JSON-API not only can you fetch related resource representations, but you can trim those related representations down using sparse fieldsets too: ``` GET /articles?include=author&fields[articles]=title,body&fields[people]=name ``` That will get you a list of articles with only the title and body fields, then the authors will be included with only their name. Endpoint-based APIs offering compound documents handle it in a myriad of ways, but again the JSON-API approach is a common one. The JSON-API approach makes some clients sad because data is "side-loaded”, which basically means included resources are jammed into a single array. Excuse the large JSON blob, but it’s important to understand the concept: ```js { "data": [ { "type": "articles", "id": "1", "attributes": { "title": "JSON API paints my bikeshed!" }, "links": { "self": "http://example.com/articles/1" }, "relationships": { "author": { "links": { "self": "http://example.com/articles/1/relationships/author", "related": "[http://example.com/articles/1/author](http://example.com/articles/1/author)" }, "data": { "type": "people", "id": "9" } }, "comments": { "links": { "self": "http://example.com/articles/1/relationships/comments", "related": "http://example.com/articles/1/comments" }, "data": [ { "type": "comments", "id": "5" }, { "type": "comments", "id": "12" } ] } } } ], "included": [ { "type": "people", "id": "9", "attributes": { "first-name": "Dan", "last-name": "Gebhardt", "twitter": "dgeb" }, "links": { "self": "http://example.com/people/9" } }, { "type": "comments", "id": "5", "attributes": { "body": "First!" }, "relationships": { "author": { "data": { "type": "people", "id": "2" } } }, "links": { "self": "http://example.com/comments/5" } }, { "type": "comments", "id": "12", "attributes": { "body": "I like XML better" }, "relationships": { "author": { "data": { "type": "people", "id": "9" } } }, "links": { "self": "http://example.com/comments/12" } } ] } ``` That article "1” has two relationship types, `author` and `comment`. These relationships might have one name, and the actual resource type could be another, so author is the relationship name but people is the data type. Cool. So, if they have `"data": { "type": "people", "id": "9" }` and `{ "type": "comments", "id": "5" }, { "type": "comments", "id": "12" }`, that means if `?include=author,comments` is in the query string, it will expand those relationships, "including" the data in the "included" section of the JSON body. These included items are all just shoved into a single array, with no hierarchy or concept of how they relate to the article. Anyone who calls this API will not just be able to call the data: ```python response = client.get('/articles?include=author,comments') article = response.body comments = article.comments author = article.author ``` Instead, there needs to be some logic that pulled out all the comments and all the authors, then in a loop you could stitch them back together. Writing this is awful, but of course loads of people have built generic abstraction layers in various languages to allow you not to have to. I used to really hate this, but respect the fact that it reduces data going over the wire, by de-duplicating the expanded forms of the same resource. E.g.: Instead of returning the same author details multiple times, it’s just there once. Side-loading is a common convention for many endpoint-based APIs, but it has nothing to do with REST. That is simply how JSON-API happens to do things and is one common standard amongst a few. An endpoint-based API can easily nest data in a more relational way just like GraphQL. I built a tool to do this in PHP years back called [Fractal](https://fractal.thephpleague.com/?ref=apisyouwonthate.com) (before going over to the JSON-API-side). I don’t think people need to spend a huge amount of time trying to make their endpoint-based APIs do all this stuff just to be cool, but I do think there are concepts in GraphQL that resonate with people who might not know they’re already available to them. As impressed as I am with GraphQL, it is not always the shiny magical hammer-for-everything some people think it is, and making informed decisions is important. Knowing about these tools and concepts can help people push back against an overzealous switch to a whole new system, because the folks advocating it were unaware that a lot of the concepts used in GraphQL are not entirely new or unique to GraphQL itself. That said, I do of course think **having all of these concepts implemented and documented in one single "package” like GraphQL is super handy**. GraphQL removes the arguing or confusion about what is "the most RESTful way to do something”, as it has a spec and example implementations. A lot of APIs that are merely RESTish could certainly be GraphQL, but an actual REST API with HATEOAS would not make sense trying to jam itself into the GraphQL paradigm. State Transfer and Query Languages are different things, but as I’ve shown, you can blur the lines a little if you’re interested. ### GraphQL vs REST: Caching URL: https://apisyouwonthate.com/blog/graphql-vs-rest-caching/ Last updated: 2023-03-24T12:28:24.000Z Recently I wrote [GraphQL vs REST: Overview](https://apisyouwonthate.com/blog/graphql-vs-rest-overview), giving a hype-free outline of the differences between [REST](https://en.wikipedia.org/wiki/Representational%5Fstate%5Ftransfer?ref=apisyouwonthate.com) and [GraphQL](https://graphql.org/?ref=apisyouwonthate.com). One section that would not have fit into that already lengthy article was caching, so I thought I'd cover this on its own. Comparing how caching is used for the two approaches is tricky, because despite common opinion, REST has nothing to do with HTTP. REST is *usually* implemented in HTTP, therefore when most folks talk about ["caching in REST"](https://apiacademy.co/2016/02/how-to-http-caching-for-restful-hypermedia-apis/?ref=apisyouwonthate.com), they are really talking about caching with HTTP. That is a [huge topic in its own right](https://www.mnot.net/cache%5Fdocs/?ref=apisyouwonthate.com). Another point to mention is that caching for HTTP-based REST, or HTTP-based RPC, is pretty much the same. In the wild I notice RPC APIs offer caching far less, but for the sake of argument let's ignore this separation and describe them both as "endpoint-based APIs"; a term used by the GraphQL folks to differentiate APIs able to leverage the entirety [of HTTP caching](https://tools.ietf.org/html/rfc7234?ref=apisyouwonthate.com) to their advantage, from GraphQL APIs which usually just use HTTP as a "dumb" tunnel. Let's have a quick look at the different types of caching that can be involved in an endpoint-based API: **Client Caching:** Also referred to as "browser caching", the client, browser, app, other service, etc. downloading the data can keep track of what was downloaded when, if that data had any expire time, ETags for the last request to allow for conditional request if data has changed, etc. **Network Caching:** Tools like [Varnish](https://www.varnish-cache.org/?ref=apisyouwonthate.com) or [Squid](http://www.squid-cache.org/?ref=apisyouwonthate.com) intercept requests that look the same (based on various configurable criteria), returning a response early straight out of memory, instead of hitting the application server. This allows allows the application server to spend more time handling other traffic. **Application Caching:** Software like Memcache, Redis, etc. can be implemented in your application, to cache various things like datastore queries, which should make responses quicker to generate. Ideally all three concepts would be used as often as possible, as doing slow things multiple times is slow, boring, unnecessary, and probably contributing to climate change or something. Whenever you mentioning caching and speed, somebody will descend from the rafters screaming "Caching should not be used for performance!" Erf. Correct, caching should not be implemented to "make the endpoint load faster", because no matter how fast the cache implementation returns, unless the cache is warmed it is still going to have that initial slow load. Doing a slow thing once, then doing that same thing quicker after, is no good for those suffering the initial slow request. API developers should always attempt to optimize endpoints to perform quickly, making sure data is structured intelligently, in a way that is quick to query, has optimized indexes, using no computed values (or are computed ahead of time, etc.). This article isn't going to confuse caching and performance, only focus on removing unnecessary cycles happening for your application server. Not doing something is usually quicker and cheaper than doing something. Each of the three caching approaches mentioned are helpful in avoiding doing stuff unnecessarily. An endpoint-based API can start off adding `Etag` and `Cache-Control` tags in the application itself. The application can control if content is cacheable or not, for how long, allowing the client to look out for these rules and handle the cache locally (an iOS app for example, remembering a result and not making the same request again). Shoving network caching (HTTP cache proxies like [Varnish](https://www.varnish-software.com/?ref=apisyouwonthate.com), [Squid](http://www.squid-cache.org/?ref=apisyouwonthate.com) or a hosted Varnish service called [Fastly](https://www.fastly.com/?ref=apisyouwonthate.com)) in front of that application server can then save that application server work. ![A request being returned early by a varnish server](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/posts/graphql-rest-caching/cache-miss.png) A request being returned early by a varnish server. — [Varnish Software](https://book.varnish-software.com/?ref=apisyouwonthate.com) ![A request failing to find a match](https://storage.ghost.io/c/b4/3e/b43eb8f8-cafd-49ee-ab1f-e4fe989d2890/content/images/posts/graphql-rest-caching/cache-miss.png) A request failing to find a match (a.k.a cache miss), and being passed on to the API server to fulfill. — [Varnish Software](https://book.varnish-software.com/?ref=apisyouwonthate.com) These tools can leverage HTTP headers like `Etag`, `Vary`, `Cache-Control` to handle cache validation, and know all the rules of HTTP, meaning this application caching can essentially be thrown in and function with very little effort from the API developers. Clients will get a speed boost without even having to implement their own client caching, even though they still could, and still should, as requesting data over the wire from the cache server is still slower than not requesting data. ## GraphQL and HTTP Caching GraphQL has been designed to easily operate in any transfer/transportation protocol: HTTP, AMQP, IMAP... It will fit easily into any protocol because instead of leveraging the existing interface of the protocol its using, it invents its own. This makes it hard to use existing caching tooling, monitoring tooling, or anything else that does not know how to inspect a GraphQl message to see what is going on. ### Network Caching Most GraphQL setups will POST against a single endpoint by default, demoting HTTP to the role of a dumb tunnel, and making network caching tools like Varnish, Squid, Fastly, etc. entirely useless as POST is not cacheable. GraphQL can be configured to run over GET, but that requires clients to change to using GET in order for network caching to start working, and there's a few more concerns. Two clients hitting `/foo` will both share a network cache, but when they have to start requesting fields one is going to request `{ a, b }` and another will request `{ b }` and a third might be `{ b, a }`. These are all different entries in a network cache, which will fill the cache up and make it hard to get a cache hit, resulting in more roundtrips for more clients. More on this later, but "just switch it to GET" is not the solution many people think it is. ### Client Caching With network caching essentially removed from the equation, the responsibility of caching falls on the API developers to do application caching, and/or client caching. The GraphQL site seems happy to suggest the client should take care of caching on their end: > In an endpoint-based API, clients can use HTTP caching to easily avoid refetching resources, and for identifying when two resources are the same. The URL in these APIs is a globally unique identifier that the client can leverage to build a cache. In GraphQL, though, there's no URL-like primitive that provides this globally unique identifier for a given object. It's hence a best practice for the API to expose such an identifier for clients to use. — Source: [graphql.org](https://graphql.org/learn/caching/?ref=apisyouwonthate.com) In an ideal world, all clients would implement client caching, but a lot don't. Many HTTP clients (gems, packages, etc. that make the actual HTTP requests, or HTTP layers for frameworks like EmberData) either make it easier, or handle it seamlessly for clients, but they utilize the same parts of the HTTP specification that cache proxies do: `Etag`, `Cache-Control`, etc. These approaches work for endpoint-based APIs, but not for GraphQL. Without that meta data, client caching automatically for GraphQL becomes seemingly impossible. Facebook has released [DataLoader](https://github.com/facebook/dataloader?ref=apisyouwonthate.com) which can make GraphQL client caching easier, but clients not using NodeJS will have to port the logic, or continue to roll their own. If client caching is ignored (seen as too complicated, or not advertised as useful well enough), an "endpoint-based" API is still provided some protection against repeated requests by utilizing network caching. Endpoint-based APIs can cache in all three ways, but GraphQL can only handle the two, and one of those kinda poorly. Maybe some GraphQL specific cache proxies exist, but this software will not be as battle tested as the HTTP cache proxies. REST is potentially onto a winner here. So long as application and network caching are actually implemented. This is not always the case, but it's highly recommended, and adding it is much easier than changing your entire API ecosystem to another paradigm. For clients, it simply involves [adding a middleware to whatever HTTP library](https://apisyouwonthate.com/blog/http-client-response-caching) they are using. ## Customization The flip side of that coin is this: the more `customisable` an endpoint-based API becomes, the higher the cache miss rate, and the less useful network caching becomes for that API. GraphQL makes you specify the fields you would like returned by default. REST APIs can do the same thing, but does not do it by default. Whatever the API is using, customization will reduce the likelihood of a cache hit. Consider this example for an endpoint-based API with network caching, that offers sparse fieldsets: 1. Client A requests `GET /turtles?fields=name,lifespan` responds in 200ms 2. Client B requests `GET /turtles?fields=name` 192ms Client B was hoping for a quicker response thanks to requesting a smaller response, which is kinda true as it shaved a bit of time off, but sadly it resulted in a cache miss as the network cache would consider those two URLs entirely different requests. If the application caching was not set up appropriately, or the data is not something that is cacheable, then both requests would be fairly slow. The same scenario for an endpoint-based API with network caching, that **does not** offer sparse fieldsets: 1. Client A requests `GET /turtles` responds in 220ms 2. Client B requests `GET /turtles` responds in 118ms Because Client A and Client B are being less picky with their responses, they can reuse the request, saving the application server from getting involved, and making the "page load quicker" or whatever is going on. If an endpoint-based API is super customisable, network caching becomes somewhat less useful, turning it into a complex micro-optimization, and forcing an API down the route of really thinking through their data caching, and data structures, just the same as GraphQL API developers are forced to do. Building a generic API that is expected to respond to all clients fairly similarly? Then using endpoint-based APIs (REST 👍) will make caching easy for everyone, and GraphQL misses out on a point here. Building a super-flexible API for a lot of different clients making loads of different requests? Either GraphQL or endpoint-based APIs will have the same limitations when it comes to application caching and structuring data, so use other pros and cons to help you decide. ## Summary However you look at it, GraphQL certainly isn't "better" at caching. It forces application developers to structure their data in a more scalable way, which is less of a feature and more of a necessity. In my experience this sort of thing goes against the grain of the MVP and Agile methodologies so many developers work within these days, and goes against the sales pitch of "GraphQL is easy for beginners" and "does everything out of the box" which is so often pitched. If you care about HTTP caching, [stick to a paradigm](https://apisyouwonthate.com/blog/picking-api-paradigm) which leverages HTTP instead of ignoring it. Even if some of your data is real-time, or private, HTTP lets you split that up over various endpoints, and thanks to HTTP/2 that's not not a performance issue like it was in the olden days of HTTP/1.1. Frontend developers stopped using images sprites, and API developers no longer need to use JSON sprites. Splitting into multiple requests via HTTP/2 is considered best practice instead of the old approach to bundling, which is what GraphQL (and [other compound document standards](https://apisyouwonthate.com/blog/lets-stop-building-apis-around-a-network-hack)) are all about. ### Representing State in REST and GraphQL URL: https://apisyouwonthate.com/blog/representing-state-in-rest-and-graphql/ Last updated: 2023-03-24T12:30:12.000Z Representing state is a complex thing. At my last two jobs, it’s been very common for APIs to be treated like "databases over HTTP”. The fields are sent up and down from the server to multiple mobile/web apps, and there’s not too much else going on. Over time, we noticed this specific problem happening over and over again: When you ask the clients to infer state from the fields alone, they often infer things differently. More than that, even if they infer the correct state *right now*, it might not be the correct state in a few months. What the hell am I going on about? An example! Let’s talk about invoices. Our API could be REST, RESTish or GraphQL, doesn’t matter. We have a basic object with a bunch of date fields: ```js { "data": { "type": "invoice", "id": "093b941d", "attributes": { "published_at": "2017-06-15 12:31:01Z", "sent_at": "2017-06-15 12:34:29Z", "paid_at": null } } } ``` We tell the clients that if there is no paid date, then it has not been paid, and the same logic applies for sent. It might look a bit like this: ```js if (model.paid_at) { status = 'paid'; } else if (model.sent_at) { status = 'sent'; } else if (model.published_at) { status = 'published'; } else { status = 'draft'; } ``` This means we can display `Status: Sent` on our various client applications, and it's likely all the apps got that right. Soon we notice a situation where the payment is initiated, but the payment bounces. Maybe we add another field, and this whole "inspect the dates” thing starts to fall apart. ```js { "data": { "type": "invoice", "id": "093b941d", "attributes": { "published_at": "2017-06-15 12:31:01Z", "sent_at": "2017-06-15 12:34:29Z", "paid_at": "2017-06-16 09:05:00Z", "payment_received_at": null } } } ``` A client inferring state from those fields would consider that invoice paid. Another client would consider it to be an unpaid invoice, as there was a failed payment. If a new "published” state was added, they’d be showing as draft. Mess all over. Of course, folks will say "well you shouldn’t change things without versioning”, but even if we started with the 2nd example of that JSON and never changed anything, multiple clients could interpret those fields differently. Changing things without versioning is sometimes known as [API evolution](https://www.mnot.net/blog/2012/12/04/api-evolution?ref=apisyouwonthate.com), and whilst you shouldn’t go running about breaking things willy nilly, you should be able to add to an API without clients exploding. GraphQL is advertising people use evolution right on their homepage, so it’s not an alien concept. #### API != Database Inferring states from dates or other arbitrary fields is awful, and it *always* goes wrong. I noticed a similar issue with Postmates Fleet saying "Waiting on verification of your profile image” on the mobile app, and the web app said "Please upload a profile image”. If I dug into the code, I’d bet they were looking for an "image verified” switch of some sort, and the web app just happened to notice a field that the mobile app didn’t. This is one of many many many reasons why folks need to stop treating an API like it is purely SQL-over-HTTP. The data can be stored like this in the *database*, but exposing it in the contract is begging for trouble, as clients will always end up with a slightly different picture of the current state. How do you avoid this? State machines! In the world of Ruby there are loads of options, but [Statesman](https://github.com/gocardless/statesman?ref=apisyouwonthate.com) is super simple, and [AASM](https://github.com/aasm/aasm?ref=apisyouwonthate.com) is cool. Let’s look at this just in code, and ignore HTTP for a second. ```ruby class InvoiceStateMachine include Statesman::Machine state :draft, initial: true state :published state :sent state :failed state :paid transition from: :draft, to: :published transition from: :published, to: [:draft, :sent, :paid] transition from: :sent, to: [:failed, :paid] transition from: :failed, to: :paid guard_transition(to: :sent) do |invoice| invoice.has_contact_info? end before_transition(to: :sent) do |invoice, transition| EmailService.new(invoice).send_contact_invoice invoice.touch(:sent_at) end after_transition(to: :failed) do |invoice, transition| EmailService.new(invoice).send_contact_failure EmailService.new(invoice).send_owner_failure invoice.touch(:failed_at) end after_transition(to: :paid) do |invoice, transition| EmailService.new(invoice).send_owner_success invoice.touch(:paid_at) end end ``` Now we can do this: ``` invoice.current_state # => "draft" invoice.allowed_transitions # => ["published"] invoice.can_transition_to?(:sent) # => true/false invoice.transition_to(:paid) # => true/false ``` This gives us a lot of ability to assert a single "status”, and you know the server has everything under control. We’ve not had to spread that logic throughout persistence layer logic, controller logic, and arbitrary classes. It’s all in the state machine, and the main thing is that we no longer have to ask the clients to try and guess what’s up with the invoice. #### Exposing State over HTTP Just like the Ruby example, we want to know the current state, and we want to know what we can do next. First thing? Shove that `current_state` property in your serializer to expose a `"status": "draft"` field in the output. This'll work fine for both RESTish APIs and those built with GraphQL. How about the "what to do next” bit? Well, this is exactly what HATEOAS is! #### HATEOAS I CALL ON THEE "Hypermedia As The Engine Of Application State” is a concept that’s ignored by many, but it’s what makes a REST API so powerful. At it’s most basic, starting to implement some HATEOAS in your API would look like this: ```js { "data": { "type": "invoice", "id": "093b941d", "attributes": { "created_at": "2017-06-15 12:31:01Z", "sent_at": "2017-06-15 12:34:29Z", "paid_at": "2017-06-16 09:05:00Z", "payment_received_at": null, "status": "published" } }, "links": { "pay": "https://api.acme.com/invoices/093b941d/payment_attempts" } } ``` The existence of that `pay` link can be used to let the various client apps know they should show the "Pay" button. If it wasn't `published` that link wouldn't be there. This theoretically works, although it is not super clear exactly what a client needs to do from the existence of this pay link alone. What HTTP method should be used? What fields need to be sent? What mime type should be put in `Accept`? There are a few ways HATEOAS can help. #### **OPTIONS + Meta Data** A client could call `OPTIONS /invoices/093b941d/payment_attempts` and get a response with metadata about the document. What actions are available. What fields can be updated. What data do those fields expect? Sometimes people make [homegrown solutions](http://zacstewart.com/2012/04/14/http-options-method.html?ref=apisyouwonthate.com), and some folks leverage tools like [JSON Schema](https://json-schema.org/?ref=apisyouwonthate.com) for the fields part. #### **Hyper Schema** Another approach is [JSON Hyper-Schema](https://json-schema.org/latest/json-schema-hypermedia.html?ref=apisyouwonthate.com), which is a draft spec. Is essentially an extension to JSON Schema, which adds `links` keywords! Instead of making your own `OPTIONS` metadata and linking to JSON Schema, this Hyper-Schema could be the entire OPTIONS response! If hiding it behind options seems weird, you can also/either place a link in the response document: ```js { "schema": "http://api.acme.com/schemas/invoice/093b941d" "data": { ``` Using metadata to let clients know what data they should send is very cool, as it offers a method for client-side validation which matches server-side validation perfectly. Now your various applications can use that JSON Schema to validate data locally before even trying the POST (saving time and reducing traffic to the server). #### **Hypermedia-friendly Formats** Instead of two requests, another approach is combining the metadata with the response document. Our example so far has been using [JSON-API](https://jsonapi.org//?ref=apisyouwonthate.com), so to continue using that: ``` "links": { "pay": { "href": "https://api.acme.com/invoices/093b941d/payment_attempts" "meta": { "method": "POST", "type": "application/json" } } } ``` This isn’t really part of the JSON-API specification, but [it is valid](https://jsonapi.org//format/?ref=apisyouwonthate.com#document-links). Inventing your own standards and conventions can be a pain in the ass, so maybe don’t bother. There’s a standard for that: [Siren](https://github.com/kevinswiber/siren?ref=apisyouwonthate.com). Check out this potential JSON response. ``` { "class": [ "invoice" ], "properties": { "id": "093b941d", "all_the_other": "fields", "so_many": "other_fields", "status": "published" }, "entities": [ { "class": [ "items", "collection" ], "rel": [ "http://acme.com/rels/pay-invoice" ], "href": "https://api.acme.com/invoices/093b941d/payment_attempts" } ], "actions": [ { "name": "pay-invoice", "title": "Pay Invoice", "method": "POST", "href": "https://api.acme.com/invoices/093b941d/payment_attempts", "type": "application/json", "fields": [ { "name": "invoice_number", "type": "hidden", "value": "42" }, { "name": "amount", "type": "number" }, { "name": "stripe_token", "type": "text" } ] } ], "links": [ { "rel": [ "self" ], "href": "http://api.acme.com/invoices/093b941d" }, { "rel": [ "previous" ], "href": "http://api.acme.com/invoices/a46c437c" }, { "rel": [ "next" ], "href": "http://api.acme.com/invoices/ca0e7f36" } ] } ``` Perfect! We know which HTTP method to use. We know what fields to send. We know the data types of the fields. We don’t know a huge amount about what to put in those fields, but it’s a start. How easy would it be to hook that response up to an "Actions” dropdown, and dynamically have the interface built out from it? You could roll out certain features to your client applications *without touching the client code*. You can finally fire Gary! There’s quite a few [other data formats](https://sookocheff.com/post/api/on-choosing-a-hypermedia-format/?ref=apisyouwonthate.com) that support hypermedia, including the awesomely named [Hydra](http://www.markus-lanthaler.com/hydra/?ref=apisyouwonthate.com). #### Actions seem unRESTy Something I’ve said [fairly recently](https://www.smashingmagazine.com/2016/09/understanding-rest-and-rpc-for-http-apis/?ref=apisyouwonthate.com) is that "actions are RPC” (‘remote procedure call’), which is… semi accurate. If an API is nothing but actions then it is probably better off as RPC. I’m building a permissions API that accepts a bunch of parameters, then returns true or false if the user has permission. That can stay as RPC. REST is all about a beautiful mixture of data, relationships and actions. Taken from the Siren homepage is this lovely quote: > It’s important to note the distinction between link relations and classes. Link relations define a relationship between two resources. > Classes define a classification of the nature of the element, be it an entity or an action, in its current representation. They continue to fit their "add item” link in with the usual expectations for collections and resources, and the add item is still `"href": "http://api.x.io/orders/42/items",`. This mixture of actions and data actually lines up rather well with what I've said in that article. A [recent talk from Mike Amundsen](http://www.amundsen.com/talks/2017-04-craftconf/index.html?ref=apisyouwonthate.com) goes through a number of topics, but specifically Hypermedia (HATEOAS) as a collection of affordances (potential actions that can taken). This quote was up in there: Information and actions, displayed up to a user through a self-documenting format of awesomeness, with a selection of links that turn a well-tuned client into a crawler instead of just being a CRUD exchange… well that’s the whole point of REST. #### To HATEOAS or not to HATEOAS We know that if you don’t have HATEOAS, [you only have a RESTish API](http://roy.gbiv.com/untangled/2008/rest-apis-must-be-hypertext-driven?ref=apisyouwonthate.com). HATEOAS is a really useful concept, which solves so many issues I’ve run into time and time again. I don’t think every API needs to use HATEOAS by any means, and simple (micro)services often wont. Regardless of needing it, I definitely think people should understand what it is. The main reason? People should know what they’re missing when they call GraphQL "REST 2.0”. GraphQL has no ability to offer HATEOAS\*, and as such is essentially a *subset* of REST, and not a "new version”. A lot of the [cool stuff you can do in GraphQL can be done in REST](https://blog.runscope.com/posts/you-might-not-need-graphql?ref=apisyouwonthate.com), but GraphQL has no HATEOAS. Maybe think about that. Sure waiting for links before working out what to do next is slower for mobile applications that need to function as blazingly fast as possible over crap networks, but REST has solutions to that, and to be honest server-to-server doesn’t always need to trim bits. If you’re really set on using GraphQL but want to make sure you’re representing state and not forcing clients to guess, at least implement a state machine and add that `status` field. Finally, if you’re not doing any of this stuff then stop calling it a REST API. RESTish will do. 👍 *Thanks to the small army of people who battled through my nonsense first drafts in an effort to make me sound intelligent. You know who you are, but especially* [*@dstockto*](https://twitter.com/dstockto?ref=apisyouwonthate.com)*,* [*@glasnt*](https://twitter.com/glasnt?ref=apisyouwonthate.com)*, and* [*@mwop*](https://twitter.com/mwop?ref=apisyouwonthate.com)*.* *\* I’m sure you could find some way to hack HATEOAS support into GraphQL, like a field with an array of potential mutators that are available, but it’ll be a bit of a mess and you’d need to request specific pieces of metadata making the queries huge. The majority of REST & RESTish APIs have ignored HATEOAS for years, so I don’t expect a stampede of GraphQL developers trying to mash it into a query language that was designed to exclude it.* ### GraphQL vs REST: Overview URL: https://apisyouwonthate.com/blog/graphql-vs-rest-overview/ Last updated: 2023-03-24T12:28:08.000Z A few months back I wrote [a comparison between RPC and REST](https://www.smashingmagazine.com/2016/09/understanding-rest-and-rpc-for-http-apis/?ref=apisyouwonthate.com) for Smashing Magazine, and now I want to talk about the differences between REST and GraphQL: the new kid on the block. [GraphQL](https://graphql.org/?ref=apisyouwonthate.com) is incorrectly considered by some to be a "replacement" to [REST](https://en.wikipedia.org/wiki/Representational%5Fstate%5Ftransfer?ref=apisyouwonthate.com). GraphQL is a newer concept, being released by Facebook publicly in 2015, whereas REST was a [dissertation](https://www.ics.uci.edu/~fielding/pubs/dissertation/top.htm?ref=apisyouwonthate.com) published by Roy Fielding in 2000, popularized by companies like Twitter (quite inaccurately) in 2006. This article aims to cover a few notable differences, and make the following points: 1. REST and GraphQL are totally different 2. GraphQL isn’t a magic bullet, nor is it "better" 3. You can definitely use both at the same time 4. GraphQL is dope *if used for the right thing* ### A few quick differences REST is an [architectural concept](https://www.ics.uci.edu/~fielding/pubs/dissertation/rest%5Farch%5Fstyle.htm?ref=apisyouwonthate.com#sec%5F5%5F3) for network-based software, has no official set of tools, has no specification, doesn’t care if you use HTTP, AMQP, etc., and is designed to decouple an API from the client. The focus is on making APIs last for decades, instead of optimizing for performance. GraphQL is a [query language](https://graphql.org/learn/queries/?ref=apisyouwonthate.com), [specification](https://facebook.github.io/graphql/?ref=apisyouwonthate.com), and [collection of tools](https://github.com/graphql/?ref=apisyouwonthate.com), designed to operate over a single endpoint via HTTP, optimizing for performance and flexibility. One of the [main tenants of REST](https://www.restapitutorial.com/lessons/whatisrest.html?ref=apisyouwonthate.com) is to utilize the uniform interface of the protocols it exists in. When utilizing HTTP, REST can leverage HTTP content-types, caching, status codes, etc., whereas GraphQL invents its own conventions. Another main focus for REST is hypermedia controls (a.k.a [HATEOAS](https://en.wikipedia.org/wiki/HATEOAS?ref=apisyouwonthate.com)), which lets a well designed client run around an API like a human runs around the Internet; starting with a search for "How to complete my tax returns", reading a perfectly relevant article, and after a few clicks ending up on BuzzFeed article about Miley Cyrus throwing Liam Hemsworth a "Weed-Themed" birthday party. If your API is not using hypermedia controls, then GraphQL could be a more relevant approach, because you [weren’t really using REST anyway](https://www.martinfowler.com/articles/richardsonMaturityModel.html?ref=apisyouwonthate.com). This article will not attempt to point out a winner, but we’re going to look at a few areas where the two differ. Don’t get mad that a lot of the sections say "it depends", because the winner in each section really depends on what your API is doing, and how. GraphQL comes out stronger in some areas, REST in others, and sometimes they’re both kinda terrible. Let’s dig in! ### Is the API More Than Data Transfer? One of the most common tasks REST APIs provide is CRUD via JSON, but it can do plenty more than that, such as [file uploads](https://philsturgeon.uk/api/2016/01/04/http-rest-api-file-uploads/?ref=apisyouwonthate.com). Uploading an image in the HTTP body can look a little something like this: ``` POST /avatars HTTP/1.1 Host: localhost:3000 Content-Type: image/jpeg Content-Length: 284 raw image content ``` Leveraging a cool part of HTTP (and therefore REST), API developers can support `application/json` requests on the same endpoint to handle the upload slightly differently, and offer URL-based uploads too: ``` POST /avatars HTTP/1.1 Host: localhost:3000 Content-Type: application/json { "image_url" : "https://example.org/pic.png" } ``` Generally the APIs I have worked on have enjoyed both, which is super handy as iOS often sends photos directly from local files, and web clients often send a URL to the user’s Facebook display picture. If we were talking about uploading videos or other large files, I would (as [this article suggests](https://philsturgeon.uk/api/2016/01/04/http-rest-api-file-uploads/?ref=apisyouwonthate.com)) switch to another approach and have a dedicated service which handles the upload, leaving the main API to only accept metadata; title, description, tags, etc. This is the approach you are forced to take with GraphQL, because you can only speak to GraphQL in terms of fields: ``` POST /graphql HTTP/1.1 Host: localhost:3000 Content-Type: application/graphql mutation addAvatar { addAvatarFromUrl(image_url: "https://example.org/pic.png") { id, image_url } } ``` Some will argue that this is more "clean", and it is, it’s very clean, but being forced to create another service is overkill for smaller images, especially early on. Another approach is to upload directly to Amazon S3, forcing a dependency on clients and potentially letting your tokens leak, or… use multipart uploads, which are a super hacky approach that depends on if the server and various clients can even support it. This is one area where REST holds strong. Some would say that REST handling CRUD and arbitrary stuff is confusing, but this is a core tenant of what makes REST so useful. A REST API can do anything, not just send fields backwards and forwards — even if that is how REST is often used. ### Both GraphQL and REST Prefer Evolution One false advertised benefit of GraphQL I’ve seen suggested (in quite a few locations) is that you "never have to version anything." The suggested approach is to add new fields and deprecate old ones, which is [a concept well known in REST as evolution](https://www.mnot.net/blog/2012/12/04/api-evolution?ref=apisyouwonthate.com). Deprecating, communicating to third-parties, monitoring usage, and removing at an acceptable time, is exactly what many REST APIs have been doing forever. REST *is* about evolvability, so don’t blame it for the bad habits of us RESTish people. Although GraphQL and REST can (and should) version via evolution just as easily, GraphQL really helps API developers out when it comes to deprecations. ### GraphQL makes Deprecations Awesome One area where GraphQL excels is to make monitoring field usage incredibly easy at a technical level. GraphQL clients are forced to specify the fields they want returned in the query: ``` POST /graphql HTTP/1.1 Host: localhost:3000 Content-Type: application/graphql { turtles(id: "123") { length, width, intelligence } } ``` Tracking this would be trivial, but a REST API acts a little differently. Whilst all REST APIs make the base endpoint available via `/turtles/123`, not all APIs offer [sparse fieldsets](https://jsonapi.org//format/?ref=apisyouwonthate.com#fetching-sparse-fieldsets): `/turtles/123?fields=length,width,intelligence`. Of those that do offer it, it's almost always optional. If a REST API client calls `/turtles/123`, then they could be using *any* field in that response. Imagine the API plans to get rid of `intelligence` (because [all turtles are geniuses](https://en.wikipedia.org/wiki/Turtle?ref=apisyouwonthate.com#Intelligence)), how do the API developers know which clients are using that field? An RPC approach is to make a new `/getTurtle2` endpoint (or `/v2/turtles/123` in a RPC API pretending to be a REST API), and tell people to use that new one. One approach in REST APIs is to email warnings to whatever email address was entered when the client signed up for OAuth tokens (like Facebook previously did), maybe offering a feature flag so various clients can flip the switch when they are ready. Another approach would be to create a new version of the resource, meaning instead of calling `GET` with `Accept: application/vnd.turtlefans.com+v1+json` they should start calling with `Accept: application/vnd.turtlefans.com+v2+json`. All of the above approaches suffer the same issue, and that is that an entire version can be overkill for a simple change, and you might be forcing the developers to look into a version upgrade without needing to. For example, if the client is requesting `application/vnd.turtlefans.com+v1+json` and the API is removing `intelligence` in v2, the API developers know not to drop v1 until the last of the clients are upgraded. Sadly, if clients are calling v1 *but not using the* `_intelligence_` *field, the API developers have no idea!* The API developers only know that the clients want v1, not what they're using of that v1 resource. GraphQL makes it easy to track specific field usage to a client, meaning API owners can reach out to only those clients using fields that are heading out, or for internal projects you could have errors thrown in development/staging environments. I had a vague brain fart about doing this latter option for non-GraphQL HTTP APIs, but have not had a chance to see it through *yet*. That could help in some situations, but would certainly be baked into things the same way it is in GraphQL. *GraphQL making field deprecation easier was a point brought to my attention by* [*Tom Clark*](https://twitter.com/itstomclark?ref=apisyouwonthate.com)*, super-smart Head of Devops at* [*WeWork*](https://www.wework.com/?ref=apisyouwonthate.com)*.* ### GraphQL Puts Client Performance First GraphQL is always the smallest possible request, whilst REST generally defaults to the fullest. It’s common practice to offer options like `?fields=foo,bar` or partials. [Google recommend doing this for HTTP APIs](https://developers.google.com/google-apps/tasks/performance?ref=apisyouwonthate.com#partial), whatever that's worth. Even if a REST API returns only a basic partial by default, there are still more bits being transferred over the wire by default, than with the GraphQL approach. If a client needs a field, they request it, and if the API adds a new field, clients don’t get it, unless they discover that field in a blog post or whatever and add it to the GraphQL query. ### REST Makes Caching Easier At All Levels Caching for HTTP, common usage in REST APIs, and different types of caching are huge topics, and something that I had to split out of this article. I’ll post a followup shortly, which will be posted on the [APIs You Won’t Hate Newsletter](https://apisyouwonthate.com/newsletter). > In an endpoint-based API, clients can use HTTP caching to easily avoid refetching resources, and for identifying when two resources are the same. The URL in these APIs is a globally unique identifier that the client can leverage to build a cache. > > In GraphQL, though, there’s no URL-like primitive that provides this globally unique identifier for a given object. It’s hence a best practice for the API to expose such an identifier for clients to use. > > *\-— Source:* [graphql.org](https://graphql.org/learn/caching/?ref=apisyouwonthate.com) REST over HTTP uses a whole pile of HTTP conventions that make existing HTTP clients, HTTP cache proxies, etc., all work easily to benefit both API clients and API servers, but with GraphQL… tough. Reorganize your data stores, use a bunch of Redis, and hope clients are caching too. ### GraphQL is a Query Language First A very fundamental difference here of course is that only one of these is a query language. REST APIs are often created initially simple, then slowly more and more query language-like features are tacked on over time. The most reasonable way to provide arguments for queries in REST is to shove them in the query string. Maybe a `?status=active` to filter by status, then probably `sort=created`, but a client needs sort direction so `sort-dir=desc` is added. Some APIs use also end up with arguments in the query string that are not related to filtering, more like options. In the GraphQL example they specify the unit they’d like to see a `height` returned in: ``` { human(id: "1000") { name height(unit: FOOT) } } ``` That’s pretty darn handy, and it stops the confusion of `?status=active` being a filter but `?unit=foot` being a display option. I have seen `?filter:status=active`, `?filter[status]=active`, etc., but this is still a bit of a mess. In these scenarios, GraphQL beats the pants off of REST APIs that try to hand-roll their own query language functionality, but I would suggest hand-rolling your own query language is a bad idea anyway. GraphQL gives you a query language syntax, and SDKs for your programming language to fetch the right models for that, but so do things like [OData](https://www.odata.org//?ref=apisyouwonthate.com), a project with the slogan "A Better Way to REST". This is another example of folks saying that REST cannot do something, just because many REST APIs don’t do it, or because its implemented poorly by many that do. Saying that, remember that the more customisation an endpoint-based API adds to the request, the fewer cache hits are likely at a network caching level, making the REST/HTTP approach less rewarding, and forcing you down the route of the application caching and database reorganisation that GraphQL forces anyway. If your API is highly customizable, it’s another ticked box for considering GraphQL. Another customisation consideration that comes up a lot is when to offer included relationships, and when to use another endpoint. This can be a difficult design choice, as you want your API to be flexible and performant, but includes used past the most trivial uses can be the opposite of that. You start off with overly simplistic examples like `/users?include=comments,posts` but end up on `/trips?include=driver,passengers,passengers.avatar,passengers.itineraries` and worse. I call this the "Mega Include of DOOOOOOM", and it is crap solution to a genuine concern, when clients strive for performance over all else. Includes are a common convention found in REST APIs, recommended in specs like [JSON API](https://jsonapi.org//?ref=apisyouwonthate.com), but actually bends the rules of REST a bit. REST would call for a HATEOAS approach, which would have you make one call to the `/trips` endpoint, then hit `"links": { "driver": "https://example.com/drivers/123" }`, and again for passengers, and again for child data of each of those passengers. In this case, it would be worse than the dreaded `n+1`, and more like `n+(2+(num passengers*2))`! Includes start off with the best of intentions, but can grow to be a bottleneck in a REST API, with unwieldy queries happening against the data store. GraphQL not only removes this design question, but forces the include approach, and forces you to consider efficient means of fetching this data. This is a big win for GraphQL, as forcing the include approach, and forcing efficient "Mega Includes of Doom", the GraphQL will be both efficient and consistent. Trying to make a REST API be include-only would be bizarre and unusable, so consistency will never work there. GraphQL will not help with your database queries, so you still need to fine tune those indexes and cache fragments of the data intelligently. If you skip this, clients will surprise you. At a previous job we had a poorly-tuned mega include from the iOS app taking about 20s+ to respond, which was bloody terrifying. We almost just had them curl down a `latest.sqlite` to process locally. 🤣 ### Scoped Includes Suck in Both Using that trip example again, a client may be including passengers but realize that includes historical passengers. Don’t want to see people who left the carpool? The client has to iterate through passengers locally, removing any models where `model.status != "active"`, which is a waste of processing on the client side. The REST way to do this would be to use `/passengers?trip=123&status=active` which is obviously more flexible, but clients will skip this due to the extra requests required. With client-side filtering being a poor choice, and the extra request not being ideal, REST API developers are often forced to add a new include: `/trips?include=activePassengers`. These are tough to pre-empt, so I was hoping GraphQL could help clients define their own scopes, filtering these includes to be the appropriate data themselves, which would help identify the scoped includes the API should add as convenience methods. It seems like GraphQL doesn’t help API developers in this instance, but there does seem to be talk of adding `@filter` to [do this in the future](https://discuss.dgraph.io/t/filters-in-graphql/831/31?ref=apisyouwonthate.com). ### GraphQL Devolves Power to Clients Another question that comes up a lot for REST API developers, is: > *Our iOS app, Android app and web app are very different from each other. How would we return different data for each client?* We all start off trying to make our REST APIs so generic they can be used by anything, but as the mega-include problem indicated, clients want and need a lot, and they’re always trying to reduce calls. Some basic solutions for outputting different data per-client in a REST API are: **Create custom endpoints:** `/iphone_snapshot`. We've done this at a past company where the mobile data was completely different to anything else. It felt dirty, was almost definitely RPC, but it got the job done. Having the supposedly generic REST API know so much about a specific client defeats the purpose a little bit, but we needed to get the data to the iPhone and that was what happened. **Create custom representations:** Using `Content-Type: application/vnd.turtlefans.com+v1+iphone+json` which had its own custom serializer would supposedly be a bit more REST in that it's just another representation, but equally odd. **Custom APIs!** This concept was/is in use by Netflix if I remember rightly, and the idea is to have one API for each of their clients. There is an Android REST API, a iOS REST API, Web REST API, etc. Each of these is tailored to perfectly match the needs of their specific teams, and makes requests back to the central Generic REST API. Requiring multiple APIs, multiple development teams, etc, is certainly out of the reach of many organizations, but it does solve the issue nicely. Or… **GraphQL!** Instead of making custom endpoints, custom representations, or custom APIs, the clients simply write their own queries. This moves the responsibility out of the hands of the API developers, and into the clients, who then get to write in their language of choice, instead of bugging the API team to write it in a different language. Whether you need this paradigm shift or not is entirely up to how similar your clients are. If you’re a private/internal API and your clients are all practically identical, then you certainly don’t want them all handling things. But, if you have a multitude of different clients, or are public (therefore have no idea how the API will be used), GraphQL quickly starts to seem more appealing. ### Why Not Use Both? The biggest oddity I notice in the "GraphQL vs REST" conversation, is the falsehood that *you must pick one*. In a world of SoA, you are likely to have multiple services, which expose multiple APIs. In the [RPC vs REST article](https://www.smashingmagazine.com/2016/09/understanding-rest-and-rpc-for-http-apis/?ref=apisyouwonthate.com) I point out that some services might be REST and some might be RPC, and you can absolutely throw some GraphQL in with your REST. One mix of REST and GraphQL could just be adding a `/graphql` endpoint to `api.whatever.com` and having that as your GraphQL endpoint on an REST API. Another, is one that has been vaguely tossed around at work, which is the idea of having one GraphQL API, acting as a gateway to our other multiple REST APIs. Having one GraphQL server act as a sort of data proxy, giving one entry point for mixed data, one Authentication scheme despite each REST API having its own "unique" approach to tokens, one HTTP call for clients — despite it hitting multiple actual REST APIs, etc., would be a damn powerful thing. Ask yourself — at the very least — these following questions: - How different are your clients from each other? - Do you trust your clients to handle caching? - Do you want "dumb clients" that act like crawlers — knowing very little about the API, or clients that own a lot of the logic and know a lot about the API, using it simply as a data transfer? - Are you ok letting go of HTTP debugging proxies, cache proxies, all the knowledge your team have around HTTP, etc? - Are you just doing basic CRUD with simple JSON documents, or will your API need file upload/download too? If your REST API is following good practices, like allowing careful [evolution](https://www.mnot.net/blog/2012/12/04/api-evolution?ref=apisyouwonthate.com) instead of [global versioning](https://www.troyhunt.com/your-api-versioning-is-wrong-which-is/?ref=apisyouwonthate.com), [serializing data](https://philsturgeon.uk/api/2015/05/30/serializing-api-output/?ref=apisyouwonthate.com) instead of returning directly from data store, implementing [sparse fieldsets](https://jsonapi.org//format/?ref=apisyouwonthate.com#fetching-sparse-fieldsets) to allow slimming down response sizes, [GZiping contents](http://checkgzipcompression.com/?ref=apisyouwonthate.com), outlining data structures with [JSON Schema](https://json-schema.org/?ref=apisyouwonthate.com), offering [binary alternatives to JSON like Protobuff](http://blog.codeclimate.com/blog/2014/06/05/choose-protocol-buffers/?ref=apisyouwonthate.com) or BSON, etc., then the advertised advantages of GraphQL seem to fall a bit short. If you need a highly query-able API, expect an array of clients that need small and different data, and can restructure your data to be inexpensive to query, then GraphQL is likely to fit your needs. Beyond these various pros and cons for GraphQL listed above, what I really enjoy about GraphQL being an option, is having a new alternative to REST when considering an API. An alternative that is well documented, with a full specification, with a lovely marketing page, with an official reference implementation in JavaScript, and which avoids some of the tricky design choices REST forces you to make. I like that many folks will no longer treat REST like a shiny unicorn, struggle to implement REST properly, then call it REST just so they look good and have a ✅ for their marketing. I just hope that not too many people treat GraphQL like a shiny unicorn instead. Swapping one false idol for another isn’t going to make the API world a better place.