Make Your API Idempotent, Avoid Ruining Clients Lives
Explore idempotency in HTTP, highlighting its importance in preventing unintended effects of repeated actions. Learn how methods like GET, PUT, and DELETE inherently support idempotency, ensuring consistent results.
Idempotency is the idea that doing something multiple times should have no different affects as doing it once, especially important for actions that can get expensive, like booking a hotel room, or sending a payment.
Idempotency in HTTP
Different HTTP methods have idempotency baked in. GET, HEAD, PUT, DELETE, OPTIONS, and TRACE can all be executed multiple times without any unintended side effects occurring.
That could mean being able to make multiple GET requests for a record without it suddenly vanishing, or doing something weird like sending emails to people. An API can still do simple things like logging a "viewed" event so you know that excuse making customer has seen their invoice and can't pretend their email isn't working.
This is important because if a connection fails or a client's timeout kicks in then they should be able to retry without fear of the item being mysterious gone. You also would not expect a GET to delete something,
PUT is idempotent by design, because the main difference from PATCH is that you are sending a complete document to be recorded entirely. The client says "here is the whole document, save that" so if you send the same request a second time it's still... that. Whether it updates the database all over again or notices that it's the same depends on the implementation, and if no triggers are being fired off then its fairly moot other than being wasteful of energy.
DELETE is idempotent because if you delete something, then a second attempt to delete it means that thing has still been deleted. Some APIs will show a 404 error saying "that could not be found" which is a bit of a weird thing to show a client who's trying to delete something and experienced a brief connection wobble triggering a retry, because if the resource is deleted and the client is trying to delete it that should really be treated as a success.
Those are all idempotent by design, but POST and PATCH are not idempotent by design; at least not by default.
Using Idempotency Keys to avoid POST spam
POST is often used to create things, but does not have to be exclusively for that. It can be used to submit form data, trigger events, run commands, things that if you executed them twice you'd expect to record the information twice.
As somebody who has cycled through ridiculously remote parts of the world then tried booking hotels on Booking.com, I've had more than a few problems, and one in particular had me buy three rooms in the same hotel. I kept booking a room, then it would say sorry that room is taken... so I'd think damn, ok, then try and buy another room. Same again... wtf, this hotel must be popular. Tries a third time... Nope! I gave up and checked into somewhere in person, and when I logged into the hotel wifi I received three emails letting me know I had successfully bought all three rooms... 🤣
I wasn't running Wireshark at the time, but I imagine something like this was going on:
POST /bookings
{
"userId": "phil123",
"hotelId": "1234",
"checkinDate": "2023-01-01",
"checkoutDate": "2023-01-02",
"roomCode": "single-room-ab1"
}
That request would venture off out into the tubes, possibly one of many required to make the booking successfully, and at some point the Booking.com client would decide things were taking too long and give it a retry. That second response of "Nope its booked" would come in a lot quicker, because yeah to be fair somebody just booked it, because even if the client gives up on the request, that doesn't mean the server did!
Here's how you fix that.
POST /bookings
Idempotency-Key: random-madeup-thing-235325
{
"userId": "phil123",
"hotelId": "1234",
"checkinDate": "2023-01-01",
"checkoutDate": "2023-01-02",
"roomCode": "single-room-ab1"
}
If the client had made up a completely random string (usually a UUID v4) then a second request using this key should reply with the same response as the first. This would have avoided making another payment, because much like caching, the server would have skipped all the functionality included in the server, and simply shown me the response that was sent to the first request.
The client could retry on my behalf several times until it worked, and if I didn't leave the page I could retry myself and it would be the same effect.
If I popped off to try and book another room, it would come up with a new Idempotency Key, so it would know that I was trying to make a new booking.
Stripe was one of the first APIs I noticed using Idempotent requests.
curl https://api.stripe.com/v1/customers \
-u sk_test_4eC39HqLyjWDarjtT1zdp7dc: \
-H "Idempotency-Key: 5mUP59iveriphkIJ" \
-d description="My First Test Customer"
Adyen use Idempotency Key too.
Go Cardless use them.
POST https://api.gocardless.com/payments HTTP/1.1
Idempotency-Key: PROCESS-ME-ONCE
{
"payments": {
"amount": 100,
"currency": "GBP",
"charge_date": "2015-06-20",
"reference": "DOLLAR01",
"links": {
"mandate": "MD00001EKBQ412"
}
}
}
They are a little different as they'll return a 409 Conflict on the second attempt, and let you know that the resource was already created.
HTTP/1.1 409 (Conflict)
{
"error": {
"code": 409,
"type": "invalid_state",
"message": "A resource has already been created with this idempotency key",
"documentation_url": "https://developer.gocardless.com/pro#idempotent_creation_conflict",
"request_id": "5f917bf9-df56-460f-a165-15d9e77414cb",
"errors": [
{
"reason": "idempotent_creation_conflict",
"message": "A resource has already been created with this idempotency key",
"links": {
"conflicting_resource_id": "PM00001KKVGTS0"
}
}
]
}
}
Ok.. seems fine. A client can see that and decide to go grab the resource id and show that as a success. Not as seamless but works all the same.
Making PATCH Idempotent
PATCH is a bit of a weird one, and honestly I had to ask the APIs You Won't Hate Slack channel why people would want to use Idempotency Keys for PATCH. As always, they came through. Thanks Evert Pot!
Generally with PATCH you're just sending a bit of JSON, and what happens is not entirely defined by default. Maybe your PATCH looks like this.
PATCH /something/some-id
{
"foo" : 1,
"baz" : 3
}
Should the omitted "bar" property be left as whatever it is on the server, or set to null? Or zero? Who knows. You can use JSON Merge PATCH to clarify that, which says "yep, leave bar alone".
Either way, when your API resources are just a bunch of JSON properties and you're updating a few of them at a time, it's essentially idempotent anyway, because if you're setting "foo" to 1 and that happens five times for some reason, foo is still going to be 1.
Where idempotency gets important for PATCH is if you're doing atomic requests, with modifications like "incrementBy": 1, where the end result of that being run accidentally run multiple times is a value far higher than intended.
If you're making atomic requests, each attempt to incrementBy: 1 (or whatever) should have it's own idempotency key, so that if a retry happens it's going to short circuit future efforts and show the previous result for any subsequent requests.
Implementing Idempotency Key
If you're using any sort of web application framework that supports middleware, there's a good chance there's a pre-built package you can install, or at least some tutorial code to copy.
Optional Keys
Seeing as idempotency keys are optional you can roll this out on your APIs tomorrow without effecting any existing API clients. You can simply document the functionality, send out emails letting people know the option is there, and rejoice as people start using it.
Each time a client implements this functionality, there will be a decline in support tickets coming into your helpdesk, which is time and money saved for everyone, and it means I won't spend my entire night in a hotel trying to explain to Booking.com how timeouts and retries work in a bid to get three hotel rooms refunded.
