How Idempotency Keys Stop Duplicate API Requests

A customer taps Pay, the spinner hangs for eight seconds on a weak train connection, and they tap again. Your server receives two identical POST requests 400 milliseconds apart and charges the card twice. Nothing in the code was wrong in the usual sense. Each request was valid. The bug was that the API had no way to know the second one was a retry. Idempotency keys exist to fix exactly that.

I have added them to payment, ordering and messaging endpoints on three different backends, and the pattern is small enough to ship in a day. The details that matter are where the key is stored, how long it lives, and what happens when two copies of the same request arrive at the same moment.

What Idempotent Means for an HTTP Endpoint

An operation is idempotent when running it twice has the same effect as running it once. HTTP already treats some methods this way. GET reads data and changes nothing. PUT replaces a resource with the same body each time. DELETE on an already deleted resource leaves it deleted.

POST is the problem. Creating an order, sending a message or charging a card each produce a new side effect every time. Clients, proxies and mobile networks all retry requests that seem to have failed, and many of those requests actually succeeded on the server before the response got lost. Without protection, every retry is a new action.

How Idempotency Keys Work Step by Step

The client generates a unique value, usually a random UUID, and sends it in a header such as Idempotency-Key with the request. The server uses that key to recognise repeats.

The request arrives. The server looks up the key in a store.
If the key is new, the server records it as in progress, performs the action and saves the response alongside the key.
If the key exists and has a stored response, the server returns that same response without doing the work again.
If the key exists but is still in progress, the server returns a conflict status, commonly 409, telling the client to wait and retry.

The client must reuse the same key for every retry of the same logical action, and generate a fresh key for a genuinely new one. That responsibility sits with the client, which is why the API documentation needs to say it clearly. Good naming and predictable behaviour here are part of designing an API other developers enjoy using.

Where to Store the Keys

The store has to be shared by every server instance, or a retry routed to a different machine will slip through.

Store	Strengths	Watch out for
Main relational database	Can write the key and the business record in one transaction	Extra write load on a busy table
In memory key value store	Fast lookups, built in expiry	Keys lost on restart unless persistence is configured
Local process memory	Trivial to code	Useless with more than one instance

For anything involving money, I store keys in the same database as the order or payment and wrap both writes in one transaction. If the transaction fails, neither the key nor the record exists, and the retry runs cleanly.

Handling Two Requests That Arrive at Once

The tricky case is not a retry five seconds later. It is two copies landing within the same few milliseconds on different servers. A naive check that reads the key and then writes it leaves a gap where both requests see an empty store.

Close that gap with an atomic insert. In a relational database, a unique constraint on the key column means only one insert succeeds and the other gets a constraint error, which you translate into a 409. In a key value store, use a set if not exists operation with an expiry. Either way, the database enforces uniqueness rather than your application code.

Store a fingerprint of the request body

Save a hash of the request body with the key. If a client reuses a key with a different body, such as a different amount, return a 422 error instead of replaying the old response. That catches client bugs that would otherwise be very hard to trace.

Why Retry Logic Alone Does Not Solve It

A common suggestion is to make clients smarter: only retry on timeouts, add exponential backoff, cap attempts at three. That advice is good, and you should follow it, but it does not prevent duplicates. It only makes them less frequent.

The client cannot know whether a timed out request reached the server. Backoff spreads retries over time but every one of them still lands as a new action. Some retries are not even sent by your client code. Load balancers, service meshes and mobile operating systems can resend requests on their own. The only place that can reliably detect a duplicate is the server, and it needs a key to do it.

Idempotency also pairs well with throttling. When clients back off after a 429, they retry with the same key, so a burst of retries never multiplies side effects. The throttling side is covered in how rate limiting protects an API without punishing real users.

Expiry, Cleanup and What to Return

Keys do not need to live forever. Twenty four hours covers almost all realistic retry windows, including a mobile app that queues requests overnight. Some payment providers keep them for longer, but storing millions of keys indefinitely is wasted space.

Set an expiry when you create the key, or run a nightly job that deletes keys older than your window.
Store the status code and response body, not just a success flag, so the replayed response matches the original exactly.
Do not store responses for server errors in the 500 range. Let the client retry those and try the work again.
Log every replay. A spike in replays usually points to a network or client problem worth investigating.

FAQ

Should GET requests use idempotency keys?

No. GET is already safe to repeat because it changes nothing. Keys are for POST and sometimes PATCH, where repeats cause new side effects.

Can I use a natural key like an order number instead?

Sometimes. If the client already has a unique business identifier, a unique constraint on it gives you similar protection. A dedicated key is more general and works for actions that have no natural identifier.

Who should generate the key, client or server?

The client. The whole point is that the client sends the same value on every retry, and only the client knows when two requests represent the same intent.

Start With Your Riskiest Endpoint

Pick the one POST endpoint where a duplicate would hurt most, usually payment or order creation. Add a key column with a unique constraint, store the response, return 409 for in progress keys and set a 24 hour expiry. Ship that, watch the replay logs for a week, and then roll the same pattern out to the rest. More server side guides are in the backend section.