How to Deliver Webhooks Reliably With Retries and Signing
A partner emails to say they never received the order.created event from last Tuesday. You check your logs and find one line: POST to their endpoint, 200 OK, 180 milliseconds. Both of you are right.

A partner emails to say they never received the order.created event from last Tuesday. You check your logs and find one line: POST to their endpoint, 200 OK, 180 milliseconds. Both of you are right. Their load balancer answered 200 and the application behind it threw an exception before the job was queued. Your system recorded a success and moved on, and the event is now unrecoverable because you kept no copy of it.
That failure mode is the normal one. Webhook delivery looks like a one line job, an HTTP POST when something happens, and then slowly turns into the part of the system that pages someone at two in the morning. Most of the pain comes from three decisions made early: no record of each attempt, a retry policy copied from a tutorial, and no way for the receiver to prove the request came from you.
Treat each attempt as a row, not a function call
The first thing to build is not the sender. It is the two tables behind it.
An events table holds the event itself: an id, the type, the payload as stored JSON, the account it belongs to, and the time it occurred. A deliveries table holds one row per subscription per event, with its own id, a status, an attempt counter, the next attempt time, and the result of the last try, meaning the HTTP status, the first 2 kB of the response body, and the duration.
That shape pays for itself the first week it exists. Support can answer the question above in ten seconds instead of grepping logs. You can replay a single delivery from an admin screen rather than asking the partner to reconcile by hand. And you can show a customer their own delivery log, which converts a long email thread into them finding their own 500 error.
Keep the payload on the event, not generated at send time. If you build the JSON fresh on each retry, a replay three hours later sends a different body from the original, the signature no longer matches what the receiver first saw, and a bug in your serialiser changes history. Serialise once, store it, send the bytes.
A retry schedule that does not punish anyone
Retrying immediately three times and giving up is the default in too many codebases. It handles a dropped packet and nothing else. Most real outages last longer than nine seconds and shorter than a day, so the useful schedule stretches across hours with growing gaps.
| Attempt | Delay after previous | Elapsed | What it catches |
|---|---|---|---|
| 1 | immediate | 0 | Normal case |
| 2 | 10 seconds | 10 s | Brief network blip, pod restart |
| 3 | 1 minute | about 1 min | Deploy rollover |
| 4 | 5 minutes | about 6 min | Database failover |
| 5 | 30 minutes | about 36 min | Expired certificate noticed and fixed |
| 6 | 2 hours | under 3 h | Someone paged out of bed |
| 7 | 6 hours | about 9 h | Overnight outage |
| 8 | 12 hours | about 21 h | Last chance before the queue is dropped |
Add jitter of plus or minus 20 percent to every delay. Without it, a thousand deliveries that failed together will retry together and arrive as a spike on an endpoint that is only just back up. Jitter costs four lines of code and prevents the retry storm that turns a recovery into a second outage.
Retry on 408, 429, 500, 502, 503, 504, on connection resets and on timeouts. Do not retry on 400, 401, 403, 404, 410 or 422, because those mean the request itself is wrong and sending it again changes nothing. A 410 Gone deserves special treatment: treat it as the receiver asking to be unsubscribed, and disable the endpoint after a couple of them. If the receiver sends a 429 with a Retry-After header, honour the header instead of your own schedule, which is the same courtesy you would want from a client of yours. The reasoning on both sides of that is in our piece on how rate limiting protects an API without punishing real users.
Signing, so the receiver can trust the body
A webhook endpoint is a public URL that accepts commands about money or accounts. Without a signature, anyone who learns the URL can post a fake payment confirmation.
The pattern that has settled across the industry is an HMAC over the raw request body with a shared secret, sent in a header along with a timestamp. Build the signed string as the timestamp, a dot, and the exact body bytes, then compute HMAC SHA-256 with the subscription's secret and send both values. The receiver repeats the computation and compares with a constant time comparison.
Three details decide whether it works in practice. Sign the raw bytes, not a parsed and re-serialised object, and document that clearly, because every framework that helpfully parses JSON before your handler sees it causes a support ticket here. Include the timestamp inside the signed string and tell receivers to reject anything older than five minutes, otherwise a captured request can be replayed forever. And support two active secrets per endpoint at once, so rotation means adding the new secret, sending both signature values in the header for a week, then removing the old one. Rotation without overlap means scheduled downtime for your partner, and they will simply never rotate.
IP allowlists get requested often and age badly. Publishing a stable set of egress addresses is a kindness for customers behind a strict firewall, but it is a weaker control than a signature and it ties your hands on infrastructure. Offer it as a supplement, never as the primary check.
The counter-take: do not promise ordering
The request I get most from integrators is ordered delivery, and the common advice is to give it to them with a per-account FIFO queue. I think that trade is a bad one, and I have regretted taking it.
Strict ordering means a single stuck delivery blocks every later event for that account. One partner whose endpoint hangs for 30 seconds on every request turns into a backlog that grows faster than it drains, and now your freshest events are hours old for a customer who has nothing wrong with their system. You have converted a localised failure into a queue-wide one.
The alternative costs the receiver very little. Put a monotonic sequence number and the resource version on every event, document that events can arrive out of order and more than once, and tell consumers to discard any event whose version is older than what they have already stored. For anything that matters, have them read the current state from your API rather than treating the payload as the source of truth. Deliver in parallel per endpoint, cap concurrency per endpoint at something like five in flight, and a slow consumer slows only itself. At-least-once delivery with versioning solves in the consumer what ordering tries to solve in the producer, and the consumer side is where the pattern already exists, as our article on idempotency keys for duplicate API requests describes.
Limits that keep one bad endpoint from spreading
A sender without limits becomes an outbound denial of service against itself. The defaults I would set on a new system:
- Connect timeout 3 seconds, read timeout 10 seconds. A receiver that needs longer than 10 seconds should accept the event and process it asynchronously, and your documentation should say so.
- Limit response reading to about 64 kB. Otherwise a receiver streaming a large error page holds your worker open.
- Per-endpoint concurrency cap and a separate worker pool from your API. Webhook delivery should never share a thread pool with requests from paying users.
- Automatic disable after sustained failure. After 24 hours of nothing but failures, mark the endpoint disabled, stop queueing new deliveries for it, and email the account owner. Keep generating events; just stop trying to push them.
- Follow no redirects. A 301 on a webhook URL is a misconfiguration, and following it can leak the signature header to another host.
The last piece is a dead letter view the customer can see, holding the events that exhausted all attempts, with a replay button and a 30 day window. Most of my support load disappeared the week we shipped that screen, because the people who needed a replay stopped needing us to do it. More on service design sits in the back-end section.
Frequently asked questions
How many retries are enough
Seven or eight attempts spread over roughly 24 hours covers nearly every outage that resolves on its own. Beyond a day, a replay from a dead letter queue is more useful than another automatic attempt, because by then someone has usually changed something on purpose.
Should a webhook payload contain the full object
Send enough to identify what changed, the ids, the type, the version and a short summary, and let the receiver fetch the rest. Full objects grow until the payload is 200 kB, they leak fields the subscriber should not see, and they go stale the moment the resource changes again.
What status code should a receiver return
Any 2xx, and 200 or 204 are the usual choices, returned as soon as the event is stored durably. A receiver that validates the business meaning before answering will reject events it should have queued, and your retries then stop for the wrong reason.
Do webhooks need a queue in front of them
Yes, once you pass a trickle of traffic. Writing the event in the same database transaction as the change itself and letting a separate worker pick it up avoids the case where the commit succeeds and the webhook never exists, which no amount of retrying can fix after the fact.
More in Back-End
Back-End
How to Run Database Migrations Without Locking Tables
The migration itself took 41 milliseconds. The outage lasted nine minutes.
Back-End
How Graceful Shutdown Stops Deploys From Dropping Requests
Every deploy at a job I had before this one left a small notch in the error graph.
Back-End
API Versioning Without Breaking Existing Clients
You renamed one field in a JSON response, deployed on a Tuesday afternoon, and by Wednesday morning a partner's mobile app was...