How Graceful Shutdown Stops Deploys From Dropping Requests
Every deploy at a job I had before this one left a small notch in the error graph. Nothing dramatic: a few hundred 502s spread over ninety seconds, then flat again.

Every deploy at a job I had before this one left a small notch in the error graph. Nothing dramatic: a few hundred 502s spread over ninety seconds, then flat again. The daily summary called it 0.3 percent of requests and nobody filed a ticket. Then the team went from two releases a week to roughly fourteen a day, and that harmless notch turned into a steady drip of failed checkouts that support could never reproduce.
The new code was not at fault. The old process was being handed requests by the load balancer at the same moment the orchestrator told it to die. That gap is what graceful shutdown closes, and almost all of the useful work happens in the two or three seconds before the process exits, not after.
What the client actually sees when a process exits
I spent years in cross-browser QA, so my instinct is to open the Network panel and watch rather than read logs. If you keep a page polling an endpoint and then trigger a rollout, you can see the failure modes separate themselves clearly.
A request that arrives after the socket closed but before the proxy noticed shows up as a 502 with a response time under 5 ms. A request that was mid-flight when the process was killed shows up as a net::ERR_EMPTY_RESPONSE in Chrome, with no status code at all, because the TCP connection died after the request headers went out and before any bytes came back. Those two are different bugs with different fixes, and the server-side logs show neither of them, since the server that would have logged them is already gone.
The timing matters more than the count. If you want to read these traces properly, the column ordering and the stalled bar are worth learning first, and we walk through that in how to read the DevTools network waterfall. In a shutdown window the giveaway is a cluster of requests whose entire duration is connection setup with zero time to first byte.
The order of operations that stops the bleeding
Kubernetes sends SIGTERM, waits for terminationGracePeriodSeconds, then sends SIGKILL. The default grace period is 30 seconds, which sounds generous and is usually not the problem. The problem is that removing the pod from the service endpoints and sending SIGTERM happen at the same time, through two independent systems, and the endpoint update has to propagate to every proxy before it takes effect. On a cluster with a few hundred nodes I measured that propagation at 1.2 to 4 seconds. Your process can finish shutting down cleanly in 200 ms and still lose requests, because traffic is still pointed at it.
The sequence that fixed it for us:
- Add a preStop hook that just sleeps. Five seconds of
sleep 5before SIGTERM is ever delivered. This feels wrong the first time you write it. It is the single highest-value line in the whole fix, because it gives the endpoint removal time to reach the proxies while the process is still happily serving. - Fail the readiness probe immediately on SIGTERM. Flip a boolean so
/readyzreturns 503 while/livezkeeps returning 200. Liveness must stay healthy or the kubelet will restart the container you are trying to drain. - Stop accepting new connections, keep serving open ones. This is
server.close()in Node, the graceful stop in your framework elsewhere. It should come after the readiness flip, not before. - Drain in-flight work with a deadline. Track open requests, wait for zero, and force exit after a timeout you choose deliberately. I use 15 seconds against a 45 second grace period, so the orchestrator never has to use SIGKILL.
- Close the things with buffers last. Database pool, metrics flush, log flush. Closing the database pool before in-flight handlers finish is how you turn clean 503s into 500s with stack traces.
Put the whole thing behind one function and call it from both SIGTERM and SIGINT. Local development is where you find out your handler deadlocks.
Keep-alive connections are the part everyone forgets
Here is the failure that cost me an afternoon. We had the preStop sleep, the readiness flip, the drain loop, all of it, and the ERR_EMPTY_RESPONSE errors dropped by maybe half instead of disappearing.
The reason is keep-alive. A proxy holds an idle TCP connection to your server, and server.close() does not touch idle keep-alive sockets. It stops the listener and waits forever for connections that are open but not currently carrying a request. Meanwhile the proxy, which has no idea anything changed, picks one of those idle sockets for a fresh request at the exact moment your drain deadline fires and the process exits.
Node added server.closeIdleConnections() in 18.2, and it belongs right after server.close(). Also set server.keepAliveTimeout below whatever your load balancer uses; an AWS ALB defaults to 60 seconds of idle keep-alive, so 65 on the application side keeps the server from being the one that closes first. In Python, the Gunicorn flag named graceful-timeout covers worker drain but not the same socket subtlety, and if you run sync workers behind nginx you want keepalive_timeout tuned on the nginx side instead.
One more detail: this all assumes clients retry safely. If a POST dies mid-flight, the caller does not know whether it landed. That is a correctness question rather than a deploy question, and the answer is the same one we give in how idempotency keys stop duplicate API requests.
Why a longer grace period is the wrong first move
The advice you will find most often is to raise the grace period. Someone sees SIGKILL in the events and pushes terminationGracePeriodSeconds from 30 to 120, or to 300 for the worker fleet. It is the wrong lever in most cases, and it makes two things worse.
It hides the real bug. A 120 second window means a process that never finishes draining looks fine, because it exits quietly on its own schedule after two minutes. You lose the signal that something is holding a connection open forever, usually a long-poll endpoint or a streaming response nobody remembered.
It also makes rollbacks slow and node drains painful. A cluster autoscaler trying to remove a node has to wait out the longest grace period on it. I have watched a 15 minute node drain that was entirely one service with a 300 second window and six replicas.
The honest version: if your requests take under 2 seconds at p99, a 45 second grace period is already more than enough, and every request you are still losing is an ordering problem, not a duration problem. Raise the window only when you have genuinely slow requests, and then cap the request itself so the two numbers relate to each other.
Measuring whether it worked
Do not trust the error rate on your dashboard, because deploy windows are short enough to vanish into a one minute bucket. Run something crude instead. Point a loop at your staging service from a box inside the same network, one request every 50 ms, and count non-200 responses across a full rollout:
for i in $(seq 1 2000); do curl -s -o /dev/null -w "%{http_code}\n" "$API_URL/healthz"; sleep 0.05; done | sort | uniq -c
Before the preStop hook we saw 38 failures out of 2000 on a six-replica rolling update. After, we saw zero across three consecutive runs, which is not proof but is good enough to stop looking. If you prefer to work from the client side, the same measurement against localhost with a proxy in front catches the keep-alive case that a direct connection will not.
Worth saying plainly: I have never seen this get to zero on a service that also holds WebSocket connections. Those need their own story, with a close frame and a reconnect that backs off, and a rollout will always be visible to someone. The rest of our back-end writing gets into the pieces around this.
FAQ
Does a preStop sleep actually delay my deploys?
By the length of the sleep, per pod, in parallel across the rolling update batch. Five seconds on a six-replica deploy with maxUnavailable of 1 adds about 30 seconds total. That has never been the slowest part of any pipeline I have worked on.
Do I need this if I run behind a managed platform?
Usually less of it, but check. Platforms that handle deregistration before signalling your process remove the need for the sleep, and the vendor docs normally say so explicitly. The keep-alive timeout and the in-flight drain are still yours to get right.
What signal should I handle, SIGTERM or SIGINT?
Both, pointing at the same handler. Orchestrators send SIGTERM; your terminal sends SIGINT. If you only wire up SIGTERM you will never exercise the path locally, and untested shutdown code has a habit of hanging.
Should health checks and readiness be the same endpoint?
No. Liveness answers "should this container be restarted" and readiness answers "should this container receive traffic". During shutdown you need those two answers to differ, and a single endpoint cannot give you that.
More in Back-End
Back-End
How to Run Database Migrations Without Locking Tables
The migration itself took 41 milliseconds. The outage lasted nine minutes.
Back-End
How to Deliver Webhooks Reliably With Retries and Signing
A partner emails to say they never received the order.created event from last Tuesday.
Back-End
API Versioning Without Breaking Existing Clients
You renamed one field in a JSON response, deployed on a Tuesday afternoon, and by Wednesday morning a partner's mobile app was...