Connection draining: stop new work without abandoning old work
Coordinate endpoint removal, protocol-level draining, application shutdown, and bounded stream lifetimes during a rollout.
On this page
The question for today
What you’ll understand
- Separate readiness withdrawal from ending established connections.
- Explain protocol-specific draining and bounded shutdown intervals.
- Build a conservative rollout timeline that includes propagation and last admitted work.
Removing a pod from new traffic is not the same event as finishing the requests already using it. A graceful rollout coordinates discovery, proxy behavior, application admission, and process termination. Long-lived streams make those boundaries especially important: a connection can remain useful after membership changes, or become a source of forced resets if the shutdown budget is shorter than its permitted lifetime.
Membership withdrawal affects future selection
A readiness change can eventually remove an endpoint from eligible discovery results, but clients and proxies observe that change after propagation. Existing TCP connections and HTTP streams may continue according to the protocol and configured pool behavior. A workload can therefore receive work during a removal tail even after the orchestrator began termination. Trace the last time new work can be admitted, rather than assuming the first readiness event is an instantaneous fleet-wide barrier. Include clients outside the mesh and any direct-address paths.
At the destination, the application and proxy must remain available long enough to handle the work their shutdown contract permits. Stopping the proxy before the application finishes can break otherwise valid responses. Stopping the application first can leave a healthy-looking proxy forwarding to an unavailable local process. Orchestration hooks, sidecar lifecycle behavior, and termination grace depend on the platform and version. Document their observed ordering and test slow startup, failed readiness, and termination under load instead of relying on container-list order.
A drain signal is protocol-specific
For HTTP/1, connection-close behavior can discourage reuse after a response. For HTTP/2, GOAWAY informs a peer about stream handling and encourages new work to use another connection, but it is not transparent migration of every existing stream. Envoy v1.40.0 documents listener drain settings and HTTP connection-manager behavior, with different support for other network filters. Draining a listener, draining an upstream pool, and stopping all listener acceptance are distinct operations. Choose the mechanism and sequence deliberately rather than treating graceful as a single boolean with identical semantics.
WebSockets, tunnels, and long gRPC streams can outlive an ordinary rollout interval. If the service requires a bounded shutdown time, define maximum stream duration, reconnect behavior, resumable application state, or an explicit forced-termination policy. A retry of a reset stream does not necessarily reconstruct its previous application state or avoid duplicate effects. Also consider connection surges when many replicas drain together. Staggering and client backoff can reduce synchronization, while preserving enough working capacity during the replacement is a separate requirement.
Verify complete requests and the failure tail
During a rollout test, record new request starts, active streams, completed responses, connection resets, retry attempts, and time until the last old replica stops. Include a request arriving near the end of propagation and a stream longer than the nominal drain period. Distinguish a successful drain from a fast termination that merely shifts errors into client retries. Check that replacement instances are both ready and actually receiving useful traffic before spending the next availability slot. A rollout controller's desired replica count does not prove end-to-end service capacity at each moment.
The queue lab depicts waiting work rather than socket draining. Reset, set λ=0 and μ=2, predict the initial four waiting units drain in two seconds, then Play or Step. Separately Reset, set λ=1 and μ=2, predict the restored four units drain in four seconds, then Play or Step. The second case represents continuing arrivals in a constant-rate toy model. It does not implement GOAWAY, active streams, readiness, or graceful shutdown. Use the difference to explain why the last-admission boundary matters, then validate the real protocol timeline separately.
Work it through
Leave room for the last admitted request
A fictional rollout allows endpoint-removal propagation up to five seconds. A request admitted at the end of that interval needs at most twenty more seconds. The plan reserves three seconds of additional margin and gives the processes thirty seconds.
The latest admitted request can begin at time five after removal starts and finish by time 5+20=25 seconds under the stated duration bound.
Adding the three-second margin requires a conservative process-lifetime budget of 28 seconds. Thirty seconds leaves two seconds beyond that plan.
A permitted sixty-second stream violates the assumed twenty-second bound. It requires a longer budget or a separate reconnect/termination contract; the thirty-second grace cannot make it complete automatically.
The thirty-second plan covers the stated bounded-request scenario with two seconds of remaining margin. It does not establish a guarantee for unbounded or longer streams.
The living notebook
Explore the model
Interactive lab · queue
Waiting work drains differently if arrivals continue
Reset, set λ=0 and μ=2, predict four waiting units drain in two seconds, then Play or Step. Separately Reset, set λ=1 and μ=2, predict drainage in four seconds, then Play or Step. This fluid backlog has no sockets, active HTTP streams, readiness, GOAWAY, or process termination.
- Fluid backlog
- 4
- Net rate · λ − μ
- −1 /s
- M/M/1 mean residence
- 1 s
Fluid backlog starts at 4 and follows max(0, 4 + (λ − μ)t). The separate M/M/1 residence-time formula assumes Poisson arrivals, exponential service and λ < μ. Parameter changes pause playback and recompute the model at the current elapsed time. Reset restores both parameters and returns time to zero. Playback starts paused. Adjust either parameter, then choose Play or Step.
Look forA bounded drain needs a defined last-admission time and a bound on remaining work.
Pause & reason
Check your understanding
Your turn
Try it yourself
A rollout has eight replicas: two are draining and one separate replica is unavailable. Find the count eligible for new work in the simplified model. If each eligible replica sustains 100 requests/s, compare capacity with demand 550 requests/s.
Take a moment to make a prediction or work it out on paper. Open a hint when you need one.
Give me a hint
Keep the categories disjoint, then multiply only the eligible count.
Compare with the solution
Eight minus two minus one leaves five replicas, giving 500 requests/s of assumed independent capacity. Demand 550 exceeds that by 50 requests/s before imbalance or retries. The draining replicas may finish old work but are excluded from new-work capacity in this model; replica counts alone do not supply the missing headroom.
Keep the idea
Take it with you
- Readiness withdrawal, last admission, stream completion, and process termination are different events.
- Draining behavior depends on protocol, component, and lifecycle configuration.
- A bounded rollout needs both a duration contract and enough remaining service capacity.
Read further
Original sources and deeper explanations to follow at your own pace.
Optional progress, saved in this browser when available. Revisit any lesson at any time.
Lesson not marked complete.