Blue-green traffic cutover
Switch between deployment pools while accounting for routing propagation, active sessions, capacity, and shared durable state.
On this page
The question for today
What you’ll understand
- Distinguish changing a traffic selector from moving existing sessions.
- Budget the overlap capacity and a stated drain interval.
- Explain why application cutback does not automatically reverse database writes.
Blue-green delivery prepares a replacement pool beside the current one and changes which pool receives new traffic. The appealing diagram is two boxes and a switch. The operational system also contains long-lived connections, caches, in-flight work, asynchronously updated routing state, and a shared database. Safe cutover defines what happens to each of those objects before retiring the old pool.
A routing update is not an instantaneous global event
The new pool needs the intended artifact and configuration, enough capacity, and health checks meaningful for real requests before promotion. A preview route can validate some behavior without proving that production traffic will have the same distribution. Changing a service selector or proxy configuration then starts a propagation process. Different nodes or proxies may observe it at different times. A controller acknowledgment and a successful synthetic probe are useful observations, but neither proves that every possible path stopped selecting the old pool at that instant.
Existing HTTP keep-alive sessions, streaming RPCs, WebSockets, and asynchronous jobs can remain associated with the old version after new connection routing changes. Define whether to drain, reconnect, finish, or cancel them, and preserve application-level guarantees when doing so. A forced process termination may interrupt a non-idempotent operation whose external effect already happened. Retry ownership and deduplication remain relevant across cutover. Do not set an arbitrary short retirement delay merely because the routing API call usually returns quickly.
Overlap consumes resources and shares state
Keeping both pools ready can require close to the sum of their capacities during the overlap, plus any surge or background work. Count CPU, accelerator memory, connection pools, load-balancer limits, and external quotas as well as replica count. A new pool that is individually healthy may overload a shared database when both versions warm caches or open connections. Capacity planning must cover the transition, not just either steady state. Verify which resources are shared and which are duplicated before claiming that blue-green always costs exactly twice as much.
Both pools may read or write the same durable state. New writes can change a format or invariant that old code cannot understand, making traffic cutback unsafe even though the old containers are still running. An expanded compatible schema and a deliberate mixed-version window can preserve reversibility; a destructive migration can remove it. A blue-green strategy therefore does not replace compatibility testing or recovery planning. Stateful consumers, scheduled jobs, and message handlers also need version coordination when they are not controlled by the same front-door route.
Observe drainage before destroying the fallback
Argo Rollouts documents delays around old ReplicaSet scale-down because traffic updates need time to propagate. Those mechanisms must be matched with application drain behavior rather than read as proof that every session is safe after a universal delay. Observe active old-revision sessions, in-flight operations, background owners, and the target routing revision. Set a maximum wait and a specified response if it is exceeded. Retaining an old pool without retaining its compatible configuration, credentials, and data contract does not preserve a usable fallback.
The queue lab isolates a drainage rate comparison with equal normalized jobs. Reset, set both λ=1 and μ=2, predict the initial four jobs drain at t=4, then Play or Step. Separately Reset, set both λ=0 and μ=2, predict drainage at t=2, then Play or Step. This illustrates stopping new assignments while service continues. It does not represent actual connection lifetimes, route propagation, database compatibility, or a traffic switch. The stochastic M/M/1 mean has its own assumptions and does not establish a worst-case drain deadline.
Work it through
Keep the old pool past the last possible assignment
In a stipulated cutover contract, routing may continue assigning old-pool work for at most fifteen seconds after the change begins. Every such assignment has an enforced maximum residual lifetime of 120 seconds. Blue and green each reserve six accelerator slots during overlap.
The latest permitted old assignment arrives at t=15 seconds. Under the enforced lifetime contract, it finishes no later than 15+120=135 seconds after cutover starts.
Retiring old capacity before t=135 could interrupt a permitted assignment. Retaining it until that bound is sufficient only if the propagation and lifetime contracts actually hold and other old work has no longer ownership.
The two steady reservations consume 6+6=12 accelerator slots during the stipulated overlap, before transition jobs. That is an overlap budget, not evidence that shared database writes are backward compatible.
A finite retirement time needs explicit bounded work and propagation. Long-lived sessions without such a bound require a separate drain or reconnect protocol.
The living notebook
Explore the model
Interactive lab · queue
Drainage needs stopped arrivals and continuing service
Reset, set both λ=1 and μ=2, predict the initial four-job backlog drains at t=4, then Play or Step. Separately Reset, set both λ=0 and μ=2, predict t=2, then Play or Step. Equal-job fluid drainage is not a route, session-lifetime, or database-state simulator and supplies no worst-case connection deadline.
- Fluid backlog
- 4
- Net rate · λ − μ
- −1 /s
- M/M/1 mean residence
- 1 s
Fluid backlog starts at 4 and follows max(0, 4 + (λ − μ)t). The separate M/M/1 residence-time formula assumes Poisson arrivals, exponential service and λ < μ. Parameter changes pause playback and recompute the model at the current elapsed time. Reset restores both parameters and returns time to zero. Playback starts paused. Adjust either parameter, then choose Play or Step.
Look forRemoving a pool safely needs evidence about remaining work, not merely an acknowledgment that new routing was requested.
Pause & reason
Check your understanding
Your turn
Try it yourself
Routing propagation is bounded by eight seconds and enforced residual assignment lifetime by forty seconds. Give the sufficient old-pool retention bound under this model, and name a condition that invalidates it.
Take a moment to make a prediction or work it out on paper. Open a hint when you need one.
Give me a hint
The last possible old assignment occurs after the propagation interval.
Compare with the solution
The bound is at least 8+40=48 seconds from cutover initiation. Unbounded streaming sessions, a router that continues assigning old work beyond eight seconds, or background owners outside the route invalidate the model. An observed bound and compatible shared state must be established rather than inferred from a configured delay alone.
Keep the idea
Take it with you
- New-traffic routing and existing-session ownership are different transitions.
- Budget overlapping resources and mixed-version durable-state behavior.
- A finite drain guarantee requires explicit propagation and lifetime bounds.
Read further
Original sources and deeper explanations to follow at your own pace.
Optional progress, saved in this browser when available. Revisit any lesson at any time.
Lesson not marked complete.