advanced24 min studyLesson 379 in Engineering

Control-plane failure: cached traffic has several expiration clocks

Separate an established data plane from new configuration, startup, endpoint freshness, credential rotation, and resource TTL expiry.

On this page

What you’ll understand

  • Explain why existing forwarding can continue during some control-plane outages.
  • Identify separate configuration, credential, freshness, and bootstrap failure horizons.
  • Distinguish average connectivity from a bound on continuous disconnection duration.

A mesh control-plane outage need not stop every request immediately, because proxies can already possess usable routes, endpoints, and credentials. That resilience is conditional. New workloads, changing endpoints, expiring resources, and certificate rotation introduce other clocks. A useful failure test identifies what each component already knows and how long that knowledge remains sufficient for the actual traffic.

Previously accepted state can outlive its publisher

Envoy's xDS protocol documents that last-known configuration can remain available when a management server is unreachable. This separates data-plane forwarding from continuous control-plane participation. However, TTL-bearing resources can expire where that feature is supported and configured, and an individual resource type determines the effect of expiration. A rejected update also does not automatically erase every previously valid resource. Inspect accepted configuration, pending dependencies, stream connectivity, and resource expiry rather than inferring forwarding behavior from the control channel's connection status alone.

Existing proxy state and new proxy bootstrap are different scenarios. A dynamically configured proxy with no usable initial listeners or routes may be unable to serve while an already configured neighbor continues. A restarted process cannot be assumed to retain another process's in-memory state; persistence or static fallback would need an explicit design. New endpoints and policy changes also cannot be learned through a disconnected management path. Keeping old routes available is useful, but a stale route to a terminated workload or an outdated authorization decision can still fail the application's requirements.

Hplanning=min⁡(Hcredential,Hresource,Hfreshness)H_{\rm planning}=\min(H_{\rm credential},H_{\rm resource},H_{\rm freshness})
The H terms are nonnegative remaining durations to explicitly selected planning deadlines under one outage scenario; an absent configured resource TTL can be treated as infinite for that term. The minimum is the earliest planning deadline, not a lower bound on actual availability. Startup, endpoint failures, and other dependencies can fail earlier.↔ Scroll wide equations horizontally.

Credentials and discovery need separate failure tests

If the unavailable component also supplies workload credentials or their signer, existing credentials can approach expiry without successful rotation. New TLS handshakes then face the actual certificate-validation rules and current trust material. Established TLS connections do not necessarily revalidate their peer's certificate before every application request, while configured connection-age or drain rules can end them. Do not assume all sessions die at the notAfter instant or that existing sessions make certificate expiry harmless. Test new and existing connections separately under the chosen transport and policy.

The identity provider, trust-bundle distribution, DNS, orchestrator, and xDS management service need not share one failure boundary. A mesh deployment can use an external CA or independent identity agents. State exactly which dependency is unavailable before assigning a horizon. A twenty-minute endpoint-resource TTL, fifty-minute credential headroom, and ten-minute operational freshness budget are three different constraints. The last is an application requirement unless a mechanism enforces it; treating all three as identical automatic proxy timers would misdescribe the system.

πdisconnected=pp+q,E[Dspell]=1q,q>0\pi_{\rm disconnected}=\frac{p}{p+q},\qquad \mathbb{E}[D_{\rm spell}]=\frac{1}{q},\qquad q>0
This is an illustrative two-state discrete chain with connected→disconnected probability p and disconnected→connected probability q per step. D_spell counts steps until leaving a newly entered disconnected state. Equal stationary availability can coexist with different mean spell durations; neither expression bounds a real outage.↔ Scroll wide equations horizontally.

Recovery should reconcile state, not only reconnect

After recovery, verify resource versions, rejected updates, warmed dependencies, endpoint convergence, and credential renewal across the fleet. A connected stream can still carry configuration that some clients reject. A burst of reconnecting proxies can load management servers and signers, so examine retry backoff and staged recovery. Include a proxy restart during the outage and a newly deployed workload that lacks prior state. Those tests reveal a different weakness than a test that disconnects only a fully warmed, unchanged service for a few seconds.

The Markov lab isolates why average availability is insufficient. Reset, set p=0.10 and q=0.40, predict first-step disconnected probability 0.10 and stationary probability 0.20, then Play or Step. Separately Reset, set p=0.01 and q=0.04, predict first-step probability 0.01 and the same stationary probability 0.20, then Play or Step. Expected disconnected spells are 2.5 versus 25 abstract steps. These exact chain probabilities do not simulate management traffic or credentials. Their unbounded geometric spell tails explain why a finite operational recovery requirement needs stronger evidence than a long-run availability fraction.

Find the earliest planning deadline

During a fictional outage, an accepted resource has twenty minutes until configured TTL expiry, a needed credential has fifty minutes of conservative validity headroom, and operations require endpoint information refreshed within ten minutes.

  1. Keep the three boundaries separate: resource lifetime is twenty minutes, credential headroom fifty, and the operational freshness deadline ten.

  2. The earliest selected planning deadline is min(20,50,10)=10 minutes. The freshness requirement does not become an automatic ten-minute proxy timer unless a separate mechanism enforces it.

  3. A new proxy with no usable initial configuration or a backend that disappears at minute one can fail earlier. Continuing cached traffic therefore cannot be advertised as a guaranteed ten-minute availability window.

The result

Ten minutes is the earliest stated planning deadline in this scenario. Actual request success still depends on the data plane, workload changes, credentials, and startup state.

Explore the model

Interactive lab

Interactive lab · markov

Equal average disconnection can hide longer outages

Reset, set p=0.10 and q=0.40, predict first-step disconnected probability 0.10 and stationary value 0.20, then Play or Step. Separately Reset, set p=0.01 and q=0.04, predict 0.01 initially and the same stationary value, then Play or Step. This exact chain has no xDS, certificate, resource TTL, or outage deadline.

Equal average disconnection can hide longer outagesExact finite-step probabilities for a two-state chain starting in state 0, not sampled trajectories. Playback advances two transitions per second. When p = q = 0 every distribution is stationary; when p = q = 1 the chain alternates forever. A single zero transition creates an absorbing state. Current readings: P(state 1) at step 0: 0%; Stationary P(state 1): 60%; Chain behavior: Monotone convergence. All parameters are adjustable with the labeled controls below.Two-state Markov distribution after 0 steps: state zero probability 1, state one probability 0.Initial distribution (1, 0) · step 0p = 0.3q = 0.201100%0%1−p1−qP(0)P(1)Monotone convergence
P(state 0)P(state 1)Stationary distribution
P(state 1) at step 0
0%
Stationary P(state 1)
60%
Chain behavior
Monotone convergence
t = 0 s

Exact finite-step probabilities for a two-state chain starting in state 0, not sampled trajectories. Playback advances two transitions per second. When p = q = 0 every distribution is stationary; when p = q = 1 the chain alternates forever. A single zero transition creates an absorbing state. Parameter changes pause playback and recompute the model at the current elapsed time. Reset restores both parameters and returns time to zero. Playback starts paused. Adjust either parameter, then choose Play or Step.

Look forA long-run fraction does not bound continuous outage duration or establish credential survival.

Check your understanding

Why can two proxies behave differently during the same management-server outage?

Try it yourself

Two illustrative chains have (p,q)=(0.20,0.80) and (0.02,0.08). Find stationary disconnected fraction and mean disconnected spell length for each. Do either provide a maximum outage length?

Take a moment to make a prediction or work it out on paper. Open a hint when you need one.

Give me a hint

Use p/(p+q) and 1/q, keeping units in abstract transitions.

Compare with the solution

Both stationary disconnected fractions are 0.20. Mean disconnected spells are 1.25 and 12.5 steps respectively. Neither gives a finite maximum: a geometric spell can last arbitrarily many steps with decreasing probability. A real outage-duration requirement needs a separate recovery design and test, not just these averages.

Take it with you

  • Cached data-plane state can survive some management outages, with explicit expiry and dependency exceptions.
  • Startup, discovery freshness, and credential rotation have different failure horizons.
  • Average connectivity does not establish a maximum continuous outage duration.

Read further

Original sources and deeper explanations to follow at your own pace.

Optional progress, saved in this browser when available. Revisit any lesson at any time.

Lesson not marked complete.