Controller leadership: a lease does not fence old work

An acting leader renews a shared record, while other candidates observe whether progress continues under the election algorithm’s timing rules. A candidate may take over when the old leader is judged unable to renew within the configured bounds. Network delay, process pauses, API availability, and clock-rate behavior affect that judgment. A Lease’s timestamps should not be naively compared using arbitrary machines’ unsynchronized wall clocks; use the library’s specified local observation and timing model rather than inventing a second election protocol around the same object.

After losing leadership, controller code should cancel and stop guarded work promptly, but cancellation is cooperative. An external call already in flight may complete later, and a stopped process can resume after another candidate starts. Releasing a lease before guarded work is actually stopped can also permit overlap. Do not infer that only one actor can affect a database merely because one current Lease holder is visible in the API. The downstream service has not necessarily checked that Lease at all.

A fencing design gives each authority epoch a monotonically increasing token and makes the downstream resource atomically reject operations from older epochs after accepting a newer one. Merely sending a token is insufficient if the receiver does not persist and enforce the check together with the effect. A Kubernetes object resourceVersion is not automatically such a token for another system, and an arbitrary increment in process memory is not a safe authority source. The token allocation and resource enforcement need their own documented correctness argument.

An idempotency key solves a different problem: repeated delivery of one logical operation should not produce duplicate effects under the receiving API’s contract. It does not necessarily stop a stale leader from issuing a different unwanted operation. Many controllers combine repeatable reconciliation, stable identities, cancellation, and leader election, adding fencing for external effects that require it. State the required safety property first. Strong exclusion at an external target is more demanding than reducing duplicate reconciliations during healthy operation.

Pause the old leader after it passes a local leadership check but before its external request completes. Let a new replica acquire the role, then resume the old request. Observe what the downstream system actually accepts. Also test cancellation during long cleanup, lease release while goroutines remain active, API request timeouts, and delayed responses. Measure time to useful reconciliation separately from time to election acquisition, because a new leader may still need cache synchronization or dependency recovery before it can make progress.

The reliability lab counts reachable participants only. Reset, set n=3 and k=1, predict two up and a majority threshold of two, then Step. That count does not run a Lease election, provide a fencing token, or stop a stale process. It helps separate the availability question from the side-effect safety question. A reachable majority can support some coordination protocols, but this diagram proves none of their behavior. Use the explicit epoch trace for the actual stale-writer argument.