Control loops and reconciliation

Kubernetes controllers use control loops to move current state toward desired state. An event is a reason to inspect state, not necessarily a complete description of everything that happened. Reconciliation should compare desired and observed resources to decide what remains. It should tolerate repeated invocation and recover after a restart without depending solely on an in-memory sequence of previous notifications.

Separate observation from authority. One controller may own a replica count while another owns certificate renewal. If both rewrite an entire object indiscriminately, one can repeatedly erase the other's work. Document owned fields, stable resource identities, and preconditions for destructive actions. A useful status record says which desired generation was observed and why progress is blocked, allowing the user to distinguish a stale report from an evaluated but unsatisfied request.

Suppose an external API creates a database but the response never reaches the controller. Retrying a blind create can allocate a second database. A stable idempotency key or an externally searchable ownership identity lets the next attempt discover the first result. Record enough intent before the side effect to recover after process death. For irreversible operations, a durable workflow and explicit compensation may be necessary; merely calling the function reconcile does not make it safe.

Reconciliation should be idempotent in effect: observing an already satisfied target should not create more changes. Use bounded retries with backoff for transient failures and clear conditions for invalid specifications. An authorization error does not become healthy merely because the controller retries it quickly. Avoid making every unrelated event trigger a broad inventory scan or an expensive external write. The platform needs both a convergent state machine and a sustainable cost per observation.

In the scalar model, a gain strictly between zero and two reduces error magnitude; gains above one alternate its sign. The endpoints zero and two do not reduce its magnitude. Actual infrastructure adds provisioning delay, noisy metrics, and discrete allocation. These effects can make an aggressive controller overshoot even when the instantaneous scalar model appears stable. Observe how long a change takes to become effective, and prevent the controller from interpreting its own unfinished operation as evidence that an identical new operation is needed.

Keep a recovery ledger: time to detect a mismatch, time waiting in the controller queue, external action time, and time to verify the result. Measure all four during a dependency outage and recovery. A low average reconcile duration can coexist with a long queue delay. Rate-limit work by tenant or dependency when necessary, and expose blocked conditions instead of generating an infinite stream of indistinguishable warnings.