Reconciliation: restore an invariant from the state you can observe

For a fictional worker-group resource, the desired specification may request five workers while observation shows three current owned workers. The controller should inspect identity, ownership, deletion state, and readiness before deciding which action is missing. A watch notification that a worker disappeared does not mean blindly create one more worker: another reconciliation may already have replaced it, or the user may have reduced the desired count. Read current state and compute the required difference instead of accumulating event-derived commands.

A converged state is a fixed point for the controller’s intended actions while the desired state and relevant dependencies remain unchanged. Reconciliation may still refresh observations, but should avoid unnecessary writes and duplicate external resources. Creating a worker is also different from that worker becoming ready; observed existence, scheduling, startup, and application readiness are separate milestones. A desired count is not a promise that capacity is available or that every attempted action succeeds within one pass.

Suppose an external create request times out after the remote service may already have accepted it. Retrying with a fresh random identity can leak a second resource. Use a stable operation identity or deterministic external name, then query or ensure the intended resource on a retry. The remote API must actually support the chosen idempotency contract. Recording a local success flag before the external effect can instead lose the effect after a crash, while recording only afterward leaves an uncertainty window that the protocol must handle.

Status should report observed facts about the relevant desired generation, rather than claim success because an API write returned. Many resource types use generation-related conventions, but the exact generation and observedGeneration semantics belong to their API definition. A Ready condition for an older desired generation is not evidence that the new configuration is ready. Avoid unconditional status updates that trigger another reconcile without changing information. Distinguish retryable dependency failures from invalid specifications that need a user-visible condition and a later change.

A reconciliation loop should limit work per pass, respect cancellation, and leave enough durable evidence to resume after a crash. External resources need an identity tied to the object’s immutable UID rather than only a reusable namespace/name. Deletion follows a different branch from ordinary provisioning, so inspect the deletion timestamp before creating more work. Protect updates against competing writers, and make every retry re-evaluate whether the desired operation still applies. Progress requires eventual useful reads and successful effects, not merely an infinite loop.

The optimization lab supplies a small fixed-point analogy. Reset, set step size η=0.5 and initial position x₀=2, predict x₁=1, then Step once. A second Step gives 0.5. Its exact gradient-descent recurrence moves toward zero under this step size; it does not reconcile API objects, count workers, or handle external failures. Changing controls preserves elapsed time, so Reset before a fresh experiment. Use the analogy to ask whether repeated application approaches the intended state, then test the actual controller’s invariants separately.