Controller tests: attack the lifecycle boundaries

A safety invariant might require at most one external allocation per object UID, preservation of another actor’s label, or no removal of a finalizer before confirmed cleanup. A liveness assertion might require eventual readiness once the API, dependencies, and capacity stay healthy and intent stops changing. Those assumptions matter: no test should demand successful external deletion while the external service rejects every request forever. Conversely, a quick timeout is not proof that a controller can never make progress; it can be weak evidence from an unsuitable observation window.

Build a controllable external fake that records operation identities and permits a request to commit remotely while its response times out. Crash or restart the controller before its local success record, then reconcile again. Assert the external cardinality and returned identity rather than only the number of HTTP calls. Multiple calls can be correct when they implement an idempotent ensure. Also test delete-and-recreate with the same namespace/name but different UID so a stale action cannot silently target the new object’s resource.

Pure unit tests can check the desired action calculation and error classification quickly. An API-server integration environment is needed for server validation, resourceVersion conflicts, subresource behavior, and installed admission configuration. The Kubebuilder envtest documentation explains that its ordinary test control plane does not run built-in controllers. Therefore owner references do not automatically trigger garbage collection there. A test expecting a dependent to disappear needs a real cluster or the relevant controller, while an envtest assertion can verify the reference metadata your code creates.

Keep asynchronous tests condition-based: repeatedly observe the required state within a declared deadline instead of using one arbitrary sleep and assuming success. Capture useful logs, object versions, UID, and operation history on failure. Use deterministic scheduling controls where possible to reproduce a race, then add stress tests for broader schedules. Passing a hundred runs of one easy schedule is not equivalent to covering a single paused-leader overlap that the system’s safety property actually depends on.

Create a table of failure locations: before finalizer persistence, after external commit but before response, after status conflict, during deletion, during watch expiration, and after leadership changes. Combine each with the relevant restart or duplicate notification rather than every imaginable combination. Include blocked permissions, invalid specs, and permanent external failures so the controller reports actionable state without hammering dependencies. Observe no-op steady state to catch status feedback loops that a single successful reconciliation would miss.

The optimization lab illustrates one narrow repeatability check. Reset, set η=1 and x₀=2, predict x₁=0, then Step once and again. The selected quadratic reaches its fixed point and remains there. Reset and set η=2.5, x₀=2 to predict first position −3 and an unstable trajectory as Step continues; out-of-range reporting is part of that model. It is not a controller test harness. The contrast helps explain why a test should exercise both a converging condition and a documented failing condition, then assert the actual system’s own invariant.