List and watch: rebuild state when history expires

In the classic list-then-watch pattern, obtain a list and its collection resourceVersion, then request changes from that boundary using the documented watch parameters. For a paginated list, follow continue tokens consistently to complete the snapshot rather than mixing pages from unrelated baselines. The collection token is server-provided and opaque; it does not count the number of objects or elapsed seconds. A list with no objects can still establish a meaningful boundary. Client libraries usually handle much of this protocol, but your controller still depends on their recovery behavior.

A local materialized view applies the returned changes and schedules affected keys. A handler should not assume an event means it is seeing the only intermediate state: later reads can already contain newer changes, and notifications may lead to coalesced work. The controller’s invariant should be computable from current relevant objects. If a product needs an immutable audit history of every action, an ordinary informer cache and reconcile queue are not substitutes for a separately specified durable event log.

When a watch request refers to history the server no longer retains, the API can return HTTP 410 Gone. The client must rebuild its view from a new list and restart watching from the new valid baseline, rather than retry the same expired token forever. Replace the obsolete cached view so objects absent from the fresh snapshot are not retained as current. A framework’s reflector can perform this recovery, but application code must remain correct when the reconstructed state differs from what individual old events would have shown. External effects therefore need the repeatable reconciliation design established in the first lesson.

A watch BOOKMARK, when requested and sent, reports progress to a resource-version boundary without carrying a complete object update. Clients cannot assume bookmarks arrive at a fixed interval or arrive at all. They are not a substitute for connection timeouts or a guaranteed liveness heartbeat. Similarly, a client-cache resync can requeue known objects without proving a fresh server list occurred. Distinguish reprocessing cached state from relisting authoritative state, and check the documented behavior of the client library version you use.

Use backoff and cancellation for reconnect attempts, avoid unnecessary independent watches for the same resource scope, and watch metrics for repeated expiration or slow processing. A short history window with many lagging clients can amplify load through repeated relists. The API’s storage and cache details can evolve, so base correctness on the documented versioned API semantics rather than treating etcd revisions as your application’s resourceVersion format. Readiness should wait for the relevant initial cache synchronization before code relies on that view.

The queue lab illustrates a processing-rate imbalance. Reset, set λ=4 and μ=3, predict backlog 4+t, then Play or Step. Reset and set both λ=2 and μ=3 to predict drainage at t=4. This fluid model has no watch tokens, list snapshots, bookmarks, or expiration rule. It explains why a slow consumer can accumulate pending work, while the actual 410 recovery still needs the discrete baseline protocol. Editing either control preserves elapsed time, so use Reset for the fresh comparison.