Control-plane storage: history, watches, and write amplification

etcd maintains a store revision that advances when the key space is modified. A transaction that changes multiple keys advances the revision once, and its generated modification events share that revision. A key also has creation revision, modification revision, and a per-key version field with different meanings. These are documented etcd concepts; they are not a license for a Kubernetes API client to parse resourceVersion numerically or infer wall-clock duration from it. The store’s logical ordering and elapsed-time measurements answer different questions.

Multi-version storage retains historical states until compaction removes superseded history according to policy. A watcher can resume only while the requested history remains available; a compacted revision produces the documented failure and requires a new valid boundary. Longer retention supports longer disconnect recovery but increases retained-history work and space. Short retention can shift cost into relists and catch-up. Choose the policy with actual client lag, restart, and recovery needs rather than minimizing one storage metric in isolation.

A rough retained-history estimate multiplies the rate of stored versions by mean version size and retention time. Actual database size includes indexes, metadata, encoding, page reuse, and workload-dependent overhead; do not present that estimate as exact on-disk usage. A status update that writes the same logical information can still generate expensive downstream activity, so compare before writing and update only useful changed facts. Separate desired-state change, status progress, heartbeat, and lease renewal rates before attributing all churn to the workload itself.

Compaction makes old history unavailable, while defragmentation reclaims backend space that can remain allocated after history removal. They are different operations. The documented maintenance procedure and availability effects matter; defragmentation on a member can block that member’s reads and writes while it runs. Quota alarms and disk latency can affect control-plane progress, so monitor logical usage, allocated size, request latency, and maintenance behavior. More storage capacity does not by itself fix high latency or uncontrolled write amplification.

An unconditional status write can wake the same controller, which writes status again, producing a self-sustaining loop. A broad watch that queues every object after any unrelated event can amplify work further. Inspect per-controller update rates, no-op reconciliations, list sizes, watch reconnects, and object payload growth. Apply bounded retries and appropriate filters without removing events needed for correctness. Capacity planning should include recovery bursts when many controllers restart simultaneously, not just the healthy steady-state average.

The queue lab isolates a producer-service mismatch. Reset, set λ=4 and μ=3, predict backlog 4+t, then Play or Step. Reset and set λ=2, μ=3 to observe drainage under the exact fluid formula. There are no etcd revisions, MVCC versions, WAL writes, watches, compaction, or defragmentation in this model. Use it only to explain why reducing unnecessary arrivals can matter as much as raising service capacity. The actual storage and fanout estimates come from measured rates and explicit units.