Work queues: coalesce keys and slow failing retries

If key A is already waiting, adding A repeatedly does not create an unbounded series of identical queued entries in this workqueue model. Once a worker gets A, the queue marks it as processing. If A changes again while processing, the queue can mark it dirty so it will be queued again after the worker signals Done. This protects against losing the notification that arrived during an in-progress pass. The next pass reads current state; it does not need the exact count of duplicate notifications.

A worker must pair every successful Get with Done through all return paths so processing state is released. On a successful reconciliation, resetting per-key rate-limit history with Forget has a different purpose from Done. A failure may schedule a rate-limited retry, while a deliberate future check may use a delay. Use the API matching the library version and controller framework; treating these operations as synonyms can strand work, retain stale failure penalties, or create tight retry loops.

A capped exponential policy can double a per-key delay after each failure up to a maximum. Add jitter where appropriate to avoid many controllers retrying a failed dependency at the same instant. The policy is not a promise that the external service recovers before the next attempt. Authentication failures or invalid specifications may need a condition and a different trigger instead of unlimited rapid retries. A shared dependency also needs a global or endpoint-aware limit; independent per-key limits can still overwhelm it when thousands of keys fail together.

Resetting failure history on success lets an unrelated later failure start from the intended initial penalty. Keep retry state bounded when keys disappear, and honor shutdown rather than draining indefinitely against a canceled context. Distinguish a queued key count from accepted business requests: one key may represent a large resource group with much work, while many notifications may represent no needed write. Measure queue age, reconcile duration, retry counts, API calls, and external effects together before adding workers.

If distinct ready keys arrive faster than they can be reconciled, waiting work grows despite perfect duplicate suppression. More workers can help independent external waits but may increase API conflicts, rate-limit pressure, or shared-lock contention. A single hot key remains serial within the stated queue model. Check whether the long pass should be broken into bounded actions and requeued, while preserving correctness under another update. Avoid a retry design that starves healthy keys behind repeated failure on one dependency.

The queue lab isolates fluid capacity rather than key bookkeeping. Reset, set λ=4 and μ=3, predict backlog 4+t, then Step or Play. For a fresh stable comparison Reset, set λ=2 and μ=3, and predict clearance at t=4. The instrument has no dirty set, Done, Forget, jitter, or exponential retry. Its separate M/M/1 residence estimate is conditional on its stochastic assumptions. Use the explicit key trace for coalescing and the fluid view only for the persistent arrival-service imbalance.