Deadlines, retries, and idempotency

An end-to-end deadline limits how long the caller can usefully wait. Each downstream attempt must fit within the remaining budget, including queueing, connection establishment, computation, and time to return a result. Propagating a relative remaining duration avoids assuming perfectly synchronized clocks across hosts. Cancellation should release queued and active work, but applications must cooperate; receiving a cancellation signal does not automatically reverse an already committed side effect. Include cleanup and response serialization in the budget instead of giving every child the full original timeout.

Choose attempt timeouts using measured latency distributions and an acceptable false-timeout rate for the relevant request class. Then test cold connections, deployment events, and dependency slowdowns. A timeout that covers only the socket read may omit DNS or TLS work in some client implementations. When the remaining budget is too short for a useful attempt, stop. Record deadline-exceeded outcomes separately from explicit permanent rejection, and retain enough trace context to determine whether the remote operation later completed.

Classify errors by whether another attempt is semantically safe and plausibly useful. Invalid input is usually permanent; a transient transport failure may be retryable only if operation identity is preserved. Let one appropriate layer own retries, cap attempts and total retry rate, and add randomized delay when many callers could synchronize. Jitter spreads attempts in time but creates no capacity. Backoff without a global or per-client retry budget can still sustain overload, particularly when a large population retries at the same capped interval.

If each of several nested layers retries independently, a single user operation can trigger multiplicative downstream attempts. An outer layer often cannot tell that the inner layer already spent its budget. Propagate attempt metadata and avoid retrying explicit overload responses when the contract says to stop. Monitor started attempts per original operation, remote resource cost, and successful useful completions. A rising success rate among individual attempts can hide that the service is spending most of its capacity repeating work that the user has already abandoned.

An idempotency key identifies a logical operation, not merely a transport attempt. Scope it to the authenticated principal and operation type, store a fingerprint of relevant input, and reject reuse with incompatible input. The deduplication record and the protected side effect must be committed atomically where possible. Otherwise a crash between the side effect and record creation permits duplication. When the effect crosses an external service boundary, use that service's idempotency facility or an explicit state machine and reconciliation process; a local cache entry cannot make a remote effect atomic.

Specify behavior for concurrent duplicates, in-progress operations, stored responses, failed attempts, and key expiration. Retention must cover the promised retry horizon; after expiration the same key may become a new operation according to the contract. The reliability lab only counts replicas and majority availability. Use it to ask where the durable idempotency record lives during failures, while remembering that a majority being up does not implement a transaction, consensus, or exactly-once external effects. Correctness comes from the protocol surrounding the record.