Reliable messaging and backpressure

At-most-once handling can discard work if progress is acknowledged before the effect is durable. At-least-once delivery retains the possibility of retries, so a crash after the effect but before acknowledgement can execute the effect again. Neither ordering alone produces exactly one external effect. An acknowledgement can also be lost after successful processing; a retry is then evidence of uncertainty, not proof that the original attempt failed. Treat timeouts as unknown outcomes and make retry safety explicit.

An idempotent operation gives repeated execution the same externally relevant result as one execution. Assign a stable key to the logical operation, not a fresh key for every attempt. In a database-backed consumer, store that key under a uniqueness constraint and perform the business mutation in the same transaction. A check followed later by an unprotected insert is racy: two workers can both observe absence. External services need their own idempotency contract or a carefully designed reconciliation process.

A transactional outbox stores the business change and a pending event in one local transaction. A relay later publishes the event; because the relay can retry, downstream consumers still need duplicate-safe handling. An inbox or deduplication table records consumed operation keys alongside local mutations. Retention matters: expiring a key before the possible replay horizon reopens the duplication window. Keys must also distinguish genuinely new operations, or deduplication will incorrectly suppress valid work.

Kafka's idempotent producers and transactions address specific log and processing boundaries. They do not automatically make an arbitrary HTTP payment or email part of the same atomic transaction. Define the boundary in the application's contract. For multi-step workflows, durable state machines and compensating actions can expose partial progress rather than pretending a cross-system rollback is always possible. A compensation is a new business operation, which can fail and require retries of its own; it is not time travel.

Backpressure slows upstream work when downstream capacity is insufficient. A queue absorbs a temporary mismatch; it does not create capacity. Bound queued bytes, item count, and age, since one large request may cost more than thousands of small ones. Choose an overload policy: reject new work, delay producers, spill to a durable tier with a known limit, or shed a lower-priority class. Make that policy visible so clients know whether and when to retry.

Retries can amplify an outage. Apply capped exponential backoff with jitter, a retry budget, and an end-to-end deadline. Distinguish transient errors from poison messages that repeatedly fail validation. Quarantine those messages with enough context to investigate, while preserving privacy and a controlled replay path. Track oldest-message age, attempts per successful logical task, dead-letter growth, and downstream saturation. The fluid queue equation is useful for planning, but burstiness, skew, and variable task duration require load testing.