Tail latency and hedged requests

Let F(t) be the probability that a child finishes within time t. If an operation requires m independent, identically distributed children launched together, the parent finishes by t only if every child does. The result explains why an apparently small child tail can become common at the parent. Independence is a strong assumption: shared storage, network paths, or synchronized garbage collection can alter the joint distribution. Use traces to retain those relationships rather than independently shuffling child samples and calling the result a production forecast.

Measure latency at the user's boundary, including queueing, connection setup, admission, and retries. Partition results by operation, payload, cache state, and relevant tenant cohort. Preserve timeout and cancellation counts: omitting incomplete requests makes the tail look artificially healthy. A load generator that waits for each response before issuing the next request can reduce offered load exactly when the service slows. Use a controlled arrival schedule and report the offered and completed workloads separately when studying overload behavior.

For an eligible read, launch a second copy at delay h only if the first is unfinished. The client accepts a valid result and cancels remaining work where possible. Under a simple model, hedge launch probability equals the original request's survival probability at h. That is an attempt-count estimate, not a CPU estimate: the abandoned original may already have consumed substantial work and cancellation may arrive after completion. Trace both copies with a shared operation identifier and separate attempt identifiers so extra resource use is visible.

Eligibility needs semantic review. A read can still trigger side effects such as billing, cache fills, or external requests. Writes require a proven deduplication or transaction boundary before duplication is safe. Bound hedge count, use an alternate failure domain when appropriate, and disable hedges under overload unless experiments demonstrate a benefit within the resource budget. An immediate duplicate to the same saturated dependency may worsen the tail for every caller. Evaluate user latency together with useful throughput, queue growth, and resources per completed operation.

When two independent copies start simultaneously, the event that both exceed a threshold has probability equal to the product of their tail probabilities. A delayed hedge has a shorter remaining window, so its expression differs. If a common outage or shared queue affects both copies, multiplying marginal probabilities is invalid. Estimate conditional tail behavior from experiments that preserve placement and load. A representative test should include the recovery interval, because a surge of duplicated work can delay clearing the backlog even after the initiating slowdown ends.

The queue lab provides a deliberately simpler warning. Raise arrival rate toward service rate and inspect the separately labeled M/M/1 mean residence time under its assumptions. Play the deterministic fluid backlog to see surplus arrivals accumulate. Neither readout is a hedging simulator or a percentile estimate, and the lab cannot represent correlated request durations. Use it to explain why spending spare service capacity on hedges has a limit, then return to measured latency distributions for an actual policy decision.