Treat timeouts as suspicions under a failure model

In a fully asynchronous model with no known upper bound on message delay or scheduling delay, a finite period of silence cannot distinguish a crashed participant from a correct but very slow one. A timeout is therefore a policy for suspecting, not a measurement that proves the remote process has stopped executing. The same silence can be observed during network partition, packet loss, scheduler stalls, long garbage collection, or overload. Local elapsed time must itself use an appropriate clock, but an ideal local timer still cannot reveal which remote cause produced the missing evidence.

Failure-detector theory separates completeness, concerning eventual suspicion of crashed participants, from accuracy, concerning mistakes about correct participants. Different classes make different eventual or perpetual promises. These properties are assumptions about the detector's outputs, not automatically delivered by choosing a timeout value. A deployment may assume that timing eventually becomes favorable enough for a protocol to make progress while still preserving safety during earlier mistakes. Keep that liveness argument separate from the resource and replication rules that prevent conflicting effects. An unsafe takeover does not become safe merely because the timeout was chosen from a plausible percentile.

Shorter thresholds can improve response to genuine failures while increasing false suspicions under ordinary delay. Longer thresholds can reduce churn but leave failed work assigned for longer. Heartbeat cadence, network paths, observer CPU pressure, and retry policy all affect the tradeoff. Measure the distribution of inter-arrival gaps for the relevant workload, including correlated pauses, rather than substituting a service's unrelated request latency percentile. A detector can be adaptive, but adaptation does not erase its model assumptions. Document what a suspect event triggers: investigation, reduced traffic, a re-election attempt, or a resource authority change.

For a narrowly specified calculation, suppose a healthy participant independently misses each heartbeat opportunity with probability p. Requiring r misses in a particular fixed group gives probability p raised to r for that group's all-missed outcome. Real gaps often correlate, so applying this formula to production without testing independence can dramatically understate false suspicions. Overlapping sliding windows also do not behave like independent groups. Specify whether the counted outcome is a candidate window, an episode of suspicion, or a transition that actually triggers recovery; those counts can differ even over the same heartbeat trace.

A protocol can use suspicion to try a new leader or redirect work while requiring separate authoritative evidence before accepting protected effects. Leases, fencing, durable generations, and quorum protocols address different pieces of that design; none follows automatically from the detector's confidence score. Recovery can also amplify the original overload by creating retries, cache misses, or concurrent replacement work. Rate-limit or stage actions according to the operational contract and examine whether the detector becomes unstable under the very load produced by its reactions. Test false suspicion as an expected failure mode rather than only testing a clean process crash.

The two-state Markov lab illustrates one abstract suspicion process while the monitored participant remains healthy. State 0 means not suspected and state 1 suspected. Reset, set both transition p=0.2 and recovery q=0.4, predict the state 1 probability after three transitions, then Step three times. The exact probabilities approach a stationary suspected fraction of one third under this model; the third-step value is 0.312. These are finite-step distributions, not sampled heartbeat traces or actual failures. A fresh parameter comparison begins with Reset and re-setting both controls, because editing alone retains the elapsed transition index.