Recovery tails include incidents that have not finished yet
Define recovery endpoints, retain right-censored observations, estimate survival carefully, and distinguish heavy-tail hypotheses from unsupported percentile forecasts.
On this page
The question for today
What you’ll understand
- Distinguish mitigation, user recovery, and full repair endpoints.
- Use right-censored durations without pretending they are completed observations.
- Explain why sparse extreme recovery data cannot justify an arbitrary tail percentile.
The longest incident may still be open when a recovery dashboard is generated. Dropping it makes the completed-incident average look better, while counting its current age as its final duration invents a recovery that has not happened. Survival analysis keeps the observation boundary explicit. Define the endpoint and censoring process before choosing a distribution or announcing how long the next severe recovery should take.
One incident can have several meaningful completion times
Detection, mitigation, restoration of user success, removal of backlog, and restoration of redundancy are different events. A service can answer requests again while remaining one fault away from another outage. Choose the endpoint relevant to the decision and record its start rule consistently. Mixing detection-to-mitigation with failure-to-full-repair durations can create a distribution that describes no actual process. Include incident classes, severity, dependencies, staffing context, and observation gaps where they affect interpretation, without slicing the data until every difficult event disappears from the population.
An open incident observed for c minutes gives a right-censored duration: recovery time exceeds c under that endpoint definition. It contributes exposure even though the final time is unknown. Survival methods usually need a noninformative censoring assumption, possibly conditional on recorded covariates. If the hardest incidents are transferred elsewhere and disappear from the dataset, censoring can be informative and the naive estimate biased. Record why observation ended. A resolved ticket is not necessarily a recovered service, and a retired service is not automatically a successful repair at its retirement timestamp.
Estimate the supported survival curve before extrapolating
The Kaplan–Meier estimator multiplies conditional survival fractions at observed recovery times, using the risk set still under observation just before each time. Right-censored cases leave the risk set without causing a recovery step. Under its assumptions, this uses more information than discarding open incidents. It still cannot identify an arbitrary far-tail percentile beyond the observed recovery support. If estimated survival remains above ten percent at the last informative time, the data do not supply a finite nonparametric 90th-percentile recovery estimate merely because a dashboard requires a number.
Heavy-tailed models can describe rare very long durations, but the name is not evidence that a particular Pareto, lognormal, or mixture model fits. Some heavy-tail families have a finite mean and infinite variance; other parameter regimes do not even have a finite mean. Empirical moments can be unstable in sparse incident samples. Plot and compare candidate survival behavior, include uncertainty, and separate fit from extrapolation. Operational changes can also invalidate iid duration assumptions, so a theoretical tail family should not silently replace evidence about how repair work actually proceeds.
Report unsupported regions and decision-relevant uncertainty
Show sample size, completed and censored counts, observation horizon, confidence information, and the region where tail estimates depend on a parametric model. A restricted mean over an observed horizon can be more defensible than an unrestricted mean requiring unsupported tail assumptions. Validate recovery forecasts on later incidents and inspect changing incident mix. For a response plan, ask how much capacity or user exposure remains while recovery is unfinished, not only whether a point estimate of duration improved. The model should support a decision whose risk tolerance is explicit.
The probability lab represents one fixed-threshold recovery question only. Reset, set BOTH p=0.8 and n=10, interpret success as recovered by a hypothetical threshold, predict all-ten probability 0.8¹⁰≈0.1074, then Play or Step. Reset and re-set BOTH p=0.95 and n=10; predict about 0.5987. Edits retain t; playback highlights exact independent counts. There are no durations, censored cases, shared incidents, or fitted tail curves. A threshold success fraction cannot identify the full recovery-time distribution or make a 90th percentile estimable beyond the data.
Work it through
Keep open incidents in the risk set
Five fictional incidents begin observation under a compatible design. Recoveries occur at two, four, and eight minutes. Two other incidents are right-censored at ten and twelve minutes. Assume noninformative censoring and no ties.
At two minutes the risk set is five and one recovers, so estimated survival becomes 4/5=0.8. At four it becomes 0.8×3/4=0.6.
At eight minutes three remain at risk and one recovers, giving 0.6×2/3=0.4. Censoring at ten and twelve removes observation without new downward recovery steps.
The observed curve does not fall to 0.1, so it does not identify a finite nonparametric 90th-percentile recovery time. The completed-only mean (2+4+8)/3≈4.67 minutes excludes the two known long unfinished durations.
The open incidents carry essential tail information even though their final durations are unknown. The supported estimate cannot be forced into an unjustified far-tail percentile.
The living notebook
Explore the model
Interactive lab · probability
Threshold recovery is not a duration model
For EACH run Reset, set BOTH p and n, predict, then Play or Step. At n=10 compare p=0.8, all-success probability ≈0.1074, with p=0.95, ≈0.5987. Edits retain t. Exact independent threshold counts contain no recovery durations, censoring, heavy-tail fit, or percentile extrapolation.
- Mean · np
- 5
- Variance · np(1−p)
- 2.5
- Total probability
- 1
These are exact binomial probabilities for independent trials with one fixed success probability. Playback tours the distribution; it does not resample outcomes. Parameter changes pause playback and recompute the model at the current elapsed time. Reset restores both parameters and returns time to zero. Playback starts paused. Adjust either parameter, then choose Play or Step.
Look forA probability of recovery by one threshold does not determine the full duration tail or erase censored observations.
Pause & reason
Check your understanding
Your turn
Try it yourself
Four incidents remain at risk just before a recovery time; one recovers and none are censored at that instant. If survival immediately before is 0.8, calculate the new Kaplan–Meier value. Does a later censoring of one case reduce it again?
Take a moment to make a prediction or work it out on paper. Open a hint when you need one.
Give me a hint
Use the recovery fraction for a downward step; censoring changes future risk sets rather than declaring a recovery.
Compare with the solution
The value becomes 0.8×(1−1/4)=0.6. A later censoring without recovery leaves the estimated survival value unchanged at that moment while reducing the risk set for subsequent calculations. It does not establish what happens beyond the observed support.
Keep the idea
Take it with you
- A recovery model needs consistent endpoints and observation rules.
- Open incidents are censored evidence, not completed times or empty records.
- Tail forecasts require explicit support, assumptions, and uncertainty.
Read further
Original sources and deeper explanations to follow at your own pace.
Optional progress, saved in this browser when available. Revisit any lesson at any time.
Lesson not marked complete.