A reliability model needs held-out failures and a defensible observation process

Mission survival, instantaneous availability, request success, and incident recovery duration have different observations and denominators. A missing monitor sample is not automatically a successful request, and a host-level outage count is not an operation-level failure fraction. Record exposure and censoring as well as events. Duplicate incident tickets can inflate counts while merged incidents can hide affected domains. Definitions should remain stable or be versioned across the dataset so an apparent improvement cannot come solely from changing what gets counted.

For an exponential lifetime model with suitable censoring, a maximum-likelihood rate uses observed failures divided by total at-risk exposure. This estimate is conditional on the constant-hazard model and observation assumptions. Zero failures over finite exposure does not establish a zero true rate; it limits what the data can say. A repairable-system intensity and a nonrepairable lifetime hazard also answer different questions. Validate the selected mechanism, not merely whether units cancel in an arithmetic expression. Keep parameter uncertainty separate from uncertainty that the entire model structure is wrong.

Choose a temporal holdout or another split matching the deployment question. Randomly mixing records from one common outage into training and validation can leak the same failure mechanism into both and exaggerate independence. Group related incidents, sites, or rollout cohorts where the claim requires generalization to new groups. Compare simple baselines and alternative dependence or duration models, using calibration and decision-relevant errors rather than selecting the most flattering metric. A model tuned after inspecting the holdout has used that evidence for fitting and needs another honest validation boundary.

Check residual patterns against time, load, age, domain, and repair phase. A constant-rate model that consistently underpredicts shared-outage windows needs more than narrower parameter intervals. Stress tests and fault injection can expose missing mechanisms, but their artificial event frequencies are not production probability estimates. Report what a test establishes: the service handles a chosen failure sequence under controlled conditions. Estimating how often that sequence occurs requires separate exposure evidence. A green test of one trace cannot certify an unbounded environment.

Document the model, parameter data, versions, assumptions, exclusions, validation period, uncertainty, and the decision it supports. Test sensitivity to common-cause probability, recovery tails, and traffic changes when those inputs influence the decision. Prefer an honest range or an unsupported-region label to an exact-looking point forecast. Monitor later calibration and define when changed architecture or workload requires refitting. A forecast can be useful within its measured envelope without becoming a permanent guarantee; keeping that envelope explicit makes later disagreement actionable.

The sampling lab isolates iid sample-mean uncertainty. Reset, set BOTH p=0.1 and n=20, predict variance 0.0045, then Play or Step. Reset and re-set BOTH p=0.1 and n=100; predict variance 0.0009. The exact binomial bars concern the sample mean; playback highlights outcomes rather than sampling data, and edits retain t. The displayed Berry–Esseen bound concerns standardized CDF error, not individual bar or continuity-correction errors; it is disabled at p=0 or p=1. There is no correlation, censoring, temporal holdout, or fitted production model. More iid samples reduce this variance without repairing structural misspecification.