Simulation calibration and external validity

Write the modeled entities, state transitions, scheduling assumptions, random inputs, and omitted mechanisms. Specify the outputs needed for a decision and acceptable error at relevant operating points. A model adequate for ranking two policies at low load may be inadequate for predicting their extreme tail latency near overload. Validate the intended use rather than the abstract label of simulator. Preserving means while removing burst correlations can change the output that matters, even when every input field has a plausible range.

Implementation verification uses invariants, hand-computable cases, conservation checks, independent implementations, and boundary tests. Check event ordering, time units, random-stream ownership, and numerical tolerances. A deterministic regression fixture can establish that a change preserved a known computation, but it does not establish that the computation represents a real system. Document the conceptual model separately from the executable so a discrepancy can be assigned to assumptions, implementation, or data. Keep stochastic replication uncertainty distinct from systematic disagreement with a physical target.

Use calibration data to estimate parameters and choose model structure under a documented procedure. Reserve independent conditions, traces, or time periods for validation that are not repeatedly used for tuning. If validation failures trigger another model revision, those failures have become development information; obtain new confirmation evidence where feasible. State how much data informed every choice. A holdout split helps expose overfitting but cannot make an unrepresentative collection suddenly match the intended operating population.

Compare predicted and observed outputs with a tolerance chosen for the intended decision. Relative error can be interpretable for positive times, while absolute error or another scale is needed near zero. Include uncertainty in both simulation and real-system measurements rather than treating noisy observations as exact truth. Test several conditions that exercise expected mechanisms, especially changes in scale or contention. Agreement at one machine size does not validate a simulator’s network contention model on a larger topology.

Plot residuals against load, size, topology, and other modeled variables. A systematic trend can reveal an omitted mechanism even when the average error is small. Use sensitivity analysis to see whether uncertain inputs can reverse a policy ranking. Distinguish interpolation inside a tested region from extrapolation into new hardware or demand regimes. External validation by another group or on another platform adds evidence, but transfer still depends on what changed and whether relevant mechanisms remain represented. Report failures as boundaries of the model’s usable domain.

The function instrument illustrates a known model’s extrapolation without pretending to validate a real simulator. Reset, set both a=0.5 and x=1, predict y=0.5, then Play or Step. Separately Reset, set both a=0.5 and x=2, predict y=2, then Play or Step. The linear rule y=0.5x matches the first observation but predicts one at the second. The fixed quadratic instrument refutes that linear extension within its own noiseless domain; it supplies no physical-system data, fitted simulator, or statistical validation certificate.