Research claim scope and limitation writing
Write conclusions that preserve the measured domain, uncertainty, comparison contract, and unresolved alternatives.
On this page
The question for today
What you’ll understand
- Connect every conclusion to a population, intervention, metric, and evidence source.
- Separate missing conditions from measured failures and uncertain effects.
- Write limitations that guide transfer decisions rather than merely soften a headline.
A precise limitation is useful engineering information. It tells a reader which deployment resembles the evidence, what assumption might fail, and which new measurement could change the decision. Limitations should therefore shape the claim itself, not appear as a vague disclaimer after a universal headline. A strong paper can make a narrow, well-supported statement and explain its mechanism clearly without pretending to cover every machine, workload, or future implementation.
Carry the experimental contract into the conclusion
Name the workload population, implementation revisions, resource contract, outcome boundary, and uncertainty method when writing the main result. Distinguish an empirical estimate from an analytic bound and a model prediction from measured hardware behavior. If a study compared complete tuned systems, do not conclude that one component alone caused the effect. If it studied a latency-quality tradeoff, do not report a speed number with the quality setting removed. The concise claim should retain the conditions that made the comparison meaningful.
A useful claim ledger links each assertion to a figure, configuration set, analysis, and assumptions. Separate what was directly tested from what is inferred through a model or proposed for future work. A single hardware model with several machines does not establish cross-architecture transfer. Ten sampled traces from one tenant do not establish coverage of ten independent organizations. Methodological prescriptions improve transparency, but neither a long checklist nor an artifact badge turns an extrapolation into an identified causal result.
Describe coverage and uncertainty without converting either into certainty
List the actual tested configurations rather than implying that every combination of reported factor levels was exercised. A Cartesian product of three hardware choices, two workloads, and two concurrency levels has twelve possible cells, but a study may test only some. A tested fraction is descriptive coverage of that chosen finite grid, not the probability of working on an arbitrary deployment. Missing cells can include the most difficult conditions, so a percentage must not conceal their identities or reasons for absence.
Report effect intervals and unfavorable cells alongside aggregates. Ten point estimates above a ratio of one do not establish that all ten underlying effects are positive when some intervals include one. Multiple comparisons and selection among configurations can also affect inference; state whether intervals are individual or simultaneous and whether the reported set was prespecified. More repetitions can narrow uncertainty within a tested condition, while adding new platforms or workloads addresses a different limitation. Precision and external validity are complementary rather than interchangeable.
Write actionable limits and competing explanations
For each material limitation, explain its direction and consequence when known. A missing high-contention workload leaves a synchronization mechanism untested; a proprietary trace limits independent population auditing; an unmodeled topology can invalidate a simulator ranking. State unresolved alternatives and propose a discriminating next experiment. Avoid claiming that a missing result is harmless merely because related cells looked favorable. A negative or uncertain observation can motivate a better-scoped contribution instead of being relegated to an appendix that contradicts the abstract.
The sampling instrument separates narrower uncertainty from wider domain coverage. Reset, set both p=0.8 and n=25, predict mean 0.8 and standard deviation 0.08 for K/n, then Play or Step. Separately Reset, set both p=0.8 and n=100, predict mean 0.8 and standard deviation 0.04, then Play or Step. These are exact independent Bernoulli mean distributions, and playback highlights bars rather than drawing new samples. Increasing n changes this fixed model’s averaging uncertainty; it does not add hardware types, traces, causal controls, or evidence that a systems claim transfers.
Work it through
Write the scope that the evidence actually supports
A planned grid contains three hardware types, two workloads, and two concurrency levels. Ten cells were tested and two were inaccessible. Their estimated baseline/candidate time ratios range from 1.05 to 1.25, but the weakest cell has an individual interval [0.98,1.12].
The planned grid has 3×2×2=12 cells. Ten tested cells give descriptive coverage 10/12≈83.33%; name the two inaccessible cells rather than treating them as successful.
The ten point estimates are above one, but the weakest cell’s interval includes one and values below it. Thus the evidence does not establish a positive underlying effect in every tested cell, even before considering simultaneous inference.
A defensible report states that ten tested configurations had estimated ratios 1.05–1.25, identifies the uncertain cell and interval procedure, and leaves the two untested conditions unresolved. It does not claim universal speedup on all twelve cells or all deployments.
The final sentence should preserve tested scope, uncertainty, missing evidence, and the exact comparison contract.
The living notebook
Explore the model
Interactive lab · sampling
More repetitions do not widen the tested domain
Reset, set both p=0.8 and n=25, predict mean 0.8 and standard deviation 0.08, then Play or Step. Separately Reset, set both p=0.8 and n=100, predict mean 0.8 and standard deviation 0.04, then Play or Step. The exact iid Bernoulli model narrows averaging uncertainty without adding platforms, workloads, or causal evidence.
- Mean sample proportion
- 0.5
- Sample-mean variance
- 0.0125
- Uniform CDF error bound
- ≤ 0.1062
Exact binomial masses for the mean of n independent Bernoulli trials, compared with normal mass between adjacent half-count boundaries; normal tails outside the finite support remain outside the bars. Playback highlights outcomes, not random samples. The Berry–Esseen bound applies to standardized CDFs, not individual bars or continuity-corrected errors. At p = 0 or 1, normal comparison and that bound are disabled. Normal-CDF evaluation uses a numerical approximation with absolute error below 7.5×10⁻⁸ (at most 1.5×10⁻⁷ per interval, before rounding). Parameter changes pause playback and recompute the model at the current elapsed time. Reset restores both parameters and returns time to zero. Playback starts paused. Adjust either parameter, then choose Play or Step.
Look forPrecision within a condition and transfer across conditions require different evidence.
Pause & reason
Check your understanding
Your turn
Try it yourself
A planned grid has four hardware types and three workload classes. Nine cells were tested, and one tested ratio interval is [0.97,1.08]. Compute grid coverage and assess a claim of improvement in every planned cell.
Take a moment to make a prediction or work it out on paper. Open a hint when you need one.
Give me a hint
Count the finite grid before interpreting the interval, and keep untested cells separate from uncertain tested cells.
Compare with the solution
The grid has twelve cells and coverage is 9/12=75%. Three conditions remain untested. The interval in one tested cell includes one and possible slowdown, so it does not establish improvement there. A universal claim over all planned cells is unsupported for both uncertainty and missing-evidence reasons.
Keep the idea
Take it with you
- Carry the domain, comparison contract, metric, and assumptions into the main claim.
- Separate descriptive grid coverage, inferential precision, and external validity.
- Write limitations that identify consequences and the next discriminating experiment.
Read further
Original sources and deeper explanations to follow at your own pace.
Optional progress, saved in this browser when available. Revisit any lesson at any time.
Lesson not marked complete.