research30 min studyLesson 990 in Engineering

Research claim scope and limitation writing

Write conclusions that preserve the measured domain, uncertainty, comparison contract, and unresolved alternatives.

On this page

What you’ll understand

  • Connect every conclusion to a population, intervention, metric, and evidence source.
  • Separate missing conditions from measured failures and uncertain effects.
  • Write limitations that guide transfer decisions rather than merely soften a headline.

A precise limitation is useful engineering information. It tells a reader which deployment resembles the evidence, what assumption might fail, and which new measurement could change the decision. Limitations should therefore shape the claim itself, not appear as a vague disclaimer after a universal headline. A strong paper can make a narrow, well-supported statement and explain its mechanism clearly without pretending to cover every machine, workload, or future implementation.

Carry the experimental contract into the conclusion

Name the workload population, implementation revisions, resource contract, outcome boundary, and uncertainty method when writing the main result. Distinguish an empirical estimate from an analytic bound and a model prediction from measured hardware behavior. If a study compared complete tuned systems, do not conclude that one component alone caused the effect. If it studied a latency-quality tradeoff, do not report a speed number with the quality setting removed. The concise claim should retain the conditions that made the comparison meaningful.

A useful claim ledger links each assertion to a figure, configuration set, analysis, and assumptions. Separate what was directly tested from what is inferred through a model or proposed for future work. A single hardware model with several machines does not establish cross-architecture transfer. Ten sampled traces from one tenant do not establish coverage of ten independent organizations. Methodological prescriptions improve transparency, but neither a long checklist nor an artifact badge turns an extrapolation into an identified causal result.

C=(D,I,M,E,A)\mathcal C=(\mathcal D,\mathcal I,\mathcal M,\mathcal E,\mathcal A)
This bookkeeping tuple records a claim’s domain D, implemented intervention I, metric M, evidence E, and assumptions A. It is a writing and audit device, not a theorem proving a claim or a universal standardized causal notation.↔ Scroll wide equations horizontally.

Describe coverage and uncertainty without converting either into certainty

List the actual tested configurations rather than implying that every combination of reported factor levels was exercised. A Cartesian product of three hardware choices, two workloads, and two concurrency levels has twelve possible cells, but a study may test only some. A tested fraction is descriptive coverage of that chosen finite grid, not the probability of working on an arbitrary deployment. Missing cells can include the most difficult conditions, so a percentage must not conceal their identities or reasons for absence.

Report effect intervals and unfavorable cells alongside aggregates. Ten point estimates above a ratio of one do not establish that all ten underlying effects are positive when some intervals include one. Multiple comparisons and selection among configurations can also affect inference; state whether intervals are individual or simultaneous and whether the reported set was prespecified. More repetitions can narrow uncertainty within a tested condition, while adding new platforms or workloads addresses a different limitation. Precision and external validity are complementary rather than interchangeable.

cgrid=∣Xtested∣∣Xplanned∣,Xtested⊆Xplannedc_{\rm grid}=\frac{|\mathcal X_{\rm tested}|}{|\mathcal X_{\rm planned}|},\qquad \mathcal X_{\rm tested}\subseteq\mathcal X_{\rm planned}
This dimensionless ratio measures tested cells in an explicitly finite planned grid. It does not measure causal identification, evidence quality, deployment success probability, or generalization beyond that grid. Repeated runs within one cell do not increase its cell count.↔ Scroll wide equations horizontally.

Write actionable limits and competing explanations

For each material limitation, explain its direction and consequence when known. A missing high-contention workload leaves a synchronization mechanism untested; a proprietary trace limits independent population auditing; an unmodeled topology can invalidate a simulator ranking. State unresolved alternatives and propose a discriminating next experiment. Avoid claiming that a missing result is harmless merely because related cells looked favorable. A negative or uncertain observation can motivate a better-scoped contribution instead of being relegated to an appendix that contradicts the abstract.

The sampling instrument separates narrower uncertainty from wider domain coverage. Reset, set both p=0.8 and n=25, predict mean 0.8 and standard deviation 0.08 for K/n, then Play or Step. Separately Reset, set both p=0.8 and n=100, predict mean 0.8 and standard deviation 0.04, then Play or Step. These are exact independent Bernoulli mean distributions, and playback highlights bars rather than drawing new samples. Increasing n changes this fixed model’s averaging uncertainty; it does not add hardware types, traces, causal controls, or evidence that a systems claim transfers.

Write the scope that the evidence actually supports

A planned grid contains three hardware types, two workloads, and two concurrency levels. Ten cells were tested and two were inaccessible. Their estimated baseline/candidate time ratios range from 1.05 to 1.25, but the weakest cell has an individual interval [0.98,1.12].

  1. The planned grid has 3×2×2=12 cells. Ten tested cells give descriptive coverage 10/12≈83.33%; name the two inaccessible cells rather than treating them as successful.

  2. The ten point estimates are above one, but the weakest cell’s interval includes one and values below it. Thus the evidence does not establish a positive underlying effect in every tested cell, even before considering simultaneous inference.

  3. A defensible report states that ten tested configurations had estimated ratios 1.05–1.25, identifies the uncertain cell and interval procedure, and leaves the two untested conditions unresolved. It does not claim universal speedup on all twelve cells or all deployments.

The result

The final sentence should preserve tested scope, uncertainty, missing evidence, and the exact comparison contract.

Explore the model

Interactive lab

Interactive lab · sampling

More repetitions do not widen the tested domain

Reset, set both p=0.8 and n=25, predict mean 0.8 and standard deviation 0.08, then Play or Step. Separately Reset, set both p=0.8 and n=100, predict mean 0.8 and standard deviation 0.04, then Play or Step. The exact iid Bernoulli model narrows averaging uncertainty without adding platforms, workloads, or causal evidence.

More repetitions do not widen the tested domainExact binomial masses for the mean of n independent Bernoulli trials, compared with normal mass between adjacent half-count boundaries; normal tails outside the finite support remain outside the bars. Playback highlights outcomes, not random samples. The Berry–Esseen bound applies to standardized CDFs, not individual bars or continuity-corrected errors. At p = 0 or 1, normal comparison and that bound are disabled. Normal-CDF evaluation uses a numerical approximation with absolute error below 7.5×10⁻⁸ (at most 1.5×10⁻⁷ per interval, before rounding). Current readings: Mean sample proportion: 0.5; Sample-mean variance: 0.0125; Uniform CDF error bound: ≤ 0.1062. All parameters are adjustable with the labeled controls below.Exact distribution of the mean of 20 independent Bernoulli variables with success probability 0.5; playback highlights outcome 0.00.250.50.75100.0580.1150.1730.23Sample mean · k / nProbability massNormal bin masses are an approximationk = 0 · exact probability = 9.54e-7
Exact binomial massNormal interval massHighlighted outcome / mean
Mean sample proportion
0.5
Sample-mean variance
0.0125
Uniform CDF error bound
≤ 0.1062
t = 0 s

Exact binomial masses for the mean of n independent Bernoulli trials, compared with normal mass between adjacent half-count boundaries; normal tails outside the finite support remain outside the bars. Playback highlights outcomes, not random samples. The Berry–Esseen bound applies to standardized CDFs, not individual bars or continuity-corrected errors. At p = 0 or 1, normal comparison and that bound are disabled. Normal-CDF evaluation uses a numerical approximation with absolute error below 7.5×10⁻⁸ (at most 1.5×10⁻⁷ per interval, before rounding). Parameter changes pause playback and recompute the model at the current elapsed time. Reset restores both parameters and returns time to zero. Playback starts paused. Adjust either parameter, then choose Play or Step.

Look forPrecision within a condition and transfer across conditions require different evidence.

Check your understanding

Which conclusion best matches ten tested cells with favorable point estimates, one interval including no improvement, and two untested cells?

Try it yourself

A planned grid has four hardware types and three workload classes. Nine cells were tested, and one tested ratio interval is [0.97,1.08]. Compute grid coverage and assess a claim of improvement in every planned cell.

Take a moment to make a prediction or work it out on paper. Open a hint when you need one.

Give me a hint

Count the finite grid before interpreting the interval, and keep untested cells separate from uncertain tested cells.

Compare with the solution

The grid has twelve cells and coverage is 9/12=75%. Three conditions remain untested. The interval in one tested cell includes one and possible slowdown, so it does not establish improvement there. A universal claim over all planned cells is unsupported for both uncertainty and missing-evidence reasons.

Take it with you

  • Carry the domain, comparison contract, metric, and assumptions into the main claim.
  • Separate descriptive grid coverage, inferential precision, and external validity.
  • Write limitations that identify consequences and the next discriminating experiment.

Read further

Original sources and deeper explanations to follow at your own pace.

Optional progress, saved in this browser when available. Revisit any lesson at any time.

Lesson not marked complete.