Resample the independent units rather than their repeated rows

Examples can share a conversation, repository, document, author, or generation template. Their outcomes may move together because they rely on the same knowledge or failure mechanism. Repeating more variants within one source can reveal local robustness while contributing less information about new sources than an equal number of independent sources would. In a simplified equal-size model, the variance inflation depends on within-cluster correlation. That model is a diagnostic calculation, not a universal formula for arbitrary overlapping groups, unequal cluster sizes, or unknown dependence.

The intended population determines which units should be considered exchangeable. For performance on new users, users may be the sampling units even when each provides many conversations. If repositories share a common upstream project, repository IDs alone may miss additional dependence. Grouping choices should follow collection and deployment knowledge, with sensitivity analyses when uncertainty remains. Saravanan and colleagues illustrate how treating nested measurements as independent can misstate uncertainty. Their hierarchical-bootstrap results concern their modeled structures; they do not justify blindly applying the same hierarchy to every evaluation dataset.

For a basic cluster bootstrap, sample cluster identities with replacement and keep each selected cluster's observed rows together. Recompute the declared statistic on that resample. For paired model comparisons, resample both models' outcomes on the same selected clusters. If the estimand gives each cluster equal weight, average cluster means. If it gives each request equal weight, recompute the request-weighted numerator and denominator; unequal cluster sizes make these different statistics. The resampling design should approximate the original sampling process rather than quietly change the target weights.

A multi-stage sample may call for a justified hierarchical bootstrap that also resamples lower-level units, while a block or other dependence-aware method may be needed for time series. There is no instruction to resample every level automatically. Cluster bootstrap intervals are generally approximations whose adequacy depends on the statistic, population, cluster count, and sampling design. A handful of highly unequal clusters can make the approximation poor. Report the number and size distribution of independent clusters, the exact resampling procedure, the interval construction, and sensitivity to plausible alternative grouping choices.

Increasing the number of bootstrap resamples reduces Monte Carlo noise in the bootstrap calculation, but it does not increase the number of independent source clusters. A resample cannot add a missing language, undo contamination, or make an old workload representative of a new population. Validate implementation with simple examples where the sampling variance is known, and retain grouping identities in the artifact. Report unresolved dependence or low support directly rather than hide it behind many decimal places from a large resampling run.

The sampling lab offers a precise counterexample to row counting, not a bootstrap implementation. For each separate view, Reset, set both p=0.5 and n to 25 or 100, predict, then inspect. Independent sample-mean variances are 0.01 and 0.0025. In the worked example, one hundred perfectly repeated rows contain only twenty-five independent Bernoulli outcomes, so the twenty-five-trial variance is the relevant toy result. The display has no clusters or resampling controls. Its role is to contrast those two explicit assumptions, while actual cluster inference requires the retained grouping and justified procedure.