Quality and speed frontier measurement

Choose quality dimensions relevant to the application, such as prompt adherence, artifact rate, diversity, composition, and human preference. A scalar metric can summarize one dimension while hiding another. For latency, name the boundary: denoiser service time, full request service, queued end-to-end response, or tail completion under load. Include text encoding, scheduler work, decode, output processing, and cold-start behavior where they belong. A reported step count alone says little about those costs when methods use different branches, internal evaluations, or model sizes.

A configuration is empirically dominated if another is no slower and no worse on the declared quality objective, with a strict improvement in at least one dimension. That conclusion is scoped to measurements and uncertainty. A point dominated on mean latency can still differ in tail latency or memory, and a single quality score may omit an application constraint. Report these additional dimensions rather than presenting a two-axis plot as proof of universal superiority. Preserve rejected or failed runs under an explicit counting policy instead of quietly dropping expensive artifacts.

Use a fixed prompt population and paired initial noise or seeds where meaningful so nuisance variation does not obscure a configuration difference. Keep model and evaluation revisions, resizing, preprocessing, precision, and output filtering fixed or record their intended changes. Repeated seeds for one prompt are not independent prompt samples. If the target population is prompts, estimate uncertainty at the appropriate prompt or cluster level. Pairing does not guarantee identical noise consumption across different stochastic samplers, so record actual initialization and random-stream policies.

For an approximately independent set of paired scalar score differences, standard error can be estimated from their sample standard deviation and pair count. This does not make a nonlinear dataset metric a per-image score. FID depends on feature-distribution estimates and has finite-sample bias that can vary by generator, as the cited primary analysis explains. Keep feature extractor, preprocessing, reference population, and sample count explicit, and use a suitable uncertainty or bias-analysis procedure. A narrow interval on an unrelated scalar proxy does not certify FID or human preference.

Sweep a justified set of complete configurations under realistic batch sizes and cancellation mix. Record actual neural evaluations, branch batch shapes, peak memory, throughput, latency distributions, and quality stratified by prompt class or resolution. Compare candidate changes with paired baselines and preserve the generation manifest so others can replay the measurement. A favorable point on one device or dataset is useful evidence at that operating region, not a guarantee for a new model, strong guidance regime, or different output size.

The sampling instrument illustrates how iid Bernoulli uncertainty shrinks with more observations. Reset, set both p=0.6 and n=20, predict standard deviation of the sample mean about 0.10954, then Play or Step. Separately Reset, set both p=0.6 and n=80, predict about 0.05477, then Play or Step. The exact binomial masses describe a stipulated binary preference probability, not actual collected ratings. The normal comparison has its documented limits, and the lab contains no image generator, FID estimator, latency experiment, or prompt-cluster model.