Diffusion evaluation: freeze the data and sampling protocol before comparing models

Deduplicate and group near-related examples before assigning training, validation, and test splits. Frames from one video, alternate crops of one image, or repeated captions can leak information across an image-level random split. Record source-group IDs, content digests, filtering, caption transformations, crop and resize rules, and dataset revisions. Split design should match the intended generalization question, such as new sources or new compositions. A split with different content difficulty is another experimental condition rather than a neutral bookkeeping change.

Keep model selection and hyperparameter tuning on the declared validation protocol, then evaluate the final comparison on a held-out test set. Preserve the autoencoder, tokenizer, condition encoder, latent scaling, and augmentation configuration. A cached latent or condition representation can accidentally carry preprocessing from another split or revision. For denoising validation, use a fixed reference target, timestep distribution, and noise policy so a changed training proposal does not redefine the reported measure. Compare raw parameter and EMA artifacts explicitly rather than silently choosing whichever looks better.

FID compares Gaussian moment approximations in a chosen feature representation. It is affected by feature extractor, preprocessing, image resizing, sample count, and the reference dataset. Clean-FID studies evaluation discrepancies caused by implementation and resizing choices. Do not compare numbers produced by undocumented pipelines as if they were identical measurements. A finite-sample score has estimation error and bias, and a lower feature distance does not prove caption alignment, absence of memorization, diversity on every data slice, or fidelity to an individual source.

Report complementary measures and human review relevant to the task, including conditional adherence, diversity, reconstruction quality when applicable, and representative failures. Freeze prompt sets, random inputs, sampler algorithm, step count, guidance formula and scale, precision, and output postprocessing. Selecting attractive samples or changing guidance only for one model confounds the comparison. A sampling improvement and a training improvement can both be useful, but distinguish them with a factorial or otherwise controlled experiment rather than attributing the full result to the loss function.

Use repeated runs or an appropriate uncertainty analysis for the measured statistic, keeping dependence and finite sample size in view. Paired seeds can help isolate a change, but generated outputs remain dependent on checkpoint and sampler choices. Report failed runs, skipped updates, compute budget, and the checkpoint-selection rule. Slice evaluation by rare content, resolution, and conditioning complexity. When several weights or guidance scales are swept, reserve fresh validation or account for the selection procedure rather than reporting only the best noisy result.

The sampling lab models a binary evaluation rate under independent identical cases. Reset, set p=0.80 and n=20, predict mean sample proportion 0.80 and variance 0.008, then Play or Step. Separately Reset, set p=0.80 and n=100, predict the same mean and variance 0.0016, then Play or Step. Playback tours exact outcome probabilities rather than generating images. It has no FID estimator, paired model scores, or human judgment; use it to see why a finite test count has uncertainty under assumptions, then analyze the actual evaluation statistic.