Architecture evaluation and ablation

A fixed-training-recipe comparison asks what happens when one component changes under an unchanged optimizer and schedule. A tuned-system comparison asks how well each alternative performs after a declared tuning budget. These are different estimands. A component may work poorly under a recipe designed for another architecture yet produce a strong tuned system. Report which question is intended, and allocate comparable search resources when making claims about best achievable performance.

Choose primary metrics and evaluation populations before inspecting final results. Accuracy, calibration, memory, latency, and energy are different outcomes; improving one does not establish dominance in the others. Fix data splits and remove overlap at the relevant identity level. Validation data can guide architecture selection, but repeated selection against a final test set converts it into another development set. Preserve a genuinely held-out assessment or describe the resulting limitation.

Removing a residual path, normalization layer, or router loss changes more than a label. It may alter parameter count, activation scale, compute, and the learning rate that works. Specify the intervention exactly and test the behavior the mechanism predicts, such as gradient scale across depth or overflow frequency. An ablation that hurts one metric shows that a component helped in that configuration; it does not prove the proposed explanation is the only possible cause.

Components can interact. If two changes only help together, testing each against a baseline can miss the benefit. A small factorial experiment evaluates the baseline, each change alone, and both together. Keep failed and divergent runs in the record under a predefined handling policy. Quietly discarding them biases a stability comparison. Use learning curves and intermediate diagnostics to distinguish faster optimization, regularization, and additional capacity rather than attributing every gain to better representations.

Repeat experiments to characterize training variability and use evaluation resampling that respects dependence, such as patient or document groups when examples are correlated. A confidence interval over test examples alone does not capture variability from retraining. A standard error over a handful of seeds relies on strong sampling assumptions and can be unstable. Report run-level values and the interval method. Multiple architecture searches also create selection effects that a single pairwise p-value does not erase.

Measure latency after appropriate warmup, with synchronization when the accelerator executes asynchronously. State batch size, sequence length, precision, device, software, and whether preprocessing or data transfers are included. Peak allocated memory and total device memory are different measurements. Present quality-resource tradeoffs rather than inventing one universal winner. Publish configurations and enough artifacts to rerun the comparison; a concise result becomes stronger when its conditions and limits are visible.