Evaluate utility without opening an unaccounted private-data channel

Running an already accounted DP model on public evaluation examples is postprocessing with public information. Querying protected holdout labels is different, even when only one accuracy number is reported. If users appear across training and holdout data, disjoint row splits do not establish disjoint protected populations for user-level accounting. Define whether metrics are per record or per bounded user contribution, and keep the privacy unit consistent with the release claim. Repeatedly inspecting private slices or choosing the best checkpoint from raw metrics is another selection path.

Keep a public development set for ordinary iteration when available and reserve private access for a declared process. The public set may be unrepresentative, so good public performance does not replace deployment-relevant private evidence. Conversely, “private” cannot mean merely hidden from the test author while raw labels are freely queried by the selection system. Version the evaluation data, model artifact, candidate protocol, decoding rules and metric randomizer so a release report describes one reproducible workflow rather than a changing mixture of tests.

For this Laplace mechanism, the probability that added noise magnitude exceeds radius r is exp(-r/b). Choosing r=b log(1/beta) gives a noise-only coverage probability 1−beta around the released value for the fixed underlying metric. This is a statement about the randomizer, not a confidence interval for future population accuracy. The dataset itself may be a finite, biased or correlated sample, and stochastic generation can introduce additional variation. Those uncertainties need their own design and cannot be folded into the privacy parameter delta.

A beta used in an error-probability statement is not a DP delta. The illustrated Laplace query has delta zero even when a noise interval has only 95% coverage. Clipping a released metric to [0,1] is postprocessing, but interval construction should account for that reporting convention. Do not release an unclipped raw accuracy alongside its privatized version as a convenience; the raw value remains a separate private-data disclosure. Likewise, error bars computed from unsanitized private statistics can themselves need protection.

Publish the protected unit, composed training and evaluation guarantees, fixed denominator assumptions, metric transformation and the uncertainty method. Include important slices only through a valid evaluation and composition plan; many slice queries are not one query merely because they appear in one chart. Evaluate operational metrics such as latency using data paths whose public or private status is explicit too. A model can have lower utility because of clipping, optimization or limited data coverage as well as added noise, so avoid assigning every regression to one parameter.

The sampling lab isolates ordinary iid accuracy variation. Reset and set both p=0.8 and n=100; predict mean 0.8 and variance 0.0016. Separately Reset and set both p=0.8 and n=25; predict mean 0.8 and variance 0.0064. Play highlights exact binomial outcomes instead of running a benchmark. This model contains no private metric query or Laplace noise and cannot compute a privacy guarantee. It helps distinguish finite-sample variability from the separate randomizer interval derived on paper.