Clip the protected contribution before aggregating gradients

For a nonzero per-example gradient g, multiply it by the smaller of one and C divided by its Euclidean norm. Leave a zero gradient unchanged. Small gradients are preserved; larger gradients retain direction while their norm becomes C. If parameters span several tensors, flat clipping uses the norm of their concatenated per-example vector. Clipping every parameter tensor independently at C instead permits a larger overall norm and therefore needs a different sensitivity or per-layer analysis.

The gradient must correspond to one protected unit’s contribution at the fixed current model state. A separable sum of per-example losses has this form. Training-time BatchNorm and other cross-example operations can make changing one record alter other records’ gradients, defeating the simple “only one bounded term changes” proof. Validate model and loss compatibility rather than assuming the presence of a per-sample tensor proves the privacy argument. Per-user clipping requires constructing the user’s full contribution, not merely renaming a row gradient.

Averaging unclipped gradients first allows a large contribution to affect the aggregate before any record-specific bound is applied. Clipping that average yields a bounded batch vector, but its privacy unit and sensitivity are different from a sum of independently clipped contributions. Even ordinary cancellation can make the two numerical results diverge. A batch-clipped mechanism can be analyzed on its own terms; it simply cannot inherit the per-example DP-SGD proof without showing the required equivalence or deriving a new bound.

Loss reduction introduces another scale. Frameworks often report a mean loss, while the clipping rule is expressed for each unaveraged example loss. Libraries may compensate for that reduction when constructing per-sample gradients. Do not divide by the batch size again without checking the implementation. Opacus v1.5.4 distinguishes per-sample clipping, summation, noise addition and later expected-batch scaling. Microbatching for memory should preserve the declared logical mechanism, including which units are clipped and how many independently noised updates occur.

Test zero, below-threshold, exactly-at-threshold and above-threshold gradients, including parameters split across tensors. Compare a trusted explicit per-example calculation with any efficient clipping implementation. Check that repeated backward passes and accumulated microbatches do not reuse stale gradients or silently add several copies. Keep model-state updates such as running statistics in scope. Clipping bounds influence but supplies no statistical privacy by itself; an uncloaked clipped vector can still reveal a participant’s data.

For the geometry, Reset the vectors lab and set both angle 0° and magnitude 3. Predict (3,0), then clip on paper at C=2 to obtain (2,0). Separately Reset and set both angle 90° and magnitude 3; predict (0,3), clipped on paper to (0,2). The instrument itself does not clip vectors or compute gradients. It illustrates that an Euclidean norm threshold limits length independently of direction and does not add noise or calculate a privacy budget.