Repeated sparse indices accumulate before nonlinear updates

For a table E with one row per identifier, a gather output y_j takes row E at index i_j. A reverse cotangent for each occurrence contributes to that same table row. If the row appears twice, both vectors add. Rows never selected receive zero under this simple gather-only model. This is a sparse linear map with respect to the table values at fixed indices, but the integer indices are not ordinary differentiable coordinates. A program that wants gradients for selecting an index needs a different explicitly defined relaxation or estimator.

A naive reverse implementation that assigns rather than adds can lose earlier occurrences. Parallel scatter-add implementations must also respect accumulation semantics, even though floating-point summation order can affect final rounding. PyTorch 2.9 documents operation-specific shape and backward restrictions and possible device nondeterminism for scatter operations. Those implementation details are separate from the exact real-valued adjoint. Test repeated indices and nontrivial vector cotangents; a dataset with unique indices can conceal a fundamentally wrong gather derivative.

PyTorch 2.9's sparse COO documentation permits uncoalesced duplicate coordinates whose values contribute by addition. Coalescing combines duplicates and orders the representation under its contract. The mathematical tensor is the sum at each coordinate, not whichever duplicate appears last. A nonlinear operation on separate fragments can differ from the operation on their sum: squaring values one and two separately gives a represented total five, while squaring the combined value three gives nine. An optimizer or custom kernel must honor the layout's semantics instead of assuming every stored coordinate is unique.

Sparse gradient support depends on the operation, layout, device, and optimizer. Embedding can request sparse gradients under its documented options, but not every optimizer accepts that representation or implements every combination. Do not infer support from the fact that a sparse tensor can be constructed. A zero stored value can also remain structurally present after cancellation; numerical zero and absent storage are different concepts. Verify supported transformations, coalescing requirements, accumulation, and optimizer state rather than declaring that sparse automatically means less memory or fewer arithmetic operations.

Compare a sparse derivative with a small dense reference after converting both to the same mathematical layout. Include repeated indices, canceled contributions, unused rows, different feature widths, and several accumulation steps. Measure index traffic and coalescing work alongside value payload, especially when the selected set is not very sparse. A representation that saves gradient storage can still require dense parameter or optimizer state. Keep the promised memory category explicit and test whether batching changes duplicate frequency or backend behavior.

The vector lab illustrates componentwise addition on paper rather than sparse storage. Reset, explicitly set BOTH angle 0 degrees and magnitude one, inspect vector (1,0), then Play or Step. Reset and re-set BOTH angle 90 degrees and magnitude two for vector (0,2). Their paper sum is (1,2), not either last assignment. Edits retain t; playback traces one selected vector and never stores both, scatters rows, or coalesces duplicates. Use the example only to inspect accumulation components; the worked gather trace determines the real indexed gradient.