Own, cache and update distributed embedding rows explicitly

For N rows of dimension d stored at b bytes per element, raw embedding payload is Ndb bytes. Optimizers can add full-sized state arrays, and a gradient buffer, master copy or checkpoint staging can increase memory further. Sparse updates mean only touched rows are updated; they do not necessarily mean the entire table or all optimizer state can be omitted from resident or backing storage. State precisely which arrays and data types a capacity estimate includes.

DGL 2.3.1 DistEmbedding exposes trainable sharded embeddings and distributed sparse optimizers. Its documented update path differs from using an ordinary framework optimizer on a local dense parameter table. More generally, a feature-based GNN can also use a learned ID embedding as one input, mixing transductive and inductive behavior. An unseen node may then lack its own trained row even if the rest of the encoder accepts new features. Define initialization and fallback behavior instead of assuming distributed storage creates inductive generalization.

Map each global row to an owner using a versioned partition policy. A worker fetches needed rows, computes contributions and routes sparse gradients to the designated update path. If a row appears several times, combine compatible contributions without losing multiplicity. Concurrent workers can produce updates from different read versions, so the system must choose synchronous, asynchronous or bounded-staleness semantics. The ownership map alone does not make a distributed training step equivalent to a sequential optimizer step.

A read cache can store an immutable serving snapshot cheaply, but caching mutable training parameters can change the algorithm unless invalidation or allowed staleness is explicit. Include model or table epoch, node namespace and row identity in the interpretation of a cached vector. During ownership migration, fence updates so an old owner cannot keep applying writes after a new owner takes over. Checkpoints should capture embedding shards, optimizer state and routing metadata under one documented recovery boundary.

Measure remote rows, cache hits by accepted version, bytes, update contention and stale-read rejection separately. A high hit rate can be harmful if the cached table epoch is wrong for the model weights. Serving precomputed graph representations has an additional dependency problem: even an unchanged node ID can need a new vector when neighbors or features change. Keep generated-representation freshness distinct from raw parameter-row caching. A rollback restores a compatible table and encoder snapshot rather than mixing whichever shards happen to be newest.

The cache lab isolates capacity-driven reuse without version semantics. Reset and set both C=3 and s=4; predict sixteen initial misses because the four-address cycle exceeds capacity. Separately Reset and set both C=4 and s=4; predict four misses and twelve hits. Requests are (i·s) mod 16 and N=min(4096,16+floor(2t)); Step adds one request until the cap. The toy stores only abstract addresses, so its hits provide no evidence about embedding freshness, optimizer ordering or a distributed owner’s consistency.