Embedding serving moves indexed bytes across a placement and cache hierarchy

For a plain table with N rows, dimension d, and b bytes per scalar, raw value storage is Ndb bytes. This omits row metadata, alignment, scales for quantization, replication, allocator overhead, and training optimizer state. A serving deployment need not carry the same extra state as training. A lookup of one full row moves db value bytes in an ideal accounting model, but hardware can fetch larger units or access compressed formats. Record whether the estimate counts logical values, memory reads, network payload, or observed physical traffic.

Count indices per request or batch before and after any valid deduplication. Repeated IDs may permit row reuse, while different feature tables with the same numeric ID remain distinct identities. An embedding bag's pooled output can be much smaller than all rows it reads. Cache hit rate must specify its denominator: lookup count, unique rows, bytes, or complete requests. A request touching many shards can still suffer a remote miss even with a high per-lookup hit rate, especially when hotness and row sizes vary.

Table-wise sharding places whole tables on selected workers and can minimize routing for a table that fits, but large or hot tables can dominate one placement. Row-wise sharding spreads IDs from a table across workers, increasing capacity flexibility while requiring routing and aggregation for multi-ID requests. Column-wise sharding splits each vector's dimensions, so one logical row may require several workers. Hybrid strategies trade memory balance, communication, and computation differently. An automated planner's recommendation still depends on its workload and topology assumptions rather than guaranteeing an optimal production tail.

Use observed frequency, bag lengths, row widths, topology, and the required aggregation path to evaluate a placement. Uniform row count does not imply balanced bytes or service demand. Replicating hot rows can reduce remote work but introduces memory and update-consistency costs. If a request waits for several shard responses, its critical path follows the last required response plus aggregation, not the mean shard latency. Version embedding values and cache keys consistently with the model and feature mapping so a fast stale row does not silently change the scoring function.

Benchmark realistic ID and bag-length distributions, including cold starts, new items, and hot-key skew. Preserve model quality when compressing rows or changing pooling and distinguish storage savings from extra decode work. Measure complete-request latency alongside bytes and per-table hit rates. Rebuilds and refreshes can change cache temperature, so compare a stated warmup regime rather than one warm candidate with a cold baseline. Current TorchRec documentation describes useful sharding and collection primitives, but a serving cache, update policy, and failure behavior remain explicit system design choices.

The cache lab provides a tiny deterministic reuse example. Reset, set both C=4 and stride s=4, predict four misses and twelve hits in the initial sixteen requests, then inspect or Play. Reset and set both C=2 and s=4 for sixteen initial misses. It begins empty and accesses (i×s) mod 16 with fully associative LRU, adds two requests per playback second, and caps at4096. Each entry has no embedding width or version. The model demonstrates reuse and eviction, not a distributed embedding benchmark, production hit distribution, or byte estimate.