Balance partitions using the communication the model actually performs

For an undirected graph represented by two message directions per edge, one cut edge can require two cross-partition messages per layer under a direct edge-message strategy. Other implementations fetch a source feature once and reuse it for several destinations, replicate boundary features or cache them across batches. Their traffic can depend more on unique remote source rows than cut edges. A partition score should state whether it approximates message bytes, feature fetches, update traffic or simply structural cuts.

Balancing the number of owned nodes helps distribute node-table memory but can leave one machine with many more edges or frequently sampled hubs. Feature dimensions can also differ by node type. Include the memory for optimizer state, adjacency, boundary replicas and batch activations when defining capacity constraints. DistDGL combines graph partitioning and distributed data access to support large training workloads, but a partitioning paper’s reported scaling does not guarantee that a new graph, sampler or machine interconnect has the same bottleneck.

If one remote source feeds many local destinations, pulling its vector once can be cheaper than sending a separate feature-sized message for every edge. Conversely, an attention or edge-conditioned message can depend on destination or edge attributes, making a shared feature fetch only part of the work. Boundary replication reduces read traffic but creates update and version-management obligations. Measure unique fetched IDs and transferred bytes per batch alongside structural cut counts. High cache hit rates can mask stale replicas unless their validation policy is also checked.

A partition map and node-ID remapping must travel together. Repartitioning can change ownership while requests are in flight, so an epoch or routing fence is needed in a design that permits live migration. Decide where gradients for moved nodes are applied and when the old owner stops accepting writes. These requirements are separate from finding a good static cut. A partition assignment that improves read locality but causes dropped or duplicated updates is not a correct optimization of the training system.

Instrument the slowest worker, remote request fanout, cross-partition bytes and waiting time rather than reporting cluster averages alone. Sampling can create transient hotspots even when full-graph degrees appear balanced. Bound concurrent requests and batches so one partition’s slowdown does not cause every worker to retain unbounded feature buffers. Communication and compute can overlap, but the overlap depends on dependencies and available resources; adding their isolated peak rates does not produce an end-to-end throughput prediction.

For a simple overload analogy, Reset the queue lab and set both arrival λ=4 and service μ=3. Predict backlog five after two Steps, since elapsed time is one second and the initial backlog is four. Separately Reset and set both λ=2 and μ=3; predict backlog three at the same time. The exact fluid model is max(0,4+(λ−μ)t). It has no partitions, packets or graph sampling and cannot calculate a real worker’s waiting-time distribution.