Topology and affinity constraints

A topology hierarchy can include device, NUMA domain, host, rack, and zone. A workload may require all workers in one domain or a particular distribution across several domains. Count compatible available resources within those boundaries. Cluster-wide totals are necessary in many cases but insufficient: four free GPUs split two per rack cannot satisfy a four-GPU single-rack requirement. Storage and network-device attachment can introduce another placement restriction even when accelerator and CPU counts fit.

Hard affinity or selection rules exclude placements; soft preferences rank otherwise feasible candidates. The difference matters during shortages. A preferred rack may be relaxable by policy, while a required device capability cannot be ignored without changing the workload contract. Keep labels authoritative and inspect which component maintains them. An incorrectly labeled node can satisfy the scheduler’s symbolic test while violating the actual hardware requirement. Explain pending decisions using both the requested domain and the inventory the scheduler observed.

Colocating tightly communicating training workers can reduce cross-domain traffic, but the actual improvement depends on the collective pattern, interconnect, and competing workloads. Spreading inference replicas can reduce exposure to one node or zone failure while increasing communication or data-access cost. A topology-spread rule expresses distribution, not a universal availability proof. Shared dependencies can still fail across every chosen domain, and a healthy replica may lack enough capacity to absorb displaced traffic.

Co-scheduled jobs can interfere through links, host memory bandwidth, storage, and device sharing. Model communication demand separately from GPU count when it is material. Measure placement-sensitive iteration time or request latency under representative contention before hardening a preference into a requirement. A rigid constraint can strand substantial capacity, while a soft rule may allow performance variance. Document which degradation is acceptable and whether the application can adapt its parallelism when the ideal topology is unavailable.

Exercise missing labels, one unavailable interconnect island, local storage restricted to a host, and affinity rules that contradict one another. A placement failure can be a correct rejection of inconsistent requirements rather than a scheduler bug. Verify replacement behavior when a replica fails: surviving capacity must satisfy the same relevant domain rules, or the recovery policy must explicitly relax them. Track how long jobs wait for preferred topology and whether that preference improves the intended workload metric enough to justify the delay.

The reliability lab deliberately omits topology. Reset, set replica count to 5 and unavailable count to 2, predict the remaining count, then Step. Repeat from Reset with both replica count at 5 and unavailable count at 3. The majority indicator reflects counts only. Describe two possible placements of the same available replicas across zones and explain why their correlated-failure exposure can differ. The instrument draws neither zones nor links and cannot validate affinity, network performance, or a failure-domain guarantee.