Preemption and victim selection

Identify which workloads may be preempted under the configured priority, queue, and protection rules. Then evaluate whether removing a candidate set releases the required device types, host resources, and topology. Four GPUs spread across incompatible domains may not help a four-GPU single-host request. Killing a low-priority job that cannot make the incoming job feasible creates disruption without achieving the purpose. Account for resources that remain owned during graceful termination or controller cleanup rather than assuming they are immediately reusable.

A victim’s termination can be delayed, fail, or be superseded by another scheduling decision. The selected incoming job is not automatically guaranteed to bind once a victim is nominated. Keep event traces linking the preemptor, victims, resource release, and eventual admission outcome. Bound repeated preemption loops so one job does not continually evict another without either making useful progress. These problems are policy and lifecycle issues as well as combinatorial selection problems.

Training interruption cost includes checkpoint or suspension overhead, restore time, and work that must be replayed from a valid saved state. A fresh successful checkpoint can reduce lost progress, but it consumes time, bandwidth, and storage and may contend with other victims. Validate what state a checkpoint contains, including optimizer and data-position state when needed. Calling a file a checkpoint does not prove that restarting reproduces the intended training computation or avoids duplicated external effects.

Availability protection also has a scope. Kubernetes PodDisruptionBudgets guide voluntary disruption mechanisms, and scheduler preemption attempts to avoid violations but does not provide an unconditional PDB guarantee. They do not prevent every involuntary failure or replace application-level capacity planning. A training gang may need protection for its useful membership rather than a simplistic replica count. Check the platform’s documented behavior and the workload controller’s recovery rules before treating a budget as an absolute prohibition on interruption.

Compare the incoming workload’s urgency and expected benefit with the full victim cost. A policy can include maximum interruptions per job, protected service classes, or checkpoint-age thresholds, but each needs observable enforcement. Record useful progress lost and time to successful resume. Reporting only that higher-priority work started can hide a large training-efficiency regression. Evaluate whether another action, such as waiting briefly for a completion or using a different compatible flavor, achieves the goal with less disruption.

Use the reliability lab for a count-only protection illustration. Reset, set replica count to 5 and unavailable count to 1, predict available replicas, then Step. Compare a new Reset with both replica count at 5 and unavailable count at 2. On paper impose an application minimum of four available replicas. The second case still has a majority of three but fails that application minimum. The instrument implements only majority counting, not PDB enforcement, preemption, checkpointing, or application capacity.