Accelerators and scheduling

An accelerator request includes device type and count, memory needs, CPU, host memory, storage throughput, and communication expectations. Some are consumable quantities; others are compatibility constraints. A binary built for one device capability may not run on another. Kubernetes exposes accelerators through device plugins or applicable allocation mechanisms. For device-plugin GPU resources, requests and limits must match when both are set; an omitted request defaults to the limit. The scheduler does not infer a model's memory peak.

Requests guide placement and reservations; utilization is what the program actually consumes over time. Under-requesting CPU can starve data loading even when the accelerator is reserved. Over-requesting whole devices creates waste when safe partitioning would suffice, but sharing introduces memory, bandwidth, and isolation concerns. Do not assume a reported percentage of device activity is equivalent to useful arithmetic efficiency. Collect application throughput and stall information to distinguish a well-used device from a busy wait or repeated failed work.

Data-parallel jobs synchronize gradients, while tensor-parallel layers may communicate several times during one forward pass. A placement that is acceptable for one can be poor for the other. Model fast interconnect groups and cross-rack links explicitly. Scheduling all necessary workers together avoids reserving three devices indefinitely while waiting for a fourth. Gang scheduling is a policy and implementation choice; simply creating several independent pods does not prove simultaneous start or topology-aware placement.

Fragmentation occurs when free resources cannot satisfy the shape of waiting requests. Eight free devices spread across four nodes are different from eight free devices within one fast group. Repacking can improve fit but interrupts running work, so evaluate checkpoint age, restart time, and queue fairness before preemption. Reserve capacity for latency-critical inference and for recovering essential services. Allowing training to occupy every available slot can make a small production failure impossible to repair quickly.

Fairness requires a declared unit: devices, device-seconds, a weighted share, or a dominant resource share. Equal job counts are misleading when one job requests one accelerator and another requests sixty-four. Priorities, quotas, queue aging, and preemption rules should be visible to users. Backfilling a small job into an otherwise idle gap can improve utilization, provided its expected duration and interruption policy do not indefinitely delay the large job that created the gap.

Track waiting time by workload class, useful accelerator-hours, completion time, failed allocations, and work lost to preemption. A scheduler can improve reported occupancy by admitting jobs that immediately contend for the same network. Compare against workload-level objectives instead. The following allocation efficiency is an accounting metric: it penalizes reserved idle time, but it does not establish fairness, numerical efficiency, or successful completion. Use it as one diagnostic alongside those outcomes.