Accelerator resource fragmentation

Per-device memory is a hard boundary for a workload that requires one device unless the application supports an appropriate distribution or offload mechanism. Adding the free bytes on unrelated GPUs does not create one larger allocation. Partitioned devices introduce supported profiles and placement geometry that depend on the hardware and driver stack. A total-memory test can pass while no compatible partition profile exists. Treat device type, profile, and topology as part of the request rather than assuming all fractional capacity is freely divisible.

Distinguish scheduler-level fragmentation from fragmentation inside a process’s memory allocator. The scheduler may see whole devices or resource claims, while a runtime sees allocations and reusable blocks within its assigned memory. A workload can fail from activation or temporary-workspace peaks even when model weights fit. Conversely, releasing a process’s unused cache may not return a scheduler-visible device allocation. Diagnose which layer owns the unusable capacity before trying a remedy that changes a different layer’s accounting.

Time-sharing, spatial partitioning, and colocating multiple processes have different memory, compute, and isolation semantics. An advertised number of accelerator units does not by itself guarantee a particular memory fraction or throughput. Check the device-allocation API and runtime contract, including which resource quantities are enforced. Host CPU, input-pipeline throughput, and network communication can also dominate observed device activity. Avoid labeling every low compute-utilization sample as reclaimable accelerator capacity without inspecting memory residency and synchronization.

Gandiva explores profiling and runtime-aware scheduling for deep learning, including migration and sharing decisions. Its useful lesson here is to measure job behavior and movement costs rather than infer them from requested GPU count alone. Workloads can change phase, and a profile from one batch size or input length may not apply to another. Sharing training with latency-sensitive inference also requires tail-latency and interference measurements. A placement that raises average device activity can still reduce useful work or violate service objectives.

Consolidation can create larger free device groups, but it may require stopping, checkpointing, migrating, or restarting workloads. Confirm that the resulting shape satisfies the waiting jobs and that the cost is worthwhile over the expected remaining run. Track stranded capacity by requested shape and compatible device class. A global free-memory percentage hides whether the pending queue needs one large device, a linked group, or many small independent instances. Changing the queue’s workload mix can alter fragmentation without changing installed capacity.

The vector lab isolates aggregate versus per-device free memory. Reset, set angle to 45 degrees and magnitude to 1, predict both components and their sum, then Step. Interpret the components as free memory on two devices in normalized units and require one full unit on a single device. For a fresh comparison, Reset and explicitly set angle to 0 degrees and magnitude to 1, predict the maximum component, then Step. This draws two numeric projections only, not GPU memory, partitions, or a consolidation algorithm.