Gang scheduling and co-scheduling

Define the minimum useful group: all workers, a minimum elastic membership, or a set of different roles such as trainers and parameter services. Its combined resource requirement includes those roles and their topology constraints. Gang scheduling checks whether enough of that group can be admitted together. A minimum-member threshold is a policy, not necessarily every replica ever requested by an elastic job. Confirm that the application can actually start and operate with the admitted membership before treating the threshold as useful progress.

Admission and physical placement are related but distinct stages in some platforms. Reserving quota does not by itself create nodes or guarantee that each pod binds successfully. After admission, workers still initialize devices, load data, discover peers, and join their communication group. A slow or failed member can delay the first collective. Observe time to useful training progress separately from time to quota reservation, pod creation, and all-workers-ready. These boundaries reveal whether the bottleneck is policy, placement, or application startup.

Partial allocation can create a resource deadlock when each job holds some devices while waiting for the rest and none releases them. Gang admission or a bounded reservation-and-rollback protocol can prevent that particular hold-and-wait pattern. Define what happens when a reservation times out or one pod fails to bind. Release stale allocations through the actual controller lifecycle rather than assuming a timeout instantly frees every device. Another job’s readiness check must not rely on resources still owned by an unresolved reservation.

A large gang can also starve if a stream of small jobs continually consumes each newly free device. Reserving future capacity, aging, backfilling, or controlled preemption can help, with different utilization and disruption costs. A scheduler that starts many small jobs quickly can still be unfair to a feasible large job. Conversely, waiting for a huge gang whose requirements exceed the entire eligible cluster can block useful work forever unless infeasibility is recognized and the queue policy handles it.

Co-scheduling can also refer to giving communicating tasks compatible execution opportunities after placement. Time-slicing one worker independently can delay the rest of a synchronous collective. Checkpointing, elasticity, and migration require runtime support and can change job semantics or performance. Gandiva and Tiresias examine different scheduling and placement choices for deep learning; their workload-specific designs do not establish that every gang policy or sharing configuration improves every model. Reproduce the assumptions before adopting a scheduling technique.

Use the queue lab to reason only about admission pressure. Reset, set arrival rate to 4 and service rate to 3, predict backlog after one second, then Step twice. Compare a new run after Reset with both arrival rate at 2 and service rate at 3, predicting before two Steps. Each run begins with four equal abstract jobs. The fluid model has no job sizes, gangs, reservations, or workers and cannot demonstrate resource deadlock; draw the partial-allocation example separately and identify who owns each device.