Scheduler simulation and trace replay

A useful trace includes job submission and eligibility times, requested resource vectors, constraints, priority or tenant identity, admission and placement decisions, start and finish boundaries, and interrupted or failed attempts. Distinguish useful runtime from elapsed allocation time. Missing rejected or cancelled jobs changes the workload population. An experiment that replays only jobs that successfully completed can systematically exclude the hardest demand. Protect sensitive identifiers through an appropriate anonymization process that preserves dependencies and resource shapes needed for the study.

Trace replay must define the arrival process. Keeping recorded submission times assumes arrivals stay fixed when policy changes. A closed user workflow may submit its next experiment only after the previous one finishes, so changed completion times can change later arrivals. Neither fixed-arrival nor feedback replay is universally right; each answers a stated question. Preserve dependency graphs where later jobs require earlier outputs. Starting a dependent training run before its data exists produces a flattering simulation of an impossible workflow.

Reusing recorded duration assumes the new placement and sharing policy do not change the workload’s effective execution time. That assumption is often false for distributed training: topology, accelerator type, worker count, interference, and preemption can alter iteration speed. Calibrate service models with representative measurements and quantify residual error. Include checkpoint, restore, migration, scheduling-cycle, and data-loading costs when the candidate policy invokes them. A simulator that treats preemption as instantaneous and free can select a policy that performs poorly once those costs are paid.

Gandiva and Tiresias motivate workload-aware evaluation, but their measured results belong to their own systems, traces, and hardware assumptions. Use them to identify mechanisms and evaluation questions rather than copy a headline improvement into your cluster forecast. Validate the simulator against held-out runs under an existing policy before trusting new-policy predictions. Then vary uncertain parameters, such as runtime error and checkpoint bandwidth, to see whether the policy ranking survives plausible model changes. A narrow tuned point is weak evidence of robustness.

Keep admission, fairness weights, hardware capacity, and workload constraints aligned between policies unless one is an intentional experimental variable. Report tail and per-tenant wait, large-gang progress, useful throughput, preemption cost, infeasible jobs, and allocation efficiency alongside average JCT. Include multiple seeds where the policy is randomized and preserve configurations for reproduction. A perfect offline oracle with future runtimes is a useful bound only if clearly labeled; it is not a deployable online baseline with equal information.

The queue lab is a simple falsifiable prediction exercise, not a scheduler simulator. Reset, set arrival rate to 4 and service rate to 3, predict the backlog at one second, then Step twice. Compare a fresh Reset with both arrival rate at 2 and service rate at 3 before two Steps. The initial four jobs and fixed rates fully specify these fluid values. Explain which trace features it omits—job shapes, tenants, topology, failures, and overhead—before transferring any intuition about backlog to a real cluster.