Queue depth buys overlap until a bottleneck buys waiting

A task can wait in application code, an asynchronous runtime, a filesystem path, software block queues, a hardware submission queue, or the device's internal machinery. Those are not interchangeable counters. Linux's multi-queue block layer organizes software submission and hardware-dispatch queues to support scalable I/O, while scheduling and request merging can change what reaches a device. Application operations can split into several block requests or combine with neighboring work. Report the boundary, request definition, and time interval before comparing queue depth across tools.

Scheduling policies trade throughput, fairness, latency, and overhead for particular devices and workloads. A policy useful for reducing mechanical seeks need not be best for an SSD or a mixed workload. Multiple hardware queues can reduce host lock contention but do not make every request independent: shared bandwidth, firmware, flash channels, and synchronization requirements still constrain service. A durability-sensitive flush can also impose dependencies that ordinary independent read benchmarks avoid. Inspect the configured scheduler and actual device characteristics rather than prescribing one universally optimal queue shape.

For a simplified set of independent requests, enough outstanding work lets the device overlap operations rather than wait for each round trip. Once a shared resource is saturated, additional depth mostly increases waiting. If throughput stops growing while residence time and unfinished work rise, the useful conclusion is not that capacity increased. For an interactive service, tail latency may worsen before average throughput reaches its maximum. Set a latency target and consider competing tenants rather than choosing depth solely from the largest IOPS value.

Request size creates another ceiling. A device capable of many small operations can still hit a byte-per-second bottleneck for large transfers, and a nominal bandwidth figure does not predict small synchronized-write latency. Keep units explicit: requests per second multiplied by bytes per request gives bytes per second only for the stated request population. CPU submission overhead, checksumming, copying, and filesystem work may become bottlenecks before hardware. Compare application and device utilization carefully; an underfed device with a saturated application thread calls for a different change than an overloaded device queue.

Run several depths while holding request size, read/write mix, address distribution, cache policy, synchronization semantics, dataset, and steady-state preparation fixed. Record achieved depth rather than only configured depth, and report completion errors as failures rather than fast successful requests. Track throughput, median and high-percentile latency, CPU use, and relevant queue counters. An asynchronous API can submit rapidly while completion lags far behind; measure completion at the boundary the user cares about. A sweep that silently disables synchronization at higher depths compares different contracts.

The queue lab provides a simpler contrast, with fluid backlog starting at four and no hardware-depth control or parallel flash channels. Press Reset, set both arrival to one and service to three, predict the stable M/M/1 mean residence of half a second, then Play or Step. For a separate smaller-margin comparison, Reset, set both arrival to two and service to three, predict one second, then Play or Step. Edits retain elapsed time and recalculate fluid backlog from the initial four, so they are not live rate changes. The separate residence formula assumes Poisson arrivals and exponential single-server service; it is not a measured SSD latency curve.