advanced25 min studyLesson 170 in Engineering

Benchmarking asynchronous I/O workloads

Design open and closed load experiments, expose coordinated omission, and separate offered load from admission, completion, and latency boundaries.

Before you begin

On this page

The question for today

What you’ll understand

Choose an open or closed arrival model that answers a concrete performance question.
Report offered, admitted, and completed work alongside rejection and timeout counts.
Measure latency from declared boundaries and identify coordinated omission in a load generator.

An asynchronous server can appear faster simply because the benchmark stops asking difficult questions. A generator that waits for every response before scheduling more work reduces its own arrival rate when the service slows. A histogram that includes only successful completions hides rejected or abandoned requests. Build the experiment around a workload hypothesis and an accounting boundary before choosing a headline throughput number. You are measuring a coupled generator, transport, and service, not an isolated event-loop keyword.

Choose the arrival process deliberately

A closed workload has a fixed client population: each client issues another request after its previous operation completes, possibly following a think delay. This is appropriate when that feedback resembles the users being modeled. An open workload schedules arrivals independently of completions, for example at a declared fixed rate or from a stochastic process. It can expose overload that a self-throttling closed population conceals. Neither model is universally correct; state which external behavior your experiment represents.

Separate intended arrival time from actual submission. If the load generator stalls during a service pause and silently skips scheduled requests, its recorded distribution omits work that would have arrived during the pause. This is coordinated omission. Record scheduling delay, preserve the intended schedule when the methodology calls for it, and report generator saturation. A corrected histogram estimates consequences of a stated arrival model; it cannot retroactively establish that a real system received requests the generator never sent.

Count every boundary and every outcome

Define offered requests at the generator, admitted requests after the service's admission policy, and completed requests at a stated terminal boundary. Report failures, rejections, cancellations, and timeouts separately, including how unfinished requests at the observation window are handled. An offered rate of one thousand requests per second is not a completed throughput of one thousand. Include backlog at both endpoints: work admitted near the end may finish afterward, so same-window differences require that accounting context.

Declare latency start and finish points. Scheduled-to-completion latency includes generator scheduling delay; submitted-to-completion latency excludes it. Server processing time excludes transport and upstream waiting. Use a monotonic clock for durations within one process and avoid subtracting unsynchronized machine timestamps. Show a distribution, not only a mean, with sample count and timeout policy. Do not silently replace timed-out samples with fast successful retries or treat the timeout threshold as the request's known eventual completion latency.

r_o=\frac{N_o}{T},\qquad r_a=\frac{N_a}{T},\qquad r_c=\frac{N_c}{T}

For a common observation window of T seconds, offered, admitted, and completed counts N_o, N_a, N_c produce rates in requests per second. Counts at different boundaries need not match; initial and final in-flight inventories and terminal outcomes explain differences.↔ Scroll wide equations horizontally.

Make comparisons reproducible and diagnostic

Hold payload sizes, connection reuse, warmup policy, concurrency limits, and timeout settings constant when comparing I/O designs. Record versions, hardware, CPU placement, dataset residency, and whether TLS or disk persistence is included. Repeat trials and show variability. Watch generator and server CPU and network utilization: a generator bottleneck can impose a false throughput ceiling while making the service look comfortable.

Use the queue lab as a prediction exercise before an experiment. Press Reset, set arrival rate to 4 and service rate to 3, predict backlog after one second, then press Step twice. For an independent underloaded observation, press Reset again, explicitly set arrival rate to 2 and service rate to 3, predict the one-second count, then press Step twice. Each run starts at four items; explain why the underloaded run still contains queued work after one second. The model is deterministic fluid flow, not a benchmark harness or an event-loop implementation. Its separate M/M/1 residence time assumes stable Poisson arrivals and exponential service; real correlated payloads and scheduler stalls can produce very different tail latency.

L_{\mathrm{scheduled}}=(t_{\mathrm{submit}}-t_{\mathrm{scheduled}})+(t_{\mathrm{complete}}-t_{\mathrm{submit}})

All timestamps refer to one request on a common monotonic client clock. Each term and their sum are durations in seconds. The decomposition separates generator scheduling delay from submitted-to-completion latency; it does not isolate server service time.↔ Scroll wide equations horizontally.

Work it through

Audit one ten-second run

During a 10-second measurement window, 1,000 requests are offered, 900 admitted, and 850 successfully completed. Exactly 100 are rejected before admission, no admitted requests fail, time out, or are cancelled, and the system begins with no in-flight work. A traced request is scheduled at 2.000 s, submitted at 2.030 s, and completed at 2.080 s on one client clock.

Divide each boundary count by ten seconds: offered rate is 100 requests/s, admitted rate is 90 requests/s, and successful completion rate is 85 requests/s. Report the hundred rejections rather than hiding them inside the completion histogram.
With no initial work and no admitted failures, timeouts, or cancellations, 900 − 850 = fifty admitted requests remain in flight at the end. Their later completion times belong to a drain report or an explicitly extended measurement policy.
The traced request waits thirty milliseconds before submission and another fifty milliseconds until completion. Its scheduled-to-completion latency is eighty milliseconds, whereas a submission-only histogram records fifty.

The result

This run demonstrates neither sustainable 100 requests/s completion throughput nor an eighty-millisecond server processing time. Boundary counts expose unfinished work, and the timestamp decomposition exposes generator delay.

The living notebook

Explore the model

Interactive lab

Interactive lab · queue

Predict overload before measuring it

Press Reset, set arrival rate to 4 and service rate to 3, predict backlog after one second, then press Step twice. For the draining comparison, press Reset again, explicitly set both rates to 2 and 3, predict the one-second count, then press Step twice. Each independent run starts at four items.

Fluid backlogCurrent time

Fluid backlog: 4
Net rate · λ − μ: −1 /s
M/M/1 mean residence: 1 s

Arrival rate λ2Service rate μ3

t = 0 s

Fluid backlog starts at 4 and follows max(0, 4 + (λ − μ)t). The separate M/M/1 residence-time formula assumes Poisson arrivals, exponential service and λ < μ. Parameter changes pause playback and recompute the model at the current elapsed time. Reset restores both parameters and returns time to zero. Playback starts paused. Adjust either parameter, then choose Play or Step.

Look forA fluid backlog explains rate imbalance. It does not generate benchmark requests, reproduce coordinated omission, or establish measured tail latency; the M/M/1 readout is valid only under its separately stated assumptions.

Pause & reason

Check your understanding

Interactive checks need JavaScript. You can still read the explanation.

Read the checkpoint explanation

Open versus closed describes how arrivals depend on completions, not whether socket operations are asynchronous. A closed workload can be appropriate, but its self-throttling must not be mistaken for behavior under independent arrivals. Excluding failures also changes the measured population.

Your turn

Try it yourself

A generator schedules one request at 1.000 s but submits it at 1.120 s; it completes at 1.150 s on the same monotonic clock. Calculate both latency definitions. If a 20-second run offers 2,400 requests and completes 2,000, can you infer the rejection count from those two totals alone?

Take a moment to make a prediction or work it out on paper. Open a hint when you need one.

Give me a hint

Use the three timestamps before interpreting the throughput counters. Requests can be pending, rejected, failed, or cancelled, and the run can start with existing work.

Compare with the solution

Scheduling delay is 120 ms, submitted-to-completion latency is 30 ms, and scheduled-to-completion latency is 150 ms. Offered and completed rates are 120 and 100 requests/s. The difference of four hundred requests does not by itself identify rejections: inspect admission counts, terminal outcome counts, and initial and final in-flight inventories. If the generator omitted scheduled requests during a pause, report that loss rather than claiming independent-arrival performance.

Keep the idea

Take it with you

Open and closed workloads answer different questions because only the latter feeds completions back into arrivals.
Report offered, admitted, completed, failed, and unfinished work with explicit window boundaries.
Measure scheduling delay and watch generator saturation to avoid coordinated omission and false capacity claims.

Read further

Original sources and deeper explanations to follow at your own pace.

Optional progress, saved in this browser when available. Revisit any lesson at any time.

Lesson not marked complete.