Service frontiers: preserve quality and latency while comparing cost

Choose a fixed cohort, traffic process, quality oracle, and client-visible latency boundary. Specify whether cost is per offered task or successful logical task, and preserve that denominator across configurations. A p95 duration means a percentile under the measured workload; it is not the maximum or a universal per-request guarantee. Repeat measurements under relevant demand classes and keep rejected or timed-out tasks visible. Otherwise a system can improve its apparent tail by dropping exactly the requests whose latency would have been measured.

Quality and latency constraints define a feasible set rather than another unconstrained score to trade away silently. A configuration below the minimum quality requirement is not an equivalent cheap service, even if its throughput is excellent. A low average latency does not satisfy a tail constraint automatically. State how uncertainty affects borderline cases: a point estimate just above a quality threshold is weak evidence if its interval crosses the boundary. Capacity, isolation, and memory limits can further exclude a configuration despite attractive measured averages.

A configuration dominates another in this three-metric comparison if it is no more costly, no worse in quality, and no slower in the stated latency measure, with at least one strict improvement. Two points can instead be incomparable: one is cheaper while the other is better and faster. The remaining frontier does not pick a unique operating policy. Requirements and an explicitly justified objective are needed to choose among incomparable feasible points. Avoid hiding those choices in an arbitrary aggregate score whose weights users never approved.

The comparison is only meaningful under compatible resource budgets, workload, accounting boundary, and metric definitions. Serving research such as DistServe studies throughput and latency tradeoffs for specified systems and workloads, not a universal cheapness ranking. Record batching timeout, maximum batch size, output length policy, cache behavior, model version, and rejection rules. These parameters can change both resource efficiency and useful output. A cheap point that uses a different quality target or excludes recovery cost belongs to another experiment until the difference is explicitly accounted for.

Sweep one policy at a time when possible and repeat at matched demand rather than compare an overloaded baseline with an underloaded candidate. Include warmup, steady-state, and failure behavior as separately identified phases. A batching timeout can lower device idle time while increasing queue residence before execution; a larger batch can also change workspace and memory pressure. Measure the client-facing result and full included effort instead of assuming that a kernel-throughput increase translates directly into lower cost per accepted task.

The queue lab supplies a narrow latency-headroom analogy. Reset, explicitly set arrival rate two and service rate three, predict, then Play or Step. Compare a separate fresh Reset with arrival rate 2.5 and service rate three explicitly set. Its M/M/1 mean residence rises from one to two seconds under the displayed formula’s assumptions; that value is a mean, not p95. The animated backlog is a separate deterministic fluid toy starting at four. There are no batches, quality scores, prices, or frontier computation, so the lab cannot locate a real serving configuration on this three-metric frontier.