Proxy cost: measure useful operations, attempts, and critical paths

The client observes a logical operation that may include local waiting, retries, network time, and application execution. A proxy's upstream-attempt duration is a different population and interval. An inclusive proxy span can overlap application work, so adding it to an inclusive application span can double-count time. Use a trace with compatible clocks or explicitly measured local durations to identify the critical path. Do not add component p99 values and call their sum the operation p99; quantiles do not generally compose that way.

Compare completed useful operations, attempted backend requests, failures, cancellations, and rejected admission together. Lower latency among the few accepted successes can hide extensive rejection. A closed-loop test client that waits before issuing its next request can reduce offered load during a slowdown, obscuring queueing that a production arrival process would create. Document offered versus achieved rate and the load-generation method. Also keep request size, protocol, streaming duration, connection reuse, and TLS handshake frequency comparable when estimating the effect of adding a proxy.

CPU seconds per second measures processor demand, not elapsed latency. Per-attempt processing includes a different cost boundary from per-connection handshakes and persistent background work; measure or amortize them explicitly. More retries can raise CPU demand without changing the original operation rate. Encryption and parsing costs depend on bytes and protocol features, while telemetry filters can add work and memory. A single requests-per-second number cannot describe an AI inference stream whose output sizes and lifetimes vary widely across requests.

Memory includes connection state, stream buffers, configuration, certificate material, telemetry queues, and allocator retention. Per-proxy idle memory multiplied by a fleet count is not a loaded-memory estimate. A node-local proxy can share some overhead while concentrating many workloads and creating a different contention boundary. Container CPU quotas and throttling can introduce long delays even when average utilization seems moderate. Measure both the resource limits available to the process and the node conditions around it, including application competition and network scheduling.

Run a matched baseline with identical application behavior and then add the selected proxy policy features in controlled steps. Separate steady reused connections from connection churn, and compare small messages with realistic payloads. Record CPU profiles, throttling, active streams, queued requests, buffer use, retransmissions, and useful completion rate. Repeat representative loads around the saturation point. A configuration that is inexpensive at low utilization can create a queue under a burst, while removing a policy feature can change security or retry semantics and invalidate an apparently faster comparison.

The queue lab isolates a specific queueing assumption. Reset, set λ=2 and μ=3, predict the separately labeled M/M/1 mean residence time is one second, then Play or Step; the fluid initial backlog drains in four seconds. Separately Reset, set λ=2.50 and μ=3, predict M/M/1 mean two seconds and fluid drainage eight seconds, then Play or Step. The stochastic mean assumes Poisson arrivals, independent exponential service, one server, and stability. It is not a measured proxy p99, and the fluid path is a different deterministic model. Explain both boundaries before applying queueing intuition to actual telemetry.