Capacity planning and cost

Start with arrival patterns, prompt and output length distributions, model mix, regions, and user-facing latency objectives. Benchmark representative distributions rather than only the shortest request. Record the largest sustained rate that meets the objective with acceptable errors and memory headroom. A service may process more tokens per second after a larger batch while violating inter-token latency. Safe capacity is tied to the objective and workload, so a single universal requests-per-second number is usually misleading.

Distinguish a demand forecast from a load test and a hardware peak. Forecasts have uncertainty; load tests cover specified scenarios; theoretical peaks are bounds under assumptions. Examine bursts, correlated retries, cold starts, dependency limits, and degraded hardware. Little's law relates average in-flight work to arrival rate and average residence time in a stable, consistently measured system, but it does not predict tail latency. Use it to reconcile measurements, not to excuse missing overload or percentile tests.

Reserve headroom for an explicit failure scenario: one replica, one node, or an entire zone. These are not interchangeable. If losing a zone also removes storage or the control plane, spare compute elsewhere may not suffice. Check load-balancer behavior and remaining dependency capacity. Keep a bounded queue and an admission policy during the transition. An autoscaler cannot instantly acquire unavailable devices, download large weights, warm kernels, or restore a cache; measure those delays and pre-provision where required.

Scaling signals should reflect the constrained work: queue age, admitted token load, active cache demand, or measured saturation, not CPU percentage alone. Use stabilization to avoid oscillation, and avoid scaling down below recovery reserve because one quiet minute looks cheap. Alternatives include workload shaping, separate interactive and offline pools, and routing simpler tasks to appropriately evaluated models. Each changes quality, isolation, or reliability assumptions and must be tested, rather than treated as a free multiplier on capacity.

Choose the business unit: a successfully completed request, an accepted document embedding, or a validated training run. Include provisioned compute, storage, networking, observability, and relevant operational costs in the same window. Separate input and output tokens when their resource behavior differs. A low cost per attempted request can conceal failures, retries, or low-quality answers. State which costs are included and whether reserve capacity is charged to the unit; otherwise comparisons can silently use different accounting boundaries.

Sensitivity analysis is more valuable than a false decimal precision. Vary demand, failure reserve, output length, cache-hit assumptions, and price. The prices below are fictional teaching inputs, not provider quotes. Model optimization can reduce compute yet increase engineering cost or shift the bottleneck to memory or storage. Preserve the workload and quality target when comparing options. More advanced planning adds stochastic demand and correlated failures, but even then a model should expose its uncertainty and be checked against observed outcomes.