Budget telemetry as a production workload with its own evidence

For event signals, measure records per second and serialized bytes per record after the chosen attributes and sampling policy. For metrics, include active series, scrape or export frequency, histogram representation, and churn. Distinguish wire-compressed bytes, backend logical ingestion, retained compressed payload, and indexes. A backend's billing or capacity unit may count a different stage from the network interface. Record decimal GB versus binary GiB and seconds versus days so a familiar-looking number does not mix incompatible denominators.

Retention multiplies demand over time, while replication and indexing can add further storage. These are separate factors whose values depend on the system and data. Measure them rather than adopting one universal bytes-per-span or bytes-per-sample constant. A sampled trace can have a variable number of spans and large attributes, so average trace size alone can hide a dangerous tail. Log-schema changes and high-cardinality attributes can alter both event size and indexing work. Capacity fixtures should include realistic distributions and controlled worst cases, not only empty or highly compressible payloads.

Measure throughput and resource use at producers, agents, gateways, exporters, and storage ingestion rather than assigning the entire path one capacity number. A CPU-heavy processor, a network quota, a finite exporter queue, or an index bottleneck can dominate independently. Scaling collectors also needs correct routing for stateful operations such as tail sampling or temporality conversion; arbitrary load balancing can split state and damage measurement. Preserve the processing contract while adding capacity, and verify achieved throughput rather than assuming a replica count implies proportional service.

Average utilization does not establish burst or failure tolerance. Choose a target utilization with explicit headroom, then check what capacity remains when a component or failure domain is unavailable. Include the spare service needed to drain queued debt after recovery. A system running at its normal arrival rate can stay permanently behind after an outage even though new data no longer accumulates faster. Admission limits or lower telemetry detail can protect service health, but every reduction changes observability and should expose its own loss or selection evidence to consumers.

Track accepted, refused, dropped, retried, and exported volume, queue capacity and age, processing latency, and last successful backend observation. Reconcile boundaries using known fixtures or heartbeats while preserving their timestamp assumptions. Collector process uptime does not prove its exporters are succeeding, and backend ingestion does not prove the data is queryable within the required freshness window. Use an independent availability check or evidence channel for critical blind spots so the same broken export path cannot be the sole authority on whether it is broken.

The queue lab illustrates a fixed service-margin comparison only. Reset, set both arrival λ=2 and service μ=3, predict drainage of the initial four jobs, then Play or Step. For a separate overload case, Reset and re-enter both λ=4 and μ=2, predict growth, then inspect. Edits retain elapsed time and recompute from the initial four rather than switch a live accumulated queue. This unbounded fluid model does not predict collector hardware, routing correctness, finite-buffer drops, prices, or query freshness. Those claims require the measured stage budgets and independent checks above.