Each label combination creates a measurement identity
Budget active series, churn, and histogram expansion while choosing bounded dimensions that preserve useful aggregation.
On this page
The question for today
What you’ll understand
- Estimate a label cross product and state when it is only an upper bound.
- Include histogram components, instance dimensions, and churn in a series budget.
- Separate bounded operational dimensions from per-event identifiers suited to other signals.
Labels make metrics useful by separating operations, locations, and outcomes. They also define identities that the storage and query system must track. Adding one convenient dimension can multiply costs across every existing combination, especially when that dimension grows with requests or users. The goal is to retain dimensions that answer recurring operational questions while measuring both the number of active identities and how quickly new ones replace old ones.
A cross product is a planning assumption
In Prometheus, a metric name and its complete label set identify a series. If a metric has several bounded label dimensions, multiplying their possible value counts gives a full cross-product estimate. It is exact only when every combination actually exists and is exported. Correlations can reduce the count: a route may support only one method, and a region may contain only some services. Conversely, hidden target labels such as instance or environment can multiply a seemingly small application-level estimate. Inspect the final exported label set rather than counting only arguments in one instrumentation call.
Unbounded identifiers such as request IDs, raw URLs containing object IDs, and user emails can create an identity per event or person. Hashing such a value does not reduce its cardinality when distinct inputs remain distinct outputs. Prefer a bounded route template or operation category for recurring metrics, with event-specific investigation in appropriately controlled logs or traces. A trace identifier in a metric exemplar serves a different role from making it a series label. Define an owner and expected maximum for each new dimension so growth is detected before it consumes the telemetry budget.
Distribution components and churn add separate costs
A classic histogram expands one label combination into multiple bucket series plus its count and sum. Include the positive-infinity bucket in the chosen bucket count so it is not forgotten or counted twice. A native histogram has a different representation and cost profile, so reuse neither the classic component count nor a guessed fixed byte cost blindly. Storage cost also includes label indexes, active head state, write-ahead logging, compression, retention, replication, and query work. Per-sample payload is only one part of the budget.
Active cardinality and churn answer different questions. A workload can keep roughly the same active count while rapidly replacing instance or deployment identities, creating substantial historical indexes and ingestion work. Long retention accumulates many identities even when each lives briefly. Measure new-series rate and historical unique sets alongside current active series. Aggregating after ingestion can make a dashboard cheaper to query without refunding the cost already paid to create and store the original identities. Decide whether dimensions should be removed or normalized before the expensive boundary when the use case permits it.
Budget dimensions with realistic query examples
Before adding a label, write the aggregation or alert it enables and estimate its full combination count. Test production-like route counts, instance counts, and deployment churn in a controlled environment. Enforce SDK or collector cardinality controls according to their actual overflow semantics, then expose overflow or dropped attributes as evidence rather than silently interpreting the reduced metric as complete. A limit can preserve process health while changing which distinctions remain observable. Include those consequences in dashboard and alert reviews so the query still measures its claimed population.
The quadratic lab demonstrates only one cross-product pattern. Reset, set both coefficient a=1 and position x=1, predict y=1, then Play or Step. Reset and re-enter a=1 and x=2, predict y=4. Interpret x as a normalized size of each of two fully populated independent label sets, so their product grows quadratically. The controls are limited to this fixed y=ax² model and do not enumerate label sets, simulate churn, or estimate storage bytes. The actual series budget needs the explicit dimensions and component counts above.
Work it through
Include histogram components in the label budget
A classic histogram has ten cumulative bucket series including positive infinity, plus count and sum. It is exported for twelve routes, three methods, four outcomes, and twenty instances. Assume every combination exists.
The label cross product is 12 × 3 × 4 × 20 = 2,880 combinations. Ten buckets plus count and sum give twelve scalar components per combination.
The family therefore produces 2,880 × 12 = 34,560 scalar series under this full-product assumption.
At a fifteen-second scrape interval, that is 34,560/15 = 2,304 samples per second before other metrics. Correlated or absent combinations can reduce the observed count, while additional labels can increase it.
A single histogram family can create 34,560 scalar series here. The estimate is about measurement identities and samples, not a complete memory or storage-cost quote.
The living notebook
Explore the model
Interactive lab · function
Two growing dimensions multiply
Reset, set both a=1 and x=1, predict y=1, then Play or Step. Reset and re-enter a=1 and x=2 to predict y=4. Interpret positive x as normalized size of each of two fully populated label dimensions. The fixed quadratic model does not inspect metrics, enumerate real label combinations, or estimate churn and storage.
- y at chosen x
- 0.608
- Tangent slope · 2ax
- 1.35
- Signed area · ax³/3
- 0.182
This lab uses only y = ax². Playback traces the curve within the fixed integration interval from 0 to x; it does not estimate numerical error or change the chosen point. Parameter changes pause playback and recompute the model at the current elapsed time. Reset restores both parameters and returns time to zero. Playback starts paused. Adjust either parameter, then choose Play or Step.
Look forDimension growth can multiply across an existing label structure; count the actual exported identities and components.
Pause & reason
Check your understanding
Your turn
Try it yourself
A scalar family uses six operations, five regions, and forty instances, with all combinations present. At a ten-second interval, compute active series and samples per second. State one cost this excludes.
Take a moment to make a prediction or work it out on paper. Open a hint when you need one.
Give me a hint
Multiply dimensions for identities, then divide by the collection interval.
Compare with the solution
The family has 6 × 5 × 40 = 1,200 active series and 1,200/10 = 120 samples per second. This does not include label-index storage, active-series memory, WAL, replication, historical churn, or other metric families. The full-product assumption also needs checking against actual valid combinations.
Keep the idea
Take it with you
- A metric's complete label set defines a series identity, and dimensions can multiply.
- Histogram components, hidden target labels, active cardinality, and churn require separate accounting.
- Bounded operational labels should justify their cost through concrete queries and explicit limits.
Read further
Original sources and deeper explanations to follow at your own pace.
Optional progress, saved in this browser when available. Revisit any lesson at any time.
Lesson not marked complete.