Metric cardinality: spot exploding series before the bill arrives

Metric cardinality: count series in review, audit with promtool and alert on new series before they inflate your bill, with tested examples.
Cardinalité des métriques : repérer les séries qui explosent avant la facture

Metric cardinality never shows up on a dashboard. It shows up on next month’s bill, or in a Prometheus server that keeps restarting because it ran out of memory. This guide starts from a practical operations decision: before a team ships a new metric, what should you check, and how do you spot a series that is exploding once it is already in production?

The method has four moves: put a number on a metric before it ships, audit a block of data with promtool, alert on the rate at which new series appear, then cap the damage at the source in the OpenTelemetry SDK. It complements the guide on Kubernetes metrics cardinality, which centres on the TSDB API and kube-state-metrics, and does not repeat it.

In brief

  • A metric’s series count is at most the product of the number of values of each label: work it out before release.
  • promtool tsdb analyze audits a block offline and ranks the hungriest labels.
  • scrape_series_added lets you alert on new series, and promtool test rules tests the alert without a server.
  • The OpenTelemetry SDK can filter attributes (attribute_keys) and cap data points, with an overflow point.

Put a number on metric cardinality before code review

The Prometheus instrumentation best practices advise keeping the cardinality of a metric below 10 and, beyond that, limiting such metrics to a handful across the whole system. They suggest looking for another solution when a metric goes over 100. These are guidelines, not bans, but they give a useful order of magnitude in review.

The arithmetic is simple. A metric with the labels route (4 values), status (3 values) and user_id (1,000 values) can reach 4 × 3 × 1,000 = 12,000 series. The third label carries the whole cost. So add one question to your pull request template: what values can each label take, and what bounds their number?

The Prometheus naming page is explicit: do not store high-cardinality dimensions such as user IDs or email addresses in labels. An identifier belongs in a trace or a log, not in a metric.

Audit a block with promtool tsdb analyze

The TSDB status API queries a running server. promtool offers the other option: analyse a data directory, for example a copy of a block taken from a server or a snapshot, without touching production. The command is tsdb analyze, described officially as “Analyze churn, label pair cardinality and compaction efficiency”.

To keep the example reproducible, I built a laboratory dataset: one http_requests_total metric with 12 legitimate series (4 routes, 3 statuses) and 400 series that also carry a user_id label. The OpenMetrics file is converted to blocks, then analysed.

promtool tsdb create-blocks-from openmetrics m.om ./db
promtool tsdb analyze ./db
Output of promtool tsdb analyze on a laboratory dataset: 412 series, with user_id at the top of the high-cardinality labels with 400 values

Read it from the bottom up. “Highest cardinality metric names” names the culprit metric, “Highest cardinality labels” the label (here user_id, 400 values against 4 for route), and “Label names with highest cumulative label value length” shows where memory goes, because value length counts. The same output also contains “Label pairs most involved in churning”, useful for spotting values that are born and die constantly. In my dataset every value is 0: the data is stable, which a cluster that redeploys is not.

Spot exploding series with scrape_series_added

A one-off audit is not enough: the explosion comes after a deployment, on a Thursday at 5 pm. Prometheus generates diagnostic series for every scrape. According to the “Jobs and instances” documentation, scrape_series_added is the approximate number of new series in the scrape, scrape_samples_scraped the number of samples the target exposed, and scrape_samples_post_metric_relabeling the number left after relabelling.

A healthy job creates few new series per scrape, apart from during redeployments. An alert on this signal warns you before memory runs out:

groups:
  - name: cardinalite
    rules:
      - alert: SeriesNouvellesParScrape
        expr: scrape_series_added > 500
        for: 10m
        labels:
          severity: warning

The threshold of 500 is an example: measure your jobs’ normal level before fixing it. Then test the rule without a server, using a test file that injects values:

The scrape_series_added alert rule in rules.yml, followed by promtool check rules and promtool test rules both answering SUCCESS

The test injects the scrape_series_added series at 10 for five minutes, then at 900, and checks that the alert is firing after 30 minutes. An alert that has never been tested is a gamble, especially with a for clause.

Cap metric cardinality at the source with OpenTelemetry

Fixing things in Prometheus is reactive. The OpenTelemetry SDK lets you decide before export. The metrics SDK specification defines an aggregation cardinality limit: if no other value is defined, 2000 is used. Beyond it, measurements go into an overflow set carrying the boolean attribute otel.metric.overflow set to true. A View can also set aggregation_cardinality_limit for an instrument, or filter attributes with attribute_keys. According to the specification, the limit is enforced after attribute filtering, so filtering first helps you stay under the cap.

I checked the behaviour with the Python SDK (opentelemetry-sdk 1.45.2): a counter receives 3,000 measurements, each with a different user.id attribute.

View(instrument_name="app.requests", attribute_keys={"http.route"})
View(instrument_name="app.requests", aggregation_cardinality_limit=100)
Output of the OpenTelemetry test script: 2001 points with no view, 1 point with attribute_keys, 101 points with a limit of 100 including the otel.metric.overflow point

With no view you get 2,001 points: the default limit of 2,000 plus the overflow point. With attribute_keys, a single point remains holding all the measurements (3,000). With a limit of 100 you get 101 points, and the overflow accumulates the remaining 2,900 measurements. The overflow keeps the total but loses the detail. A cap limits the damage, but only removing the attribute solves the problem. I have not verified these options in other languages: availability varies by SDK.

To place the two ecosystems side by side, see OpenTelemetry and Prometheus as complementary tools.

Best practices: a five-line review guardrail

  • List the possible values of each label in the pull request and multiply.
  • Refuse any identifier (user, request, session) as a metric label or attribute.
  • Set sample_limit on jobs, knowing that exceeding it fails the whole scrape.
  • Alert on scrape_series_added and test the rule with promtool test rules.
  • Replay promtool tsdb analyze every quarter on a recent block.

The same reasoning applies to logs: the article on Loki labels and cardinality covers the equivalent for a log index, and the one on Datadog custom metrics shows how billing is computed at one vendor. On the operations side, alerting on nodes, pods and quotas remains a separate matter.

Sources: Prometheus instrumentation best practices, OpenTelemetry metrics SDK specification.

Frequently asked questions

What cardinality is acceptable for a metric?

The Prometheus documentation gives guidelines: under 10 values per metric, only a few metrics above that, and a rethink of the dimensions beyond 100. The right threshold depends on your series budget, which you have to measure.

What happens when the OpenTelemetry SDK limit is reached?

New attribute combinations are aggregated into a single point carrying otel.metric.overflow=true. The total stays exact, but the per-attribute detail is lost for those measurements.

Should I use sample_limit to protect myself?

It is a useful safety net but a blunt one: according to the documentation, if the number of samples after relabelling exceeds the limit, the entire scrape is treated as failed. Combine it with the alert on new series.

Conclusion

Controlling metric cardinality means moving control earlier: count in review, audit with promtool, alert on the rate of series creation, cap in the SDK. The outputs shown come from a laboratory dataset, not a production environment: repeat the measurements on your own data before fixing thresholds.

span.observer is an independent blog, not affiliated with Dynatrace LLC. Product names mentioned belong to their respective owners; configurations are examples to adapt and test before any production use.

Total
0
Shares
Leave a Reply

Your email address will not be published. Required fields are marked *

Previous Post
Migrer Jaeger v2 : passer au Collector OpenTelemetry sans perdre de traces

Migrate to Jaeger v2: moving to the OpenTelemetry Collector without losing traces

Related Posts