Histogram metrics at ingest: distribution-based latency SLOs with OpenPipeline

Histogram metrics in OpenPipeline: build distribution-based latency SLOs and percentiles from existing spans and logs, with no re-instrumentation needed.
Métriques histogrammes à l'ingestion : des SLO de latence fondés sur la distribution avec OpenPipeline

Performance dashboards still very often show an average. It is reassuring, readable and almost always misleading. Take a service where 90% of calls respond in 50 ms and 10% in 1.5 seconds: the average comes out at 195 ms. That figure describes neither the satisfied users nor the ones who are waiting. It hides precisely the population that will open a ticket or abandon their basket. That is exactly what the histogram metrics described here correct.

SRE practice settled the question long ago: a service level objective is expressed as a distribution (“99% of requests under 800 ms”), not as an average. You still need data that preserves that distribution. That is the job of histogram metrics, and Dynatrace SaaS now lets you produce them directly at ingest with OpenPipeline, from the spans and logs you already collect.

This article explains what this type of metric brings, when it is better to use it rather than a query on raw data, and how to fit it into an SLO approach.

Average, summary, histogram: three ways to lose (or keep) information

The single value

Average, minimum, maximum: these aggregates reduce thousands of observations to one number. Once stored on their own, there is no going back. You will never know what share of requests exceeded a given threshold.

The precomputed summary

Some instrumentation libraries compute quantiles on the client side, then export them. The problem appears as soon as you have several instances: the p99 of instance A and the p99 of instance B do not let you calculate the overall p99. Quantiles do not add up.

The histogram

A histogram places each observation in an interval (a “bucket”) and counts occurrences per interval. The shape of the distribution is preserved: the bulk of fast requests, the slow tail and everything in between. Because they are counters, histograms from several instances can be merged, and percentiles can be calculated on the server side across the whole fleet.

On the format side, the OpenTelemetry specification defines two variants: the explicit-bucket histogram and the exponential histogram, whose boundaries adapt to the range of observed values. Support differs by ingest path: through OTLP ingest, Dynatrace stores the buckets of explicit histograms, but only min, max, sum and count for exponential ones. Before building an SLO, check the Dynatrace histogram documentation to see which format is kept for your source.

Query the raw data or extract histogram metrics?

Nothing stops you from calculating a percentile directly on the spans stored in Grail:

fetch spans
| makeTimeseries percentile(duration, 95), by:{endpoint.name}

This approach remains perfect for one-off exploration. However, each run rereads all the spans for the period. On a dashboard tile refreshed every minute, over a thirty-day window or in a continuously evaluated SLO, that reprocessing costs time and resources.

With a histogram metric extraction rule in OpenPipeline, the distribution is calculated once, at ingest. The same analysis becomes a simple time series query:

timeseries percentile(span.api.duration, 95), by:{endpoint.name}

Important point: no re-instrumentation is needed. Neither the SDKs, nor the OpenTelemetry collectors, nor the application code change. OpenPipeline works on the span and log streams that already arrive in Dynatrace. The resulting metrics are stored in Grail and can be used in DQL, Dashboards, SLO definitions and anomaly detection, with the same retention and access controls as any other metric.

A simple rule for choosing: if a distribution question comes up regularly (permanent tile, SLO, alert), it deserves a histogram metric; if it is asked once during an investigation, querying the raw data is enough.

In practice

Step 1 — Identify the distributions that matter

List the measurements your users or your SLOs are really sensitive to: public API call duration, job processing time, machine learning model inference time, or a numeric value present in logs (message size, queue delay). Start with two or three candidates, not the whole catalogue.

Step 2 — Check the source data

Before creating the rule, make sure the field exists and is numeric across the target scope. For spans, an exploratory query like this lets you validate the scope and the useful dimensions:

fetch spans, from:now()-1h
| filter isNotNull(endpoint.name)
| summarize appels = count(), p95 = percentile(duration, 95), by:{endpoint.name}
| sort appels desc
| limit 20

For a value extracted from logs, the field must first be available in numeric form, for example thanks to a parsing processor earlier in the same pipeline.

Step 3 — Create the extraction rule in OpenPipeline

In the OpenPipeline configuration for the relevant stream (spans or logs), add a histogram-type metric extraction. You define a metric key (here span.api.duration, replace it with your own), select the source field, and OpenPipeline records each value as a histogram data point. For spans, the Sampling Aware Histogram Metric variant accounts for sampling. Choose the dimensions you keep with care: each extra dimension multiplies the number of series. The OpenPipeline documentation details the available options.

Step 4 — Query the metric

Once the data has arrived, several percentiles can be calculated from the same metric, without rereading the raw data. The choice between p50, p90, p95 or p99 is made at query time rather than at instrumentation time. To plot the median and the tail on the same chart, combine several aggregations in the timeseries command.

Step 5 — Base an SLO on the distribution

The classic SLO question is the inverse of the percentile: not “what is the p99 latency?”, but “what proportion of requests completes under 500 ms?”. The new percentRank() aggregation function answers that question: given a threshold, it returns the estimated fraction of observations below it. It is the natural basis for a distribution-based latency SLO. Check the DQL documentation for its exact syntax.

What else the distribution reveals

Beyond SLOs, keeping the shape of the data changes how you read incidents. A bimodal distribution often points to two distinct code paths, for example a cache that does or does not respond. A tail that lengthens after a deployment signals a regression that the median does not show yet. An alert set on a high percentile of the histogram metric can therefore spot a widening tail, where a threshold on the average would stay silent; the anomaly detection documentation lists the available models.

Best practices

  • Ban the average from latency SLOs: express them as a proportion under a threshold or as a percentile.
  • Limit dimensions: endpoint and service are often enough; avoid high-cardinality identifiers.
  • Keep the raw data for exploration: the metric answers recurring questions; spans and logs remain the reference for detailed analysis.
  • Compare before switching: over a test period, compare the percentile from the metric with the one calculated on raw data to validate the rule.
  • Document metric keys: a clear naming convention avoids duplicates between teams.

Frequently asked questions

Do you need to change your instrumentation to get histogram metrics?

No. OpenPipeline works on the spans and logs that already arrive in Dynatrace: neither the SDKs, nor the OpenTelemetry collectors, nor the application code change.

When is it better to query the raw data?

For a question asked only once, during an investigation, a query on spans or logs is enough. A histogram metric is worthwhile as soon as the question keeps coming back: a permanent tile, an SLO or an alert.

How do you express a latency SLO from a histogram?

As a proportion below a threshold rather than as an average. The percentRank() aggregation function returns the estimated fraction of observations below a given threshold; the DQL documentation gives its exact syntax.

Conclusion

Histogram metrics reconcile two needs that were long at odds: the precision of analysis on raw data and the speed of a pre-aggregated metric. By producing them at ingest, OpenPipeline makes them available without touching existing instrumentation. For a team that steers its reliability through SLOs, it is a chance to finally replace reassuring averages with figures that describe the experience of every user, including the worst served. To get to grips with the syntax before touching your own environment, the Dynatrace Playground offers sample queries on histogram metrics.

Further reading: Create histogram metrics at ingest with OpenPipeline (Dynatrace blog).

span.observer is an independent community blog, not affiliated with Dynatrace LLC. Dynatrace is a trademark of Dynatrace LLC. SaaS features evolve quickly: check their availability in your environment and in the official documentation.

Total
0
Shares
2 comments
Leave a Reply

Your email address will not be published. Required fields are marked *

Previous Post
Dynatrace MCP + Microsoft Teams : un agent Copilot Studio branché sur le serveur MCP

Dynatrace Microsoft Teams integration: a Copilot Studio agent connected to the MCP server

Next Post
Ingestion logs : détecter le silence, les trous de couverture et les excès de volume

Dynatrace log ingestion: detecting silence, coverage gaps and excess volume

Related Posts