OpenTelemetry traces reach Dynatrace, logs too, but the counters are nowhere to be found. No error in the application, nothing obvious in the Collector. In most cases the cause is OTLP metrics delta temporality, or rather its absence: the application sends cumulative sums, while Dynatrace expects deltas.
Goal of this tutorial: understand the difference between delta and cumulative in two minutes, spot the problem, then fix it in the SDK (Java, Python, Go) or in the Collector, and check that the metrics arrive. The content applies to both Dynatrace SaaS and Dynatrace Managed, which expose the same OTLP API.
If OTLP export is new to you, start with setting up the OpenTelemetry Collector for Dynatrace.
OTLP metrics delta temporality versus cumulative
A request counter receives 10 requests per minute. The exporter can send it in two ways:
| Minute | Cumulative | Delta |
|---|---|---|
| 1 | 10 | 10 |
| 2 | 20 | 10 |
| 3 | 30 | 10 |
With cumulative, each point holds the total since the process started. With delta, it holds only the increase since the previous point. Most OpenTelemetry SDKs export cumulative by default, a choice designed for Prometheus. The Dynatrace OTLP API, however, expects counters and histograms as deltas. The cumulative points concerned are not ingested.
Gauges have no temporality and pass through unchanged. UpDownCounters are accepted with both temporalities: delta ones become counters, cumulative ones become gauges. Cumulative histograms, on the other hand, are rejected just like cumulative counters. Details in the OTLP metrics ingestion documentation (“About OTLP metrics ingest” page).
Prerequisites
- An API token with the
metrics.ingestscope. - Your environment’s OTLP endpoint (
https://<env-id>.live.dynatrace.com/api/v2/otlpon SaaS,https://<your-domain>/e/<env-id>/api/v2/otlpon Managed). - A recent OpenTelemetry SDK, or a contrib Collector (the processor used below is part of the contrib distribution).
Step 1 — Confirm the diagnosis
Temporarily add the debug exporter to the Collector to see the temporality actually sent:
exporters:
debug:
verbosity: detailed
service:
pipelines:
metrics:
receivers: [otlp]
exporters: [debug, otlphttp/dynatrace]
In the Collector logs, look for the AggregationTemporality line of your counters. The value Cumulative confirms the problem. Without a Collector, the OTLP API response may report rejected points (a partial success response): turn on the SDK exporter logs to see it.
Step 2 — Fix it in the SDK
Java, Python, .NET, Node.js: one environment variable
export OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE=delta
This variable is part of the OpenTelemetry specification and is read by the Java (including the Java agent), Python, .NET and Node.js SDKs. Check that the SDK version you use supports it. For Java, it is already included in the OpenTelemetry Java agent configuration.
Java: in code, if you build the exporter yourself
import io.opentelemetry.exporter.otlp.http.metrics.OtlpHttpMetricExporter;
import io.opentelemetry.sdk.metrics.export.AggregationTemporalitySelector;
OtlpHttpMetricExporter exporter = OtlpHttpMetricExporter.builder()
.setEndpoint("https://abc12345.live.dynatrace.com/api/v2/otlp/v1/metrics")
.addHeader("Authorization", "Api-Token " + System.getenv("DT_API_TOKEN"))
.setAggregationTemporalitySelector(AggregationTemporalitySelector.deltaPreferred())
.build();
Critical line: deltaPreferred() switches counters and histograms to delta. Note that the full URL, including /v1/metrics, is required when you configure the exporter in code.
Go: a temporality selector
Recent Go OTLP metric exporters also read this variable, but older versions ignore it. For explicit behaviour regardless of the version, declare the selector:
import (
"context"
"go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp"
sdkmetric "go.opentelemetry.io/otel/sdk/metric"
"go.opentelemetry.io/otel/sdk/metric/metricdata"
)
exp, err := otlpmetrichttp.New(ctx,
otlpmetrichttp.WithTemporalitySelector(
func(sdkmetric.InstrumentKind) metricdata.Temporality {
return metricdata.DeltaTemporality
}),
)
This example forces delta for every instrument. For an UpDownCounter, delta makes it appear as a counter; if you prefer a gauge (current value), keep it cumulative in the selector.
Step 3 — Or fix it in the Collector
When you do not control the applications (third-party libraries, Prometheus receiver), convert in the Collector with the cumulative_to_delta processor:
processors:
cumulative_to_delta: {}
batch: {}
exporters:
otlphttp/dynatrace:
endpoint: https://abc12345.live.dynatrace.com/api/v2/otlp
headers:
Authorization: "Api-Token ${env:DT_API_TOKEN}"
service:
pipelines:
metrics:
receivers: [otlp, prometheus]
processors: [cumulative_to_delta, batch]
exporters: [otlphttp/dynatrace]
Critical lines: cumulative_to_delta (the new name of the former cumulativetodelta, kept as a deprecated alias) is stateful. It keeps the last value of each series in memory. Consequences:
- all data for a given series must go through the same Collector instance (no random load balancing across several instances);
- after a restart, the first point of an already existing series becomes the reference and is not exported (default
initial_value: autobehaviour); - memory grows with the number of series: watch cardinality.
Place the processor before batch, as above.
How to check it works
- In the
debugexporter logs, temporality now showsDelta. Then remove thedebugexporter. - In Dynatrace SaaS, open a Notebook and run (replace with your metric name):
timeseries requetes = sum(http.server.request.count), interval: 1m
| fieldsAdd total = arraySum(requetes)
The line should be flat for constant traffic (10 per minute in our example), not a ramp that climbs forever. A ramp would mean a cumulative value read as a delta. The metric name in Dynatrace may differ slightly from the OpenTelemetry name: look it up in the metrics list. On Managed, use the data explorer with the same visual check.
Troubleshooting
- Still nothing after switching to delta: check the token’s
metrics.ingestscope and the endpoint URL (a 404 often means a duplicated or missing/v1/metrics). - Isolated spikes after each restart: a
cumulative_to_deltaprocessor behind a load balancer receives points of the same series on several instances. Route by series or fix it in the SDK. - Only some metrics arrive: some instruments stay cumulative, for example histograms, which are then rejected (a cumulative UpDownCounter does arrive, as a gauge). Compare the
debugexporter output instrument by instrument. - Doubled values: the same metric is sent both directly from the SDK and through the Collector. Keep a single export path.
Best practices
- Fix it as close to the source as possible: an SDK set to delta avoids Collector state.
- Keep the same temporality for every application on a platform, documented in your standards.
- For latency SLOs, send histograms as deltas: see histogram metrics with OpenPipeline.
- A metric lost in silence is a blind SLI: add a no-data alert on critical metrics.
Frequently asked questions
Why does Dynatrace prefer delta?
Delta is stateless on the receiving side: each point stands on its own, however many instances are sending. There is no need to track every process restart.
My Prometheus metrics are cumulative: what should I do?
Scrape them with the Collector’s Prometheus receiver, then apply cumulative_to_delta as in step 3.
Are gauges affected?
No. A gauge represents an instantaneous value (memory used, queue length) and has no aggregation temporality.
Conclusion
OTLP metrics delta temporality is a one-line setting, but forgetting it makes whole counters vanish. Check temporality with the debug exporter, switch the SDK to delta or convert in the Collector, then check the shape of the lines. Your SLIs then rest on complete numbers.
span.observer is an independent community blog, not affiliated with Dynatrace LLC. Dynatrace is a trademark of Dynatrace LLC. Features evolve quickly: check their availability in your environment and in the official documentation.