Your services send their traces over OTLP, but your reliability targets remain numbers on a wiki page. Without an OpenTelemetry SLO wired to that data, nobody knows whether the service keeps its promise, or how much error budget is left before freezing releases.
Goal of this tutorial: create two SLOs in Dynatrace from OpenTelemetry spans, one for availability and one for latency, written in DQL, then check their values. The content targets Dynatrace SaaS (Grail, Service-Level Objectives app). One section gives the alternative for Dynatrace Managed.
If you are still unsure which indicators to measure, first read how to choose SLIs: the practical method.
What we measure
- Availability: share of server requests that are not in error. Target: 99.5% over 28 days.
- Latency: share of server requests served in under 500 ms. Target: 95% over 28 days.
Both SLIs are “good events / valid events” ratios. This format makes the error budget readable: 0.5% of requests may fail over the window.
Prerequisites
- SaaS on the Grail platform, with the Service-Level Objectives app and permissions to read spans and write SLOs (exact list in the documentation).
- OpenTelemetry spans already in Grail, with the
service.nameattribute. If nothing arrives, follow the OpenTelemetry traces not showing checklist. - Server spans (
span.kind == "server") with a status set by the instrumentation.
Creating an OpenTelemetry SLO step by step
Step 1 — Explore the spans in a Notebook
Before writing the SLI, check the available fields:
fetch spans, from: now() - 1h
| filter service.name == "checkout" and span.kind == "server"
| summarize total = count(),
errors = countIf(span.status_code == "error"),
p95 = percentile(duration, 95)
Critical lines: span.kind == "server" keeps only incoming requests, not outgoing calls. span.status_code carries the OpenTelemetry status. If your HTTP spans are not marked as errors on 5xx responses, filter on http.response.status_code >= 500 instead. The exact field names in Grail are described in the Semantic Dictionary documentation: check them before going further.
Step 2 — Write the availability SLI
The SLO app expects a time series that returns a percentage. So we turn the spans into series with makeTimeseries:
fetch spans
| filter service.name == "checkout" and span.kind == "server"
| makeTimeseries {
total = count(),
errors = countIf(span.status_code == "error")
}, interval: 1m
| fieldsAdd sli = 100 * (total[] - errors[]) / total[]
In the Service-Level Objectives app, create an SLO with a custom SLI, paste the query, then set the target to 99.5 and the window to 28 days. The expected shape (result field name, query timeframe handled by the app) may change: compare with a template offered by the app.
Step 3 — Write the latency SLI
fetch spans
| filter service.name == "checkout" and span.kind == "server"
| makeTimeseries {
total = count(),
fast = countIf(duration < 500ms)
}, interval: 1m
| fieldsAdd sli = 100 * fast[] / total[]
We measure a ratio of fast requests rather than a percentile. A percentile does not add up over 28 days; a ratio does. Target: 95. Create this second SLO the same way.
Step 4 — Keep query cost under control
An SLO re-evaluates its query regularly. Querying every span of a high-traffic service is expensive in Grail reads. Two leaner options:
- extract a metric from the spans with OpenPipeline, then write the SLI with
timeseries; - use the OpenTelemetry metric
http.server.request.duration(histogram) if your SDK exports it.
Example with an extracted metric (name to adapt to your pipeline):
timeseries {
total = sum(otel.checkout.requests),
errors = sum(otel.checkout.requests, filter: { status == "error" })
}
| fieldsAdd sli = 100 * (total[] - errors[]) / total[]
See metrics and histograms with OpenPipeline for the extraction.
What about Dynatrace Managed?
Managed has neither Grail nor DQL. Ingested OpenTelemetry spans become service requests there: a classic SLO can then rely on a metric expression, for example (100)*(builtin:service.errors.server.successCount:splitBy())/(builtin:service.requestCount.server:splitBy()), filtered on the service. Check in the documentation for your version that the OpenTelemetry service does expose these metrics.
Step 5 — Read the error budget
With 99.5% over 28 days and 2 million requests in the window, the budget allows 10,000 failed requests. If 4,000 have already failed halfway through the window, 40% of the budget is consumed: the pace is healthy. If 8,000 have failed in one week, the team slows down releases and prioritises reliability. This rule, written in advance and agreed with product, avoids debates during the incident.
How to check that it works
- Run each SLI query in a Notebook over 24 hours: the
slicurve must stay between 0 and 100, with no gaps. - In the SLO app, the status and remaining error budget are displayed for both SLOs.
- Generate errors in a test environment (for example an endpoint that returns 500): the availability SLI must drop within minutes.
Best practices
- Filter on valid requests: exclude health checks and synthetic traffic from the denominator.
- Keep the latency threshold aligned with user experience, not with current performance.
- Normalise
service.namein the Collector so the SLO survives a deployment rename. - Add burn rate alerts at creation time: see SLO burn rate alerts in Dynatrace.
- To roll these SLOs out to all your services, version them: see generating Dynatrace SLOs with Terraform.
Frequently asked questions
Do I need OpenTelemetry metrics, or are spans enough?
Spans are enough to start. For a high-traffic service, a derived metric is cheaper and faster to evaluate.
Why 28 days rather than a month?
A 28-day window always contains four full weeks: each weekday weighs the same, which makes SLOs comparable from one period to the next.
What if the SLI goes above 100 or becomes empty?
An empty SLI signals no traffic in the interval (division by zero). An SLI above 100 signals a badly filtered numerator. Go back to step 1 and compare the counters.
Conclusion
Two DQL queries are enough to go from OpenTelemetry traces to availability and latency SLOs. Start with one critical service, validate the numbers in a Notebook, then add the burn rate alerts.
Recommended next step: present these two SLOs at a team review, together with the release freeze rule tied to the error budget. An SLO that product shares and understands is worth more than a perfect SLO that only on-call looks at. Then review the thresholds after a month of real data.
Keep the Notebook with your exploration queries: it becomes the reference when someone asks why the SLI dropped, and it shortens the next investigation.
span.observer is an independent community blog, not affiliated with Dynatrace LLC. Dynatrace is a trademark of Dynatrace LLC. SaaS features evolve quickly: check their availability in your environment and in the official documentation.
1 comment