Dynatrace vs Grafana SLO: a hands-on comparison with Prometheus

Dynatrace vs Grafana SLO: the same 99.5% SLO built with Prometheus, then in Dynatrace SaaS and Managed, to decide where your source of truth should live.
SLO Dynatrace Grafana : le comparatif avec Prometheus mis en pratique

Your team already runs Prometheus and Grafana, and management has just bought Dynatrace. The first question in the SRE meeting: where should reliability targets live? A Dynatrace vs Grafana SLO comparison on a feature sheet is useless; you need to implement the same SLO on both sides and look at what each costs in effort, maintenance and on-call noise.

Goal: build an identical availability SLO (99.5% over 30 days, HTTP requests without 5xx errors) first with Prometheus and Grafana, then in Dynatrace, and derive a decision grid. On the Dynatrace side, the article covers Dynatrace SaaS (Grail SLOs, DQL) and Dynatrace Managed (classic SLOs based on metric selectors).

For the basics of error budget and burn rate maths, keep SLO burn rate alerts at hand.

The problem: two sources of truth for one SLO

Two tools that compute the same availability with different numbers is the worst case: every incident turns into an argument about measurement instead of a blameless review. The choice is less about the tool than about where the reference SLI lives.

Prerequisites

  • Prometheus 2.x (or compatible) scraping a request metric, here http_server_requests_total with a code label; adapt to your instrumentation’s real name.
  • Grafana 10 or later, with the Prometheus data source.
  • A Dynatrace environment (SaaS or Managed) monitoring the same service through OneAgent or OpenTelemetry.
  • A Dynatrace API token with read and write rights on SLOs.

Step 1: the SLO on the Prometheus and Grafana side

Prometheus has no notion of SLO: you build it with recording rules. Here is the error ratio over two windows, then a fast burn alert.

# slo-checkout.rules.yml
groups:
  - name: slo-checkout
    rules:
      - record: slo:checkout_errors:ratio_rate5m
        expr: |
          sum(rate(http_server_requests_total{job="checkout",code=~"5.."}[5m]))
          /
          sum(rate(http_server_requests_total{job="checkout"}[5m]))
      - record: slo:checkout_errors:ratio_rate1h
        expr: |
          sum(rate(http_server_requests_total{job="checkout",code=~"5.."}[1h]))
          /
          sum(rate(http_server_requests_total{job="checkout"}[1h]))
      - alert: CheckoutFastBurn
        expr: |
          slo:checkout_errors:ratio_rate1h > (14.4 * 0.005)
          and
          slo:checkout_errors:ratio_rate5m > (14.4 * 0.005)
        for: 2m
        labels:
          severity: page
        annotations:
          summary: "Checkout is burning its error budget 14.4 times too fast"

Critical lines: 0.005 is the error budget of a 99.5% SLO. The double condition (1 h and 5 min) avoids paging for a spike that is already over. Validate the file before loading it:

promtool check rules slo-checkout.rules.yml

In Grafana, a Stat panel shows 30-day availability:

1 - (
  sum(increase(http_server_requests_total{job="checkout",code=~"5.."}[30d]))
  /
  sum(increase(http_server_requests_total{job="checkout"}[30d]))
)

A query over 30 days of raw data is expensive on a large Prometheus; in production, base it on intermediate recording rules. Generators such as Sloth or the Grafana Cloud SLO feature produce these rules from a specification; check what your Grafana offering includes.

Step 2: the same SLO in Dynatrace SaaS

On SaaS, the SLO relies on Grail. Start by validating the SLI in a Notebook with DQL:

timeseries {
    total = sum(dt.service.request.count),
    failed = sum(dt.service.request.failure_count)
  },
  filter: dt.entity.service == "SERVICE-0123456789ABCDEF",
  from: now()-30d
| fieldsAdd sli = 100 * (1 - (arraySum(failed) / arraySum(total)))
| fields sli

The dt.service.request.* metric names are the Grail service metrics; check them in your environment, as they depend on the source (OneAgent or OpenTelemetry). Then create the SLO in the SLO app from this query, with a 99.5 target and a 30-day window. The full walkthrough is in OpenTelemetry SLOs with DQL.

Step 3: the same SLO in Dynatrace Managed

On Managed, the SLO is a configuration object based on a metric expression, created in the UI or through the Environment API v2:

curl -s -X POST "$DT_ENV/api/v2/slo" \
  -H "Authorization: Api-Token $DT_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "enabled": true,
    "name": "checkout - availability",
    "metricName": "checkout_availability",
    "metricExpression": "(100)*(builtin:service.errors.server.successCount:splitBy())/(builtin:service.requestCount.server:splitBy())",
    "evaluationType": "AGGREGATE",
    "filter": "type(\"SERVICE\"),entityId(\"SERVICE-0123456789ABCDEF\")",
    "target": 99.5,
    "warning": 99.8,
    "timeframe": "-30d"
  }'

The body schema may vary with the cluster version: compare it with the reference on docs.dynatrace.com. To version these objects, prefer Dynatrace SLO with Terraform.

Dynatrace vs Grafana SLO: the hands-on comparison

Criterion Prometheus + Grafana Dynatrace
SLI definition PromQL, rules written by hand or generated DQL (SaaS) or metric selector (Managed)
Instrumentation Up to you (exporters, libraries, OpenTelemetry) Automatic OneAgent, or OpenTelemetry
Error budget calculation To build (rules, dashboards) Native in the SLO object
Burn rate alert Alertmanager, multi-window rules Metric event or SLO app alert
Link to root cause Manual, through traces and logs if correlated Davis problem, Smartscape topology
Main cost Engineering time, long-term storage Licence, consumption

SRE reading: Prometheus gives full control at the price of rule maintenance toil. Dynatrace reduces that toil and shortens MTTR through the direct link between an SLO violation and its probable cause, but you depend on its data model.

How to check that it works

  1. Prometheus: promtool check rules passes, then the /rules page shows the slo-checkout group as OK.
  2. Dynatrace SaaS: the DQL query returns a value, and the SLO tile shows a status and an error budget.
  3. Dynatrace Managed: GET /api/v2/slo?evaluate=true returns the SLO with a populated evaluatedPercentage.
  4. Comparison: over the same window, both availabilities should stay close. A gap of more than a few tenths of a point points to a different definition (HTTP codes counted, health checks included, service scope).

Best practices

  • One source of truth per SLO: the other tool displays, it does not recompute.
  • Document the SLI in one sentence (“checkout requests without 5xx, excluding health checks”) and apply it to both tools.
  • Alert from the tool that owns the SLO, to avoid double pages.
  • Version Prometheus rules and Dynatrace SLOs in the same repository.

Frequently asked questions

Can Dynatrace SLOs be shown in Grafana?

Yes, through a Dynatrace data source plugin for Grafana or through the metrics API. Check the plugin’s compatibility with your version and environment type.

Should we drop Prometheus when adopting Dynatrace?

No. Dynatrace can ingest Prometheus metrics. The question is where the SLO lives, not which tool to remove.

What should a small team choose?

A team with no time to maintain rules benefits from owning the SLO in the tool that computes the error budget natively. A team already fluent in PromQL can stay on Prometheus.

Conclusion

Implementing the same SLO on both sides settles the debate better than a feature list. Choose the tool that minimises your toil and MTTR, then stick to a single definition.

span.observer is an independent community blog, not affiliated with Dynatrace LLC. Dynatrace is a trademark of its owner. Grafana and Prometheus are trademarks of their respective owners.

Total
0
Shares
Leave a Reply

Your email address will not be published. Required fields are marked *

Previous Post
SLO Dynatrace Managed : configuration, limites et API (tutoriel)

Dynatrace Managed SLO: configuration, limits and API (tutorial)

Next Post
Réduire bruit, alertes et fatigue d'astreinte : régler Davis et les profils d'alerte

Reduce alert noise in Dynatrace: tune Davis and alerting profiles for calmer on-call

Related Posts