Dynatrace SLO Terraform: one SLO per service, generated at scale

Dynatrace SLO Terraform: generate one availability SLO per service from a YAML file with for_each, reviewed in a merge request, on SaaS and Managed.
SLO Dynatrace Terraform : un SLO pour chaque service, généré en masse

Three SLOs created by hand is manageable. Eighty services, each with an availability SLO, is toil: repeated clicks, diverging targets and SLOs forgotten for every new service. Managing your Dynatrace SLO Terraform configuration turns this manual work into a versioned list reviewed in a merge request.

Goal of this tutorial: generate one availability SLO per service from a single configuration file, with the Dynatrace Terraform provider and a for_each loop, then check the result. The content relies on classic SLOs (Settings 2.0), available on Dynatrace Managed and Dynatrace SaaS. One section flags the option of the new Grail SLOs on SaaS.

Before automating, make sure your indicators make sense: see how to choose SLIs.

The principle

  • A services.yaml file lists the services and their target.
  • Terraform reads this file with yamldecode and creates one SLO resource per entry.
  • Adding a service = adding three lines and passing a code review. Removing a line deletes the SLO.

Prerequisites

  • Terraform 1.5 or later.
  • The dynatrace-oss/dynatrace provider (Terraform registry).
  • An API token with the scopes the dynatrace_slo_v2 resource requires: slo.read, slo.write, settings.read and settings.write. Check the list in the provider documentation.
  • Services with a common tag, for example service:checkout, set by your tagging rules (see management zones and tagging).

Generating Dynatrace SLO Terraform resources step by step

Step 1 — Configure the provider

terraform {
  required_providers {
    dynatrace = {
      source = "dynatrace-oss/dynatrace"
    }
  }
}

provider "dynatrace" {}

The provider reads the URL and token from the environment. Never write them in the code:

# SaaS
export DYNATRACE_ENV_URL="https://<env-id>.live.dynatrace.com"
# Managed
export DYNATRACE_ENV_URL="https://<your-domain>/e/<env-id>"
export DYNATRACE_API_TOKEN="dt0c01.XXXX"

Pin a provider version (version = "~> 1.x" with the tested version) to avoid an unexpected schema change.

Step 2 — Describe the services in a file

# services.yaml
checkout:
  tag: "service:checkout"
  target: 99.9
  warning: 99.95
cart:
  tag: "service:cart"
  target: 99.5
  warning: 99.7
search:
  tag: "service:search"
  target: 99.5
  warning: 99.7

This file becomes the catalogue of your objectives. The product owner can review it without knowing Terraform.

Step 3 — Write the resource with for_each

locals {
  services = yamldecode(file("${path.module}/services.yaml"))
}

resource "dynatrace_slo_v2" "availability" {
  for_each = local.services

  name               = "Availability ${each.key}"
  enabled            = true
  custom_description = "Server requests without error / requests. Managed by Terraform."
  evaluation_type    = "AGGREGATE"
  evaluation_window  = "-30d"
  filter             = "type(\"SERVICE\"),tag(\"${each.value.tag}\")"
  metric_name        = "availability_${replace(each.key, "-", "_")}"
  metric_expression  = "(100)*(builtin:service.errors.server.successCount:splitBy())/(builtin:service.requestCount.server:splitBy())"
  target_success     = each.value.target
  target_warning     = each.value.warning

  error_budget_burn_rate {
    burn_rate_visualization_enabled = true
    fast_burn_threshold             = 14.4
  }
}

Critical lines:

  • for_each = local.services: one instance per YAML key, addressed by name (dynatrace_slo_v2.availability["checkout"]). Unlike count, removing a service does not shift the others.
  • filter: the entity selector restricts the SLO to services carrying the tag. Inner quotes are escaped.
  • metric_name: must be unique; it is used to expose the SLO as a metric.
  • fast_burn_threshold: aligned with the fast alert described in SLO burn rate alerts.

Attribute names depend on the provider version. The safest route: create an SLO by hand, export it with the provider’s export tool, then compare with your code.

Step 4 — Plan and apply

terraform init
terraform plan -out slo.plan
terraform apply slo.plan

The plan must announce exactly one creation per service in the YAML. Review it in a merge request before the apply, ideally from a CI pipeline.

What about the new Grail SLOs on SaaS?

On SaaS, SLOs from the Service-Level Objectives app (based on DQL) are managed by another provider resource, separate from dynatrace_slo_v2. Its name and attributes have changed between versions: check the provider documentation before using it. The for_each pattern stays the same; only the SLI definition changes, with a DQL query like those in creating an OpenTelemetry SLO.

Step 5 — Add a latency SLO to the same catalogue

The same pattern extends to a second SLI type. Add an optional latency_ms field in the YAML, then filter the map so the latency SLO is only created for services that declare it:

locals {
  latency_services = {
    for name, svc in local.services : name => svc
    if try(svc.latency_ms, null) != null
  }
}

A second dynatrace_slo_v2 resource with for_each = local.latency_services then reuses the step 3 structure, with a latency metric expression suited to your version (check the available response time metrics in the documentation). The catalogue stays single: one service, one entry, all its objectives.

How to check that it works

  1. terraform state list shows one resource per service: dynatrace_slo_v2.availability["cart"], and so on.
  2. A second terraform plan reports “No changes”. If not, an attribute is normalised by the API: align your code with the returned value.
  3. In the Service-level objectives screen, each SLO shows a status and an error budget, not “no data”. A “no data” almost always signals a missing or misspelt tag in filter.

Best practices

  • Store the Terraform state in a remote backend with locking, never on a laptop.
  • Forbid manual changes to managed SLOs: the “Managed by Terraform” note in the description helps the team.
  • Run terraform plan every night in CI to detect drift.
  • Keep targets in the YAML, not in the HCL: the discussion is about objectives, not code.
  • For the rest of the automation, see automating with the cluster API.

Frequently asked questions

Can I import SLOs already created by hand?

Yes, with an import block (Terraform 1.5+) or terraform import, using the setting’s identifier. Add the entry to the YAML first so the address exists.

How do I generate the list of services automatically?

The provider offers an entities data source that lists services. It is handy, but one SLO per service without human review quickly produces SLOs without an owner. Prefer a reviewed YAML.

Does this code work on an offline Managed cluster?

Yes, if the machine running Terraform can reach the cluster API and has the provider in a local mirror.

Conclusion

One YAML file, one resource and a for_each loop are enough to give every service an SLO, with code review and history. The toil disappears, and each new service arrives with its objective from day one.

Recommended next step: connect this repository to your CI pipeline, with an automatic plan on each merge request and a manual apply after approval. Start with three pilot services, check that the SLOs show real data, then open the catalogue to the other teams.

Keep the YAML catalogue short and readable: it is the document product and SRE will discuss each quarter.

span.observer is an independent community blog, not affiliated with Dynatrace LLC. Dynatrace is a trademark of Dynatrace LLC. SaaS features evolve quickly: check their availability in your environment and in the official documentation.

Total
0
Shares
Leave a Reply

Your email address will not be published. Required fields are marked *

Previous Post
Choisir ses SLI : la méthode pratique, des requêtes DQL aux objectifs

How to choose SLIs: a practical method, from DQL queries to objectives

Next Post
Budget d'erreur release gate : bloquer une mise en production avec Dynatrace (SaaS et Managed)

Error budget release gate: block a production release with Dynatrace (SaaS and Managed)

Related Posts