Error budget release gate: block a production release with Dynatrace (SaaS and Managed)

Error budget release gate: a Dynatrace Workflow computes the remaining budget and blocks the CI/CD deployment when it runs low, on SaaS and Managed.
Budget d'erreur release gate : bloquer une mise en production avec Dynatrace (SaaS et Managed)

Friday, 4 pm: a release goes out while the service has already used 90% of its error budget for the month. Nobody checked, because that number lives on a dashboard the pipeline never reads. An error budget release gate fixes this: the CI/CD pipeline asks Dynatrace before it deploys and stops if the budget is too low.

Goal of this tutorial: build a release gate that computes a service’s remaining error budget, compares it with a threshold and returns a verdict your pipeline can act on. The Dynatrace SaaS version uses a Workflow with a DQL task and a JavaScript task. The Dynatrace Managed version, which has no Workflows app, reads the SLO through API v2.

This tutorial assumes an SLO already exists. If it does not, start with SLO burn rate alerts in Dynatrace, which walks through the budget maths.

How an error budget release gate works

The error budget is what the SLO allows you to miss. With 99.9% over 30 days, the budget is 0.1% of requests. While budget remains, the team ships. When it is nearly gone, the team slows down and prioritises reliability. This is the error budget policy described in the SRE literature.

The gate enforces that policy without a meeting:

  • Remaining budget ≥ 25%: the deployment continues.
  • Remaining budget < 25%: the deployment is blocked, except for reliability fixes approved by on-call.
  • High burn rate over the last hour: also blocked, because an incident is in progress.

The thresholds are examples. Agree them with the product team and write them into your error budget policy.

Prerequisites

  • SaaS: a Grail environment, access to the Workflows app, permission to read metrics. To start the Workflow from CI: an OAuth client or platform token allowed to run and read Workflows (scopes such as automation:workflows:run and automation:workflows:read; check the exact names in the documentation).
  • Managed: an existing classic SLO, an API token with the slo.read scope, and curl + jq on the CI runner.
  • A service monitored by OneAgent or OpenTelemetry, with steady traffic. For the OpenTelemetry route, see the OpenTelemetry Collector for Dynatrace.

Steps in Dynatrace SaaS

Step 1 — Write the budget DQL query

Test the query in a Notebook first. It computes the 30-day SLI, the remaining budget and the burn rate over the last hour:

timeseries {
    total = sum(dt.service.request.count),
    failures = sum(dt.service.request.failure_count)
  },
  from: now() - 30d,
  filter: { dt.entity.service == "SERVICE-1234567890ABCDEF" }
| fieldsAdd t = arraySum(total), f = arraySum(failures)
| fieldsAdd sli = 100 * (t - f) / t
| fieldsAdd budget_restant = 100 * (1 - (f / t) / 0.001)
| fields sli, budget_restant

Critical lines: 0.001 is the error rate a 99.9% SLO allows. budget_restant (remaining budget) is 100 when there were no errors and goes negative once the budget is exceeded. For the recent burn rate, duplicate the query with from: now() - 1h and compute (f / t) / 0.001. If you already have an SLO in the Service-Level Objectives app, keep the same SLI definition so the gate and the SLO show the same numbers.

Step 2 — Create the Workflow

  1. Open Workflows and create a Workflow named release-gate-checkout.
  2. Keep an on-demand trigger: the pipeline will start it.
  3. Add an Execute DQL Query task named budget with the query from step 1.
  4. Add a Run JavaScript task named verdict that depends on budget.

Code for the verdict task:

import { execution } from '@dynatrace-sdk/automation-utils';

const SEUIL = 25; // minimum remaining budget in %

export default async function ({ execution_id }) {
  const ex = await execution(execution_id);
  const res = await ex.result('budget');
  const ligne = res.records[0];
  if (!ligne || ligne.budget_restant === null) {
    throw new Error('No data: gate closed as a precaution');
  }
  const budget = Number(ligne.budget_restant);
  if (budget < SEUIL) {
    throw new Error(`Remaining budget ${budget.toFixed(1)} % < ${SEUIL} %: deployment blocked`);
  }
  return { budget, decision: 'go' };
}

Critical lines: an exception fails the task, and therefore the Workflow execution. The pipeline only has one state to read. Missing data closes the gate: a silent SLI must never count as a green light. The exact shape of the DQL task result may change; print it once with console.log(res) to confirm the records path.

Step 3 — Call the gate from the pipeline

A bash example that works in GitLab CI, GitHub Actions or Jenkins:

#!/usr/bin/env bash
set -euo pipefail
BASE="https://${DT_ENV}.apps.dynatrace.com/platform/automation/v1"
AUTH="Authorization: Bearer ${DT_PLATFORM_TOKEN}"

EXEC_ID=$(curl -sf -X POST "$BASE/workflows/${WF_ID}/run" \
  -H "$AUTH" -H "Content-Type: application/json" -d '{}' | jq -r '.id')

for i in $(seq 1 30); do
  STATE=$(curl -sf "$BASE/executions/${EXEC_ID}" -H "$AUTH" | jq -r '.state')
  case "$STATE" in
    SUCCESS) echo "Gate open"; exit 0 ;;
    ERROR|CANCELLED) echo "Gate closed ($STATE)"; exit 1 ;;
  esac
  sleep 10
done
echo "Timeout: gate closed"; exit 1

Critical lines: .id is the execution ID; .state goes through RUNNING and then a final state. Check the paths and state values in the Workflows documentation and the Automation API reference. The timeout also closes the gate, again as a precaution.

The equivalent step in Dynatrace Managed

Managed has no Workflows. The pipeline reads the classic SLO directly through Environment API v2:

SLO=$(curl -sf "https://${DT_DOMAIN}/e/${DT_ENV_ID}/api/v2/slo/${SLO_ID}?timeFrame=CURRENT" \
  -H "Authorization: Api-Token ${DT_API_TOKEN}")
BUDGET=$(echo "$SLO" | jq -r '.errorBudget')
echo "Error budget: $BUDGET"
# errorBudget is expressed in SLI percentage points (e.g. 0.08 against a 99.9 target)
awk -v b="$BUDGET" 'BEGIN { exit !(b >= 0.025) }' || { echo "Gate closed"; exit 1; }

The errorBudget field returns the margin between the SLI and the target, not a percentage of the initial budget. With a 99.9 target, the maximum margin is 0.1, so 25% of the budget is 0.025. Confirm the field’s meaning and the timeFrame parameter in the SLO API documentation for your cluster version.

How to check it works

  1. Passing case: run the Workflow by hand. The execution should succeed and the verdict result should show decision: "go" with the budget.
  2. Blocking case: temporarily set SEUIL to 101. The execution should fail with the “deployment blocked” message and the pipeline script should exit with code 1.
  3. No-data case: replace the service ID with one that does not exist. The gate should close.
  4. Audit trail: in Workflows, the executions tab keeps every verdict. On Managed, keep the CI job output.

Then restore the real values. A test that has never seen the gate closed proves nothing.

Best practices

  • Provide an explicit override (pipeline variable, manual approval) for reliability fixes: blocking the fix for an incident makes MTTR worse.
  • One SLO per user journey, not per microservice. A service’s gate reads the SLO of the journey it serves.
  • Combine remaining budget and recent burn rate: a large budget does not protect you from an incident in progress.
  • After a block, run a blameless review: the gate flags a risk, it does not name a culprit.
  • For richer checks (several objectives, before/after comparison), look at Site Reliability Guardian in the documentation, which covers the same need on SaaS.
  • Manage your Workflows as code: see the Workflows and automation module.

Frequently asked questions

Why not read the SLO directly in the SaaS Workflow?

You can, if your SLO exposes its values through DQL or a dedicated action. The explicit query has one advantage: the calculation is visible and testable in a Notebook. Keep a single SLI definition to avoid two versions of the truth.

Which threshold should block a release?

Start loose (25%, for example) and adjust after a few weeks. A threshold that is too strict blocks often, and the team ends up bypassing the gate.

Does the gate replace tests?

No. Tests validate the code you ship; the gate checks that the service can take on new risk. They complement each other.

Conclusion

An error budget release gate turns the error budget policy into an automatic rule: the pipeline reads Dynatrace and decides. Start with one critical service and a cautious threshold, test the blocking case, then extend to other teams. For the SRE culture behind this practice, read AI and reliability: what SREs need to know.

span.observer is an independent community blog, not affiliated with Dynatrace LLC. Dynatrace is a trademark of Dynatrace LLC. SaaS features evolve quickly: check their availability in your environment and in the official documentation.

Total
0
Shares
Leave a Reply

Your email address will not be published. Required fields are marked *

Previous Post
SLO Dynatrace Terraform : un SLO pour chaque service, généré en masse

Dynatrace SLO Terraform: one SLO per service, generated at scale

Next Post
OneAgent OpenTelemetry doublons : éviter les spans en double et le contexte cassé

OneAgent OpenTelemetry duplicate spans: how to avoid them and keep context intact

Related Posts