Istio Dynatrace: monitor the service mesh with Dynatrace (Istio traces and metrics)

Istio Dynatrace setup: export Envoy traces over OTLP through the Collector, collect istio_requests_total metrics and follow the mesh in Dynatrace with DQL.
Istio Dynatrace : superviser le service mesh avec Dynatrace (Istio, traces et métriques)

Your Kubernetes cluster runs Istio: every request goes through an Envoy proxy that knows its latency, its status code and its destination. Yet during an incident the team opens Dynatrace and sees only part of the path. An Istio Dynatrace setup deserves better: the data already exists in the mesh, it only needs to be routed. This Dynatrace Istio guide shows how.

The mesh is often the first place where degradation shows: cascading retries, rejected mTLS, a sidecar timeout. Without its traces and metrics, diagnosis relies on guesswork and MTTR grows.

The goal of this tutorial: send spans from the Envoy proxies over OTLP to Dynatrace through an OpenTelemetry Collector, collect Istio’s standard metrics, then check everything with DQL. It applies to Dynatrace SaaS and Dynatrace Managed (OTLP ingest on Managed depends on your version; DQL queries assume Grail, so SaaS).

Istio Dynatrace: what the mesh can provide

Signal Source in Istio Path to Dynatrace
Traces (proxy spans) Envoy, through an OpenTelemetry provider declared in the mesh configuration OTLP to the Collector, then to the Dynatrace OTLP API
Metrics The sidecars’ Prometheus endpoint (istio_requests_total, istio_request_duration_milliseconds…) Scraped by the Collector or by the Dynatrace Operator (annotations)
Access logs Envoy access logs Your usual log collection (OneAgent or Fluent Bit)

An important reminder: proxy spans alone cannot rebuild a complete trace. Each application must propagate the context headers (traceparent) from the incoming request to its outgoing calls, otherwise every hop becomes an isolated trace.

Prerequisites

  • Istio installed with istioctl (the telemetry.istio.io/v1 API is available in recent versions).
  • An OpenTelemetry Collector in the observability namespace, exposing OTLP gRPC on 4317, as in deploying the Collector on Kubernetes.
  • A Dynatrace API token with the openTelemetryTrace.ingest and metrics.ingest scopes, stored in a secret.
  • Applications that propagate W3C context (OpenTelemetry instrumentation or OneAgent).

Step 1: declare the Collector as a tracing provider

Add an OpenTelemetry extension provider to the mesh configuration:

cat <<EOF | istioctl install -y -f -
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  meshConfig:
    enableTracing: true
    extensionProviders:
    - name: otel-tracing
      opentelemetry:
        port: 4317
        service: otel-collector.observability.svc.cluster.local
EOF

The name otel-tracing is up to you; it is referenced in the next step. The service field must match the Collector service’s DNS name exactly.

Step 2: enable tracing with the Telemetry API

A Telemetry resource in Istio’s root namespace enables the provider for the whole mesh:

apiVersion: telemetry.istio.io/v1
kind: Telemetry
metadata:
  name: mesh-tracing
  namespace: istio-system
spec:
  tracing:
  - providers:
    - name: otel-tracing
    randomSamplingPercentage: 10

randomSamplingPercentage sets the share of requests traced by the proxy. Start low in production (1 to 10%) and tune by volume. If applications already make the sampling decision, the proxy honours the decision carried by the incoming header. For finer sampling, keep the decision in the Collector (see tail sampling).

Step 3: export to Dynatrace: Istio traces and metrics through the Collector

In the Collector configuration, add the Dynatrace OTLP HTTP exporter and a Prometheus receiver that scrapes the sidecars:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
  prometheus:
    config:
      scrape_configs:
      - job_name: istio-sidecars
        scrape_interval: 30s
        metrics_path: /stats/prometheus
        kubernetes_sd_configs:
        - role: pod
        relabel_configs:
        - source_labels: [__meta_kubernetes_pod_container_name]
          regex: istio-proxy
          action: keep
        - source_labels: [__address__]
          regex: ([^:]+)(?::\d+)?
          replacement: $$1:15090
          target_label: __address__

processors:
  batch: {}
  cumulativetodelta: {}

exporters:
  otlphttp/dynatrace:
    endpoint: https://${env:DT_ENV_ID}.live.dynatrace.com/api/v2/otlp
    headers:
      Authorization: "Api-Token ${env:DT_API_TOKEN}"

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlphttp/dynatrace]
    metrics:
      receivers: [prometheus]
      processors: [cumulativetodelta, batch]
      exporters: [otlphttp/dynatrace]

Critical lines: port 15090 exposes Envoy statistics in Prometheus format; $$ escapes $ in the Collector configuration; cumulativetodelta is needed because Dynatrace expects delta counters over OTLP. On Managed, the endpoint becomes https://<domain>/e/<env-id>/api/v2/otlp. The Collector’s service account must be allowed to list pods.

If you would rather let the Dynatrace Operator scrape, use the metrics.dynatrace.com/* annotations described in collecting Prometheus metrics on Kubernetes. Do not combine both methods on the same pods: the metrics would be counted twice.

How to check it works

  1. Generate some traffic, then look at the Collector logs: kubectl -n observability logs deploy/otel-collector | grep -iE "error|refused" should return nothing.
  2. Check that proxy spans are present:
    fetch spans, from: -15m
    | filter isNotNull(istio.canonical_service)
    | summarize spans = count(), by: { istio.canonical_service }
    | sort spans desc

    If the attribute has another name in your version, inspect a raw span with fetch spans | limit 5.

  3. Check the mesh metrics, split by service and status code:
    timeseries requests = sum(istio_requests_total), by: { destination_service, response_code }, from: -1h

    The share of 5xx series over the total gives the per-service error rate SLI. Adjust the metric key to the name you see in the metrics browser: depending on the ingest path, a suffix may be added.

Good practices

  • Base your SLOs on server-side mesh metrics (reporter="destination") to avoid counting each request twice.
  • Limit cardinality: drop useless labels before export rather than storing millions of series.
  • Keep a known, documented sampling rate; an unannounced change skews week-on-week comparisons.
  • Monitor the Collector itself: if it goes down, the mesh goes silent.

Frequently asked questions

Isn’t OneAgent enough without configuring Istio?

OneAgent instruments the application processes in the pods and propagates its own context, so it sees service-to-service calls. Envoy spans and metrics add the proxy’s view (retries, mTLS, sidecar timeouts). The level of native Envoy support depends on the version: check the documentation before duplicating signals.

Why are my traces broken at every hop?

Almost always because an application does not propagate traceparent from the incoming call to the outgoing one. The proxy does create its spans, but cannot link two requests the application has not linked.

Do these steps work in ambient mode?

Ambient mode replaces sidecars with ztunnel and waypoint proxies; ports and metric sources differ. The principle (OpenTelemetry provider, Collector, OTLP export) still holds, but adapt the scraping to that architecture.

Conclusion

An OpenTelemetry provider in the mesh configuration, a Telemetry resource, a Collector that scrapes the sidecars and exports over OTLP: the Istio mesh becomes visible in Dynatrace, with SLIs based on every request. The next retry storm will be read in a trace, not guessed.

span.observer is an independent community blog, not affiliated with Dynatrace LLC. Dynatrace is a trademark of its owner. Options and attributes change: check them in the Istio documentation and in the Dynatrace documentation.

Total
0
Shares
Leave a Reply

Your email address will not be published. Required fields are marked *

Previous Post
Fluent Bit Dynatrace : envoyer les logs Kubernetes pas à pas

Fluent Bit Dynatrace: shipping Kubernetes logs step by step

Next Post
Dynatrace Application Security (AppSec) : scan des vulnérabilités à l'exécution

Dynatrace Application Security (AppSec): runtime vulnerability scanning in production

Related Posts