Your Kubernetes cluster runs Istio: every request goes through an Envoy proxy that knows its latency, its status code and its destination. Yet during an incident the team opens Dynatrace and sees only part of the path. An Istio Dynatrace setup deserves better: the data already exists in the mesh, it only needs to be routed. This Dynatrace Istio guide shows how.
The mesh is often the first place where degradation shows: cascading retries, rejected mTLS, a sidecar timeout. Without its traces and metrics, diagnosis relies on guesswork and MTTR grows.
The goal of this tutorial: send spans from the Envoy proxies over OTLP to Dynatrace through an OpenTelemetry Collector, collect Istio’s standard metrics, then check everything with DQL. It applies to Dynatrace SaaS and Dynatrace Managed (OTLP ingest on Managed depends on your version; DQL queries assume Grail, so SaaS).
Istio Dynatrace: what the mesh can provide
| Signal | Source in Istio | Path to Dynatrace |
|---|---|---|
| Traces (proxy spans) | Envoy, through an OpenTelemetry provider declared in the mesh configuration | OTLP to the Collector, then to the Dynatrace OTLP API |
| Metrics | The sidecars’ Prometheus endpoint (istio_requests_total, istio_request_duration_milliseconds…) |
Scraped by the Collector or by the Dynatrace Operator (annotations) |
| Access logs | Envoy access logs | Your usual log collection (OneAgent or Fluent Bit) |
An important reminder: proxy spans alone cannot rebuild a complete trace. Each application must propagate the context headers (traceparent) from the incoming request to its outgoing calls, otherwise every hop becomes an isolated trace.
Prerequisites
- Istio installed with
istioctl(thetelemetry.istio.io/v1API is available in recent versions). - An OpenTelemetry Collector in the
observabilitynamespace, exposing OTLP gRPC on 4317, as in deploying the Collector on Kubernetes. - A Dynatrace API token with the
openTelemetryTrace.ingestandmetrics.ingestscopes, stored in a secret. - Applications that propagate W3C context (OpenTelemetry instrumentation or OneAgent).
Step 1: declare the Collector as a tracing provider
Add an OpenTelemetry extension provider to the mesh configuration:
cat <<EOF | istioctl install -y -f -
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
meshConfig:
enableTracing: true
extensionProviders:
- name: otel-tracing
opentelemetry:
port: 4317
service: otel-collector.observability.svc.cluster.local
EOF
The name otel-tracing is up to you; it is referenced in the next step. The service field must match the Collector service’s DNS name exactly.
Step 2: enable tracing with the Telemetry API
A Telemetry resource in Istio’s root namespace enables the provider for the whole mesh:
apiVersion: telemetry.istio.io/v1
kind: Telemetry
metadata:
name: mesh-tracing
namespace: istio-system
spec:
tracing:
- providers:
- name: otel-tracing
randomSamplingPercentage: 10
randomSamplingPercentage sets the share of requests traced by the proxy. Start low in production (1 to 10%) and tune by volume. If applications already make the sampling decision, the proxy honours the decision carried by the incoming header. For finer sampling, keep the decision in the Collector (see tail sampling).
Step 3: export to Dynatrace: Istio traces and metrics through the Collector
In the Collector configuration, add the Dynatrace OTLP HTTP exporter and a Prometheus receiver that scrapes the sidecars:
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
prometheus:
config:
scrape_configs:
- job_name: istio-sidecars
scrape_interval: 30s
metrics_path: /stats/prometheus
kubernetes_sd_configs:
- role: pod
relabel_configs:
- source_labels: [__meta_kubernetes_pod_container_name]
regex: istio-proxy
action: keep
- source_labels: [__address__]
regex: ([^:]+)(?::\d+)?
replacement: $$1:15090
target_label: __address__
processors:
batch: {}
cumulativetodelta: {}
exporters:
otlphttp/dynatrace:
endpoint: https://${env:DT_ENV_ID}.live.dynatrace.com/api/v2/otlp
headers:
Authorization: "Api-Token ${env:DT_API_TOKEN}"
service:
pipelines:
traces:
receivers: [otlp]
processors: [batch]
exporters: [otlphttp/dynatrace]
metrics:
receivers: [prometheus]
processors: [cumulativetodelta, batch]
exporters: [otlphttp/dynatrace]
Critical lines: port 15090 exposes Envoy statistics in Prometheus format; $$ escapes $ in the Collector configuration; cumulativetodelta is needed because Dynatrace expects delta counters over OTLP. On Managed, the endpoint becomes https://<domain>/e/<env-id>/api/v2/otlp. The Collector’s service account must be allowed to list pods.
If you would rather let the Dynatrace Operator scrape, use the metrics.dynatrace.com/* annotations described in collecting Prometheus metrics on Kubernetes. Do not combine both methods on the same pods: the metrics would be counted twice.
How to check it works
- Generate some traffic, then look at the Collector logs:
kubectl -n observability logs deploy/otel-collector | grep -iE "error|refused"should return nothing. - Check that proxy spans are present:
fetch spans, from: -15m | filter isNotNull(istio.canonical_service) | summarize spans = count(), by: { istio.canonical_service } | sort spans descIf the attribute has another name in your version, inspect a raw span with
fetch spans | limit 5. - Check the mesh metrics, split by service and status code:
timeseries requests = sum(istio_requests_total), by: { destination_service, response_code }, from: -1hThe share of
5xxseries over the total gives the per-service error rate SLI. Adjust the metric key to the name you see in the metrics browser: depending on the ingest path, a suffix may be added.
Good practices
- Base your SLOs on server-side mesh metrics (
reporter="destination") to avoid counting each request twice. - Limit cardinality: drop useless labels before export rather than storing millions of series.
- Keep a known, documented sampling rate; an unannounced change skews week-on-week comparisons.
- Monitor the Collector itself: if it goes down, the mesh goes silent.
Frequently asked questions
Isn’t OneAgent enough without configuring Istio?
OneAgent instruments the application processes in the pods and propagates its own context, so it sees service-to-service calls. Envoy spans and metrics add the proxy’s view (retries, mTLS, sidecar timeouts). The level of native Envoy support depends on the version: check the documentation before duplicating signals.
Why are my traces broken at every hop?
Almost always because an application does not propagate traceparent from the incoming call to the outgoing one. The proxy does create its spans, but cannot link two requests the application has not linked.
Do these steps work in ambient mode?
Ambient mode replaces sidecars with ztunnel and waypoint proxies; ports and metric sources differ. The principle (OpenTelemetry provider, Collector, OTLP export) still holds, but adapt the scraping to that architecture.
Conclusion
An OpenTelemetry provider in the mesh configuration, a Telemetry resource, a Collector that scrapes the sidecars and exports over OTLP: the Istio mesh becomes visible in Dynatrace, with SLIs based on every request. The next retry storm will be read in a trace, not guessed.
span.observer is an independent community blog, not affiliated with Dynatrace LLC. Dynatrace is a trademark of its owner. Options and attributes change: check them in the Istio documentation and in the Dynatrace documentation.