Monitor OpenTelemetry Collector internal telemetry in Dynatrace (SaaS and Managed)

Monitor OpenTelemetry Collector health: expose its internal telemetry, ship it to Dynatrace and alert on refused data, queue saturation and failed exports.
Superviser collector OpenTelemetry : sa télémétrie interne dans Dynatrace (SaaS et Managed)

Your OpenTelemetry Collector carries every trace, metric and log you rely on. When it saturates, refuses data or fails to export, you lose visibility without a single error on the application side. To monitor OpenTelemetry Collector health is to protect the pipeline that feeds your SLOs, your alerts and your incident analysis.

The goal of this tutorial: enable the Collector’s internal telemetry, send it to Dynatrace, then add three simple checks on refused data, failed exports and the sending queue. It applies to Dynatrace SaaS (metrics in Grail, DQL queries) and to Dynatrace Managed (OTLP ingest through ActiveGate, Data Explorer and classic metric events). Differences are flagged at each step.

If your Collector does not yet send data to Dynatrace, start with the OpenTelemetry Collector configuration guide for Dynatrace.

The problem: a pipeline blind to itself

A Collector that drops exports does not raise a visible incident. Spans vanish, charts show a dip, and the on-call engineer hunts for an application outage that does not exist. The result is a longer MTTR and SLOs computed on incomplete data. The good news: the Collector already exposes its own metrics. You only need to collect them.

Prerequisites

  • A recent OpenTelemetry Collector (the otelcol-contrib distribution or the Dynatrace distribution). This tutorial assumes version 0.100 or later; the service.telemetry syntax has changed over time, so check your version’s changelog.
  • A Dynatrace API token with the metrics.ingest scope (OTLP metric ingest).
  • Your environment URL: https://{environment-id}.live.dynatrace.com on SaaS, or https://{activegate}:9999/e/{environment-id} on Managed.

Step 1: expose the Collector’s internal telemetry

The service.telemetry block controls the Collector’s own telemetry. The most robust option is to expose metrics in Prometheus format locally, then read them back with the Prometheus receiver. That way they follow the same pipeline as your other metrics.

service:
  telemetry:
    logs:
      level: info
    metrics:
      level: detailed            # basic, normal or detailed
      readers:
        - pull:
            exporter:
              prometheus:
                host: 0.0.0.0
                port: 8888

Key lines: level: detailed adds useful queue and batch metrics; the pull reader publishes metrics on :8888/metrics. On older versions, the equivalent was address: 0.0.0.0:8888 under metrics.

Step 2: scrape those metrics and send them to Dynatrace

Add a Prometheus receiver that scrapes port 8888, then a dedicated pipeline. Dynatrace expects OTLP metrics with delta temporality: the cumulativetodelta processor converts Prometheus counters, which are cumulative. This is covered in the article on OTLP metric temporality.

receivers:
  prometheus/self:
    config:
      scrape_configs:
        - job_name: otelcol-self
          scrape_interval: 30s
          static_configs:
            - targets: ["127.0.0.1:8888"]

processors:
  cumulativetodelta: {}
  batch: {}

exporters:
  otlphttp/dynatrace:
    endpoint: https://{environment-id}.live.dynatrace.com/api/v2/otlp
    headers:
      Authorization: "Api-Token ${env:DT_API_TOKEN}"

service:
  pipelines:
    metrics/self:
      receivers: [prometheus/self]
      processors: [cumulativetodelta, batch]
      exporters: [otlphttp/dynatrace]

On Managed, replace the endpoint with https://{activegate}:9999/e/{environment-id}/api/v2/otlp. Keep the token in an environment variable, never in the file.

A classic trap: if the Dynatrace exporter fails, the internal metrics fail with it. Add an external check as well, for example the health_check extension probed by Kubernetes.

extensions:
  health_check:
    endpoint: 0.0.0.0:13133

service:
  extensions: [health_check]

Step 3: pick the metrics that matter

The Collector publishes dozens of series. For on-call, four families are enough:

  • otelcol_receiver_accepted_spans and otelcol_receiver_refused_spans: what comes in and what is refused (with _metric_points and _log_records equivalents);
  • otelcol_exporter_sent_spans and otelcol_exporter_send_failed_spans: what goes out and what fails;
  • otelcol_exporter_queue_size and otelcol_exporter_queue_capacity: how full the sending queue is;
  • otelcol_process_memory_rss and otelcol_process_cpu_seconds: process health.

Watch the naming: depending on the version and the Prometheus round trip, counters often carry a _total suffix (for example otelcol_exporter_send_failed_spans_total). Check the exact names on /metrics before writing queries.

How to check it works

Start locally, on the Collector host or pod:

curl -s http://127.0.0.1:8888/metrics | grep -E "otelcol_exporter_(sent|send_failed)_spans"
curl -s http://127.0.0.1:13133/

You should see non-zero counters and a health response. Then, on Dynatrace SaaS, open a Notebook and run:

timeseries failed = sum(otelcol_exporter_send_failed_spans_total),
           sent = sum(otelcol_exporter_sent_spans_total),
           by: { exporter },
           interval: 5m

If the query returns nothing, look up the real metric name with fetch dt.metrics | filter contains(metric.key, "otelcol"); availability of this command depends on your environment, see the DQL documentation. On Managed, open Data Explorer and type otelcol in the metric selector.

Step 4: alert on the right signals

Three alerts cover the essentials without noise:

  1. Failed exports: send_failed above zero for 10 minutes. That is real data loss.
  2. Queue nearly full: queue_size / queue_capacity above 80% for 10 minutes. The Collector will soon refuse data.
  3. Refused input: rising refused_spans, often the memory_limiter protecting the process.

On SaaS, build these alerts with Davis Anomaly Detection from a DQL query, for example:

timeseries q = max(otelcol_exporter_queue_size), cap = max(otelcol_exporter_queue_capacity),
           by: { exporter }, interval: 1m
| fieldsAdd ratio = q[] / cap[] * 100

On Managed, create a metric event (Settings, Anomaly detection, Metric events) on the ingested metric with a static threshold. Then route these alerts like any other, through an alerting profile or a Workflow, as described for burn rate alerts.

Best practices to monitor OpenTelemetry Collector instances

  • Isolate the internal pipeline: a dedicated metrics/self pipeline keeps a fault in your business metrics from hiding Collector health.
  • Tag each instance: add service.instance.id and the cluster name with the resource processor, so you know which Collector is struggling.
  • Size the queue: tune sending_queue and retry_on_failure on the exporter, then check the fill ratio under load.
  • Do not use detailed everywhere: it increases series volume. Keep it where it helps diagnosis.
  • Double the check: the health_check probe catches a dead Collector, which its own metrics cannot.

On Kubernetes, the same settings apply through Helm values: see deploying the OpenTelemetry Collector on Kubernetes. And if traces go missing while the Collector looks healthy, follow the missing OpenTelemetry traces checklist.

Frequently asked questions

Can the Collector send its internal telemetry directly over OTLP?

Yes, recent versions accept a periodic reader with an OTLP exporter under service.telemetry.metrics. The syntax still changes between versions; the Prometheus approach shown here is more stable. Check your version’s documentation on opentelemetry.io.

Are these metrics billed?

Yes, they are custom metrics ingested like any other. Volume stays low if you keep a 30-second interval and the normal level outside diagnosis.

Does the Dynatrace Collector distribution change anything?

The principle is the same: service.telemetry is part of the Collector core. Just check that the components used here (Prometheus receiver, cumulativetodelta processor) are included in your chosen distribution.

Conclusion

An unmonitored Collector is a silent single point of failure. With a dedicated internal pipeline, four metric families and three alerts, you know within minutes whether data loss comes from the Collector or the application. That means fewer false leads during incidents, and SLOs built on reliable data.

span.observer is an independent community blog, not affiliated with Dynatrace LLC. Dynatrace is a trademark of its owner.

Total
0
Shares
Leave a Reply

Your email address will not be published. Required fields are marked *

Previous Post
Attributs OpenTelemetry Dynatrace : service.name, enrichissement et allow-list

OpenTelemetry attributes Dynatrace uses: service.name, enrichment and allow-list

Next Post
Site Reliability Guardian : valider une release automatiquement sur vos SLO (Dynatrace SaaS)

Site Reliability Guardian tutorial: validate every release against your SLOs (Dynatrace SaaS)

Related Posts