Your OpenTelemetry Collector carries every trace, metric and log you rely on. When it saturates, refuses data or fails to export, you lose visibility without a single error on the application side. To monitor OpenTelemetry Collector health is to protect the pipeline that feeds your SLOs, your alerts and your incident analysis.
The goal of this tutorial: enable the Collector’s internal telemetry, send it to Dynatrace, then add three simple checks on refused data, failed exports and the sending queue. It applies to Dynatrace SaaS (metrics in Grail, DQL queries) and to Dynatrace Managed (OTLP ingest through ActiveGate, Data Explorer and classic metric events). Differences are flagged at each step.
If your Collector does not yet send data to Dynatrace, start with the OpenTelemetry Collector configuration guide for Dynatrace.
The problem: a pipeline blind to itself
A Collector that drops exports does not raise a visible incident. Spans vanish, charts show a dip, and the on-call engineer hunts for an application outage that does not exist. The result is a longer MTTR and SLOs computed on incomplete data. The good news: the Collector already exposes its own metrics. You only need to collect them.
Prerequisites
- A recent OpenTelemetry Collector (the
otelcol-contribdistribution or the Dynatrace distribution). This tutorial assumes version 0.100 or later; theservice.telemetrysyntax has changed over time, so check your version’s changelog. - A Dynatrace API token with the
metrics.ingestscope (OTLP metric ingest). - Your environment URL:
https://{environment-id}.live.dynatrace.comon SaaS, orhttps://{activegate}:9999/e/{environment-id}on Managed.
Step 1: expose the Collector’s internal telemetry
The service.telemetry block controls the Collector’s own telemetry. The most robust option is to expose metrics in Prometheus format locally, then read them back with the Prometheus receiver. That way they follow the same pipeline as your other metrics.
service:
telemetry:
logs:
level: info
metrics:
level: detailed # basic, normal or detailed
readers:
- pull:
exporter:
prometheus:
host: 0.0.0.0
port: 8888
Key lines: level: detailed adds useful queue and batch metrics; the pull reader publishes metrics on :8888/metrics. On older versions, the equivalent was address: 0.0.0.0:8888 under metrics.
Step 2: scrape those metrics and send them to Dynatrace
Add a Prometheus receiver that scrapes port 8888, then a dedicated pipeline. Dynatrace expects OTLP metrics with delta temporality: the cumulativetodelta processor converts Prometheus counters, which are cumulative. This is covered in the article on OTLP metric temporality.
receivers:
prometheus/self:
config:
scrape_configs:
- job_name: otelcol-self
scrape_interval: 30s
static_configs:
- targets: ["127.0.0.1:8888"]
processors:
cumulativetodelta: {}
batch: {}
exporters:
otlphttp/dynatrace:
endpoint: https://{environment-id}.live.dynatrace.com/api/v2/otlp
headers:
Authorization: "Api-Token ${env:DT_API_TOKEN}"
service:
pipelines:
metrics/self:
receivers: [prometheus/self]
processors: [cumulativetodelta, batch]
exporters: [otlphttp/dynatrace]
On Managed, replace the endpoint with https://{activegate}:9999/e/{environment-id}/api/v2/otlp. Keep the token in an environment variable, never in the file.
A classic trap: if the Dynatrace exporter fails, the internal metrics fail with it. Add an external check as well, for example the health_check extension probed by Kubernetes.
extensions:
health_check:
endpoint: 0.0.0.0:13133
service:
extensions: [health_check]
Step 3: pick the metrics that matter
The Collector publishes dozens of series. For on-call, four families are enough:
otelcol_receiver_accepted_spansandotelcol_receiver_refused_spans: what comes in and what is refused (with_metric_pointsand_log_recordsequivalents);otelcol_exporter_sent_spansandotelcol_exporter_send_failed_spans: what goes out and what fails;otelcol_exporter_queue_sizeandotelcol_exporter_queue_capacity: how full the sending queue is;otelcol_process_memory_rssandotelcol_process_cpu_seconds: process health.
Watch the naming: depending on the version and the Prometheus round trip, counters often carry a _total suffix (for example otelcol_exporter_send_failed_spans_total). Check the exact names on /metrics before writing queries.
How to check it works
Start locally, on the Collector host or pod:
curl -s http://127.0.0.1:8888/metrics | grep -E "otelcol_exporter_(sent|send_failed)_spans"
curl -s http://127.0.0.1:13133/
You should see non-zero counters and a health response. Then, on Dynatrace SaaS, open a Notebook and run:
timeseries failed = sum(otelcol_exporter_send_failed_spans_total),
sent = sum(otelcol_exporter_sent_spans_total),
by: { exporter },
interval: 5m
If the query returns nothing, look up the real metric name with fetch dt.metrics | filter contains(metric.key, "otelcol"); availability of this command depends on your environment, see the DQL documentation. On Managed, open Data Explorer and type otelcol in the metric selector.
Step 4: alert on the right signals
Three alerts cover the essentials without noise:
- Failed exports:
send_failedabove zero for 10 minutes. That is real data loss. - Queue nearly full:
queue_size / queue_capacityabove 80% for 10 minutes. The Collector will soon refuse data. - Refused input: rising
refused_spans, often thememory_limiterprotecting the process.
On SaaS, build these alerts with Davis Anomaly Detection from a DQL query, for example:
timeseries q = max(otelcol_exporter_queue_size), cap = max(otelcol_exporter_queue_capacity),
by: { exporter }, interval: 1m
| fieldsAdd ratio = q[] / cap[] * 100
On Managed, create a metric event (Settings, Anomaly detection, Metric events) on the ingested metric with a static threshold. Then route these alerts like any other, through an alerting profile or a Workflow, as described for burn rate alerts.
Best practices to monitor OpenTelemetry Collector instances
- Isolate the internal pipeline: a dedicated
metrics/selfpipeline keeps a fault in your business metrics from hiding Collector health. - Tag each instance: add
service.instance.idand the cluster name with theresourceprocessor, so you know which Collector is struggling. - Size the queue: tune
sending_queueandretry_on_failureon the exporter, then check the fill ratio under load. - Do not use
detailedeverywhere: it increases series volume. Keep it where it helps diagnosis. - Double the check: the
health_checkprobe catches a dead Collector, which its own metrics cannot.
On Kubernetes, the same settings apply through Helm values: see deploying the OpenTelemetry Collector on Kubernetes. And if traces go missing while the Collector looks healthy, follow the missing OpenTelemetry traces checklist.
Frequently asked questions
Can the Collector send its internal telemetry directly over OTLP?
Yes, recent versions accept a periodic reader with an OTLP exporter under service.telemetry.metrics. The syntax still changes between versions; the Prometheus approach shown here is more stable. Check your version’s documentation on opentelemetry.io.
Are these metrics billed?
Yes, they are custom metrics ingested like any other. Volume stays low if you keep a 30-second interval and the normal level outside diagnosis.
Does the Dynatrace Collector distribution change anything?
The principle is the same: service.telemetry is part of the Collector core. Just check that the components used here (Prometheus receiver, cumulativetodelta processor) are included in your chosen distribution.
Conclusion
An unmonitored Collector is a silent single point of failure. With a dedicated internal pipeline, four metric families and three alerts, you know within minutes whether data loss comes from the Collector or the application. That means fewer false leads during incidents, and SLOs built on reliable data.
span.observer is an independent community blog, not affiliated with Dynatrace LLC. Dynatrace is a trademark of its owner.