Module 8 — OpenTelemetry AI observability: from OTLP traces to MCP agents

OpenTelemetry AI observability in Dynatrace SaaS: send OTLP data directly or via a Collector, trace LLM calls and tokens, and see where MCP agents fit in.
Module 8 : OpenTelemetry/IA, des traces OTLP aux agents MCP

Course: Dynatrace SaaS · Module 8/8 — OpenTelemetry AI observability · Duration: ~90 min · Level: advanced

Objectives:

  • Position OpenTelemetry relative to OneAgent and choose between the two, or combine them.
  • Send traces, metrics and logs over OTLP to Dynatrace, directly or through a Collector.
  • Read an LLM application trace and its attributes (model, tokens, latency).
  • Understand where MCP fits in an agent architecture and what to observe there.

Prerequisites: the whole course, especially modules 3 (DQL) and 6 (logs); some knowledge of Python or another instrumentable language; a trial environment and an API token.

This last training module covers OpenTelemetry and AI observability in Dynatrace SaaS: how to bring in standardised signals from applications that OneAgent does not instrument, then how to apply the same principles to generative AI applications, from model calls to agents that use MCP. The two topics are linked: LLM observability today relies heavily on OpenTelemetry.

By the end of this module, you will have covered the whole chain: collect, query, visualise, alert, process logs, automate and, finally, integrate open sources.

OpenTelemetry AI observability: why an open standard?

OpenTelemetry (OTel) is a CNCF project that defines APIs, SDKs, a protocol (OTLP) and naming conventions for traces, metrics and logs. Its benefit: instrument once and be able to send the data to the backend of your choice. The following table helps you decide:

Situation Recommended approach
Hosts and clusters you administer OneAgent, for automatic instrumentation and the full topology.
Code already instrumented with OTel OTLP export; OneAgent can also enrich the OTel spans of the processes it monitors.
Environments where no agent is possible (serverless, appliance, poorly covered language) OTel SDK and OTLP export.
LLM applications and agents Specialised OTel instrumentation libraries, alongside OneAgent on the infrastructure.

For a complete example of correlation between browser, services and hosts, see end-to-end OpenTelemetry.

Sending data over OTLP

Direct export

Dynatrace exposes an OTLP endpoint over HTTP. OTel exporters are configured with standard environment variables:

export OTEL_SERVICE_NAME="order-api"
export OTEL_EXPORTER_OTLP_ENDPOINT="https://<environment-id>.live.dynatrace.com/api/v2/otlp"
export OTEL_EXPORTER_OTLP_PROTOCOL="http/protobuf"
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Api-Token <token>"

The token must carry the ingest permissions matching the signals you send (traces, metrics, logs). Check the exact URL and permissions in the documentation: they differ depending on whether or not you go through an ActiveGate.

Through an OpenTelemetry Collector

In production, you usually put a Collector in between: it receives data from the applications, enriches it, filters it, batches it and exports it. It also centralises the token, which no longer has to be distributed to every service. Dynatrace publishes its own Collector distribution, but the community distribution works too. A configuration skeleton:

receivers:
  otlp:
    protocols:
      grpc:
      http:
processors:
  batch:
exporters:
  otlphttp:
    endpoint: "https://<environment-id>.live.dynatrace.com/api/v2/otlp"
    headers:
      Authorization: "Api-Token ${env:DT_API_TOKEN}"
service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlphttp]

A point to watch with metrics: Dynatrace expects delta temporality. If your SDKs produce cumulative metrics, configure them as delta or convert them in the Collector.

Querying what arrives

Once the data is received, it is queried in Grail like everything else:

fetch spans, from:now()-30m
| filter service.name == "order-api"
| summarize total = count(), avg_duration = avg(duration), by:{span.name}
| sort total desc

Observing an LLM application

A generative AI application raises new questions: which model answered, how many tokens were consumed, how long generation took, which tools the agent called. OpenTelemetry defines “GenAI” semantic conventions to standardise this information in spans, with attributes such as gen_ai.request.model, gen_ai.usage.input_tokens and gen_ai.usage.output_tokens. These conventions are still evolving: check the exact names your library produces.

Libraries such as OpenLLMetry automatically instrument the clients of the main providers and frameworks. In Dynatrace SaaS, these traces feed views dedicated to AI observability and remain queryable in DQL, for example to track token consumption by model:

fetch spans, from:now()-24h
| filter isNotNull(gen_ai.request.model)
| summarize input_tokens = sum(gen_ai.usage.input_tokens),
            output_tokens = sum(gen_ai.usage.output_tokens),
            by:{gen_ai.request.model}

The pillar article end-to-end AI observability details the quality and evaluation challenges, beyond performance alone.

And where does MCP fit in?

The Model Context Protocol (MCP) standardises the way an AI agent accesses tools and data. It comes into this course at two levels. First, as something to observe: each MCP tool call by an agent is a step that can fail or slow down, and it is worth tracing it as a span. Second, as an interface to Dynatrace: the Dynatrace MCP server lets an assistant query your data, for example run a DQL query or list problems, from an editor or a messaging tool. To get started, read the Model Context Protocol explained, then, for a concrete Dynatrace case, Dynatrace in Microsoft Teams with the MCP server.

Hands-on exercise

Step 1 — Create a token

In your trial environment, create an API token with OTLP ingest permissions for traces and metrics. Expected result: a token stored in an environment variable, never in the code.

Step 2 — Instrument a small application

Take a demo application (for example a small Python service) and enable OpenTelemetry automatic instrumentation for its language, with the OTEL_* variables above. Expected result: the application starts without export errors.

Step 3 — Check that spans arrive

Generate a few requests, then run the fetch spans query filtered on your service.name. Expected result: spans visible within a few minutes.

Step 4 — Add an LLM call

If you have a provider key or a local model, add a model call instrumented by a GenAI-compatible library. Expected result: a span carrying the model name and the number of tokens, found with the consumption query.

Quiz

  1. What is OpenTelemetry’s standard export protocol?
    A. SNMP · B. OTLP · C. Syslog · D. JMX
  2. What is the main advantage of a Collector between applications and Dynatrace?
    A. It replaces Grail · B. It centralises processing, batching and credentials · C. It removes the need for a token · D. It generates the dashboards
  3. Which metric temporality does Dynatrace expect over OTLP?
    A. Cumulative · B. Delta · C. It does not matter · D. Monthly
  4. What does MCP standardise?
    A. The log format · B. An AI agent’s access to tools and data · C. Metric storage · D. The network protocol between OneAgent and the cluster

Answers

  1. B — OTLP carries traces, metrics and logs over gRPC or HTTP.
  2. B — the Collector enriches, filters and batches the data and keeps the token in one place.
  3. B — cumulative metrics must be configured as delta or converted.
  4. B — MCP describes how an agent discovers and calls tools and data sources.

Further reading

← Module 7: Workflows and automation · Course overview · Back to the course overview →

Frequently asked questions

Do you have to choose between OneAgent and OpenTelemetry?

No. OneAgent remains the simplest option on infrastructure you control; OpenTelemetry covers the rest and brings portability. The two can be combined in the same environment.

Are LLM prompts and responses sent in the traces?

That depends on the library and its configuration. This content can be sensitive: disable its capture or mask it, for example in the Collector or with OpenPipeline, if your rules require it.

What comes after this course?

Consolidate on a real project: instrument a service, build its dashboard, its alerts and a workflow. The official training and certifications then let you validate what you have learnt.

This module closes the Dynatrace SaaS course. You now know how to bring standardised signals into Grail, link them to your services and monitor AI applications with the same tools as the rest of your system.

span.observer is an independent community blog, not affiliated with Dynatrace LLC. This course does not replace the official Dynatrace University training and certifications. Check features against the official documentation for your version.

Total
0
Shares
2 comments
Leave a Reply

Your email address will not be published. Required fields are marked *

Previous Post
Module 7 : Workflows, automatisation des opérations

Module 7 — Dynatrace Workflows automation for operations

Next Post
Configurer le collector OpenTelemetry Dynatrace : traces, métriques et logs en OTLP/HTTP

OpenTelemetry Collector Dynatrace setup: traces, metrics and logs over OTLP/HTTP

Related Posts