OpenTelemetry Python FastAPI: instrument an API for Dynatrace without OneAgent

OpenTelemetry Python FastAPI: send traces, metrics and logs over OTLP to Dynatrace without OneAgent, on SaaS or Managed, then verify ingestion.
OpenTelemetry Python FastAPI : instrumenter une API vers Dynatrace sans OneAgent

A FastAPI service that slows down without leaving a trail is an on-call nightmare: the logs say “200 OK”, the users say “it’s slow”. Instrumenting OpenTelemetry Python FastAPI gives you traces, metrics and shared context, without installing OneAgent on the host, which is often impossible on managed containers or serverless platforms.

Goal: by the end of this tutorial, your FastAPI service sends its spans and metrics over OTLP to Dynatrace, directly or through a Collector, and you know how to verify ingestion. It applies to Dynatrace SaaS (Grail storage, DQL queries) and to Dynatrace Managed (the environment’s OTLP endpoint, analysis in the classic screens). Differences are flagged at each step.

If your spans do not show up at the end, keep the troubleshooting guide OpenTelemetry traces not showing in Dynatrace close at hand.

The problem: an opaque Python API

Without instrumentation, a slow request looks like any other. You see neither the 800 ms outbound HTTP call nor the unindexed SQL query. MTTR grows because the on-call engineer rebuilds the path by hand. With traces, the duration of every step is one click away, and you can build a reliable latency SLI.

Prerequisites

  • Python 3.10 or later (minimum required by current OpenTelemetry SDK versions), FastAPI and Uvicorn.
  • A Dynatrace SaaS or Managed environment with OTLP ingestion available.
  • An API access token with the openTelemetryTrace.ingest, metrics.ingest and logs.ingest scopes (confirm the exact names in your token creation screen).
  • Optional: an OpenTelemetry Collector, recommended in production (see the OpenTelemetry Collector on Kubernetes).

Instrumenting OpenTelemetry Python FastAPI step by step

Step 1: a minimal application

# main.py
from fastapi import FastAPI
import httpx

app = FastAPI()

@app.get("/health")
async def health():
    return {"status": "ok"}

@app.get("/orders/{order_id}")
async def get_order(order_id: int):
    async with httpx.AsyncClient() as client:
        r = await client.get("https://httpbin.org/delay/1")
    return {"order_id": order_id, "upstream": r.status_code}

Step 2: install the OpenTelemetry packages

pip install fastapi uvicorn httpx \
  opentelemetry-distro \
  opentelemetry-exporter-otlp-proto-http

# installs the instrumentations matching the libraries it detects
# (fastapi, httpx, requests, sqlalchemy...)
opentelemetry-bootstrap -a install

Critical point: use the HTTP/protobuf exporter. The Dynatrace OTLP endpoint accepts OTLP over HTTP; gRPC goes through a Collector.

Step 3: configure export to Dynatrace

export OTEL_SERVICE_NAME="orders-api"
export OTEL_RESOURCE_ATTRIBUTES="service.version=1.4.2,deployment.environment.name=prod"

# SaaS
export OTEL_EXPORTER_OTLP_ENDPOINT="https://<environment-id>.live.dynatrace.com/api/v2/otlp"
# Managed (instead of the line above)
# export OTEL_EXPORTER_OTLP_ENDPOINT="https://<your-domain>/e/<environment-id>/api/v2/otlp"

export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Api-Token%20<your-token>"
export OTEL_EXPORTER_OTLP_PROTOCOL="http/protobuf"
export OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE="delta"
export OTEL_TRACES_EXPORTER="otlp"
export OTEL_METRICS_EXPORTER="otlp"
export OTEL_LOGS_EXPORTER="otlp"
# logs: the opentelemetry-instrumentation-logging package (installed by bootstrap)
# attaches the handler by default in recent versions

Three lines deserve attention:

  • OTEL_EXPORTER_OTLP_HEADERS: the space between Api-Token and the token is encoded as %20, otherwise the header is misread.
  • OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE=delta: Dynatrace expects delta temporality; cumulative metrics are rejected. Details in OTLP metrics temporality.
  • OTEL_SERVICE_NAME: this is the service name in Dynatrace. Without it you get unknown_service.

The exporter appends /v1/traces, /v1/metrics and /v1/logs to the base endpoint itself. Do not write them into OTEL_EXPORTER_OTLP_ENDPOINT.

For logs, the old OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED=true variable enables the SDK LoggingHandler, deprecated since opentelemetry-sdk 1.40. With recent versions, leave it unset: the logging instrumentation installs its own handler (can be disabled with OTEL_PYTHON_LOG_AUTO_INSTRUMENTATION=false). On older versions, the old variable is still required.

Step 4: start the instrumented application

opentelemetry-instrument uvicorn main:app --host 0.0.0.0 --port 8000

Do not use --reload with auto-instrumentation: the restarted child process does not always inherit the instrumentation. With several workers (Gunicorn), initialise the SDK after the fork, in a post_fork hook, otherwise the background exporters misbehave.

Step 5 (alternative): explicit instrumentation in code

If you prefer to control initialisation, for example to add business spans, configure the SDK in code:

# telemetry.py
from opentelemetry import trace
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor
from opentelemetry.instrumentation.httpx import HTTPXClientInstrumentor

def setup_tracing(app):
    resource = Resource.create({
        "service.name": "orders-api",
        "service.version": "1.4.2",
    })
    provider = TracerProvider(resource=resource)
    # endpoint and headers are read from the OTEL_EXPORTER_OTLP_* variables
    provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter()))
    trace.set_tracer_provider(provider)
    FastAPIInstrumentor.instrument_app(app, excluded_urls="health")
    HTTPXClientInstrumentor().instrument()
# in main.py
from opentelemetry import trace
from telemetry import setup_tracing

setup_tracing(app)
tracer = trace.get_tracer("orders")

@app.post("/orders")
async def create_order(payload: dict):
    with tracer.start_as_current_span("validate-order") as span:
        span.set_attribute("order.items", len(payload.get("items", [])))
    return {"created": True}

Critical line: excluded_urls="health" removes health probes from traces. They skew latency percentiles and cost volume for nothing.

Step 6: go through a Collector in production

Point the application at the local Collector (http://otel-collector:4318) and let the Collector hold the token. You gain batching, retries and a single configuration point. The Collector then exports with otlphttp:

exporters:
  otlphttp/dynatrace:
    endpoint: "https://<environment-id>.live.dynatrace.com/api/v2/otlp"
    headers:
      Authorization: "Api-Token ${env:DT_API_TOKEN}"

How to check that it works

  1. Application side: for a local test, set OTEL_TRACES_EXPORTER=otlp,console. Spans are printed to standard output; if they appear there but not in Dynatrace, the issue is the export (URL, token, proxy).
  2. Generate traffic: for i in $(seq 1 20); do curl -s localhost:8000/orders/$i; done.
  3. Dynatrace SaaS: open the Distributed Tracing app, filter on the orders-api service, or run in a notebook:
    fetch spans
    | filter service.name == "orders-api"
    | summarize count(), avg(duration), by: {span.name}
    
  4. Dynatrace Managed: open the distributed traces view (exact name depends on your version) and filter on the service. The OTel service may take a few minutes to appear.
  5. HTTP errors: a 401 or 403 in the exporter logs points to a wrong token or scope; a 404 to a wrong endpoint URL.

Best practices

  • Do not mix OneAgent and the OTel SDK on the same process without thinking: duplicates are likely (see OneAgent and OpenTelemetry duplicate spans).
  • Set service.name, service.version and the environment: they make deployments comparable and speed up diagnosis.
  • Keep the token out of the code: environment variable or Kubernetes secret.
  • Link your logs to traces to jump from an error to its request: see OpenTelemetry logs with trace_id.
  • Build a latency SLI on server spans, then a burn rate alert.

Frequently asked questions

Do I need OneAgent to see a FastAPI application in Dynatrace?

No. OTLP export is enough for traces, metrics and logs. OneAgent additionally brings automatic host and process discovery; without it, enrichment relies on the resource attributes you provide.

Why do my metrics not arrive when traces do?

Usually the temporality is still cumulative or the token lacks the metrics ingest scope. Check the temporality preference variable and the scopes.

Does auto-instrumentation work with Gunicorn and several workers?

Yes, provided the SDK is initialised in each worker after the fork. The OpenTelemetry Python documentation covers this case; test it before production.

Conclusion

A few environment variables and one launch command are enough to make a FastAPI API observable in Dynatrace, on SaaS and Managed alike. Start with auto-instrumentation, verify ingestion, then add business spans where on-call needs them. For endpoint details, refer to the Dynatrace OpenTelemetry documentation and the OpenTelemetry Python documentation.

span.observer is an independent community blog, not affiliated with Dynatrace LLC. Dynatrace is a trademark of its owner.

Total
0
Shares
Leave a Reply

Your email address will not be published. Required fields are marked *

Previous Post
Workflow Dynatrace Jira ou ServiceNow : ouvrir un ticket automatiquement sur problème

Dynatrace workflow Jira integration: open a ticket automatically on a problem (and ServiceNow)

Next Post
Migrer Managed vers SaaS : plan, outils et pièges de la bascule Dynatrace

Migrate Dynatrace Managed to SaaS: plan, tools and pitfalls

Related Posts