Your security team lives in Splunk, your SRE team increasingly works in Dynatrace, and management asks whether you need both. The Dynatrace vs Splunk question then comes back at every contract renewal, usually phrased too broadly: “which one is best?”.
The two vendors do not start from the same place. Splunk was built around indexing and searching machine data (the SPL language), then added an observability offering, Splunk Observability Cloud, and APM with Splunk AppDynamics. Dynatrace starts from APM and application topology, and today stores traces, metrics, logs and events in Grail, queried with DQL.
This article offers a method to compare on evidence: a scorecard of measurable criteria, an OpenTelemetry proof of concept that sends the same logs to both platforms, and questions to ask before signing. No prices are quoted: they depend on your volumes and your contract. On the Dynatrace side, the proof of concept targets Dynatrace SaaS; differences with Dynatrace Managed are flagged.
Dynatrace vs Splunk: two different starting points
Summarising each product fairly avoids comparing a log search tool with an APM platform. The table below is based on both vendors’ public documentation.
| Criterion | Dynatrace | Splunk |
|---|---|---|
| Products in scope | One platform: Dynatrace SaaS (Grail) or Dynatrace Managed (self-hosted) | Splunk Enterprise (self-hosted), Splunk Cloud Platform, Splunk Observability Cloud, Splunk AppDynamics |
| Collection | OneAgent, Dynatrace Operator on Kubernetes, native OTLP ingestion | Forwarders and HTTP Event Collector (HEC) for the Splunk platform; Splunk Distribution of the OpenTelemetry Collector for Observability Cloud |
| Query language | DQL on Grail (SaaS); metric selector and classic search in Managed | SPL and SPL2 for the Splunk platform; SignalFlow for Observability Cloud |
| Root cause analysis | Causal analysis based on the Smartscape topology: Dynatrace Intelligence in SaaS, Davis AI in the Managed documentation | Correlation through searches, dashboards and detectors; ITSI for the service view |
| Hosting | SaaS, or Managed in your own data centre | Self-hosted Splunk Enterprise, or cloud offerings |
In practice: if your main need is log search for security and audit, the comparison is mostly with Splunk Enterprise or Cloud Platform. If it is APM and incident diagnosis, it is with Splunk Observability Cloud or AppDynamics, covered in the Dynatrace vs AppDynamics comparison. For the same method applied to another vendor, see Dynatrace vs Datadog.
Proof of concept prerequisites
- A Dynatrace SaaS trial tenant and a Splunk instance (test Enterprise or Cloud Platform) with HEC enabled and a dedicated index.
- A test Kubernetes cluster with an application instrumented with OpenTelemetry.
- An OpenTelemetry Collector, contrib distribution, which includes the
splunk_hecexporter (see deploying the Collector on Kubernetes). - A Dynatrace access token with the
logs.ingestscope (plusopenTelemetryTrace.ingestandmetrics.ingestif you also test traces and metrics), and a Splunk HEC token.
Step 1: duplicate logs to both platforms
The comparison only makes sense on identical data. The Collector receives logs over OTLP and sends them both to the Dynatrace OTLP endpoint and to the Splunk HTTP Event Collector.
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
http:
endpoint: 0.0.0.0:4318
processors:
batch: {}
exporters:
otlphttp/dynatrace:
endpoint: https://${env:DT_ENV_ID}.live.dynatrace.com/api/v2/otlp
headers:
Authorization: "Api-Token ${env:DT_API_TOKEN}"
splunk_hec:
token: "${env:SPLUNK_HEC_TOKEN}"
endpoint: "https://splunk.example.internal:8088/services/collector"
source: "otel"
sourcetype: "otel"
index: "poc_observabilite"
service:
pipelines:
logs:
receivers: [otlp]
processors: [batch]
exporters: [otlphttp/dynatrace, splunk_hec]
Critical lines:
Authorization: "Api-Token …": a classic Dynatrace token is passed with theApi-Tokenprefix. In Managed, the URL becomeshttps://{your-domain}/e/{your-environment-id}/api/v2/otlp.- HEC
endpoint: the default port is 8088 on Splunk Enterprise; on Splunk Cloud Platform, the Splunk documentation states 443 by default. Check your own instance. index: an index dedicated to the proof of concept keeps this data apart from production and makes volume easier to measure.
Before starting the Collector, test HEC on its own. The Splunk documentation requires the Splunk keyword before the token in the header:
curl https://splunk.example.internal:8088/services/collector/event \
-H "Authorization: Splunk ${SPLUNK_HEC_TOKEN}" \
-d '{"event": "test POC observability"}'
The expected response is {"text": "Success", "code": 0}. To go further on Kubernetes log collection with Dynatrace, see Fluent Bit and Dynatrace.
Step 2: ask the same question in SPL and in DQL
The query language is the on-call engineer’s daily tool: compare it on real questions. Example: “how many errors per service over the last hour?”. In Dynatrace SaaS, with DQL:
fetch logs, from: now() - 1h
| filter loglevel == "ERROR"
| summarize errors = count(), by:{service.name}
| sort errors desc
Write the SPL equivalent against the proof-of-concept index, then score readability, run time and how easily the result becomes a chart or an alert. More useful DQL queries are collected in our operational DQL examples.
One criterion to test: moving from a trace to its related logs without writing a query. Do the same journey in Splunk and time it.
Step 3: measurable criteria
- Time to diagnosis: inject a failure (slow dependency, HTTP 500 after a deployment) and measure the time to root cause, by someone who does not know the scenario.
- Trace–log correlation: do logs carry the
trace_id, and can you navigate from one to the other? See OpenTelemetry logs with trace_id. - Query effort: number of queries on-call engineers get right first time.
- Volume and retention: daily ingested volume for the same stream, configured retention, cost projected from the usage screens.
- Uses beyond observability: if security (SIEM) depends on Splunk, this criterion often outweighs all the others.
How to check that it works
- Dynatrace:
fetch logs, from: now() - 15m | summarize count()returns a non-zero count after sending test logs. - Splunk: a search on the
poc_observabiliteindex over the same period returns a comparable volume. - The Collector shows no export error:
kubectl logs deploy/otel-collector | grep -i error. - If volumes diverge, check filters and sampling before drawing conclusions.
Best practices
- Separate the “security logs” decision from the “application observability” decision: the users are not the same.
- Keep OpenTelemetry as the instrumentation foundation: the Collector lets you change destination without touching code.
- Have the scorecard approved before the proof of concept by the on-call team and by security.
- Ask for written offers on the same projected volume; for Dynatrace, cross-check them with DPS consumption tracking.
- Write a blameless, factual report, with no “winner” chosen in advance.
Frequently asked questions
Can we keep Splunk for security and Dynatrace for observability?
Yes, it is a common architecture. The Collector can route each stream to the right destination. The hidden cost is double operational effort and the risk of diagnoses split across two tools: decide which logs go where, and document it.
Does Splunk Observability Cloud accept OpenTelemetry?
Yes. Splunk publishes its own Collector distribution, under the Apache 2.0 licence, which exports to Splunk APM and Splunk Infrastructure Monitoring. Dynatrace also offers its distribution and accepts standard OTLP.
Which one is cheaper?
There is no general answer: the billing models do not use the same units. Compare two written offers on the same scope and project them over three years.
Conclusion
Dynatrace vs Splunk is not a contest between two identical products: Splunk has historically excelled at machine data search, Dynatrace at APM and topology. A proof of concept fed by the same Collector, questions asked in both languages and a measurable scorecard lead to a defensible decision, even when the right answer is to keep both for distinct uses.
span.observer is an independent community blog, not affiliated with Dynatrace LLC or Splunk. Dynatrace and Splunk are trademarks of their respective owners. Features and offerings change: check them in each vendor’s official documentation (Dynatrace, OTLP API, Splunk, HTTP Event Collector).