A deployment goes to production and, a few minutes later, kubectl get pods shows the same status on several replicas: CrashLoopBackOff. Just above it, a last-state field reads OOMKilled. This guide covers the CrashLoopBackOff OOMKilled Dynatrace workflow for one of the most frequent Kubernetes incidents, and one of the most poorly diagnosed: someone raises the memory limit and the problem comes back the following week.
Every restart drops in-flight requests, empties caches and burns the service’s error budget. The back-off delay grows with each failure: the longer you wait, the more available capacity melts away. A methodical diagnosis cuts MTTR and stops a memory leak from being disguised as a simple YAML tweak.
The goal of this tutorial: tell the causes of a CrashLoopBackOff apart, confirm an OOMKilled, then use Dynatrace to find the affected workloads, compare consumption with limits and alert before the outage. The DQL queries apply to Dynatrace SaaS (Grail, Kubernetes app). On Dynatrace Managed, the same approach goes through the classic Kubernetes pages and the data explorer.
CrashLoopBackOff and OOMKilled: two different signals
CrashLoopBackOff is not a cause: it is the waiting state of a container that the kubelet restarts after successive failures, with a growing delay. The cause lies elsewhere. OOMKilled is one of them: the kernel killed the process because the container exceeded its memory limit (exit code 137).
| Last termination reason | Exit code | Lead |
|---|---|---|
OOMKilled |
137 | Memory limit too low, memory leak, load spike |
Error |
1 or other | Start-up exception, missing configuration or secret |
Completed |
0 | Process that exits when it should keep running |
| Failed liveness probe | 137 or 143 | Probe too aggressive, slow start-up |
Exit code 137 alone is not enough: a container killed by a liveness probe also receives a signal. Only the OOMKilled reason confirms a memory overrun.
Prerequisites
kubectlwith read access to the namespace concerned.- The cluster monitored by Dynatrace Operator (see Connecting Dynatrace Operator to a cluster), with Kubernetes monitoring enabled.
- On SaaS: access to Notebooks and the Kubernetes app. On Managed: access to the Kubernetes pages and the data explorer.
Diagnosing a CrashLoopBackOff OOMKilled Dynatrace case step by step
Step 1: read the container state
kubectl get pods -n shop
kubectl describe pod payment-7d9c6b5f8-x2kq4 -n shop
In the describe output, look for the container’s Last State block: Reason: OOMKilled and Exit Code: 137 confirm the memory hypothesis. Also note Restart Count and the Limits section. To go straight to the point:
kubectl get pod payment-7d9c6b5f8-x2kq4 -n shop \
-o jsonpath='{.status.containerStatuses[*].lastState.terminated.reason}'
Step 2: read the logs of the previous run
The current container has just restarted: its logs are nearly empty. The ones that matter belong to the killed instance.
kubectl logs payment-7d9c6b5f8-x2kq4 -n shop --previous
An OOMKilled rarely leaves a clear error message: the kernel kills the process without warning. A start-up exception (missing variable, refused connection), on the other hand, shows up here and points to another cause.
Step 3: find every affected workload with DQL
kubectl shows one pod. Dynatrace shows the whole estate. In a SaaS Notebook, the dt.kubernetes.container.oom_kills metric counts out-of-memory terminations:
timeseries oom = sum(dt.kubernetes.container.oom_kills, rollup:sum),
by:{k8s.namespace.name, k8s.workload.name},
from:-7d
| fieldsAdd total = arraySum(oom)
| filter total > 0
| sort total desc
rollup:sum adds up the events in each interval; arraySum reduces the series to one total per workload. A single workload at the top of the list points to an application problem; several workloads on the same node suggest node pressure instead.
Step 4: compare consumption with the limit
Overlay the memory actually used (working set) and the configured limit:
timeseries {
used = sum(dt.kubernetes.container.memory_working_set),
limit = sum(dt.kubernetes.container.limits_memory)
},
by:{k8s.workload.name},
filter: k8s.namespace.name == "shop",
from:-24h
| fieldsAdd ratio = used[] / limit[] * 100
Read the shape of the curve, not just its peak:
- steady climb up to the limit, then a drop: memory leak. Raising the limit only delays the next OOMKilled;
- stable plateau close to the limit: limit undersized for normal load;
- short spikes correlated with traffic: sizing to review against peak load, or an unbounded cache.
For the CPU equivalent, the article Effective CPU requests of Kubernetes pods explains the logic of effective resources.
Step 5: read the Kubernetes events
Kubernetes events ingested into Grail give you the timeline (back-off, probe failures, evictions):
fetch events, from:-24h
| filter event.provider == "KUBERNETES_EVENT"
| summarize n = count(), by:{event.name}
| sort n desc
Then add a filter on the namespace or pod to rebuild the incident sequence. Exact event field names may vary: display one raw record first to spot them.
Step 6: fix, then protect
Depending on the diagnosis:
resources:
requests:
memory: "512Mi"
limits:
memory: "768Mi"
- Undersized limit: adjust
requestsandlimitsbased on the working set observed at peak load, with headroom. - Managed runtime (JVM, Node.js): align the heap size with the container limit, otherwise the runtime believes it has the whole node’s memory.
- Memory leak: open an application ticket with the Dynatrace chart attached. OneAgent memory profiling, where available for the technology, helps locate allocations.
How to check that it works
kubectl get pods -n shop -w: theRESTARTScolumn must stop increasing once the fix is deployed.- The step 3 query, run again over the last few hours, no longer returns the workload.
- The step 4 ratio stays below a comfortable threshold (for example 80%) during a traffic peak.
- In the Kubernetes app (SaaS) or the workload page (Managed), the workload no longer has an open health problem.
Good practices
- Alert on the trend, not the incident: an alert on the working set / limit ratio above 90% warns you before the first OOMKilled. Keep it rare and actionable so it does not add to the noise.
- Always set memory limits: a container without a limit can push the whole node into memory pressure and get its neighbours evicted.
- Realistic probes: a
startupProbestops an overly strict liveness probe from killing a slow-starting service, which looks like a CrashLoopBackOff without OOM. - Blameless postmortem: keep the investigation’s DQL queries in a shared Notebook; the next on-call shift will save time. Log pattern analysis helps spot the messages that precede the termination.
Frequently asked questions
Why is the pod OOMKilled when the node has free memory?
Because the limit applies to the container, not to the node. As soon as the container exceeds its own limit, the kernel kills it, even if the node has several gigabytes free.
The CrashLoopBackOff status disappears and comes back: what is happening?
The kubelet restarts the container after each back-off delay. While it runs, the status switches back to Running. Look at the restart count and the last termination reason rather than the instantaneous status.
Do these queries work on Dynatrace Managed?
No, DQL syntax and Grail are specific to SaaS. On Managed, use the workload’s Kubernetes pages, the data explorer and the events shown on the pod entity.
Conclusion
A CrashLoopBackOff OOMKilled is diagnosed in three moves: confirm the reason with kubectl, measure the scale and shape of consumption with Dynatrace, then fix the real cause. Add a trend alert and the next outage becomes a sizing ticket, not a night-time wake-up call.
span.observer is an independent community blog, not affiliated with Dynatrace LLC. Dynatrace is a trademark of its owner. Metric and screen names change: check them in the official documentation and the Kubernetes documentation.