tooolbox/blog/Java Memory Errors on Linux: Understanding and Taming the OOM Killer
troubleshooting2025-07-10·12 min·7/348

Java Memory Errors on Linux: Understanding and Taming the OOM Killer

A deep dive into Linux Out-Of-Memory killer behavior with Java applications, including diagnosis, prevention, and production-grade solutions.

Introduction

There is a special kind of panic that sets in when a production Java application vanishes from the process list without a trace. No stack trace, no graceful shutdown, no chance to flush logs. Just gone. If you have ever stared at a terminal wondering where your JVM went, chances are the Linux OOM Killer was involved. In this post, I will walk through how the OOM Killer interacts with Java, how to diagnose when it strikes, and how to prevent it from happening again.

Environment

The scenarios described here apply to Linux systems running Java 11 through 21, on kernels 4.x through 6.x. I have personally encountered these issues on Ubuntu 22.04 LTS, Rocky Linux 9, and Amazon Linux 2023. The examples assume a server with 8 GB of RAM running a Spring Boot application with moderate traffic.

uname -r
# 5.15.0-91-generic

java -version
# openjdk version "21.0.2" 2024-01-16

Problem

The first symptom is usually a missing Java process. You check your monitoring dashboards and see a gap. Your application is not responding. You SSH into the server and run:

ps aux | grep java
# No java processes found

Then you check the kernel logs:

dmesg | grep -i "oom\|killed"
# [12345.678901] java invoked oom-killer: gfp_mask=0xcc0(GFP_KERNEL), order=0, oom_score_adj=0
# [12345.678950] Out of memory: Killed process 4567 (java) total-vm:8388608kB, anon-rss:6291456kB

The OOM Killer selected your Java process because it was the largest memory consumer. The total-vm value of approximately 8 GB matches what you would expect from a JVM configured with -Xmx6g.

Analysis

The OOM Killer activates when the kernel determines that the system is critically low on memory and cannot reclaim enough through page cache eviction or swap. The kernel scores each process using oom_score and terminates the one with the highest score.

Java applications are frequent victims because:

  1. JVM heap memory is allocated as anonymous pages that cannot be easily reclaimed
  2. -Xmx settings often claim a large percentage of available RAM
  3. Native memory allocations (Netty buffers, JNI, mapped files) add to the footprint beyond what the heap reports

You can check the current OOM scores:

for pid in $(pgrep java); do
  echo "PID $pid: $(cat /proc/$pid/oom_score) ($(cat /proc/$pid/oom_score_adj))"
done

The kernel's memory behavior can be inspected:

cat /proc/meminfo | grep -E "MemTotal|MemFree|MemAvailable|SwapTotal|SwapFree"
# MemTotal:        8053064 kB
# MemFree:          123456 kB
# MemAvailable:     456789 kB
# SwapTotal:       2097152 kB
# SwapFree:        2000000 kB

If MemAvailable is consistently near zero, your system is a candidate for OOM termination.

Solution

There are several layers of defense:

1. Configure JVM memory correctly

Never allocate more than 70-75% of system RAM to the heap. For an 8 GB server, limit the heap to 5-6 GB and account for metaspace, thread stacks, and native buffers.

java -Xmx5g -Xms5g \
     -XX:MaxMetaspaceSize=512m \
     -XX:ReservedCodeCacheSize=256m \
     -jar application.jar

2. Set the OOM score adjustment

Protect critical processes by lowering their OOM score:

echo -1000 > /proc//oom_score_adj

Or in systemd:

[Service]
OOMScoreAdjust=-900

3. Enable cgroup memory limits

Use cgroups to isolate the JVM memory:

mkdir /sys/fs/cgroup/java-app
echo "6g" > /sys/fs/cgroup/java-app/memory.max
echo $JAVA_PID > /sys/fs/cgroup/java-app/cgroup.procs

4. Monitor and alert

Set up monitoring to alert when available memory drops below a threshold:

#!/bin/bash
AVAILABLE=$(grep MemAvailable /proc/meminfo | awk '{print $2}')
THRESHOLD=1048576  # 1GB in kB
if [ "$AVAILABLE" -lt "$THRESHOLD" ]; then
  echo "ALERT: Memory critically low" | mail -s "OOM Risk" admin@example.com
fi

Lessons Learned

The OOM Killer is not a bug. It is a safeguard that prevents the entire system from freezing. Treating it as such, rather than fighting it, leads to more resilient systems. Always size JVM heap relative to actual system capacity, monitor memory trends rather than just current usage, and understand that native memory allocations exist outside the heap. A well-tuned Java application on Linux should rarely, if ever, be killed.

This blog does not accept any external sponsorships, affiliate marketing, or ad revenue.