Tracking Memory Leaks on Linux: Tools and Techniques

Memory leaks are silent killers of long-running applications. A Python data pipeline I maintain started consuming 8GB of RAM after running for 3 days. Finding the leak required a systematic approach with the right Linux tools. This post documents that process.

Environment

• OS: Ubuntu 22.04 LTS
• Application: Python 3.10 data pipeline (with C extensions)
• Symptoms: RSS grows from 200MB to 8GB+ over 72 hours

$ python3 --version
Python 3.10.12

$ free -h
              total        used        free      shared  buff/cache   available
Mem:           15Gi       4.2Gi       8.1Gi       256Mi       3.0Gi       10Gi
Swap:         2.0Gi          0B       2.0Gi

The Problem

The application's memory usage grew linearly over time:

# Day 1
$ ps aux | grep pipeline | grep -v grep
joel     12345  2.1  4.5 2100000 920000 ?  Sl   09:00   1:23 /usr/bin/python3 pipeline.py

# Day 2
$ ps aux | grep pipeline | grep -v grep
joel     12345  3.5  22.1 4200000 4500000 ?  Sl   09:00   8:45 /usr/bin/python3 pipeline.py

# Day 3
$ ps aux | grep pipeline | grep -v grep
joel     12345  4.2  45.3 8400000 9200000 ?  Sl   09:00  15:23 /usr/bin/python3 pipeline.py

Analysis

Step 1: Monitor Memory Growth Over Time

Create a monitoring script:

#!/bin/bash
# /opt/scripts/mem-monitor.sh

PID=$1
LOGFILE="/var/log/mem-monitor-${PID}.csv"

echo "timestamp,rss_mb,vsz_mb,mem_percent" > "$LOGFILE"

while kill -0 "$PID" 2>/dev/null; do
    TIMESTAMP=$(date +%Y-%m-%d_%H:%M:%S)
    PS_OUTPUT=$(ps -p "$PID" -o rss=,vsz=,%mem= 2>/dev/null)
    RSS=$(echo "$PS_OUTPUT" | awk '{print $1}')
    VSZ=$(echo "$PS_OUTPUT" | awk '{print $2}')
    MEM=$(echo "$PS_OUTPUT" | awk '{print $3}')
    
    RSS_MB=$((RSS / 1024))
    VSZ_MB=$((VSZ / 1024))
    
    echo "${TIMESTAMP},${RSS_MB},${VSZ_MB},${MEM}" >> "$LOGFILE"
    sleep 60
done

Run it:

$ sudo /opt/scripts/mem-monitor.sh 12345 &
$ cat /var/log/mem-monitor-12345.csv | tail -5
timestamp,rss_mb,vsz_mb,mem_percent
2025-07-05_14:00:00,450,4200,2.9
2025-07-05_14:01:00,452,4200,2.9
2025-07-05_14:02:00,455,4200,3.0
2025-07-05_14:03:00,458,4200,3.0
2025-07-05_14:04:00,461,4200,3.1

Step 2: Examine Memory Maps with pmap

# Show memory mapping of the process
$ sudo pmap -x 12345 | tail -20
Address           Kbytes     RSS   Dirty Mode  Mapping
00007f8a1c000000   10240   10200   10200 rw---   [ anon ]
00007f8a22000000    8192    8180    8180 rw---   [ anon ]
00007f8a2a000000    4096    4088    4088 rw---   [ anon ]
00007f8a30000000    2048    2040    2040 rw---   [ anon ]
...
total kB         8400000 8200000 8100000

Large anonymous memory regions that keep growing indicate the leak.

Step 3: Check /proc/meminfo for the Process

# Memory status of the process
$ sudo cat /proc/12345/status | grep -i mem
VmPeak:  8600000 kB
VmSize:  8400000 kB
VmRSS:   8200000 kB
VmData:  8100000 kB
VmStk:      8192 kB
VmExe:      4096 kB
VmLib:    120000 kB

VmData growing while VmLib stays constant points to heap allocation growth.

Step 4: Use valgrind for C Extensions

If the leak is in C code:

# Build with debug symbols
$ gcc -g -O0 -o my_extension my_extension.c

# Run with valgrind
$ valgrind --leak-check=full --show-leak-kinds=all --track-origins=yes \
    ./my_extension 2>&1 | head -50
==12345== Memcheck, a memory error detector
==12345== HEAP SUMMARY:
==12345==     in use at exit: 4,096 bytes in 1 blocks
==12345==   total heap usage: 1,234 allocs, 1,233 frees, 892,160 bytes allocated
==12345==
==12345== 4,096 bytes in 1 blocks are definitely lost in loss record 1 of 1
==12345==    at 0x4C2FB0F: malloc (vg_replace_malloc.c:309)
==12345==    by 0x401234: process_data (my_extension.c:45)
==12345==    by 0x401300: main (my_extension.c:78)

Step 5: Use AddressSanitizer for Runtime Detection

# Compile with AddressSanitizer
$ gcc -fsanitize=address -g -o my_extension my_extension.c

# Run - ASan will report leaks automatically
$ ./my_extension
=================================================================
==12345==ERROR: AddressSanitizer: heap-buffer-overflow on address ...
    ...
==12345==HINT: this could be a classic malloc() bug

Step 6: Python-Specific Memory Profiling

# Install memory profiler
# pip install memory_profiler

from memory_profiler import profile

@profile
def process_batch(data):
    result = transform(data)  # Memory usage tracked here
    return result

# Run with: python -m memory_profiler pipeline.py

Output:

Line #    Mem usage    Increment   Line Contents
================================================
     3    200.0 MiB    200.0 MiB   @profile
     4                         def process_batch(data):
     5    450.0 MiB    250.0 MiB       result = transform(data)
     6    200.0 MiB   -250.0 MiB       return result  # Memory not freed here?

Solution: Fix the Leak

The leak was in a circular reference in the data pipeline:

# BEFORE: Circular reference prevents garbage collection
class DataProcessor:
    def __init__(self):
        self.cache = []
        self.parent = self  # Circular reference!
    
    def process(self, data):
        self.cache.append(data)
        return self  # Returns self, accumulating references

# AFTER: Break the circular reference
class DataProcessor:
    def __init__(self):
        self.cache = []
    
    def process(self, data):
        self.cache.append(data)
        if len(self.cache) > 1000:
            self.cache = self.cache[-500:]  # Trim cache
        return data  # Return data, not self

Also fixed a C extension leak:

// BEFORE: Memory allocated but never freed
void process_record(Record *rec) {
    char *buffer = malloc(rec->size);
    memcpy(buffer, rec->data, rec->size);
    // buffer is never freed!
}

// AFTER: Free after use
void process_record(Record *rec) {
    char *buffer = malloc(rec->size);
    memcpy(buffer, rec->size, rec->data, rec->size);
    // ... process buffer ...
    free(buffer);
}

Lessons Learned

1. Monitor RSS growth over time before diving into profiling tools - it establishes the pattern.
2. Use pmap -x to identify which memory regions are growing.
3. valgrind is excellent for C/C++ but too slow for production - use it in development.
4. Python circular references are a common cause of leaks that gc.collect() can help diagnose.
5. Set memory limits on services using systemd's MemoryMax= to prevent OOM from killing the host.

This blog does not accept any external sponsorships, affiliate marketing, or ad revenue.