Recovering from 100% Disk Usage on Linux

A full disk on a production Linux server can bring everything to a halt. Databases stop writing, applications crash, and you cannot even SSH in sometimes. I learned this the hard way when a logging misconfiguration filled up a 50GB volume overnight. This guide covers the emergency recovery process and long-term prevention.

Environment

• OS: Ubuntu 20.04 LTS
• Disk: 50GB root partition, 200GB data volume
• Trigger: Uncontrolled debug logging from a Python application

$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1        50G   50G     0 100% /
/dev/sdb1       200G   45G  145G  24% /data

The Problem

The server became unresponsive. SSH connections dropped, cron jobs stopped running, and the application threw database connection errors:

$ ssh admin@server
ssh_exchange_identification: read: Connection reset by peer

# After forcing a connection via console:
$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1        50G   50G     0 100% /

$ sudo systemctl status postgresql
● postgresql.service - PostgreSQL Cluster 14
     Active: failed (Result: exit-code)
     ...
Jul 15 03:22:01 server postgresql[1234]: FATAL: could not write to file "pg_wal/...": No space left on device

Analysis

Step 1: Identify What Is Consuming Space

# Top-level overview
$ sudo du -sh /* 2>/dev/null | sort -rh | head -10
18G     /var
12G     /opt
8.5G    /usr
5.2G    /home
3.1G    /tmp

The /var directory is the clear culprit at 18GB.

Step 2: Drill Down into /var

$ sudo du -sh /var/* 2>/dev/null | sort -rh | head -10
15G     /var/log
2.1G    /var/lib
800M    /var/cache

Logs are consuming 15GB inside a 50GB partition.

Step 3: Find the Largest Log Files

$ sudo find /var/log -type f -size +100M -exec ls -lh {} \;
-rw-r----- 1 syslog adm 4.2G Jul 15 03:22 /var/log/syslog
-rw-r----- 1 syslog adm 3.8G Jul 15 03:22 /var/log/syslog.1
-rw-r----- 1 www-data www-data 2.1G Jul 15 03:22 /var/log/nginx/access.log
-rw-r--r-- 1 root root 1.5G Jul 15 03:22 /var/log/journal/...

Step 4: Check for Deleted-but-Open Files

This is a critical step many people miss. Even after deleting a file, if a process still holds it open, the disk space is not freed:

$ sudo lsof +L1
COMMAND    PID  USER   FD   TYPE DEVICE SIZE/OFF NLINK NODE NAME
syslogd   1234  syslog  1w  REG    8,1  4482134016  0  /var/log/syslog (deleted)
nginx     5678  www-data 2w  REG  8,1  2254856192  0  /var/log/nginx/access.log (deleted)

This confirms the problem. Files were deleted but processes still hold references.

Solution

Step 1: Emergency Space Recovery (Immediate)

Truncate the largest files without restarting services:

# Truncate syslog (keeps file handle, frees space)
$ sudo truncate -s 0 /var/log/syslog
$ sudo truncate -s 0 /var/log/syslog.1

# Truncate nginx access log
$ sudo truncate -s 0 /var/log/nginx/access.log

# Verify space freed
$ df -h /
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1        50G   32G   18G  64% /

Step 2: Clean Up Old Logs

# Remove old rotated logs
$ sudo find /var/log -name "*.gz" -type f -delete
$ sudo find /var/log -name "*.1" -type f -delete
$ sudo find /var/log -name "*.old" -type f -delete

# Clean journal logs older than 3 days
$ sudo journalctl --vacuum-time=3d

# Clean apt cache
$ sudo apt-get clean
$ sudo apt-get autoremove

# Clean tmp
$ sudo find /tmp -type f -atime +7 -delete

Step 3: Identify What Caused the Log Explosion

$ grep -c "DEBUG" /var/log/syslog
12847593

$ grep "DEBUG" /var/log/syslog | head -5
Jul 15 03:20:01 server myapp[9999]: DEBUG: Processing record 12847591
Jul 15 03:20:01 server myapp[9999]: DEBUG: Processing record 12847592
Jul 15 03:20:01 server myapp[9999]: DEBUG: Processing record 12847593

A debug logging statement was left enabled in production. Each record generated a log line, and the application processed millions of records daily.

Step 4: Fix the Configuration

# Before (debug logging enabled)
import logging
logging.basicConfig(level=logging.DEBUG)

# After (proper log level for production)
import logging
logging.basicConfig(level=logging.WARNING)

Step 5: Set Up Log Rotation

$ sudo cat > /etc/logrotate.d/myapp << 'EOF'
/var/log/myapp/*.log {
    daily
    rotate 7
    compress
    delaycompress
    missingok
    notifempty
    size 100M
    postrotate
        systemctl reload myapp > /dev/null 2>&1 || true
    endscript
}
EOF

Step 6: Add Disk Space Monitoring

#!/bin/bash
# /usr/local/bin/disk-monitor.sh

THRESHOLD=80
USAGE=$(df / --output=pcent | tail -1 | tr -d '% ')
HOSTNAME=$(hostname)

if [ "$USAGE" -gt "$THRESHOLD" ]; then
    echo "WARNING: Disk usage at ${USAGE}% on ${HOSTNAME}" | \
        mail -s "Disk Alert: ${HOSTNAME}" admin@example.com
fi

Add to crontab:

$ crontab -e
0 */6 * * * /usr/local/bin/disk-monitor.sh

Lessons Learned

1. Never use DEBUG logging in production without a size-bounded log rotation policy.
2. Monitor disk usage proactively with alerts at 70%, 80%, and 90% thresholds.
3. Check lsof +L1 when disk space does not free after deletion - deleted-but-open files are a common trap.
4. Use truncate -s 0 instead of rm for files held open by running processes.
5. Separate partitions for /var/log to prevent log explosions from killing the root filesystem.

This blog does not accept any external sponsorships, affiliate marketing, or ad revenue.