Keeping logs from eating the disk

A small VPS with a 20 GB disk ran out of space. Nothing had been installed recently. The cause was one log file that had grown for years without ever being rotated.

Find it

du -xh / 2>/dev/null | sort -rh | head -20
journalctl --disk-usage

Empty it without breaking the writer

Deleting a file that a daemon still has open doesn't free the space until the daemon restarts. Truncate it instead:

truncate -s 0 /var/log/big.log

Cap the journal

Create /etc/systemd/journald.conf.d/size.conf:

[Journal]
SystemMaxUse=200M
MaxRetentionSec=1month

systemctl restart systemd-journald
journalctl --vacuum-size=200M

Rotate the stray file

Anything that writes its own file under /var/log needs a logrotate rule, or it grows forever. /etc/logrotate.d/big:

/var/log/big.log {
    weekly
    rotate 4
    maxsize 50M
    compress
    missingok
    notifempty
    copytruncate
}

copytruncate is for programs that don't reopen their log on a signal. Check the rule with a dry run:

logrotate -d /etc/logrotate.d/big

Better still: if a program has a debug or verbose logging option turned on, turn it off. That's usually where the gigabytes come from.