Self Hosted (part 4) : Logs

Self Hosted
linuxlog rotationlogsred hatrhel

Intro

This post will show how to make journald logs persist on the hard drive and how to rotate SELinux audit logs.

Persist journald logs

By default, journald will not persist logs on the hard drive. The logs are saved in the volatile memory (/run/log/journald) and are lost on system reboot which makes debugging harder when system reboots due to errors.
There are two ways to make journald logs persistent:

  1. Create a directory /var/log/journal. Once the directory is created, journald will automatically start writing logs to the directory.
    mkdir -p /var/log/journal
  2. Edit /etc/systemd/journald.conf file and set Storage to persistent. Do it manally with vim or nano
    [Journal]
Storage=persistent
    or use sed command
    sed -i 's/#Storage=auto/Storage=persistent/' /etc/systemd/journald.conf
    Finally, restart the journald service with
    systemctl restart systemd-journald.service
    NOTE: Make sure the key and value are not in quotes otherwise it will not work and errors will be shown when journald starts.
    MORE: https://access.redhat.com/solutions/696893

Rotate journald logs

By defaut, journald will rotate logs and use a maximum of 10% of partition and leave at least 15% free space for other uses, both capped at 4GB.
I will leave my settings at defaults, but if needed these values can be changed with SystemMaxUse and SystemKeepFree in /etc/systemd/journal.conf
MORE: https://www.freedesktop.org/software/systemd/man/journald.conf.html

Rotate SELinux audit logs

By default, the SELinux audit logs will not be rotated and sooner or later there will not be enough space for more logs. Depending on the low disk space action set in the config file, the system might stop booting up. The error messages in journald logs will be something like:

Audit daemon is low on disk space for logging
The audit daemon is now halting the system

To rotate audit logs based on size, set max_log_file_action to rotate in /etc/audit/auditd.confg

max_log_file_action = rotate

To implement time based instead of size based log rotation: https://access.redhat.com/solutions/661603

Thank you for reading. Check out the other parts in the series below.

Posts in this series