BlogLinux & scripting
Centralized logging with journald and rsyslog
Make the journal persistent, forward it to a central host, and keep logs you can actually search after an incident.
systemd's journal is structured, queryable, and already running on every modern Linux box — but by default it is in-memory and vanishes on reboot, which is useless the moment you need it. Two changes make it incident-ready: persist it, and forward it somewhere central.
Make the journal survive reboots
/etc/systemd/journald.conf
[Journal]Storage=persistentSystemMaxUse=2GMaxRetentionSec=30day
journalctl -u sshd --since "1 hour ago" -p warningFailed password for invalid user admin from 203.0.113.9structured, filterable by unit, priority, and timeForward to a central host
A log on a box that got compromised is a log you cannot trust. Ship it off the host with rsyslog.
/etc/rsyslog.d/50-forward.conf
# forward everything to the central collector over TCP*.* @@logs.acme.internal:514
Watch the rate limiter
journald drops messages past RateLimitBurst per RateLimitInterval. Under a flood you lose exactly the lines you want — tune these on noisy hosts.