Rsyslog stops logging completely under high load
Posted by scottchiefbaker@reddit | linuxadmin | View on Reddit | 3 comments
I need to troubleshoot a service problem so I enabled debug logging on that service. This results in 500+ syslog lines hitting the logs per second. After a couple minutes of logging rsyslog completely stops logging. The process is still running and it doesn't give any indication of WHY it stopped logging.
I see a handful of these in the log files (even when things are running normally), so I'm not sure if this is related or not. I've read through the URL it mentions and I don't see anything obvious.
Oct 27 14:23:59 ns1 rsyslogd[54222]: imjournal: journal files changed, reloading... [v8.2412.0-1.el10 try https://www.rsyslog.com/e/0 ]
Oct 27 14:23:59 ns1 rsyslogd[54222]: imjournal: journal files changed, reloading... [v8.2412.0-1.el10 try https://www.rsyslog.com/e/0 ]
Oct 27 14:24:14 ns1 rsyslogd[54222]: imjournal: journal files changed, reloading... [v8.2412.0-1.el10 try https://www.rsyslog.com/e/0 ]
Oct 27 14:24:14 ns1 rsyslogd[54222]: imjournal: journal files changed, reloading... [v8.2412.0-1.el10 try https://www.rsyslog.com/e/0 ]
Oct 27 14:24:14 ns1 rsyslogd[54222]: imjournal: journal files changed, reloading... [v8.2412.0-1.el10 try https://www.rsyslog.com/e/0 ]
Oct 27 14:25:11 ns1 rsyslogd[54284]: imjournal: journal files changed, reloading... [v8.2412.0-1.el10 try https://www.rsyslog.com/e/0 ]
FWIW when I don't have "debug logging" enabled I don't have any problems with rsyslog stopping logging. It runs for weeks/months at a time without any poking.
This is a Rocky 10 server if it matters.
xdice@reddit
If it is rate limiting (and it kinda seems that way), you could try adding this to your rsyslogd config (in /etc/rsyslog.conf, or /etc/rsyslog.d/<your_filename.conf)
Disable rate limit.
$imjournalRatelimitInterval 0 $imjournalRatelimitBurst 0
That will disable rate limiting. I wouldn't leave this in place permanently if it does work for you.
spudlyo@reddit
What's it doing when it's not logging? When a process doesn't tell you what it's doing, and when it's doing something you don't like, that's typically when you break out strace and log a few thousand syscalls and see what it's doing.
jaymef@reddit
check if its rate limiting
journalctl -u systemd-journald -p warning journalctl --verify