A bit lost about logging in general, especially rsyslog
Posted by 420829@reddit | linuxadmin | View on Reddit | 20 comments
I've been studying rsyslog, but I'm still having trouble understanding what its real-world usage pattern looks like in companies that actually use it.
From what I understand, rsyslog acts more as a log transporter/router, and in many cases journald is the component actually collecting the logs. What confuses me is that a lot of modern applications no longer use the syslog() syscall directly and instead write to stdin/stdout.
In these cases, what have you been seeing in current Linux administration practices? Do people usually rely on imuxsock, imjournal, or some combination of both?
Also, if anyone here works with rsyslog in enterprise environments, I'd really appreciate some broader context on how this logging infrastructure is typically designed and operated in real-world setups.
forbiddenlake@reddit
rsyslog is old. typically journald is being used, and systemd is capturing stdin/stdout.
And in real shops, logs are collected by Fluent/Alloy/OpenTelemetry and sent to Loki/Elastic.
420829@reddit (OP)
For local log collection, I understand that journald is the more appropriate choice. But from what I’ve seen on the official rsyslog website, they still seem very up to date when it comes to event distribution and forwarding. I’ll take a look at Fluent later on. Right now I’m trying to learn the more “native” approaches first so I can build a solid foundation, and hopefully the knowledge will be transferable.
justin-8@reddit
Honestly I haven't seen anywhere serious using rsyslog in production in 10-12 years. Especially not now that practically all flavors of Linux targeting enterprise use are on systemd now and have journald. You just pick a log forwarding solution and a log centralization one and connect it.
E.g. if you're on a cloud you might pick their native one, or a third party saas, or run elastic, graylog, splunk, etc and use their forwarding tools to send logs to it.
chock-a-block@reddit
Grep is old. Does that mean we should stop using it?
On e very valuable use of rsyslog is sending only the important parts of logging to elasticsearch.
kai_ekael@reddit
Rather use grep/awk/perl than elastic any day.
chock-a-block@reddit
You and I would. Your CTO wants cheap, easily substituted labor. It’s what drives products like elasticsearch.
kai_ekael@reddit
Luckily, not my "CTO", since I don't have one. 😃
kai_ekael@reddit
Journald is a snot-nosed POS. First thing I turn off/re-route on my stuff.
"Oh, sorry, too many log entries, so I dropped them." whack
vivaaprimavera@reddit
For central log collection... Since we can't forget that both journald and syslog might be needed... What is the best practice, sending journald directly to the central point or sending the journald logs also to syslog and handling the collection with one of the solutions that you have mentioned?
On services that log to stout, journald collects those, what is bugging me is how to send to the central location for permanent storage/processing.
420829@reddit (OP)
Exactly. From what I’ve understood, journald is better suited for log collection, while rsyslog mainly handles basic normalization and forwarding. Journald even creates syslog-formatted logs for stdout/stderr events and sends them through the Unix socket at
/run/systemd/journal/syslog, which honestly seems like a pretty consistent and modern approach rather than something outdated.Topfiiii@reddit
We use rsyslog in our SIEM environments to collect logs from different sources e.g. Linux servers without log collector agent, network devices and hypervisors via UDP, TCP and TLS. Rsyslog writes the logs to the disk and the SIEM solution then indexes the log files. you can also use two servers with keepalived to build a redundant setup.
420829@reddit (OP)
Very cool, I liked this approach. It makes sense to use rsyslog for things that aren't necessarily Apps because of its high compatibility.
Rxyro@reddit
Rxyro@reddit
Feature | journald | rsyslog
---|---|---
Log format | Binary (structured) | Plain text
Structured logging | Native support | Requires config/plugins
Default in Debian 13 | ✅ Yes | ❌ No (optional)
Storage location | /var/log/journal or /run/log/journal | /var/log/syslog, /var/log/auth.log, etc.
Remote forwarding | Needs external tools | Native TCP/UDP/TLS/RELP support
Log rotation | Automatic (size/time-based) | Manual via logrotate
Query tool | journalctl with rich filters | grep, awk, traditional tools
Performance under load | High (indexed binary) | Can bottleneck
pobrika@reddit
We use rsyslog coupled with greylog and n8n handles a large volume of data but it does eat disk space in our situation. These logs need to be archived for many years, we recently moved the logging to a zfs proxmox server and were getting 2.4x compression with increased read write speed overall.
chock-a-block@reddit
I am in a high compliance environment where extensive logging is required.
Rsyslog is fantastic for ticking the compliance boxes and routing the important bits to monitoring.
I’m the first person to admit the syntax requires time and effort.
CopperNik@reddit
We use it to collect web server logs to clickhouse. Rsyslog => vector => clickhouse. This chain can handle a way more logs than others we tested on the same hardware
420829@reddit (OP)
That’s nice to hear. I’ve been really enjoying rsyslog so far — it seems very solid at what it does, and I get the feeling it’s still very relevant today. In your environments, do you usually forward local logs to rsyslog through journald?
CopperNik@reddit
No, web server sinks logs through unix socket directly to rsyslog
TheDevauto@reddit
Historically it was used when you had a logserver that acted as a central collection point for system and app logs. Monitoring tools would run there and forward evemts to an event management server.
Today there are better tools, but in some cases rsyslog can be used in say a dmz where you can send logs back for another tool to grab. Very niche, but it can be done.