Any rsyslog gurus in the house?

Posted by JustCallMeBigD@reddit | linuxadmin | View on Reddit | 9 comments

I am trying to collect and organize logs from my Windows servers on my syslog server.

The syslog server is using rsyslog, and my Windows servers send their events to it through SolarWinds Event Log Forwarder for Windows.

Ideally, I would like to have a folder for each server, and within that folder will be a log file for security events, a file for windows events, a file for Active Directory events, etc.

As I have it now, my rules are filtering all events from a particular system into a dedicated file, and it's ridiculously painful trying to extract anything useful from them in a timely manner.

I am trying to set up a dynamic file naming structure and filtering rules to handle this, but what I have isn't working and I don't understand why/where I went wrong.

This is what I currently have:

template(name="SolarWindsDynamicPath" type="list") {
   constant(value="/var/log/syslog/servers/")

   property(name="hostname")
   constant(value="/")

   property(name="$now")
   constant(value="-")

   property(
       name="msg"  
       regex.expression="MSWinEventLog#[0-9]+#([A-Za-z0-9 ]+)"
       regex.submatch="1"
       regex.nomatchmode="FIELD"
       caseconversion="lower"
   )

   constant(value=".log")
}

template(name="CleanLogLine" type="list") {
   property(name="timestamp" dateFormat="rfc3339")
   constant(value=" ")
   property(name="hostname")
   constant(value=" ")

   property(name="rawmsg" controlcharacters="drop")
   constant(value="\n")
}
 
if ($msg contains "MSWinEventLog") then {
   action(type="omfile" dynaFile="SolarWindsDynamicPath" template="CleanLogLine")
   stop
}

It passes the rsyslogd syntax check, but it doesn't work and my server logs are just going into the generic 'warn' log file specified in rsyslog.conf.

Any advice is appreciated!

[-]

showbizusa25@reddit

The dangerous phase is when your logging setup becomes more complicated than the thing you’re trying to monitor.

[-]

Been running syslog servers for decades. Honestly, just put it all into one log, you life will be so much better. At first it seems like you would want to do all this fancy segregation based on host, but it's just way more complexity with marginal benefit. Realize that with one combined log, rotation is simpler and getting "single node logs" back out only takes grep if you need it. On the plus side, with everything in one log you can very easily extract cross-cluster events with a single grep. Monitoring for known event signatures, again, one log to watch. You can do it the hard way or the easy way.

[-]

JustCallMeBigD@reddit (OP)

I appreciate the insight! After dicking around with this all afternoon, I'm tending to agree with your sentiment.

KISS

[-]

libertyprivate@reddit

This is the type of thing ai does well

[-]

kai_ekael@reddit

`grep interest1 /var/log/biglog | grep -v notthat | awk '{print $whatever}' | sort | uniq -c | sort -n | mail -s 'Bug Count' lead-developer@mule`

Yeah, that big file is just so useless.

[-]

scottchiefbaker@reddit

We do something similar and sort logs by incoming IP address:

#####################################################
# Template definitions
#####################################################

template(name="foobar" type="string" string="/var/log/foobar-%$now%.log")

#####################################################
# Umask of files created
#####################################################

$FileCreateMode 0600

#####################################################
# Rules
#####################################################

if $fromhost-ip == "10.1.1.9" then {
        action(type="omfile" DynaFile="foobar")
        stop
}

This had the added benefit of automatically rolling the log files each day.

[-]

chock-a-block@reddit

Looks like you are mixing things that maybe don’t work together. Simplify your first template by removing

property(name="$now")
   constant(value="-")

Take out the event log decoder for now so the path ends with the hostname.

I would delete everything except dropping control characters in your second template.

Also, you didn’t mention what Distribution you are using. Might be a selinux issue.

[-]

JustCallMeBigD@reddit (OP)

Thank you! My syslog server is openSUSE Leap 16; I've edited the op to add that info.

I'll try your suggestions and let you know how it went.

[-]

JustCallMeBigD@reddit (OP)

Negative, logs are still going to /var/log/warn

Here's what I pruned the config down to:

template(name="SolarWindsDynamicPath" type="list") {
   constant(value="/var/log/syslog/servers/")
   property(name="hostname")
}

template(name="CleanLogLine" type="list") {
   property(name="rawmsg" controlcharacters="drop")
   constant(value="\n")
}

if ($msg contains "MSWinEventLog") then {
   action(type="omfile" dynaFile="SolarWindsDynamicPath" template="CleanLogLine")
   stop
}

Suppose it's also worth mentioning that this config is in a separate file at
/etc/rsyslog.d/02-winservers.conf