Create Wazuh Monitors Cause 500 Internal Server Errors

Posted by IndyPilot80@reddit | sysadmin | View on Reddit | 7 comments

I have a problem that I've been chasing for a long time and can't seem to resolve. When creating a new monitor, it usually ends up in 500 Internal server errors. I've created a "per document" monitor. After I save the monitor, I immediate get a "Failed to run the query - Request Timeout after 30000ms". In watching "top" for the CLI, java CPU usage jumps to about 800% and stays there. This usually causes a "500 internal server" error which requires a reboot. In /etc/wazuh-indexer/jvm.options, I've increased the Xms and Xmx to half of the physical RAM size and this doesnt help. Any suggestions?

Reply to Post

7 Comments

[-]

SevaraB@reddit

You’re getting the 500 because of the connection timeout. Are you sure the monitor agent has access/permissions to the thing you’re trying to monitor?

[-]

IndyPilot80@reddit (OP)

Yes, I don't see why it wouldn't I have another monitor setup for almost the exact same thing and it works properly. In other words, for example, the one monitor that works is for a custom rule that checks for 3 failed login attempts in 30 seconds. The new monitor I'm trying to add is for another rule I've created that monitors for 5 failed login attempts in 5 minutes. So, essentially the same rule/monitor but different times/parameters.

[-]

SevaraB@reddit

We run into this with Grafana- you’re drowning the query in noisy data with the 5-minute timespan. You can try upping the timeout interval, but if scraping 5 minutes of logs is taking longer than 30s, then you’re logging *way* too much via a single collector. You’ll need to distribute your log collection or make some difficult decisions about how much you can realistically retain.

[-]

IndyPilot80@reddit (OP)

Well, after digging much deeper, I think I've been setting up stuff wrong due to my own ignorance. I'm really dumbing this down but alert/monitors are mainly used for aggregate data and rules are for alert on certain events. Thus, when trying to make a monitor based on a rule, it was swamping the database basically trying to "gather" all of the data on that one rule id. I was mainly using a monitor because A. it is an alert that I need to acknowledge and B. I can set it up to send an email. When, in reality, I should be using a rule with the alert_by_email flag.

[-]

IndyPilot80@reddit (OP)

I must be doing something wrong. I tried it in my lab, which has wazuh and only two agents connected to it with a minimal amount of data, and am getting the same issue.

[-]

IndyPilot80@reddit (OP)

Now, correct me if I'm wrong. But, based on that assumption, if I change the rule from 5 minutes down to 30 seconds of scraping, it theoretically shouldn't error out because I have a 60sec rule that works. If so, I tried it and still get the 500 error.

[-]

WokeHammer40Genders@reddit

Wazuh is a fickle beast You will most likely find better success in the forums. Make sure to gather more detailed logs