Does anyone else hate Splunk?

Posted by bobert3275@reddit | sysadmin | View on Reddit | 50 comments

I am setting up Splunk and the sheer amount of effort it takes to get things right is astonishing. I don’t want to collect all these logs. But to configure that part and to get the agents running right with proper addons, etc, it sucks.

Does anyone have a proper resource for setting up the server, Linux systems, Windows workstations and servers to send the logs to? I simply want to send logs to it and access those logs when needed. There’s so many config files

[-]

nitroman89@reddit

We have a guy that is the Splunk guy. I use Ansible to deploy it on the Linux servers. We use Automox,Lansweeper,ManageEngine to deploy it on the Windows machines.

[-]

crowEatingStaleChips@reddit

I mean I managed to set up a splunk environment as part of a homelab a couple years ago and I am dumb as shit, so it can't be that bad.

[-]

bobert3275@reddit (OP)

Setting it up is simple. Optimizing it and getting past the initial setup like security hardening, license optimization, proper log parsing for different technologies, etc is where the frustration comes in.

[-]

crowEatingStaleChips@reddit

Makes sense. Good luck out there.

[-]

naked-and-famous@reddit

It's 2026, are you doing this by hand or are you using a bot to do it? Because you should be using a bot (along with Terraform or Pulumi, using --dry-run) to manage it.

[-]

DickStripper@reddit

We have 4 dedicated Splunk Admins. It’s a heavy, heavy tool but wicked powerful for large orgs that can leverage it with 1 guy that can write complex SPL. Without a reg ex SPL expert you will never be using Splunk properly. I see in real time everyday how powerful it is but you need a very expensive guy to write code. When you have that guy the possibilities are endless. I was a skeptic but now I see what our guy does and it’s beyond any infra management capabilities I’ve ever seen but it’s very, very heavy and requires intense expertise.

[-]

conrad_curze@reddit

I have used Splunk and plain Elasticsearch. I liked Elasticsearch is much better.

[-]

awetsasquatch@reddit

I like it for investigations but when I work with the Splunk admin it seems like a massive headache to do anything with to properly configure.

[-]

Coupe368@reddit

You configure the deployment server to tell the forwarders what to send to splunk, then you just install the forwarder on the boxes and point them to the ds. The DS downloads the config to the machines and there you go.

What agents are you trying to setup?

[-]

bobert3275@reddit (OP)

Universal forwarder.

[-]

Coupe368@reddit

Did you setup the deployment server yet? Forwarders only need to be installed and pointed at the DS.

Don't do it individually for every machine, what a nightmare that will be to maintain.

[-]

FarToe1@reddit

I looked at the price of it. Then looked at Graylog and how it was free.

[-]

scotticles@reddit

graylogs is better anyways.

[-]

Andronike@reddit

Nah I love it - git gud

[-]

Secret_Account07@reddit

It’s funny because we use a subpar product for our large env but some of our smaller customers use Splunk. I’ve always wanted splunk though

[-]

naked-and-famous@reddit

Yes, it's great when it's setup well.

[-]

sullivanmatt@reddit

Having been a Splunk cluster admin early in my career, let me say to anyone considering buying it: just go get Datadog's logs product. You won't save money (lol), but it's at least way, way less headache.

[-]

fumar@reddit

I use Datadog for logging and I have to use Splunk for logging in a client environment and the difference is night and day. Splunk is basically unusable.

[-]

shoobedoodoo@reddit

I've set up clusters too on CentOS with several indexers, search heads, separate deployment and licensing servers. Then deployed the forwarder via sccm to windows boxes. It was one of projects I enjoyed the most. I'm currently managing a single instance for a local org which has a small perpetual license.

[-]

SevaraB@reddit

You’re going to run into the same thing no matter what monitoring stack you choose- it’s as simple or as complex as your environment. If you’re looking to monitor more systems that are different from each other, you’ll have to configure more than, say, just pulling in data from an rsyslog receiver and an SNMP poller.

[-]

SandeeBelarus@reddit

I love Splunk. (As an end user).

[-]

bobert3275@reddit (OP)

I love it as an end user too lol

[-]

Hi_Im_Ken_Adams@reddit

The problem is that you are supporting Splunk as a secondary responsibility on top of your other work when you really should have a dedicated Splunk Admin taking care of it.

[-]

bobert3275@reddit (OP)

This might be true. I’ve spent hours on Splunk trying to configure it. Then get it to a working state. Don’t touch it for a bit and come back to it only to find it in some broken state. It’s a lot to manage

[-]

IdealParking4462@reddit

Then you hit all the limits in the query language. Skip it and go Sentinel.

[-]

Tex-Rob@reddit

You know how some software feels like the person who made it thinks like you? I find this to be most evident with things like CAD software. Like for me, Fusion 360 is super intuitive, manipulating stuff feels intuitive. Splumk represents the opposite for me, nothing makes sense and it feels like it’s intentionally obtuse.

[-]

bobert3275@reddit (OP)

I completely agree. Yes it is powerful, but to get there seems like a battle. I cover a lot different technologies with ease but for some reason my brain cannot accept the way Splunk does things

[-]

smooth_criminal1990@reddit

Have you downloaded the addons for Windows and *NIX? Put them on your indexes and search heads? And pushed them out to agents as needed?

[-]

bobert3275@reddit (OP)

Yes. Each step seems like it requires a mini education. Installing everything as is and calling it a day works fine out of the box. It’s when we are hardening it and attempting to fix little annoying things that I despise. I don’t want a ton of application logs using up the license. I don’t want noisy logs clogging up my license. It’s just a lot of little things that I have to think about that’s annoying about it. Like why do I need to accept your license every time I update? Just run lol

[-]

Odd-Anywhere2130@reddit

Very difficult product we had to use Splunk support and still had issues. This product requires a team of high end engineers to keep going You may want to consider Netwrix fairly inexpensive and easy to maintain

[-]

linuxelf@reddit

I recently switched from Splunk to OpenObserve in my homelab. Running the free version of Splunk, it became nagware to get me to upgrade to an enterprise license. It's probably great stuff if you want to pay for it.

[-]

Lordnerble@reddit

no, my bro works for splunk. pay his ass.

[-]

coomzee@reddit

Yes, it always spunks itself on a Friday afternoon. It is so expensive that it was cheaper for Cisco to acquire it than buy a licence, is the cherry on top.

[-]

TerrorBite@reddit

Oh is that why they did it

[-]

firedocter@reddit

I noped out the moment I saw that the localy hosted license still charged by usage.

[-]

Independent-Sir3234@reddit

The learning curve is brutal. I spent two weeks getting a custom sourcetype to parse correctly, only to find out the defaults already handled most of what I needed. Once it clicks though it's genuinely hard to go back to grepping through raw logs.

[-]

AngleBackground157@reddit

What’s your biggest pain point with Splunk—getting agents deployed, parsing logs, or just maintaining the config over time? We use Anomalog to monitor our app and catch errors in real time, and we also use Datadog and Sentry for broader observability.

[-]

bobsbitchtitz@reddit

Splunk is way better than the alternatives

[-]

Wonder1and@reddit

You should likely start by rolling universal forwarders to simplify things. There's a lot of settings still in config files. Deploying on your own without help will be frustrating and likely cost more of your time than a quick consulting engagement would.

[-]

techvet83@reddit

Also, be aware that the universal forwarders, whether Linux or Windows, periodically need updating to resolve vulnerability findings.

[-]

Hollow3ddd@reddit

ThreatLocker is silently breaking into SIEM scene. Along with some applications based VPN features and token protection services.

It’s becoming a pretty valid target, but if you are waist deep already, it’s not a hard plunge

[-]

weaver_of_cloth@reddit

I've been a splunk admin for several years now, both enterprise on-prem and on the forwarder support side. I also run our syslog aggregators for networking and, well, hyper-v logs (was VMware logs, in the before times). I go hot and cold on it. I put splunkforwarder on all servers that my department creates, and it was announced last week that it has to go on all our org's servers.

All of which is to say that I do a LOT of config along with the install, including setting facls, setting up a script to modify facls as needed, the deployment server subscription, and cronjobs for the script. I share my playbook across the organization, and support other sysadmins as needed.

I am very happy not to be running an on-prem installation, for sure.

[-]

funky_bebop@reddit

I wish we had splunk. I miss using it.

[-]

KrimsonBinome@reddit

Hate is a strong word but I dont care for splunk mostly because it requires me to know yet another language \syntax for something that should be a meta search.

Tbf it is very powerful and logs are only a bit of what it does

[-]

ludlology@reddit

If it helps lessen your pain, pretty much all log-ingestion systems whether SIEM or otherwise are a pain in the ass. Its good practice though because the skills generally transfer to other similar platforms.

If you don't have bandwidth get a consultant.

[-]