Never assume. Yes, I mean that.

Posted by SuperTechnoDunce@reddit | talesfromtechsupport | View on Reddit | 11 comments

TL,DR: I'd have to make assumptions about the reader to summarize this post. I now know better.

I've spent several years prior to this entertained by some of the excellent stories by users such as Gambatte, Lawtechie, Airz23 ^((what about the keyboards?)), Mr_Cartographer, et cetera - but didn't expect that I'd end up adding to the pile so soon. That being said, this was a pretty entertaining one that I'm sure the more senior members here will get a chuckle out of.

Background for you lovely folks:

Shortly before I arrived at $Facility, the recording team started having issues with an external sound card, likely driver-related. Enterprise sound cards designed for Windows 10 do not always play nice with Windows 11 - thanks for nothing, Microsoft. The exact source of the problem has been difficult to diagnose and fix; in the meantime, a Mac Studio was dropped in as a stopgap,, which brought its own issues.

We use $SANSoftware at $Facility. For those not familiar with the term - a SAN is essentially a very fancy drive mapping solution. $SANSoftware lets our users mount any network drive they are authorized to, and potentially create or resize new virtual volumes if given the appropriate permissions. The entire system is composed of multiple RAID clusters that have a expiry date set by the vendor, to avoid cascading failure with aging drives.

Now. On with the tale!

$Mentor and I had been aware of an issue with $SANSoftware on the new Mac for a while now - we could view the virtual drives, but not mount them. Given that it was on a separate VLAN from the physical servers and all of the other machines running $SANSoftware, $Mentor had made the assumption that we simply needed IT to assign it an IP within the same VLAN as the rest of the SAN hardware and users.

"Hang on," you say. "You're supposed to be IT. IT manages storage, backups, et cetera."

Normally, yes. Here? Apparently not. There are a large number of blurry lines in the sand at $Facility as to where different departments' purview begins and ends. Generally, machines are domain-joined, with a few users outside IT such as $Me and $Mentor having administrator permissions on domain machines. However, a lot of our equipment comes from specialized vendors and is pretty sensitive to any kind of meddling with firewall configurations, thanks to various various thing-over-IP protocols - not to mention the occasional PCIe FPGA hardware and other fun obscure things that IT simply can't account for. These machines are not domain-joined and are instead managed by us on local accounts, or by other users who know what they are doing. User competence is shockingly high at $Facility thankfully, which saves us a lot of work.

Back to the story. Over the course of a few days, $Mentor contacted IT and requested an IP for the Mac on the same VLAN as our SAN. Instead of doing so, IT conducted a few tests with $SANSoftware and determined that the SAN was accessible from any VLAN on campus except the public-facing network - and sent a few pointers back along with their result, one of which was to double-check the permissions for the Mac's user.

So we did. In fact, we spent the next three hours troubleshooting. We learned two things from it:

Then I caught something out of the corner of one eye. As $Mentor moused over the "connect" button for one of the drives, a tooltip popped up: Failed to create mountpoint.

Oh. Great. I think we all know where this is going now.

Sure enough, 5 minutes of Google-Fu later and there it was - a little blurb in the system settings indicating that $SANSoftware's extension had been blocked from loading at boot. No privileged extension equals no ability to create mountpoints. One reboot later and a bit of searching to figure out exactly where $SANSoftware was mounting the drives, we had a working solution.

Now. Where does the assumption come in? Well, according to $Mentor, this is a known issue with the install. One of the more competent users had taken care of installing $SANSoftware to save us some time - and didn't know that it was necessary to explicitly allow the software's extension to load. We assumed the install was fully functional, and that it therefore must be a networking issue.

One underrated feature of working in a mixing room is the soundproofing. You can swear at whatever misbehaving technology you like all day, and nobody will hear you. Suffice to say there was a healthy dose of that at the end of our troubleshooting session, followed by some healthy chuckling on my part. I've read so many tales of assumptions turning out poorly, and even after all that was still bitten by a basic one.