IT didit
Posted by CheezitsLight@reddit | talesfromtechsupport | View on Reddit | 73 comments
We make a wireless, police radio-based alarm system with network connection. Thousands of them in the field. The system is fully supervised, monitors everything, even has a months-long battery backup. It's a critical piece of life safety equipment that saves lives in basically every courthouse, hospital and schools.
It runs off a "wall wart" that plugs into an AC outlet. The transformer has a hole at the top for a security screw that's difficult to remove. So it must be plugged in an outlet in the bottom, then screwed into the electrical plate center screw hole. It's basically secure, hardened, locked and monitored by IT and the police. It can even push direct to 911 systems, bypassing operators to direct officers instantly.
We always install it, which is basically bolt it down, plug it in and tighten that one screw, turn the key, and then teach them how to use it.
A few months after one routine install they called and said it had quit working. Asked us to fly in and fix it. It's a $2,500 charge. So off I go.
It's unplugged. Someone in IT
had unscrewed it, and plugged something else in. In a locked IT closet.
Easy fix. Unplug their box, move it to the top plug and screw mine in the bottom.
Then the police remember that for two months it has spoken over their radio that it was on battery power. Every hour. They thought it meant it was working. And IT had ignored every email saying the system was on battery power.
Cruxwright@reddit
Was it IT that directed you to install the kit in the server room? Did you notify IT that you installed this important piece of equipment in the server room? Did you inform IT of what notifications to expect and actions to take?
Server rooms are like the domain of IT. They are responsible for all equipment in there and its security. Finding some black box plugged in with no notice of what it is? Yeah, give that thing a scream test.
tobascodagama@reddit
This is a really obnoxious pattern I've noticed. A different team hires a contractor to install some kind of technology. The contractor puts it in an IT closet because that's where technology goes. No documentation, and we're lucky if they even tell us they put it there.
(And it's always security; they think because they have access to the room they can just go in there and do whatever.)
The one I've been dealing with lately is a beeping UPS. Dead battery. It's not our UPS, but of course the "hey, there's beeping from the switch room" call comes to us. And because it's not our UPS, we don't have spares in stock or documentation about what kind of batteries to order, but because it's "tech" it's now our job to ID the model and buy the spares...
ThunderDwn@reddit
That's why IT - and only IT - control the swipe cards authorised to get into the room. Contractor wants access? Ask IT. And explain why, exactly and with documentation.
(Yes, I am such a bastard, and I do that. I pay the price in running the access control system for the entire office - but that's a small price to pay).
dustojnikhummer@reddit
Amen to that. If you want access to my (as in a place I'm legally responsible for, not that I own it) server room I'm coming with you and will observe you.
AlemarTheKobold@reddit
This sounds like a perfect scream testing scenario; get whoever owns it to scream so you can write it down. This is usually done by unplugging, lol
CheezitsLight@reddit (OP)
IT installed it. I don't know if they were part of training.
joshg678@reddit
Update the messages to say “THIS IS A PROBLEM PLEASE FIX IT BY MAKING SURE ITS PLUGGED INTO A WORKING OUTLET OR YOU WILL LOOSE CONNECTION!!!!!!” That Might help. Maybe…
Dougally@reddit
"YOU! YES YOU! GET I. T. TO FIX ME BEFORE I BURN THIS MOTHERFUCKING PLACE TO THE MOTHERFUCKING GROUND"!
SteveDallas10@reddit
I read that in Samuel L. Jackson’s voice.
dustojnikhummer@reddit
It would be a capital offense if you didn't.
djfdhigkgfIaruflg@reddit
/me hearing this over radio while being 10 blocks away.
Good. But where the fuck are you? Stupid machine
That_Ol_Cat@reddit
Tell me you used the rest of the day to go see a local attraction.
Wells1632@reddit
Sadly, a beeping UPS that is solely controlled by IT in a locked room that they only have the keys for is common. There is one close to my office that has been beeping for months now. We report it every couple of months, but they just ignore what we say. It is the UPS for the network switch for the building.
ShotFromGuns@reddit
An error message that can't be understood by the people it's addressed to is the same as no error message at all. Hopefully the recording has been updated with unambiguous instructions to investigate the cause of the loss of power.
(Obviously the other contributions to this failure were more severe, including and especially IT ignoring the warning emails. But if you're going to have redundancy, you need to actually have redundancy.)
SeraphiM0352@reddit
I don't think this is an error message issue.
This is an education issue. I'm sure the message was easily understood. The just didn't have or didn't pay attention to the training that explained that, despite having a long battery back up, it was not intended to function long term without a direct power connection.
Police ignorance is somewhat 'understandable' (but not acceptable), ITs has no excuse for their ignorance
ShotFromGuns@reddit
The message demonstrably wasn't easily understood, since the people who heard it thought it meant the opposite of what it was supposed to convey (i.e., a confirmation that the system was functioning correctly, not a warning that something was dangerously broken).
The point of this kind of error message is that it shouldn't require education to understand.
A simple "low battery" message is the kind of thing that makes sense on the first pass, to a tech person who understands the implications. Once it's been demonstrated to fail horribly in actual use, it needs to be redesigned to actually convey the correct information to users.
SeraphiM0352@reddit
I stand by what I said.
This is a training/education issue. They were likely informed of the features of the device, including that it has a direct power connection, a long lasting battery back up, and what kind of messages to expect. Not to mention the training/operating materials likely left behind.
The Police's first hint should have been when they started getting radio messages they weren't receiving before hand. They can then check their materials if they are uncertain of the meaning.
This also applies to the IT people who should have received all the same but includes an inherent deeper understanding of technology.
Failures are all abound but the message contents would be the least of them...
ShotFromGuns@reddit
With that attitude, I hope you never work anywhere like aviation, where it will absolutely get people killed.
DracoBengali86@reddit
Right, because a stick shaker unambiguously tells you how to fix the issue, not just that there is one. That's why not a single pilot has ever responded to a stick shaker by pulling back... Oh wait, they have.
SeraphiM0352@reddit
This isn't aviation.
Bring other industry specific requirements into a discussion outside that industry is irrelevant.
But as someone who has worked both government physical security and IT, I can tell you this is how it works.
As stated in another comment, at some point responsibility and accountability shifts to the users...
djfdhigkgfIaruflg@reddit
This. Programmes suck at writing good error message
zimmerframeRaces@reddit
Surely "This device has been disconnected and is running on backup battery. Please investigate immediately" is more likely to see action when dealing with many independent IT teams. Clear, unambiguous messages that call for a specific action or better yet, include actionable steps, are issues that get resolved quickly. Don't trust staff to remember the training you gave them five years ago and don't make it harder than it has to be.
CheezitsLight@reddit (OP)
It just said battery low. Not capable of knowing it was unplugged. This was back in 8 bit times and 64k bytes. The flash stored analog levels and speech was very limited.
fatmanwithabeard@reddit
I've never seen error messages with actionable content work in general usage.
I've gotten people to call support more often for silly test messages than I have for messages that have exact instructions to do so.
"Frob the whatsitwhosit" generated correct actions (calling support)
"Call support at 800-555-1234, report SN 123456 has failure state 16" gets people trying fixes that they think they remember from previous calls.
Jonathan_the_Nerd@reddit
This reminds me of a story I read a long time ago. A programmer had written software that supported a safety-critical system. If a strange error occurred, he needed to know right now. So that happened, the software would say something like, "General error. Please sacrifice a goat and two chickens to continue." This would confuse the user and make them call the programmer for help.
xyzzytwistymaze@reddit
Shut 'er down she's pumping mud!
fatmanwithabeard@reddit
Yeah. I had a friend accidently send a compile with a joke error message (meant for an overzealous tester) to the prod team.
"A PEBKAC has occurred."
Once the acronym was explained to the user, they made a formal request to keep it.
(Problem Exists Between Keyboard And Chair)
jasapper@reddit
My aging father consistently reports any ID10T errors reported by the macro I built for him.
GolfballDM@reddit
I love this. I wonder if I can persuade my boss to let me put some "whimsy" into my error messages.
CheezitsLight@reddit (OP)
Very true. But it's hardwired to say battery low.
SeraphiM0352@reddit
At some point responsibility and accountability shifts to the users.
OP is a vendor who provided the equipment and training (likely at government request). After receiving adequate training it is the responsibility of both the IT dept and Police dept to maintain their own readiness through regularly scheduled training
androshalforc1@reddit
Its a group psychology issue, one that is taught in first aid courses.
If everyone is responsible then no one is responsible. It’s why you don’t say someone call 911, and someone else get me a first aid kit. Everyone will stand around thinking someone else is doing it. You call out you in the blue shirt call 911, you in the red shirt get me a first aid kit.
In this case you have a system that can bypass 911 operators And direct officers but it can’t be programmed to send a message to an IT manager?
SeraphiM0352@reddit
Except that IT was getting messages but they ignored them.
This isn't exactly an "everyone is responsible" issue. There are two specific groups that are responsible and both groups are responsible for similar things already.
androshalforc1@reddit
Ahh i misread that i thought it was just broadcasting on the police radio.
Connect-Preference@reddit
Uhh, yeah, but a sticker on the wall wart stating "Emergency Police Radio. Do not unplug!" might have helped.
NixIsia@reddit
Actually the error was understandable, but your breath smells like farts in real life and everyone notices.
gadfly1999@reddit
I don’t think it was telling the police that it was on backup power over their radios. They just observed that it was still working for months after being unplugged. The ignored email was the one saying it was on battery.
ThePants999@reddit
Eh?
> the police remember that for two months it has spoken over their radio that it was on battery power
djfdhigkgfIaruflg@reddit
A totally out of context message over radio. And I'm 100% sure it wasn't mentioning the device's location.
And it's the police's fault. Sure Jan 🤦
gadfly1999@reddit
Yes. And?
djfdhigkgfIaruflg@reddit
I'm picturing the police's car radio announcing "warning, unit running on battery power", and the policeman scratching their head wondering wtf is happening
Distinct_Reality1973@reddit
I work in communications, and emerg services, I've never heard of such a device. We must not have any here.
CheezitsLight@reddit (OP)
It's a billion dollar industry. Try lynx systems dispatcher or secure tech systems.
JaschaE@reddit
So this system directly uses police radio in case of alarm, emergency...and being unplugged. Are they synced or can I DDOS police radio with your devices and a screwdriver? Shitty design, imho.
J_Landers@reddit
Welcome to wireless comms. The industry is like that... a lot.
Or asking for pairing of Bluetooth earbuds with a radio. Seems to upset a looooot of people.
JaschaE@reddit
Pardon my ignorance, but why is the earbud thing so touchy? Vastly different frequencies, very different protocolls, I don't see how either could interfere with the other?
I am reasonably certain that hooking your alarm to frequencies reserved for first responders here in Germany AND broadcasting malfunctions of your equipment would lead to all of your devices being hounded down and the Federal-Network-Agency knocking on your door to have a VERY expensive conversation. The last company having such a conversation, that I know off, sold "water energizers" for esotherics. And instead of doing the sensible thing, addding a couple LED that blink a reassuring "I am working" and pocketing the money of the willfully ignorant, they created radio-beacons that disrupted amateur radio up to 20km away.
CheezitsLight@reddit (OP)
The police department is the customer. Schools have to go through a PSAP which is an approved way to call 911. Except when the county puts them in all schools or courthouses. They own the police department that runs the courthouses and jails and schools.
JaschaE@reddit
Thanks for the explanation and this beautiful sidenote of dystopia.
"They own the police department that runs the courthouses and jails and schools."
CheezitsLight@reddit (OP)
Haha yes. It does sound weird. But true for the mayor of most cities and in counties it's the the county commissioner.
But elected civilian oversight is important. I think it's worse when an elected sheriff has no one over them. Then you get abusers such as slow response times so the other departments puck up the slack. , therefore no bad headlines, and decades of being re elected. While pocketing long distance phone fees and commissary fees from the county courthouse.
SteveDallas10@reddit
You live in the southern US?
CheezitsLight@reddit (OP)
Yes, where half the sherriffs are corrupt
AdreKiseque@reddit
If your idea of a DDOS is a warning message over radio every hour and emails to IT, and requires getting into server rooms of government buildings, I don't think they have much to worry about.
JaschaE@reddit
No, that would be DOS , and a bad one at that, but I saw OP mentioned schools. And a 2.5k service fee, so if I go around cutting the ones in schools, famous for their tech departments and full coffers, I can at least create an alarm fatigue.
AdreKiseque@reddit
And you think it's trivial to get into various schools' server rooms with a screwdriver undetected?
JaschaE@reddit
I mean, people get in with assault rifles with some frequency, so...
I have no desire, nor geographic ability, to mess with these devices. I do maintain that some alarm system sending malfunction-messages across emergency channels is shit-tier design.
StorminNorman@reddit
Theyre not usually real successful with the undetected bit though...
grunkle_dan78@reddit
sheesh. kinda discouraging that the technology hasn't changed much in the 30 years since I installed security systems. well, that's not entirely true. the backup now lasts months instead of 8ish hours. and those transformers... I had an unfortunate incident with one and an outlet cover plate. the plate was metal, but had been painted over so many times it looked and felt like plastic. as I plugged the transformer in, the plate crossed the prongs and I became the ground rod for a very cranky circuit. burned a big starburst pattern on the wall and knocked me out for a few minutes. fun times.
Gadgetman_1@reddit
That's not IT. That's someone employed to pretend to be IT.
lesethx@reddit
There are incompetent people in every profession
No-Procedure5991@reddit
Spelled "Google" correctly two out of three times on their resume.
Jonathan_the_Nerd@reddit
That's funny because Google (the company) was a misspelling. The founders named it after the number 10^100, but they got it slightly wrong.
resonantfate@reddit
To be fair, it isn't like they could have googled the correct spelling.
PalaceOfStones@reddit
Infuriating Technician.
imilnes@reddit
I Tried
PalaceOfStones@reddit
Intern Trouble.
CharcoalGreyWolf@reddit
Insufferable Twit
Gadgetman_1@reddit
That's management qualifications.
CharcoalGreyWolf@reddit
If you change one vowel, yes
YouSayToStay@reddit
Unfortunately every job has to have someone who is the worst at it. The best you can hope for is that they learn and grow, and then it becomes someone else's turn.
If they don't learn/grow, generally they get removed from the job pool and hopefully find something else they can excel at. You just have to pray it's before too much damage is done :p
that_one_wierd_guy@reddit
so management then
Zealousideal-Top672@reddit
-
bob152637485@reddit
What?
ephemeralmiko@reddit
Did they say anything when you "fixed" it?