A tale of two breakers
Posted by Own-Cupcake7586@reddit | talesfromtechsupport | View on Reddit | 43 comments
A few years back, my company got a call from one our customers. “Machine is down. Throwing error codes. Need someone ASAP.”
Nothing out of the ordinary, such is life in service. Unfortunately this customer is several states away, and has minimum training requirements to even get through the door. But duty calls, so next day sees me on a plane.
So day one is spent flying and driving to a hotel. Day two is spent going through training and talking through what could be causing the issues. Based on the error description, we determine what parts they have that might be useful and gather them from the warehouse. All set for day three.
Day three I finally get hands on the machinery. Start troubleshooting. Find that a brake is not releasing, causing the error. Fair enough, that was one of the issues I expected. Keep working through the issue…
Guy standing next to me as I’m on a ladder, “Hey, should this breaker be off?”
Background time. This customer had a very particular procedure for this piece of equipment. At the start of every shift, the operator had to climb onto the machine, walk down the walkway (it’s a big machine), and open up the *fourth* electrical enclosure to turn on a breaker to enable the machine. At the end of his shift, he had to climb onto the machine, walk to the *fourth* enclosure and turn off that same breaker. This ensured that a proper walkdown was being done every shift.
We knew about this during the design phase. The salesman suggested “hey, there are several breakers in these panels. If there’s one that you need to manipulate twice every shift, we can move it out to the cover so you can access it without having to open anything up.”
“No,” says the customer, in their infinite wisdom, “the process is procedure-driven. We’ll do it our way.”
Fast-forward to me, 3 days into an out-of-state service trip, staring at a little breaker in the *third* electrical enclosure. “No. No that breaker should not be off.”
One little flip of a switch later, and the machine is right as rain. No errors, no problems. Just an easy mistake that cost a lot of money, and which was just waiting to happen. If only someone had warned them…
Day four saw me back on a plane, with a stupidly funny story to tell.
Hawteyh@reddit
I dont think I can ever complain anymore about having to drive 20 minutes to take out a power supply and put it back in.
You win this one :)
Schrojo18@reddit
Breakers aren't designed with the same actuation count as switches/isolators. They should not have been using it regularly, they should have had an isolator installed at ground/operator level where it could be turned on and off safely and reliably.
Neumanium@reddit
I used to work on big dangerous semi-conductor equipment. It had thermal hazards, electrical hazards, RF hazards, and crush hazards. We had a complete and accurate lock out tag out procedure to make the equipment safe for maintenance. We used rotary switch breakers for all isolations, they were designed to be used for safety.
Using normal breakers all the time is a disaster waiting to happen. Whoever thought this up at the customers end is creating a disaster waiting to happen, breaker failure even and arc flash depending on the current and voltage load.
hicow@reddit
Every retail job I ever had (and I had a fair number), the procedure for turning the lights off before we left for the night was cutting most of the breakers in the box. Part of opening in the morning was turning all the breakers back on. Most if not all didn't even have a cover on the breaker box, and nearly all had a label over at least one of the breakers warning to never turn it off
TheChance@reddit
Only in big box stores and warehouse clubs and so forth, and only because otherwise you'd have like 100 light switches.
Neumanium@reddit
I am going to make an assumption here that the shield around the breakers at your place of employment. This cover around the breaker is there to protect from the flash. Just because you use a breaker to turn off the lights and everyone probably does, does not make it a safe decision. Big equipment for manufacturing more likely then not is probably is significantly larger currently load then lighting circuits in a retail business. The breakers were probably 15 amps each at most, the equipment I used to work on the smallest breakers were 3 phase 240 volt at 50 amps.
You are basically comparing apples to oranges in terms of danger from a failure. Danger is still there but a lot less danger in terms of overall risk. Look up arc flash on YouTube, trust me it is nothing to mess with.
commentsrnice2@reddit
More like comparing kiwis to watermelons
hicow@reddit
I'm not comparing anything. Simply making an observation
RayEd29@reddit
Sounds like you did work like my nephew does. He's an industrial electrician. Scariest thing he ever said was "You know, it'd be nice to have a job where you don't suddenly catch fire."
tgrantt@reddit
My high school gym didn't have light switches at all, just used the breakers. Twenty years later, when I learned a little about electrical, I realized "That's a poor idea."
Own-Cupcake7586@reddit (OP)
Fully agree. There was more than one poor decision involved here.
spiritsarise@reddit
But, you know, “our procedure.”
Severe_Ad_5914@reddit
Should Have, Could Have: 0
Our Procedure: 1
KelemvorSparkyfox@reddit
This sub would be a lot emptier if people in charge made good decisions.
Severe_Ad_5914@reddit
There are a LOT of subs on reddit that would be a lot emptier if people in charge made good decisions.
androshalforc1@reddit
This sounds counter productive. If I’m reading the post correctly the breaker wasnt being switched to turn the machine on and off, It was being switched to force the operator to walk the length of the machine and do a visual inspection.
Putting in a switch at the operators level completely bypasses the reason they were switching the breaker in the first place.
Practical-Shape2325@reddit
Having it at ground level would not have solved the problem of forcing operators to walk the equipment in order to be sure they were visually inspecting everything. Now, they could have just had some sort of scan/log/button thing up there to confirm you actually walked up, but moving the switch wouldn't have solved the problem they were trying to fix.
leitey@reddit
And generally*, operators shouldn't be in electrical enclosures. Only trained and authorized technicians. Most likely with the power off. If it must be done while powered (people will die if it's powered off, like certain areas of a hospital) then only while wearing proper arc flash safety gear.
Dustquake@reddit
Makes me feel a lot better about calling the local tech about the freezer when the problem was just morning management had flipped the breaker for cycle counting.
RockyMoose@reddit
My colleague once took a 2 hour airplane to push a button.
"Are you sure the server will not power on? The button is recessed so you might need push it firmly or even use the end of a pen to press the button."
Customer: "I know how to push a button. This server will not turn on. It's dead. Send your technician immediately with a replacement server."
... and you can guess the rest of the story
OldGeekWeirdo@reddit
Was it one of those switches with a guard and a hole that *requires* a pen to push it? (Unless you have really skinny little fingers).
azaz0080FF@reddit
Or a key to open the front of the server unless you use a pen
ecp001@reddit
>“the process is procedure-driven. We’ll do it our way.”
That decision, along with suggested clarification, should have been included in any reference material available to the CS personnel dealing with the customer. Even though revenue is increased by dealing with stupidity, the long-term relationship has a better chance of survival if the customer is not confronted with an expensive resolution of dumb-ass errors.
Sandy_W@reddit
"Guys, this is what the management figures mean by 'agile'. Until he said that, we were a proud engineering company, determined to provide the best solution we could. Now, however, we make the 'agile pivot' to a proud sales team, determined to help them spend as much money as we can talk them into. Good engineering practices are no longer appropriate for this client. Make sure this pivot gets noted in all support documents."
ecp001@reddit
...and to make sure the customer realizes the value of our top-notch, A-one, super-duper customer service, "elevate" the call to the rep next to you who will claim to be a manager, confirm the severity of the problem and authorize the speedy dispatch of a knowledgeable, skilled technician who will have the tools, parts, and supplies to readily resolve the issue.
kandoras@reddit
The procedure was for the regular operator to, twice a day, open up a live electrical enclosure and flip a breaker?
For me, that wouldn't have even been a question of "Do you guys really want it like this?"
It would have been "We can do it my way, that won't eventually get someone killed, or you can do it your way that will. And this conversation is being recorded for when that happens and you get sued so I can say to a judge that I had no fucking part in this other than warning you."
Own-Cupcake7586@reddit (OP)
I’m not sure they were operating the breaker live/ under load. They may have used an upstream disconnect to make sure the panel was dead. I was less concerned with their stupid procedures and more focused on resolution.
Throwaway_Old_Guy@reddit
These two paragraphs contain the most pertinent information.
Putting aside the breaker debate, the Employer left room for the error to occur by not specifically marking the panels clearly. OP doesn't say, however, I surmise the breaker in the third electrical enclosure was in the same location as the one in the fourth electrical enclosure (although there may have been only one in each, just the panels not clearly marked)
Own-Cupcake7586@reddit (OP)
Correct. Very similar breakers in the lower left corners of enclosures 3 and 4 (and 2, for that matter). Definitely asking for trouble, lol.
Leonie-Lionheard@reddit
Other possibility: you can write an error message for that error.
Error handling should be a step in the design process and in later testing.
Own-Cupcake7586@reddit (OP)
Our machines are less “smart” then you may be imagining. The error message boiled down to “motor no spin,” but didn’t give much beyond that.
KelemvorSparkyfox@reddit
I used to support a beast of an interface between two ERP systems. It had its own message file for errors - most of which were cribbed from the ERP systems in question. I think there were more entries in it than are produced by a Stargate.
My absolute favourite was, "ITEM NOT FOUND AT THIS LEVEL OF UNIQUENESS", which still makes me chuckle.
Leonie-Lionheard@reddit
Ah. Okay, I get that for such machines. Still would be nice to have.
Own-Cupcake7586@reddit (OP)
No doubt.
ethnicman1971@reddit
If you knew that the suggestion was made to the customer to put the breaker that needed to be switched off and on twice every shift and that they declined, and you knew that the machine has multiple breakers, why wouldn't you have the client verify the position of each breaker? And then have a second person verify that what the first person told you was correct? This would have saved 3 wasted days for you.
Own-Cupcake7586@reddit (OP)
Hindsight is always 20/20. I didn’t learn any of that background until afterward. But yes, you are ultimately correct.
ethnicman1971@reddit
well damn. That sucks :). So then it is the fault of the sales person and the original implementer who should have added those details in the runbook.
New_Crow3284@reddit
So you don't ask for pictures of the inside of the cabinets, and there is no breaker detection in the plc software.
Own-Cupcake7586@reddit (OP)
The industry in question doesn’t love sharing pictures, and no, no breaker detection in the PLC.
tkguru8@reddit
And I hope they had to prepay for the cost to have someone come on site. That'll teach them or at least penalize them.
Own-Cupcake7586@reddit (OP)
Oh yeah, they paid well. Not sure if they actually learned anything, though.
atomicsnarl@reddit
Hope you got per diem out of it as well!
Own-Cupcake7586@reddit (OP)
Absolutely. I also get my hourly rate for travel. I’m one of the few people that gets a smile out of a delayed flight.