The disappear fault

Posted by Ndog4664@reddit | talesfromtechsupport | View on Reddit | 19 comments

Time for a couple more badly written stories, words are hard and I never went to college. You get what you get.

Get some doordash, maybe some Adderall or whatever your vise is and enjoy

My job is tech support related but not directly. I work on anything from servers, networking, to automation(belts, motors, bearing) and PLCs. I'm a jack of all trades and definitely a master of none.

:The disappearing fault

So one day Operations calls us due to an output module fault. It looks like 7 modules lost communication. Well we first check the com cables, 2 40pin cables that create a loop for 4-19 modules. They seemed fine but Admittedly avoid these cables because I hate them, bulky, bad retention mechanism, and likely to have more problems just from touching them. All the cables for controlling coms, gates, actuators, and safety loop go to a backplane that slots into the main control PCB. So we replaced the main PCB, nothing happens except for even more faults. Then we got a second one, kinda worked just different faults this time. So we got a third one, most faults gone except one but communication is back for everything. At this point I called a remote SME, system matter expert. Who says to swap the board with another module to see if the fault follows the main board or stays with the module. One problem, it does nether, it just disappeared. Doesn't make sense to me but it's gone and the machine works.

Main lesson learned, just expect all your parts are bads.

:When the SME is wrong

So Operations calls about a machine intermittently stopping for a safety loop fault that never calls out where the fault is in the machine. The machine will act good when not processing but after 30s to 5m it will fault out. We arrive and start looking at interlocks but couldn't find anything. We keep pressing start till we get the fault to show up and not immediately disappear. We checked a 24v safety aux contact attached to a relay, even though it didn't test bad it's so common we replaced it. After checking every interlock we can't figure it out. So we call the remote SME. One piece of info I did have to diagnose the issue is the aux contact has no power. Letting the SME know this he first says to replace it, I told him I already did, also told him I'm not trained on the equipment though so I'm at a lost and need schmatics. He emailed me the schematics and also wanted me to follow the schematics after the aux contact which didn't make sense to me because it wasn't getting power. I felt I should trace where the power comes from and see where I get it back. So I lied to him that I would then with the self confidence of a stupid person, I did my own thing. Found power going into the safety loop at the breakers, the breakers have the ability to tell the computer if they're tripped, but not coming out. Started shaking the connectors on the back of each one until I found one that would make the machine go ready and not ready just from wiggling it, yes I wore gloves if anyone from OSHA is asking. After the machine was down for 5 hours, one aux contact and one breaker later it was fixed.

To explain what was happening, the machine vibrates when running due to motors, bearing and belts. This vibration would cause the tab inside the breaker to disconnect momentarily causing the machine to stop due to the safety loop opening.

:when the senior tech doesn't compute

So on day shift they had a machine go down due to the output modules not communicating. The senior tech(for day shift) found the module not communicating and replaced the board but still wasn't able to fix it. Even shift with the 2 most senior techs out of everyone refused to touch it. Finally when i came in on night shift and was questioning why we had a machine down i decided i would look at. Still not trained but from knowledege of last time I asked if anyone configured the board. All evidence pointed to no. So one call to the SME for the document on dip swich configuration and crawling inside the machine later the machine worked...

Unsure if anyone else shares my frustration but fixing stuff more experienced and trained people shouldv'e give mangement unreasonable expectations of your ablities. I love solving problem, i don't love being put on a pedistol.

Btw the down time of that machine probably cost $150k-300k

:how to solve a random persons problems from 500 miles away.

So the techs at my company have a facebook group for memes but also for help when SMEs are no help.

A person in another state posted they have had a machine down for over 7 days. The machine would only fault out if you tried to run it. With the fault being a communication fault from the operator PC to the on site OCR server, Optical Character Recognition. The issue was they could ping the server, and PC and server would show connected in their respective software. They even ran a new cable from the switch to no avail. I guess no one on site or the SME thought to actually see what the switch was reporting. I had access to see the monitoring of every facility just not make switch configurations. I was bored and looked them up and saw a ton of errors. The port was configured correctly, so most likely bad port.

So I messaged the guy. We got me =me, tech= guy from that facility, and supervisor = his boss

Me- hey, i saw your FB post i think it's the switch port

Tech- we are going to reboot again

Tech- I'm going to make a group chat with my supervisor

---new chat---

Me- hi, i think this is an issue with the IDF switch, do you guys have anyone with cisco CLI training and log in.

Supervisor- i think so but he hasn't logged in awhile

Supervisor- SME says to check switch at machine, we replaced it but that didn't work. SME now says to replace IDF switch

Me- before stopping all operations lets just try another port

Tech- we need a ladder

Me- i see the switch lost power recently, did you guys have a power outage.

Supervisor- actually yes, thats when the problems started

Me- please take the cable from port 6 and plug it into port 8

---Note, port 6 is for the machine having problems, port 8 is for a machine that is working

---23 messages and 4 hours later of being ignored

Me- please take the cable from port 6 and plug it into port 8

Supervisor- that worked we think port 6 is bad

Me- plug the cable ftom port 8 into port 6 and see if it faults out too

Supervisor- it does

Me- that comfirms 6 is bad, have your tech open and cofigure another port and label port 6 as bad.

Supervisor- thank you!


Moral of the story sometimes you need to repeat yourself i guess. Still working on being assertive.

On the plus side this interaction helped me pass the interview to become a SME, just waiting for an open postion.

:the normal tech support call

So us machine techs are only supposed to fix anything related to machinery and their functions "processing infrastructure side". We consider anything not related "Lan side"(printers and supervisor computers).

One problem, one onsite "Lan side" tech covered like 6 plants almost all 120 miles apart. They could drive 5 hours for one call and responce time is like 2 weeks.

Due to how over streched this guy was, even though he didn't want my help, and my interest in tech I would help when i can. It was against the union contract but keeping the bosses happy was in everyones interest. I mainly would just help with printer problems and was well known by management for solving printer problems. After the print server/directory failed i was the only one to get the printers working while we waited for it to get fixed. Anyways here's the story.

-Over the dispatch radio

Supervisor- hey OP can you help me with the printer by machine 9

Me- On my way

---i arrive stage left

Supervisor- I can't get the printer to print, i think it's broken

Me- please bring up what you're trying to print

Me- press print or ctrl p please

Me- can you select the printer labeled "printer by machine 9" please instead of "print to pdf"

--- exit stage right as it starts printing ---

When i was asked to work on their networking though i would say, "only if you can provide a network diagram/topology" . I perfectly well knew they couldn't because they never made one for their side of the network. Their network closet was an actual birds nest. Like you had to walk on the cables to get to the rack, like the rack looked like vines covering a tree and all the walls. There was more un used cables ran in there then used ones. Patch panels, what patch panels. Idk how it looked like that for only having lile 8 switches, 2 firewalls, and 2 routers.

Grammerly broke like half way through this so sorry not sorry.