I am a fiber optic engineer. And this datacenter will be the death of me. Recognizing the big words will bore the crap out of most people, I'll give the abridged version.

This particular datacenter does not train or equip it's employees properly, and as a result, anytime they have a slightly complex problem, they make it worse by trying to fix it.

Then they call me for help, despite the fact I don't even work for them. I am their OSP vendor, meaning I fix problems outside and between buildings. Inside their own building is supposed to be their own problem.

I receive an email. This infamous datacenter tells me they're experiencing an "ORL" issue. That's optical return loss. It means the connection is too shiny and too much light is returning the way it came, backwards, instead of going forwards through the fiber optic cable to the destination.

I tell them it means their connectors are dirty and to clean them with proper cleaning supplies. Fun fact: they do not have proper cleaning supplies.

Days later they follow up, telling me the issue is now a 10dB degrade. That gives me pause. That could actually be an issue between the buildings I would be responsible for. A degrade means the light going from one building to the other is too dim when it arrives, some of it has been lost. 10dB is not a small amount, it means 90% of the light is missing and only 10% is getting to the destination.

I show up. I start my troubleshooting by asking a lot of questions. The answers I get confused me. The equipment readings I get confused me. Finally I realize what's happened.

On a previous trip I told them they had bad patch cords, they would fall out of their plugs if you so much as breathed at them.

Following my advice, they replaced them with proper cables. So far so good.

But one of the circuits didn't come back up fully. They properly diagnosed the new cable was dirty and needed cleaning.

Mistake number 1- I told them the cleaning supplies they had on hand were terrible, and likely to make things worse. Standard cleaning solution is isopropyl alcohol. I was told that was a hazard and they needed to use an alternative cleaner. This alternative cleaner is a mixture of Propyl Acetate, an industrial solvent that is itself flammable and emits hazardous vapours that are also flammable. Ethanol, which is literally fuel. And- Isopropyl alcohol!

This cleaning agent leaves behind some oily residue that- causes ORL issues because it's shiny. So they cleaned this fiber over and over and failed every time, and concluded the problem must be elsewhere- completely ignoring that I had already told them their cleaning supplies sucked and were incredibly inadequate.

Now if their cleaning supplies weren't awful, that might be a reasonable conclusion. So they performed a loopback test.

The connection normally goes equipment, patch cord, rack, cable leaving building, rack in next building, patch cord, equipment. A loopback test is purposefully looping the patch cord back to the transmitting equipment by connecting it to itself at the rack.

When you do this, you add an attenuator. This is a special plug that adds loss. The system is set up assuming you'll lose so much light going from building to building, so that short range connection needs to be 'dimmed' a bit. That's what the attenuator is for.

Mistake number 2: they connected the attenuator, and forgot where they put it.

The 10dB degrade I was sent to repair was caused by a 10dB attenuator they installed, and couldn't remove, because they forgot where they put it. And on top of that- they still needed me to clean their damn panel for them.

The remote support team knew all of this, and through tactical lies of omission, made me think I might actually need to fix something that was my responsibility. Instead they used my competence to fix shit their internal team doesn't have the training or equipment to deal with: which are basic essential functions of any datacenter team.

These lazy bastards tricked me into troubleshooting diagnosing and repairing their own shit because they can't be bothered to train and equip their own employees to do their jobs.

I put my foot down and told them this is our last courtesy dispatch. For future calls involving this datacenter, we are working strictly to the terms of our contract.