I'm a corporate instructor, I mostly teach folks how to setup/use/troubleshoot storage arrays. Sometimes I get sent to customers sites to run classes there. Its pretty common that when I end up onsite I end up owning all the problems of the site. This is the story of one of those times.

So I show up and after the preliminaries I ask where the system is. I like to get eyes on the system they have before class so I can make reference to it during class. Maybe emphasize some things or leave out stuff they don't have.

Here through, nobody knows where it is. The system had been bought by a previous management administration after which everybody left. The new people don't know anything about it.

Okay, well where are the network switches? We can start there. Nobody knows.

So I start at one of the client systems, the network cable goes up into the ceiling so I climb up on a desk move a ceiling tile and see which way the cable goes and the great search is on.

Eventually I find the network switch in an innocuous data closet. There is 1x 10Gb link coming off of it which doubtless leads to my system.

I eventually find the storage system in a rack hiding behind a rack of coats in the hallway to the garage. When I finally move the coat rack I can hear the fans screaming. The machine itself looks like it's wearing a fur coat. The dust is literally 2" thick. This machine has been in use for 4 years, no one has touched it...

The site doesn't own a vacuum cleaner, the cleaning company brings their own, so I walk across the street to Wal-Mart and buy a shop vac. I put some wire around the vacuum nozzle to try to get at least some kind of grounding. We then spend 3 frickin hours deep cleaning everything in the rack. I figure if I'm there it would be cruel not to clean everything.

After the deep clean the system works again, all the demons have been purged. When I suggest they need to keep the system clean they look at me like I'm a lunatic. "Nobody's got time for that."

Amazingly 2 years later I'm called back to the same place to run another class. I really didn't want to go but they've asked for me specifically. They also want to book the hotel directly. This is unusual and somewhat concerning as it probably means they want me to stay in some fleabag motel.

Turns out they've had another change of management. This time its only management all the lower level guys are the same. The new managers have bought a new system, they've moved it to the data closet which now has redundant power, cooling and filtration and is cleaned on a regular schedule.

Of course their system is working perfectly now, its amazing what a good management structure can do...