Network security breach - but not as you know it
Posted by mywifeshubby@reddit | talesfromtechsupport | View on Reddit | 53 comments
I'm in the Johannesburg area of South Africa, where I have been doing network and IT support (among other things) for a variety of clients, some of which are located in a bad part of town. And in Jo'burg terms, 'bad' is, indeed, pretty bad.
Recently I got a distress call from one of them. After the break the boss had arrived at the office early, and was unable to log into the network server. His office is on the ground floor. So I drove down there as usualy, with all car doors locked, my head on swivel and the local private security response guys on speed dial but fortunately no mishap this time) and made my way to the server room on the second floor.
Where the cause of the problem became immediately apparent: some person or persons unknown had gotten in through the roof and grabbed what they could. The server wasn't down, it was gone.
Welcome to Johannesburg, where the biggest network security risk is not theft of just data, but theft of your network infrastructure itself!
Fortunately the boss (who was a classic car afficionado) firmly believed in the 'spare wheel' approach and had kept their previous server in his junk room at home. One hurried trip there later I found myself trying to dust off and revive an ancient Compaq Proliant of 1998 vintage- the sort of kit that has its plastic front bezel and panels discoloured to a bright shade of yellow. With the aid of some compressed air to blow out the cobwebs, two ancient hard drives that I keep in my on junkbox for just such an occasion (this is Africa after all, so one does meet a need for obsolete hardware parts every now and then) and a lot of grunt I managed to get it going in time before work started again the next morning. It was so slow it needed a tow rope, seeing as a few old 100baseT hubs was the best we could do on such short notice, but it did get them going until proper kit could be sourced from the insurance claim. I shudder to think what their premium must be like now, in that part of town.
IT support in downtown Jo'burg... Never a dull moment!
lovethebacon@reddit
Oh sweet I'm going to start checking Gumtree for a cheap server. What specs on it?
BTW, bobshop has some decently priced second hand enterprise gear to keep for spares. Plenty of Cisco switches for no more than a few thousand and similarly ancient but functional rackmount servers.
Stryker_One@reddit
And you might even find a bookshelf on there.
Toratchi888@reddit
I wonder if it is worthwhile to camouflage newer technology by putting older stuff in front of it. Put the server in a room called "broom closet", and put the old server in the open, put some blinking lights on it, give it a new paint job, etc.
KenseiSeraph@reddit
Camouflage can definitely work. Years ago my family had a break in and the robbers took my brother's PC but not mine. Mine had the case off (cooling issue I think) and was in the options menu for a game so the robbers must have thought that it looked like it was broken and I was in the process of repairing it.
Those same robbers were unable to figure out how to unscrew the cable for the monitor and so decided to just cut the cable and take it.
georgiomoorlord@reddit
DVI cables.. sounds right.
Z4-Driver@reddit
Or VGA
himitsumono@reddit
My comedic dylexia rendered that as "Oy VGA".
Computers got Yiddish now. Who knew!
WhoSc3w3dDaP00ch@reddit
i know someone who hid(es?) his laptop in a closed pizza box...
His home was broken into. The thieves stole other stuff but missed the $2K laptop on the kitchen counter...
MikeSchwab63@reddit
I put pizza boxes on my windshield before snowstorms, food side out.
Silent_Ad_8672@reddit
Now I'm thinking about the fresh prince of bel air episode where Will would hide his books in a pizza box, and got jumped because the other guys were hungry 😂
OldMetalHead@reddit
Good thing the thieves weren't hungry.
evilmonkey853@reddit
Or take the guts out of the old servers and put the new smaller servers inside…
ozzie286@reddit
You guys assume that thieves know enough about servers to recognize old vs new equipment. I'll pretty much guarantee they were just grabbing anything that looked techie that they could get their hands on, regardless of age.
JaapieTech@reddit
Bru this stuffs got copper inside lets take it
See also - stealing high-voltage electrical cables directly from live pylons
Smith6612@reddit
> Bru this stuffs got copper inside lets take it
Discovers the cable is actually Fiber.
Ich_mag_Kartoffeln@reddit
That happened near where I live a few years ago. Thieves periodically steal quantities of wire from the railway signalling system for the copper.
Over time (as other upgrades have happened) parts of the signalling system have been upgraded to fibre optic.
One day the thieves went to nick the copper wires, stuck a 15" demo saw through the signal wiring -- only to discover it was fibre optic.
It took most of the day to repair the damage and test the signalling system before that train line could resume operation.
mywifeshubby@reddit (OP)
That happened close to my house just last week. Half the area is without fibre right now. Which is why I still use fixed wireless. It's fast enough for me and a damn sight more reliable than fibre in my neck of the woods. 😄
vinyljunkie1245@reddit
And railway lines
Head_Razzmatazz7174@reddit
I can remember stories of finding would be thieves near the base of power poles who had been electrocuted. It doesn't happen often, but it does happen.
MichigaCur@reddit
Came across it a couple of times back I the early 2000s.
Had a site... best we could figure out was one guy had his hands in the generator for something, another guy cut the mains. Severed Mr generator guys forearm... Trail of blood back and forth to a set of tracks. Give him credit for still working on stealing shit until he collapsed. Either that or he freaked out and buddy finished the job once he collapsed... But he made at least 3 trips between the tracks and the generator.
One of the more wild things I've seen when copper was through the roof was they'd take heavy duty pickups wrap a chain around the coax cut the lines at the equipment and gun the truck trying to rip the coax down.
NotYourReddit18@reddit
Yeah, using a big old case as camouflage for the newer, smaller server inside is only the first step.
Then you bang it up a bunch and add scratches and stains so that it does look old and battered to everyone.
Now you place it next to a cable conduit so that you can cut a small hole in it's side to route network and power from the case directly into the conduit without it being visible from the outside.
The next step is bolting the old case into place so that it can't be easily moved anymore.
And after placing the new server inside in powering it on you then close up the side panel and lock it shut with a case lock. Welding is an alternative, but makes opening it back more difficult.
The finishing touch is putting a bunch of boxes (at best filled with random junk so that the thieves won't want to take them) around the old case so it can't be seen anymore.
mywifeshubby@reddit (OP)
And then they simply break everything trying to get to the hardware they know must there. Trust me, with these guys you can't win...
ozzie286@reddit
Never weld a case with electronics inside of it, you're just asking for trouble.
mywifeshubby@reddit (OP)
Exactly this. \^\^\^
Also, often the damage they cause during the break-in exceeds the cost of the kit they steal. Post-heist repairs usually cause more downtime than replacing the vanished hardware.
Weird_Technology_282@reddit
Right, a "bait" server, and the real one hidden in a secret server room!
ratsta@reddit
My boss spent 20+ years in Papua New Guinea and worked for a telco for a while. PNG is a Pacific island due north of Australia. It's a beautiful, tropical paradise that was fought over by various imperial powers and finally became an independent nation in 1975. The tribal cultures are protected by its constitution and only 15% of people live in the cities. Not sure about resources but it's location at the Australian end of Oceania (with SE Asia at the other end), makes it very important for commerce.
My boss has regaled us with many stories of the shenanigans that happen here as a result of interaction between traditional cultures and modern commerce. Just yesterday we were talking about about some of these issues.
He said that major communications lines go from AU to PNG to Guam then on to the US. It's 300-500km from the south side of the island to the north side so the logical way would be to bring the cable in from the south, run it overland to the north then back undersea on its way to Guam. Only one small problem... the land owners quite reasonably demand compensation for clearances and have demonstrated that they're quite happy remove infrastructure that gets placed without consent AND much of the territory is contested at a local level which means even if an agreement was reached with one group, another would come along and cut things up. It was actually cheaper and safer to run a chain of undersea cables from south to north around the coastline, touching in at various locations along the way.
Some time in the 90s he was once called to investigate why a cell tower on a mountain top appeared offline. If you google "PNG highlands" you'll see some utterly gorgeous landscape and understand the difficulty of maintaining high tech infrastructure in the region. There are no roads these locations; everything needs to be done by helicopter.
So Bob takes a whirlybird on a one hour flight. They had great difficulty finding the location. You'd think it was hard to miss a cell tower, particularly when you have GPS. Then they discovered the reason. There was no tower, just a concrete pad.
Of course there was no cell signal, what with there being no tower, so all he could do was snap a few photos and tell the pilot to head back. When he called in with the news, his boss unleashed a stream of profanity to make a sailor blush. Bob learned that this was the third tower in a 50km radius to simply disappear. In each case, despite the lack of roads, thieves had made off with an entire tower, two diesel generators, the control box and the fuel shed!
DiodeInc@reddit
They stole the whole fucking tower???? How the fuck did they not die???
ratsta@reddit
Maybe the thieves had guys who knew how to demolish towers? It was a cell tower, not electricity, so gravity is the main enemy.
DiodeInc@reddit
Lol if they had put that energy into engineering jobs, they'd have materials and skills to build cell towers :P
ratsta@reddit
Story of humanity, right? Starts back at school. Some people put more effort and skill into avoiding study than it would take to actually learn and earn their qualifications.
This is a comic from 2008 that has stuck in my mind: https://wondermark.com/c/414/
DiodeInc@reddit
Very true
LeahInShade@reddit
This is a fascinating story! Should definitely be in the list of the biggest things ever stolen! (There are a few such compilations online, but Idon't recall them listing a PNG tower 😁)
evangael@reddit
I, as naive as I am, couldn't imagine people in the networking world having to face such problems whereas my view of a datacenter or equipment rooms are those of heavily secured facilities where even a fly needs to get valid credentials to even buzz around.
Cool story 😄
DiodeInc@reddit
Man, you wouldn't want a fly in your server room. That shit gets into a server vent and potential boom
JaschaE@reddit
There is a VERY large Multinational in my country. Chances are you have heard of them ~~and cursed their name~~. They run a datacenter in one of the richest areas of the country. In a suburb of a affluent city. Their Security is tight. Single person entry, everybodys ID checked, credentials verified against "expected visitors" list, just stopping short of shoving a flashlight up your ass upon exit... the works.
Fairly recently, a large number of their hard-drives found their way to ebay. Not entirely sure how.
mywifeshubby@reddit (OP)
In my experience that's usually an inside job. YMMV of course.
JaschaE@reddit
Thankfully not my problem. Sadly not my extra income.
NotPrepared2@reddit
Must be Oracle!
JaschaE@reddit
Wrong continent\^\^
mywifeshubby@reddit (OP)
Or MICROS\~1
dreaminginteal@reddit
Could be any of the multinationals, frankly....
masterventris@reddit
There is always one person for whom the security is a little bit more lax.
And that is the head of security....
Claidheamhmor@reddit
:D :D :D
I mean, ouch. Are we talking Marshalltown bad, or Hillbrow bad, or Doornfontein bad?
Through the roof too. Those guys are inventive.
mywifeshubby@reddit (OP)
Jeppestown bad. (I'm seriously considering firing that particular customer; they're the last one in that area and I'm more and more inclined to treat the entire Jo'burg CBD as a no-go zone. Being hijacked there once was enough.
Claidheamhmor@reddit
Gotcha. Been there to get spare parts before.
mywifeshubby@reddit (OP)
I strongly suggest you get those spares from somewhere else from now on. Empty buildings in that area are stolen brick by brick, and there are shacks on the track where the railway used to be (the rails having been stolen long ago). Taxi wars, hijackings and muggings are common there these days. I'm taken to avoiding the entire area. Frankly I can't understand why businesses are still located there; if it were me I'd have pulled up stakes and moved elsewhere. Even rent-free the cost of doing business there must be higher than elsewhere due to the high levels of crime alone.
But I'm straying off-topic. 😄
vinraven@reddit
This is when you setup the network server at a more secure remote location.
mywifeshubby@reddit (OP)
Except that connectivity in those parts is terrible. So remote access is slow, intermittent or (often) entirely absent.
Frankfrombluvelvt@reddit
My friend lived in a rough neighborhood in the states, someone dropped a live transformer from a telephone pole, needed a fix bad, I guess.
AllSeeingAI@reddit
Ah South Africa. Every time I think I know what to expect from a story about it, I'm still surprised.
zeus204013@reddit
Chappie robbed that!!! /s
zeus204013@reddit
South Africa, land of the Elon...
I saw DISTRICT 9, maybe the alien stole the hardware, maybe to assemble a ship to return home!!!
/s
Challenge: What's more dangerous?
_ A bad place of Johannesburg _ The worst of El Conurbano (in Buenos Aires province). Land of Kici-love.
MoneyTreeFiddy@reddit
TL;DR, A Part Hide is still happening in South Africa.