What A Day This Has Been... and not over yet!

Posted by Individual_Fun8263@reddit | sysadmin | View on Reddit | 39 comments

Production manger stops me this morning and says he needs me to wipe and reinstall server OS on a box on our ABB manufacturing/production network. Anything else I need to do he says tell him and he'll clear it off my schedule. I don't normally support the production network, but help support them for some specific tasks. So okay, sure, I already have an ISO of the server OS here. Better go get the updated drivers from HP. HP's web site says: - "HPE regrets to inform you that we are unable to act on your access request at this time due to technical issues we are currently experiencing with user validation. To proceed please submit a site support request for assistance and we will help you shortly." Hmm, okay, I think I have a slightly older version around somewhere. So no problem. Just gotta put these on to a USB stick: - Corporate security just implmented a new policy over the weekend blocking external USB devices. When I brought up last week that IT needs USB on a regular basis, they just said they would deal with exceptions as needed. Manager approved an exception for me, but won't be done until tomorrow. Alright, well they said the policy was for workstations and laptops, so I'll just jump the files off my PC to a USB stick on a server: - Corporate accidentally applied the USB policy to my servers as well. Worst case, I'll copy the files off my PC through a jump box over to the ABB production network and then to the server from there. - Since the server is a backup, it has no live network connections. Alright, patched iLO port to the network, maybe we can connect and set it up through iLO. - iLO is not configured and there is no iLO software installed. Alright at least I can get on the console, so will just reboot it and set up the iLO. - DHCP is not enabled on the network (rightuflly so, because security) Maybe I can just patch the iLO over to my office network for now. - No ports available on the switch (no way I'm disconnecting anything that is related to production!) Only thing I know I can disconnect is this jump box. That worked and iLO got an address. - Now waiting for some other user who need to the jump box to start complaining.

[-]

SikhGamer@reddit

...and no one in that business realises how much that "security" is holding them back.

dlucre@reddit

I guess it might also hold back an adversary, but yeah, massive productivity losses all around in this story. But if it prevents the network from being nuked from North Korea I guess it's worth it.

Yeah but it doesn't. You know it doesn't. They know it doesn't. But it's all a tick box exercise.

AxisNL@reddit

If you have the SPP image on a stick, you should be able to boot from that, unless usb is disabled in the bios. And I would advise them to have every ilo connected and up to date, and for example in a tool like hpe oneview (free in monitoring mode). If shit hits the fan, you don’t want to think about all this mumbo jumbo 😂

visibleunderwater_-1@reddit

Yeah, I would hate to see what happens if there is some crypto-related security event and a system administrator can't use USB. The whole "We will deal with it when it comes up" is a bullshit answer, cause now it HAS come up and is still awaiting implementation after managerial approval.

I'd document all this into the ticket, add the original requestor into the CC, and shoot it back off as "on hold, waiting on business". At this point the OP is trying to circumvent ISSEC policy; it's not worth potentially loosing his job over it.

KickedAbyss@reddit

Your issue started when you said HPE.

vCentered@reddit

I'd have copied the ISO to my laptop, slapped an IP on the ilo, configured the NIC on my laptop to be on the same network and cabled directly to it. Should work well enough to install the OS.

jamesaepp@reddit

OP have you ever heard of paragraphs before?

CountGeoffrey@reddit

stuff of legends

hankhillnsfw@reddit

Worked at 3 it enterprise orgs.

All had strict usb device policies.

It’s standard operating model in 2024…get with it.

mfinnigan@reddit

`DHCP is not enabled on the network (rightuflly so, because security)`
That's definitely not security, just a rake you've left on the lawn to trip on and get the handle embedded in your face.

placated@reddit

It’s the same mentality that will argue until they are blue in the face as to why hardcoding interface speed and duplex is “best practice”.

osxdude@reddit

disabling DHCP is not security just FYI

mercurygreen@reddit

The thinking is if an idjit (not an intruder) tries to plug in nothing works so they're confused and go try elsewhere.

Also, if it's the server VLAN I generally want everything to NOT need DHCP to start, so... static addresses. Since there shouldn't be any dynamic addresses, lack of DHCP just means "This is not setup completely."

I know, it's old school thinking, but if it stops ONE manager that thinks he knows better...

Can confirm that it’s old school thinking.

The USB thing is what amuses me. It's like someone watched "Firewall (2006)" and thought the ONLY way to transfer large amounts of data is via USB...

easier2say@reddit

Really? I thought it was. I'd like to know your point of view in this

Your physical security controls should be strong enough to prevent unauthorized people from accessing network ports.
It would be trivial for a bad actor to discover an IP to use.
A reasonably sized organization should have a NAC solution which is the actual solution to this problem.

Disabling DHCP is at best security through obscurity at the cost of much more network administrative overhead.

pyrrhios@reddit

It absolutely is. Unless you're using DHCP for pre-mapped MAC addresses.

OCAU07@reddit

I'm curious on this one.

Lets forget physical security about getting access to the switch.

If I have a switch and multiple vlans configured on various ports without Dhcp on the switch. If someone unpatched a port and connects a device, how is not having Dhcp not security?

ianpmurphy@reddit

It would typically take no more than a few minutes to work out the subnet manually.

one could simply open Wireshark. something somewhere can blast out an ARP request or multicast or something else that reveals an IP on the network. even trial and error is also an option since there are only a few IP blocks reserved for private LAN use so it wouldn't be too hard to iterate through them and find what you need eventually

cosmos7@reddit

Yeah I've never understood the reasoning here. Trivial to sniff / ARP to figure out some network details.

puffpants@reddit

800xa?

PrettyAdagio4210@reddit

Installing a server OS on a production box during office hours is indeed a bold strategy, Cotton.

Agreeable-Comfort567@reddit

gif

buy_chocolate_bars@reddit

How's it a production box without an OS on it?

GMginger@reddit

How is it a production box if the iLO hasn't been configured / licensed too?

Individual_Fun8263@reddit (OP)

It's a failover/secondary server.

mortsdeer@reddit

This is the way.

Gotcha. Sounds like a day ending in Y to me, all the same.

Capable_Tea_001@reddit

Not a useful failover solution yet though is it 😜

medium0rare@reddit

Have you tried using the SUM tool to patch the system? I believe it can do windows drivers on configured windows nodes and firmware updates on the server. Might have trouble getting the ISO if HPE's site is being even more garbage than usual.

pdp10@reddit

Strongly consider putting your drivers for machine builds into an artifact repository, if second or third party drivers are needed.
If DHCP or auto-IP reduces security, then your environment is baking in some bad assumptions.
You can still run a DHCP server with DHCP Reservations if everything is hardcoded.