Need help with BMC / ESXI Reset on a Hitachi advanced server DS120
Posted by ChaoticTech@reddit | sysadmin | View on Reddit | 17 comments
Story:
I recently started a new job taking over for a system admin that documented nothing literally nothing no passwords no network diagrams etc.
The biggest problem:
I learned on day 1 we are currently locked out of our esxi environment running on a Hitachi advanced server DS120.
This server is apparently running our entire critical infrastructure and to rebuild it would be extremely expensive due to the medical vendors that would need to get involved.
Additionally these systems haven't been rebooted in years and we don't know if there are backups anywhere in the environment.
Solutions I'm considering:
I opened a ticket with Hitachi support but don't know when I'll hear back from them.
I checked Hitachi documentation online and the answer is unclear. Their official documentation is vague and I couldn't find and videos on YouTube about what's happening behind the scenes during a BMC reset.
I want to reset the BMC to then reset the ESXI password but it's unclear the impact this will have on the virtual machines. Gemini said if I hold down the (I) button on the front panel for 30-60 seconds it would reset the IP and credentials but it's unclear as to whether the VMs will be impacted.
Normally I wouldn't blink twice to try something like this but if it does impact the VMs that becomes a very sudden and abrupt outage with the only recovery path forward being a very expensive rebuild alongside vendors.
Resetting both the BMC and esxi virtual environment seems like the quickest path forward but because I've never worked with this hardware before I'm unsure how it will behave or impact the production virtual environment.
On the front panel there is also a reset button but it's unclear what the reset button does.
Also contemplating buying N-able to perform a system level backup and then restoring it on our other production esxi hosts.
TLDR:
Has anyone ever reset the BMC on a Hitachi advanced server DS 120 using the (I) button and did it impact your production environment if you did?
MrYiff@reddit
As others have suggested, you likely need to reinstall ESXi in order to regain control over the root account, this can normally be done safely as the installer should detect the existing datastore and prompt you to keep it.
I would always recommend that you check the state of your backups before attempting this so you have an option if something goes wrong.
Another consideration you may need is drivers for your server, I've not touched Hitachi servers before so I'm not sure if they provide something similar to Dell and HP where there is a special ESXi install ISO that contains all the relevant Dell/HP drivers and software.
St0nywall@reddit
Why don't you live migrate (vMotion) the VMs to another ESXi host?
After they are vMotioned you can rebuild the server and add it back into the cluster.
All-in-all could be a days worth of work at most.
ChaoticTech@reddit (OP)
I"m locked out of one of my ESXi environments via direct host and it's not being managed by vcenter so I'm unable to vmotion from one host to the other.
St0nywall@reddit
Are you not licensed for vCenter or was it never setup or you can't access it?
What version of ESXi is running on the host?
ChaoticTech@reddit (OP)
It was never setup for that environment. I'm not sure of the version but it would be a flavor of 6 or 7.
St0nywall@reddit
There is a procedure for 6.5 and below (I think) that allows you to change the root password. Doesn't always work though.
Anything newer, like 6.7, 7 or 8 requires you to do a "re-install". Think of it as an inplace upgrade but with the same version and build. It requires the server to be restarted, so all VMs will need to be shutdown.
ChaoticTech@reddit (OP)
Thanks for trying to help me out. I'll keep that in my back pocket for Monday assuming I get the scheduled down time I'll need. Enjoy the rest of your weekend!
FreakySpook@reddit
One thing to check before you go the nuclear option as well is see if there is an 'ESX Admins' security group in your AD, and if there is a computer object for the ESXi host. It may have been domain joined and you can log in with a user who is a member of that specific group.
St0nywall@reddit
No worries, wish I could have done more for you.
Good luck! :)
zaphod777@reddit
I would make sure that you have system level backups of the VM's no matter what path you take.
pdp10@reddit
If the BMC needs to be reset from console firmware after a reboot, then you'll need to take an outage window. There's no question that this can be done one way or another, but you may need an outage.
This "(I)" button is obviously for the firmware/BMC. It's probably only affecting the IP address of the BMC, but you should take an outage to do this.
ESXi password recovery depends on several factors. Apparently, editing the on-disk password is only practical up to ESXi 6.7. I'd take an outage for most anything that you're going to attempt here, emphatically for anything where the host must be in "maintenance mode".
mcapozzi@reddit
I had to reset almost every iDRAC in my last job, resetting iDRACs, BMCs, and ILOs are harmless procedures.
doglar_666@reddit
I cannot speak to helping with your technical query but, to me, this reads like a serious problem for "the business", not you, as IT. The financial mathematics should be being run RE: legal route to gain the credentials from the previous admin vs total time, effort and cost of the rebuild. My understanding is that handover of credentials is something your company can demand from the previous employee.
If I were in your position, I would refuse to touch prod. I would want a signed understanding from management that it's already compromised and at risk, with an unknown amount of downtime being inevitable. Without that assurance, you are setting yourself up to be the fall guy for the previous admin's sins.
ChaoticTech@reddit (OP)
Thank you for your perspective these were things I was already contemplating but hearing somebody else echo the sentiment is genuinely helpful thank you for your time today.
doglar_666@reddit
I understand it's very easy for me to make such statements, given I have no stake in the outcome and will feel no repercussions should things go south. I just feel it is always worth stating when such posts appear, as it is easy to lose perspective of what is reasonable when it is your lived reality. Your employer allowed this mess to happen, so are complicit in any negative outcomes, even if they don't wish to admit it. You are just the messenger.
I wish you the best of luck, minimal downtime, and that "shooting the messenger" isn't your employer's MO.
itworkaccount_new@reddit
Resetting the BMC will have no impact on the running VMs. However if you don’t know your VWware host root credentials and need to reset those the process depends on the version you are running and there is no “supported” way.
The supported way is a reinstall of ESX and your VMs will go down during this. During install it will detect your existing datastore. Tell it to preserve that during install and you’ll be able to re-register all the VMs and power them back on after the install completes.
ChaoticTech@reddit (OP)
Thank you for the information I figured resetting the BMC wouldn't have impact but I really need to make sure I get this right.