For the multiple keystrokes issue, if every other possibly have been covered, firmware update, running hard reset, then check for network latency.
This happened in the past for us, and a faulty switch was the problem in one data center.
Also, I would assume that the servers running Linux, since this was posted here. Why would you not have ssh access to them? How are they updated/managed/monitored then?
You will need ssh access to the server(s) in order to properly troubleshoot this.
Since you specified iDRAC9, then you should check if you are passing the correct parameters to the kernel.
I don’t think it is network latency but that’s a good suggestion to check. I’ll also compare the serial setup to yours, thanks!
Systems are Linux, and OS and iDRAC are all up to date.
As for why no ssh - the sites local policy. The administration team on site has ssh access on that network, but I’m not located there but they bring me in to fix issues they can’t solve locally.
No VPN allowed before anyone suggests that. Web access is all that is allowed. They carry over patches and updates from a remote location that is part of their organization.
I’ve discussed something like Kasm so I can at least get a virtual desktop over the web, but that is pending approval.
It is a weird situation but they pay well so here I am.
So the latency wasn't the issue, confirmed that with some testing with the network folks on site.
Linux console settings were also correct and nothing I could change there seemed to have any effect.
I've asked the local site guys to setup a test where setup one of these iDRACs behind the same security stack but locally on the LAN to see if they can replicate my experience. While we didn't find any latency issues, I still have some questions on if the security devices are having any impact.
Thanks for the update. I appreciate it. Maybe run a tcpdump and see if you can spot anything not normal. I think that's all can be done at this point. It could be some weird firewall policy that is causing that, since there was no network latency detected, or maybe one of the network devices on that path having an issue.
Ok, dumb suggestion because you probably already thought of it or it's not possible...
Reverse SSH?
IDRAC -> shell -> SSH towards a host you control you can SSH
SSH that host -> SSH into the reverse, boom you have an SSH connection that dies off when the IDRAC is closed.
(Or you can tmux it so if your webshell dies you don't have to redo it all, just remember to shut it off when you log off)
That’s a good idea but I can’t use it here as these hosts can’t ssh back outside of their network. The network these live on is very very very locked down. I had to make sure I had a fixed IP/IPv6 so they could get me an exception for access but nothing beyond 443/tcp is allowed.
And remember I don’t have to do daily management of these systems, I’m just the one they call on to fix things when it is beyond their own staff’s capabilities.
Cause if they don't can you sneak the reverse SSH out to your port 443, and you iptable it to their pub IP on your side, it's not really different than a long lasting HTTPS?
Glad you don't have to deal with them daily cause that would be maddening... The less I interact with IDRAC the happier my life is! 😂
Like, the risk is legit a big incident that's urgent and you get stuck with a shitty IDRAC console single connection when you could actually do work correctly with the correct tools... But at this point it's a shadow IT vs manglement issue...
Honestly, because I suspect they're actually using a "nobody said we couldn't give console access in the mgmt VLAN", you're probably in red tape territory... But the way safer option is a bastion with something like Teleport that does proper auditing, ecdsa keys passwordless, with potentially a 2FA on the SSH (Teleport does that too, but I bet every other product does too, it's the one name I remember)... Double that with a keepalived or BGP on a fabric network for redundancy, access restricted to your pubIP on the VIP, box in your own special VLAN that routes only to "your" hosts they can firewall it to hell and back, let only 22/tcp and disable port binding on the bastion... They can even shut the thing off and turn it on when they need you, if it's a VM it takes 30sec to boot you'd not even be done with your coffee.
(Not that I don't think you've not thought of that, but if you wanna show the boss that "random stranger on the internet with a very "securing" username agrees with me", feel free ;) I was a platform engineer for a few years, and managed our bastions (just openssh tho, we were considering Teleport), for context)
I’m fairly sure these hosts have no outbound access to anything outside of their network. And they do have a fairly large security stack doing who knows what to any traffic coming in or out. The other commenter mentioned network lag so I’ll be hitting up their security guys to see what latency might be getting introduced by the stack.
A large portion of their servers are air gapped from everything - they bring over data to process, patches, code, etc. The on site guys have zero access from outside - if someone calls in an issue, needs data moved “right now”, etc they have to come in. They also get paid well but I’d never sign up for that job.
It is a weird situation for sure, but that’s the way they want it.
Had to deal with that stuff for a bit and hated it from day two.
But yeah, someone else already suggested the good ol' reset in the teeth and upgrades, and once it got to that we usually ended up filing an RMA or decommissioning (depending if new or not), cause they're such a pain!!
As for why no ssh - the sites local policy. The administration team on site has ssh access on that network, but I’m not located there but they bring me in to fix issues they can’t solve locally.
Why can't you use a jumpbox or something? Giving you iDRAC access poses more of a risk than ssh.
I have the same problem with my idracs, i just dont use them often enough to be troubled by it. But seeing as it is just a vnc, can you use a different client?
researcher7-l500@reddit
For the multiple keystrokes issue, if every other possibly have been covered, firmware update, running hard reset, then check for network latency.
This happened in the past for us, and a faulty switch was the problem in one data center.
Also, I would assume that the servers running Linux, since this was posted here. Why would you not have ssh access to them? How are they updated/managed/monitored then?
You will need ssh access to the server(s) in order to properly troubleshoot this.
Since you specified iDRAC9, then you should check if you are passing the correct parameters to the kernel.
This worked for me.
buzzsawcode@reddit (OP)
I don’t think it is network latency but that’s a good suggestion to check. I’ll also compare the serial setup to yours, thanks!
Systems are Linux, and OS and iDRAC are all up to date.
As for why no ssh - the sites local policy. The administration team on site has ssh access on that network, but I’m not located there but they bring me in to fix issues they can’t solve locally.
No VPN allowed before anyone suggests that. Web access is all that is allowed. They carry over patches and updates from a remote location that is part of their organization.
I’ve discussed something like Kasm so I can at least get a virtual desktop over the web, but that is pending approval.
It is a weird situation but they pay well so here I am.
researcher7-l500@reddit
I appreciate the explanation. If this was fixed, please keep the thread updated.
buzzsawcode@reddit (OP)
So the latency wasn't the issue, confirmed that with some testing with the network folks on site.
Linux console settings were also correct and nothing I could change there seemed to have any effect.
I've asked the local site guys to setup a test where setup one of these iDRACs behind the same security stack but locally on the LAN to see if they can replicate my experience. While we didn't find any latency issues, I still have some questions on if the security devices are having any impact.
researcher7-l500@reddit
Thanks for the update. I appreciate it. Maybe run a tcpdump and see if you can spot anything not normal. I think that's all can be done at this point. It could be some weird firewall policy that is causing that, since there was no network latency detected, or maybe one of the network devices on that path having an issue.
Loveangel1337@reddit
Ok, dumb suggestion because you probably already thought of it or it's not possible...
Reverse SSH?
IDRAC -> shell -> SSH towards a host you control you can SSH SSH that host -> SSH into the reverse, boom you have an SSH connection that dies off when the IDRAC is closed.
(Or you can tmux it so if your webshell dies you don't have to redo it all, just remember to shut it off when you log off)
buzzsawcode@reddit (OP)
That’s a good idea but I can’t use it here as these hosts can’t ssh back outside of their network. The network these live on is very very very locked down. I had to make sure I had a fixed IP/IPv6 so they could get me an exception for access but nothing beyond 443/tcp is allowed.
And remember I don’t have to do daily management of these systems, I’m just the one they call on to fix things when it is beyond their own staff’s capabilities.
Loveangel1337@reddit
Deep Packet Inspection too I guess?
Cause if they don't can you sneak the reverse SSH out to your port 443, and you iptable it to their pub IP on your side, it's not really different than a long lasting HTTPS?
Glad you don't have to deal with them daily cause that would be maddening... The less I interact with IDRAC the happier my life is! 😂
Like, the risk is legit a big incident that's urgent and you get stuck with a shitty IDRAC console single connection when you could actually do work correctly with the correct tools... But at this point it's a shadow IT vs manglement issue...
Honestly, because I suspect they're actually using a "nobody said we couldn't give console access in the mgmt VLAN", you're probably in red tape territory... But the way safer option is a bastion with something like Teleport that does proper auditing, ecdsa keys passwordless, with potentially a 2FA on the SSH (Teleport does that too, but I bet every other product does too, it's the one name I remember)... Double that with a keepalived or BGP on a fabric network for redundancy, access restricted to your pubIP on the VIP, box in your own special VLAN that routes only to "your" hosts they can firewall it to hell and back, let only 22/tcp and disable port binding on the bastion... They can even shut the thing off and turn it on when they need you, if it's a VM it takes 30sec to boot you'd not even be done with your coffee.
(Not that I don't think you've not thought of that, but if you wanna show the boss that "random stranger on the internet with a very "securing" username agrees with me", feel free ;) I was a platform engineer for a few years, and managed our bastions (just openssh tho, we were considering Teleport), for context)
buzzsawcode@reddit (OP)
I’m fairly sure these hosts have no outbound access to anything outside of their network. And they do have a fairly large security stack doing who knows what to any traffic coming in or out. The other commenter mentioned network lag so I’ll be hitting up their security guys to see what latency might be getting introduced by the stack.
A large portion of their servers are air gapped from everything - they bring over data to process, patches, code, etc. The on site guys have zero access from outside - if someone calls in an issue, needs data moved “right now”, etc they have to come in. They also get paid well but I’d never sign up for that job.
It is a weird situation for sure, but that’s the way they want it.
Loveangel1337@reddit
Ah shit, that does look like a headache!
Had to deal with that stuff for a bit and hated it from day two.
But yeah, someone else already suggested the good ol' reset in the teeth and upgrades, and once it got to that we usually ended up filing an RMA or decommissioning (depending if new or not), cause they're such a pain!!
Good luck anyhow!
Hotshot55@reddit
Why can't you use a jumpbox or something? Giving you iDRAC access poses more of a risk than ssh.
buzzsawcode@reddit (OP)
Because their management said no - I don’t have control over their policies. I agree that a jump box would be ideal but I have to play by their rules.
aenae@reddit
I have the same problem with my idracs, i just dont use them often enough to be troubled by it. But seeing as it is just a vnc, can you use a different client?
buzzsawcode@reddit (OP)
That’s a great suggestion, let me see if I can get that to work.