How do you remotely support on-prem deployments?
Posted by Durovilla@reddit | sysadmin | View on Reddit | 53 comments
Been asked by a few customers for on-prem deployments, and I'm pulling my hair trying to figure out how to best handle remote support.
When something breaks, what are you supposed to do? SSH in? VPN? Pretty new to this stuff, so I would really appreciate some ideas or pointers!
replicatedhq@reddit
Fwiw Replicated helps with exactly this - we have a platform that helps to handle support, license management, security, etc for on prem deployments of your software. But, I am not focused on selling you anything - just want to help - and we do have an open source product, troubleshoot.sh, that can help you handle this. Totally free, and used by a ton of orgs. It will essentially help you and your customers to create and share support bundles that have all the debugging info you need to manage the on prem installs and upgrades.
Good luck!
DigiInfraMktg@reddit
SSH/VPN works fine at first, but it can become painful very quickly (especially when the network itself is broken).
What usually works better long-term:
A big mindset shift is going from “how do I log into this box” to considering how to manage all your sites if/when something breaks at 2am. You’ll thank yourself later if you design for that early.
Inn0centSinner@reddit
You would tell the customer that they need to set you up with VPN access of their choice and have the necessary permissions to gain access to whatever servers you need to get to. Customers should have at least have a small in-house IT staff to maintain their own networks to set up remote users. Gawd help them if they don't.
zerassar@reddit
Usually you VPN into their network and then connect from there. Unless it's a mediated screen sharing system like logmein/TeamViewer then depending on config VPN might not be needed.
Whatever you do, done go blasting holes in their public firewall so you can have direct access to idrac/ssh/rdp/vnc etc. those should not be public facing. If they are what you're using you need to vpn or something into the network first
uptimefordays@reddit
It depends on your size and scale. If you’re a hyperscaler with 200,000 servers, you probably use out of band management with limited direct access to production hosts. For small companies with 20 servers, probably RDP or ssh directly.
excitedsolutions@reddit
Outbound connecting rmm. This way it relies on the least amount of pieces to be working (just internet, dns resolution and firewall).
Alive-Back-4843@reddit
Each job I’ve had (3) has had an RMM solution. You should have a VPN for the network regardless imo
Durovilla@reddit (OP)
Which RMM software would you recommend?
oddball667@reddit
The big names I know are Kaseya, connectwise, and bombgar
TeamViewer maybe
The right one for you depends on size and budget most likely
cookerz30@reddit
Stay far away from Kaseya
anonymousITCoward@reddit
I'm really not liking vsa x, i find myself calling it a piece of shit more and more often now... can't speak to datto rmm, never used it...
slugshead@reddit
Listen to this admin. They're not wrong in that statement.
oddball667@reddit
I haven't used it in a while, they get shitty or was I too green to know it was shitty?
natflingdull@reddit
NinjaOne isnt bad but its been a few years. I agree with folks ITT that an RMM is a must have here.
Also depends on the OS of the endpoints, the type of server infrastructure on site (if at all), etc.
When I had remote deployments I would configure a fortigate firewall and switch, work with an MSP to ship out equipment and install it, leveraged autopilot for desktop/laptop builds and MDM. Depending on the importance and employee count of the new office I would have some facetime for “opening day” and either fly out or drive to location, use the opportunity to verify the overall config and get a feel for the end user experience, make changes for future deployments if needed.
The last step isn’t a requirement though. There are plenty of modern solutions designed specifically for this use case
Alive-Back-4843@reddit
I use Immy right now which is pretty kickass. Used to use connectwise and Splashtop, former is better
Emotional_Garage_950@reddit
Are we talking endpoints? virtual servers? physical servers?
Durovilla@reddit (OP)
EC2 instances
Festernd@reddit
ec2 instances are cloud, not on prem... so what are you doing?
Emotional_Garage_950@reddit
Depends on what you consider “on prem” imo. We run legacy DCs in Azure cloud, to me that is very much a “on prem” setup, the premises just so happens to be the cloud…
Festernd@reddit
Correct me if I'm incorrect, but ec2 is Amazon's cloud compute offering -elastic compute cloud. By definition it's cloud, not on-prem.
... Ah the guy I replied to edited their previous response to include physical servers.
A legacy DC in Azure is still cloud, so the remote access is largely dictated by the cloud provider.
Emotional_Garage_950@reddit
RMM or remote access client = easiest, VPN with RDP/SSH might be cheaper but more involved setup
KimJongEeeeeew@reddit
This post inspires so little confidence.
If you’re running a business and selling solutions into existing infrastructure, this is absolutely the sort of thing you should have nailed down well before ink meets paper.
OneSeaworthiness7768@reddit
OP’s (hidden) post history is all about promoting their AI coded tools. So little confidence seems like the right assessment. They’re probably selling services they have no idea how to support properly.
Durovilla@reddit (OP)
That's why I'm asking...
I wanna offer the right setup so that I'm prepared when something fails. I haven't deployed anything yet
Throwaway_WiGuy@reddit
Deploy a jump box (server/workstation) into their environment and use that as your workstation, install all of your tools, etc.. on that server/workstation and RDP into it once you have VPN connected. This mean anything you do you will be on their network/vLAN and if you need to start a job, etc.. it will run on that machine and not your local workstation. You can save your downloads and documents on that as well.
Ph886@reddit
Remote into the machine via your preferred software of choice to fix it. This is how major companies do it.
Durovilla@reddit (OP)
I want to SSH into the machines, but they are behind a VPC with no public IP. Do I have to ask my customers to setup a bastion host or VPN?
HorseShedShingle@reddit
Tailscale to an exit node within their network (probably a VM you setup for this exact purpose) and then use Tailscale to remote to anything on that network via the exit node.
jsiwks@reddit
You can use something like Pangolin VPN which enables you to deploy site connectors that do NAT traversal so no public ip is required. It also lets you manage multiple sites pretty easily by just dropping a connector in each on-prem location and defining resources
Calm_House8714@reddit
Yes, you need access to a device on the local network, or access to the local network (VPN). Either one would let you SSH into something that accepts SSH from it's LAN.
I would want remote access to the device itself if applicable. It'd help to know what you're working on. (Are these servers? End user computers Firewalls? Switches? Routers?)
Durovilla@reddit (OP)
Mainly EC2 instances in a VPC
Nuronus@reddit
No, don’t make this complicated, just get a standardized access model rather than creating a new system for every client. There are three options: accessing through a virtual private network (VPN; e.g., OpenVPN or Cisco AnyConnect), a bastion host or jump host, or utilizing services such as Tailscale/ZeroTier, where the server reaches out to you. The first option will work but will be a pain for each client’s IT staff; the second will be more secure and easier to control; and the third is definitely the easiest to implement initially because it avoids firewall issues altogether. If the company is a startup, I have seen it use Tailscale as its only method unless a larger client requires a VPN connection.
However, whatever you decide to do, ensure you lock it down, allow access only via SSH keys (not password-based), limit access to certain users, and enable session logging. Moreover, inform clients about what will happen and what their responsibilities are before they begin working with you to prevent any future issues. The primary issue companies face is creating an ad hoc, temporary solution for access that eventually becomes a permanent headache.
natflingdull@reddit
Great response
Durovilla@reddit (OP)
Thank you SO much this is super useful
the_red_raiderr@reddit
iDRAC/ILO to allow you to physically power a server on or access system console, then a decent remote tool to do stuff on the server. If you get one decent server you can use it as a host, then run several virtualised servers on the hardware using something like Hyper-V. If you want redundancy, two servers in different locations with something like failover clustering. Then it scales up/out from there.
GLHF
natflingdull@reddit
Second the iDrac if you have physical servers. Its a lifesaver
gptbuilder_marc@reddit
Figuring out remote support for on-prem deployments before the first breakage is exactly the right instinct because you do not want to be solving this problem during an incident. The standard approaches are VPN tunnels SSH bastions or remote access tools like Tailscale or WireGuard depending on how much the client controls their firewall. More importantly: do your customers have someone internal who can physically be at the server in an emergency or are you expected to handle everything remotely?
ErrorID10T@reddit
Secure remote access to network stack, either IP restricted to firewall WAN or by VPN. Remote control software like ScreenConnect. iDRAC or iLO for bare metal control in case of OS issues. Always have multiple options to connect in case of issues.
If you have to go onsite to fix anything other than a hardware failure you're probably doing something wrong. That's assuming you're managing the entire stack though. If it's just one server or something, throw ScreenConnect or similar on it and call it good. You can always contact their IT department for assistance if your VM crashes or something.
slugshead@reddit
Guacamole in docker is quite nice.
Acceptable_Mood_7590@reddit
The answer really is depends on your budget. VPN is no compromise. Then firewall, Then you need your ADC Cisco or f5 etc and if you want to micro segment traffic there are options too
TerrificVixen5693@reddit
Depends, BMC like iLo or iDRAC or IPMI is good. If they don’t have that I used RDP or a comparable software for remote support.
Real-Patriot-1128@reddit
For physical servers, ensure you can remote into their bmc/ilo etc… able to rdp into servers…. Etc. establish those 2, then you only go onsite for hardware failures…
Unable-Entrance3110@reddit
For servers I ensure that critical servers have an enterprise (for remote console) iDRAC. For non-critical servers, basic iDRAC is fine (for power on/off).
For desktops, I use FixMe.IT for workstation GUI console access or Enter-PSSession for remote SSH-like command line console access.
I also have some remote PDUs that can turn off/on power outlets which critical equipment remote access equipment and firewalls are plugged in to.
I have a "back door" VPN that only IT can access that gives us pretty broad access, but it requires that it be set up and tested on a registered, domain-joined computer prior to needing it.
Otherwise, I just use the same remote desktop solution (SonicWALL's Cloud Secure Edge) that everyone else in the org uses for remote access.
easyedy@reddit
I connect to the firewall via VPN, and then I can access nearly all machines, including RDP for Windows systems. I support a client like that and have good experience with it. For VPN, I use 2FA. I can also reboot servers or Windows clients without losing connection to the remote site. I can also maintain multiple internal connections at the same time.
GBICPancakes@reddit
I manage a lot of various clients with various methods. In general I prefer to VPN into the office to look at firewall, switches, and servers. Basically "everything in the closet or server room".
I have about 40 separate VPN connections I maintain, using multiple different protocols/firewalls/systems, depending on what the client has installed. Rather than trying to run Cisco/Sonicwall/OpenVPN/Wireguard/etc. all separately, I use a third-party VPN management app that manages the connections and lets me easily toggle them on/off as needed.
That's assuming they're big enough to have a proper firewall and have on-premises servers and complex networks (like a school district or larger business client).
For smaller locations without servers I usually don't have VPN (and often just use Unifi gear there so I can manage the switches and wifi stuff via the remote access console at ui.com)
For user workstations, I use Teamviewer. But any RMM or management software will do.
unknwnerrr@reddit
Are you consulting? Use an rmm
man__i__love__frogs@reddit
VPN or RMM.
Durovilla@reddit (OP)
Which ones would you recommend?
man__i__love__frogs@reddit
The one on their business firewall, or NinjaRMM since you sound like a small shop.
benuntu@reddit
Splashtop for remote desktop support. I run Unifi gateways at remote sites with site-to-site VPN connections, so can also remote in to servers, switches, etc. to manage infra.
No_Investigator3369@reddit
Technically a secure data center doesn't allow the people who configure to walk in the DC without an escort and under strict guidelines. Sign off on having a technician be able to bring an IP kvm into the DC. You tell them the ports to plug into, they do the physical part, you do the brain part.
ccosby@reddit
When i worked in a msp we had had an agent. Would hit the vpn if i needed access to something like a lights out card.
Now where i work we dont have servers at the offices. Have out of band management devices though at the branches(open gear). They have cell backup and give me serial access to the switches and firewalls. Networking gear is connected to a smart pdu which is also connected to the open gear. I can power cycle the switches remotely if someone writes a bad config, get in via console if needed etc.
Haven’t had to use the out of band stuff much but it’s been hella useful when we have needed it.
Double_Confection340@reddit
We use endpoint central to remote into the machines.