That time I had to SSH into a Roomba to fix a VPN issue
Posted by votekick@reddit | talesfromtechsupport | View on Reddit | 74 comments
It’s been a while since I posted a story, but this one came up in conversation the other day and I figured it was worth sharing.
Back during Covid, when everyone was working remotely, I had an issue escalated to me from our helpdesk.
They’d already gone through the usual steps — repairing the connection, reinstalling the client, testing other credentials — but nothing worked. The user would hit connect, enter their password, and the moment it connected, it would immediately disconnect.
Now, I’ve learned not to blindly trust “I already tried that” because I’ve been burned before when someone skipped the less-obvious step. So, I started checking things myself.
Some background: a few of our older clients had set up their own networks before we came on board. Normally, when we take over, we standardize things — readdress the network, VLAN off cameras and guest Wi-Fi, that sort of thing. But this particular client never went through that process. Their office at the time was literally just a converted residential house, with desks in every room.
That meant their office network was still on 192.168.1.x — the same subnet as the user’s home network.
I ran an IP scan and noticed a device on 192.168.1.254, which happened to be the same address as their office firewall. So the moment the VPN connected, traffic defaulted to the local device instead of tunneling through, and the connection dropped.
The device didn’t have a web interface, and a MAC lookup just came back as some generic manufacturer. But it did respond on Telnet and SSH. After some questioning, we figured out what it was: their robot vacuum cleaner that the user’s husband had set up. Apparently, you’re only supposed to manage it through the app, which explained the lack of a web interface.
I ended up finding default credentials online, SSH’ing into the thing, and readdressing it to resolve the issue.
To this day, I still enjoy watching people’s expressions when I ask:
“Did I ever tell you about the time I had to SSH into a Roomba to fix a VPN issue?”
TL;DR:
When you onboard a client, push harder to change their office network so it’s not sitting on the default subnet.
Disturbed_Bard@reddit
And that is why I immediately never ever set up client's office subnets as 192.168.1.0 or 10.0.0.0
Asking for trouble when VPNs start coming into the equation.
djingrain@reddit
192.168.69.x ftw
SeanBZA@reddit
I have seen 192.168.68.x, not quite there yet. Huawei routers default to 192.168.8.1 as the default, and have a reasonably decent DHCP in them as well. Very popular by me, and they do support 5G well. Gave away 2 older ones that still do 4G and 3G, as they have good use as routers in poorer signal areas even without an external antenna.
i3inaudible@reddit
Yeah Huawei is great. You hardly even notice when they report back to China
techtornado@reddit
Psh!
198.18.0.0/15 is where it’s at
JeffTheNth@reddit
Who needs 131,071 device IPs for a HOME network? How many cameras do you have set up on that compound, man??? :D
dlist925@reddit
I’m a 172.16.69.x enjoyer, myself.
TheThiefMaster@reddit
IPv6 helps a lot in this case. If you deploy FD:: site addresses for the company, then it essentially never gets a conflict with home users (who will at best have FE80:: local addresses and a 2001:: public address, but never FD::)
Disturbed_Bard@reddit
That's assuming home users are using IPv6 at all...
Problem is and I admit even I, adding NAT64 and DNS64 into the equation does complicate troubleshooting a bit if your staff members have no experience with IPv6. I have a tiny bit of experience as I use it in my own home network environment but it's still very rare to see it being used in an organisation setting due to greybeards and juniors coming in are barely even trained on basic network troubleshooting and lack the initiative to want to even try learning it.
oloryn@reddit
My home network is on 172.17.113.0/24, just for that reason. Very few people use 172.16.0.0/12, so I'm unlikely to run into duplicate addresses over a VPN (and I have had to VPN into client's networks from home).
CrazyAlbertan2@reddit
You just won the internet for today.
atmanm@reddit
Not gonna lie, you had me with just the title
meski_oz@reddit
Certainly sucked me in
JeffTheNth@reddit
this comment needs more love
Weird1Intrepid@reddit
I'm not sure that's appropriate when a Roomba is involved
the_thrillamilla@reddit
Duct tape a knife on for violence, yes, of course. But just perhaps, duct tape on a vibrator, for love?
fcewen00@reddit
I mean, how can you not click on a title like that.
MoreRopePlease@reddit
Why not change the firewall IP address?
anubisviech@reddit
Or change DHCP rules. Because why on earth would a Roomba have a fixed IP?
400HPMustang@reddit
In my house everything that’s not a portable device gets a DHCP reservation. IoT devices are kind of stupid and cheap ones don’t handle renewing leases very well sometimes so give them a reservation and let them keep their IP.
Ephemeral-Comments@reddit
You have IOT devices? In your own home?
I'm disgusted on behalf of Sarah Connor.
I barely have my printer on wifi, and then I have a camera and a remote-controlled shotgun pointed at it just in case it makes a weird noise.
But really, even my wifi-controlled thermostat is just a raspberry pi with some homegrown code. No IOT bullshit in my home.
400HPMustang@reddit
The majority of my smart home tech is local communication only, mostly Zigbee so not IP based and the stuff that is IP based only connects to the internet when it absolutely has to be that way. There's just something nice about having things happen without user intervention.
Ephemeral-Comments@reddit
Skynet agrees :-)
SeanBZA@reddit
Yes i fixed Ip addresses for printers, the bonus being that, as they were HP devices, replacing the printer meant you did not need to update drivers on any of the computers talking to them, as the new printer was quite happy to speak PCL4, PCL5 and thus the clients did not need the hell of setting up printer settings again on the POS software, which was incredibly touchy on how it printed, and also had a few preset printers for various output options, so that assorted reports and invoices would print properly. new HP printer, just set to the same IP statically, masked out in a DHCP range as well, and thus you had no problems with having to change every single setting to the new one, it just worked.
I was printing from my computer using the HP 4MV driver, to a HP4200, because it worked, and the reporting back of PCL gave options, like double sided, as the printers supported it automatically. also worked under Linux as well, with zero problems, just was a lot faster to print compared to the original 4MV printer.
Also worked with the Tektronix Phasor 340, despite it not being HP, because it also spoke PCL, amongst other languages, and PCL was a shed load faster to be able to generate a complete page, so I would send a print job, and walk to the printer, and be there just as the first page finished collecting the wax image off the imaging drum, and popped out of the top. that printer i had a setting of all prints being with a full black background, because there was no difference in cost per page for having a full black background, nd black wax blocks were supplied free with any colour, and you never used them up. Still have packs of them around, despite the printer being long obsolete, simply because I use them as crayons.
ethnicman1971@reddit
It would not need a fixed IP address. If DHCP assigned it the address it would keep that address as long as it was able to renew the lease in time. So if the Roomba got the assignment from one DHCP server and the firewall had a static address stored in a different table on a different server it would always have a conflict.
techforallseasons@reddit
So....that is USUALLY how it works; but some crummy IoT things don't DHCP correctly. I've see devices have internal renew / release timers that are static to the device and I've seen devices that simply only request an IP and hold it without ever renewing.
For devices that follow DHCP spec correctly - you are correct; but the world being as it is; reservations help with alot of crummy equipment that we are stuck supporting.
masterxc@reddit
Yeah, the DHCP lease is more or less a suggestion per spec, so you can simply squat on the IP and ignore the expiration, much to the dismay of the rest of the network. ARP tables help with that problem at least, but really depends on network configuration.
Agret@reddit
Pretty weird for the office to be using the regular residential IP range rather than being a 10.0.x based network
smashedbotatos@reddit
Exactly this, and who gives clients instructions to put the VPN on their router so all their home devices are using it?
A roomba, would not connect if just the clients laptop/desktop was on the VPN. It had to be set up on the router to connect ALL devices to the work VPN.
laplongejr@reddit
Roomba wasn't connected. It was an IP conflict issue.
The work network was telling the client computer to use the firewall at 192.168.1.x (VPN network), so the computer was contacting the roomba at 192.168.1.x (physical network)
Agret@reddit
The Roomba didn't connect to the VPN, it was using the same IP as the VPN gateway as they were both 192.168.1.254. when the client PC is connected to the work VPN it has 2 adapters both on the same IP range so it was getting confused which of the 2 devices it should be trying to talk to (duplicate IPs)
dnabsuh1@reddit
If it is a small office, and they don't know how to set things up, it's not that unusual- they take the defaults the router gave and start working.
Aln76467@reddit
Why not throw out the firewall and replace it, just to be sure?
MerionesofMolus@reddit
Instructions unclear, replaced firewall with Roomba.
JaschaE@reddit
The replacement was a sweeping success.
alnyland@reddit
Cleaning up all the packets, on a schedule
created4this@reddit
Well perhaps, but there was that one time when it went through the IPTable and spread it all over the kitchen. Nervous dogs and Roombix don't mix
406highlander@reddit
"Hey, this new firewall sucks"
ratshack@reddit
underpowered comment
Nuka-Crapola@reddit
I’m pretty sure that’s how the client got into this mess
techtornado@reddit
Don’t throw out the roomba, nature abhors a vacuum
Stryker_One@reddit
Replaced firewall. No Internet now, but I the double wall steel construction does make me feel WAY more protected against fire spreading into the office.
Shinhan@reddit
More work to reconfigure the entire network than to just change a single roomba.
wicked_one_at@reddit
Pretty common issue, but the story gets cool because it is a roomba
Sarke1@reddit
This 100% LLM written.
votekick@reddit (OP)
Yeah, ChatGPT fixed some grammar because I've had people complain at my punctuation because I usually just don't care.
Story is real though with some tweaks to avoid unnecessary detail.
ghostgurlboo@reddit
The fact you can SSH into a Romba at all is sending me
HenkPoley@reddit
iRobot Roomba’s do not run SSH.
Overall-Tailor8949@reddit
Well hell, I was expecting a story about Captain Stabby!
To_Go_Back1984@reddit
Right?! I'm kinda disappointed; I was hoping for a story about Captain Stabby being the creator of Skynet
BuffyVR@reddit
Simply turn off the roomba when you need to login to the firewall.
Problem solved! /s
jeffrey_f@reddit
Well, at least it was a clean connection.......
Ba Da, Tss!
I'll see myself out!
JaschaE@reddit
You can tell IT is a trade on par with plumbing and carpentry, because you post a way you resolved an issue and everyone in the comments knows a better, nay, the correct way to do it xD
skiing123@reddit
True, NGL I did think of a way to change the IP without doing ssh and was thinking of commenting
LupercaniusAB@reddit
LOL, this is perfect. Because I’m a stagehand who does automated lighting systems I lurk in both this and r/electricians. You are right 100% correct.
AlabasterWitch@reddit
So to clarify - their roomba was using a static ip reserved for the vpn? Making sure i understand lol
Ph1User@reddit
Same default subnet in Office and home, Roomba had the same private IP that their Office firewall had, so instead of the connection going to the FW with VPN, it went to the home Roomba and got dropped.
Tegumentario@reddit
Couldn't the firewall ip be changed?
DraconianFlame@reddit
How dare you ask an honest question! Clearly you should know all aspects of IT before enjoying funny stories.
/uj
Firewall IPs are more or less static due to dependencies. If you change it, you'll have to change things behind it. Also who is allowed to change comes into play as well (ISP, cloud, etc)
Sorry people are mean.
super5aj123@reddit
Did people get their comments deleted? The only responses to them I'm seeing are yours, and a guy saying it's like moving the counter instead of a toaster, which seems more like a joke than being mean.
DraconianFlame@reddit
He had like 20 downvotes.
super5aj123@reddit
Ah, that makes sense. Thanks! I was reading the comments and wondering if I was just missing something or had somebody blocked, lol.
Nevermind04@reddit
Yes, you could. But this is like moving the kitchen counter instead of moving a toaster.
ideaguyken@reddit
Or in this case, like moving the wall instead of moving the vacuum
Dark54g@reddit
Yep, pretty typical.
sferau@reddit
Why not just update the DHCP server?
Hurricane_32@reddit
The Internet of Shit strikes again.
LeomundsTinyButt_@reddit
Nah, that one is on whoever set up the office network, not on the Roomba. No matter what device had that IP, it would have caused the same problem. Doesn't matter if the static IP was set on the device or the router, or even if it was an unlucky random assignment by DHCP.
I'm no sysadmin, but even I know a VPN using the default subnet most routers ship with is a horrible idea.
apVoyocpt@reddit
Which vacuum has a default and open ssh? I have a Roborock s5 and I had to uart my way into ssh by interrupting uboot. But the cool thing is it runs a normal Ubuntu and there is an open software called valetudo which you can install to disconnect from the roborock cloud. Long story short I can ssh into my vacuum too.
Also: the vacuum surly ran on dhcp. You could have assumed a new IP at the router and that’s it?
votekick@reddit (OP)
Don't recall, but I think it was actually one of those serial number = password situations.
SeanBZA@reddit
I was thinking that you logged into the robot, and used the on board tools to access the local network to log in to the router instead, and do the work there as if you were on the local network.
ratshack@reddit
Somehow using a Roomba as a jump box does not sound cooler than what OP actually did.
I feel like it should… but OP’s issue stands tall right there with it.
u/votekick not sure if I’ve ever before said this unironically but… CSB!
MrCatSquid@reddit
I can’t believe no one has pointed out this a completely AI written story. Especially on this subreddit. Come on guys.
Geminii27@reddit
Ouch. I take it that the user was remoting in and their home workstation was being assigned an IP from the office network - something like 192.168.1.103, maybe - while simultaneously also being on something like 192.168.1.8 from their home network? And for some reason, the remote workstations were being allowed to act as packet forwarders...