In a previous job, I did support for a company that makes automatic people counting devices. I've mentioned these on a few posts now, but basically they're ceiling mounted and count people via infrared across a couple of virtual counting lines. Anyway, they're quite advanced being network accessible and also have a (low res) camera view for remote setup and support.

So, a customer buys about 30 of these to install in their building to count people in and out, as well as on and off each floor via the different stairways and elevator lobbies. Basically a building utilisation project.

Everything is working fine and everything is accessible on their main network and counting pretty accurately. The customer is happy.

So fast forward a few months and we're informed that they're upgrading their network equipment and as part of the change everything's going on a new sub net. This means the static IP details of each device must be changed (they don't allow dchp). Usually you'd just log into each device on the current network, make the changes you need, and then once the main network changes are done the devices will just join the new network and all will be well. But this wasn't what the customer wanted. They wanted no interruption at all. A decent bit of coordination would have meant that was possible, but the guy running this project was a bit vague about timescales of each element.

So, for one reason or another it was decided that they would go around to each device locally with a laptop and patch lead, and change each one, verify it works and then move onto the next one. Ok fine, it's your time.

Little did I know, they'd asked their inhouse IT department to do the work. The first I hear is when I get a phone call saying that they're at the first device and they need help. Ok no worries, there is a network reset button on the back. Just hold that for 5 seconds and it will go temporarily to a known IP address. You can then connect your laptop, enter in the new details and voila. At this point I'm told they did a full reset by holding down the reset button for 20 seconds. Oh dear. You've not only permanently lost the network settings but every other setting too. This means it will need setting up again to count properly, and send the data to the right server etc etc. Not a 30 second job.

I look on our server and see we have automatically backed up the device settings so it's not that bad. They'll just need to connect to this one on the default IP, upload the backup file, then make the new IP changes.

As mentioned before each device has a built in camera, so when the guy connects, I casually mention that if they're stood right under the device with their laptop, they should now be able to see themselves. "No" , is the reply. "In fact that looks like the elevator lobby from down stairs".

OK well that shouldn't be possible as you're physically connected to this one. "oh no, we're not, we're on the WiFi". At this point I realise what they've done. Instead of resetting each one and making the changes, in turn. They've reset every single one so that they'll all on the default IP and it's just random chance which one they make a connection to.

At this point I simply email them every backup file that I have and tell them they're on their own. Essentially they either need to physically turn every device off and turn them on one at a time to make the changes, or they need to make a random connection, try to work out which one it is and hope they can make the changes before the connection drops.

They did manage to fix it as I saw all the devices come back only eventually, but their no interruption ended up being a two day interruption.