What’s the oldest machine you’re still keeping alive in production and how do you troubleshoot it when it acts up?
Posted by WritHerAI@reddit | sysadmin | View on Reddit | 22 comments
Genuinely curious how others handle this. I keep running into old boxes XP, NT, embedded stuff from the early 2000s still running something critical, where the one person who actually understood the system left years ago.
When one of these starts misbehaving, how do you even begin? Real docs, or tribal knowledge and gut feel? What’s the worst “this 20-year-old machine is down and nobody knows why” moment you’ve been through, and how did it end?
Looking for war stories more than solutions tell me the messy ones.
BladeCollectorGirl@reddit
I run into this with industrial systems running applications that aren't being upgraded or capable of being replaced. Things like pumps, sensors inside pavement, et cetera.
I virtualize when I can (vBox, VMWare, whatever works). That gets past hardware issues. In some cases, in-place Windows upgrades work. Also, create a duplicate system in case something blows up.b
WritHerAI@reddit (OP)
Makes sense virtualizing plus a spare covers the hardware dying. I’m more curious about the other side: when one of those systems starts misbehaving in software weird errors, a service that won’t start, something a rollback doesn’t obviously fix how do you figure out what’s actually wrong? Is that always you, or can others on the team handle it?
BladeCollectorGirl@reddit
Me. I'm the principal engineer, I train others.
What happens is that we have it isolated from other systems to reduce lateral attacks, and lots of backups. Even system restore points as last resort. Where possible, storing profiles on a server vs local.
In one client, deploying Druva for backup. Also a list of critical services and startup order.
Ok_Size1748@reddit
Tru64 Unix 5.1 running in some Alpha ES40 boxes with memory channel & Oracle Parallel Server.
Still running. Slow as hell by nowadays standards, but keeps the lights on here for some archival purposes.
boot dka anyone?
Kumorigoe@reddit
Sorry, it seems this comment or thread has violated a sub-reddit rule and has been removed by a moderator.
Do Not Conduct Marketing Operations Within This Community.
Your content may be better suited for our companion sub-reddit: /r/SysAdminBlogs
If you wish to appeal this action please don't hesitate to message the moderation team.
mixxituk@reddit
Nice try hacker
thebigshoe247@reddit
I did have an NT4 machine which was "mission critical" but nobody could tell me what it did. It's been about 2 years now and nobody has said anything...
cptlolalot@reddit
You have to start with solid data/os backup.
I posted recently about this here https://www.reddit.com/r/sysadmin/s/hwNDAEJe6A
After the data is safe, it's all about obsolete hardware. If you're lucky enough to not be using random proprietary ISA cards. I would attempt to set up a VM and get it working in place of bare metal. If you can get that working, you're pretty safe with good backups and documentation.
I had a couple of old ISA cards reverse engineered at great cost only to remain reliant on old mobo, CPU and ram.
Best bet is to try to migrate to modern hardware and software but often the people who suggest that have never had to support stuff this old and come up against the same limiting factors.
My advice:
Get solid hard disk backups (see above link). Keep the machines clean (blow them out once a year) Keep an eye on fans/cooling. Old machines won't tell you if a PSU fan dies. Put the machines on a UPS/power conditioner. If the PSU is decades old and standard, install a new one. Label the machine to indicate it's fragility and the importance to handle gently.
yyg-linux@reddit
My soul
One_Monk_2777@reddit
And i dont get paid to troubleshoot myself, we assign that to the bartender
ZachVIA@reddit
As far as hardware goes, we have a dev VM host that my personal work VM runs on. It’s dog shit slow. I was curious so I looked up the CPUs on it. You can buy them on eBay for $4 (11 year old CPU).
bigpacks@reddit
I haven't found any XP machines in years
But I did just learn today about the Y2038 Problem ( https://en.wikipedia.org/wiki/Year_2038_problem ).... So, OP has about 11 years to figure out how to deal with these old systems
cptlolalot@reddit
Dos 6.22
Data backups, crossed fingers and prayers
NoThatsNotPasta@reddit
centos 6 box before it went to stream. We migrated most the tasks away from it over the years, but it still hangs about as a print and LDAP server.
To think of it, i don't think it's really gone wrong, it seams quite content sitting there, humming away, not a care in the world - lucky bastard. Services have stopped randomly, but they normally start straight back up again without issue. The persistent data is kept on a network drive, so we suspect that could cause niggles, but there is a migration plan for... August? I think? for it, so it'll be on a new box which gets forgotten.
ITjoeschmo@reddit
A safe approach is to start by making an image of the drive, then exploring if you could spin that image up in a hypervisor as VM, if there is external hardware then also passing through USB/serial/whatever to support.
jort_catalog@reddit
You don't all need to reply to this obviously fake question
XelfinDarlander@reddit
I worked in manufacturing for a long while. Ran into that a lot, such as industrial equipment running on Windows 2000 or XP. For me, I just created an image of the disk, and kept as much old replacement hardware available as possible. It was super common that it was so old that we didn't have any installation media available or the equipment manufacturer was out of business. Most of the time it never mattered that it was old and not getting patches as it was never connected to the network.
Que_Ball@reddit
Industrial CNC type equipment. Lathes, mills, etc.
Image the drive, keep / hoard spares and a bag of old serial cables. Remember that compact flash to IDE adapters exist as well as industrial grade CF cards are far better than holding on to old spinning disks. Makes cloning and keeping a spare drive on hand easy since that old dos software tends to corrupt it's own data files easily. We have an local industrial control specialist who is also good at figuring out old PLC and relay based systems.
RavenousTitan818@reddit
There's a Windows 2008 Server for an application I've been told for the past 10 years is in the process of being replaced.
DerpSillious@reddit
I think you work at my old Job.
Do you also have a dedicated server to run one departments "Application" so it does not use all the resources and crash the laptops and desktops of users, and have to find out that the "Application" is just an excel workbook with 125 pages, all linked with equations, Macros, and VB Scripts?
Sarian@reddit
That's a tale as old as time.
Jaaames_Baxterrr@reddit
I FINALLY got a Windows Server 2003 switched off last year. It took ages for the team to migrate from it. I didn't ever want to touch it in fear of it breaking. When something like that exists, and they've been warned multiple times to get over to the new system, if it fails it's on them.