Truck wash computer too bored to die, news at 11.
Posted by Stellapacifica@reddit | talesfromtechsupport | View on Reddit | 80 comments
Alright, my loves. I've got a yummy one for you today.
This comes by way of my partner, who worked security for a couple years. I'll tell it as he told me:
It would have been end of 2020, early 2021, when I was at the [big name distribution company you have definitely heard of] site one day a week. They had a truck wash station which doubled as both the security office and an OSHA violation, and which had in its bowels a Machine:tm:.
This machine ran the truck wash mechanically, and had done so since time immemorial, by which I mean the late 90s. How late? Not nearly late enough. This beleaguered box of technological decay, which somehow merited a UPS, but not a power switch, was still running its original licensed copy of Win 95.
The thing is, this building was (and is) classed as vital infrastructure meant to withstand disaster, up to and including full loss of power for sustained intervals. So this poor computer, meant only to power cycle in times of dire crisis, had not met such crisis since its installation. Between the diesel generators and its own aged UPS, to all my bored record-searching in dead of Sunday pre-dawns, there exists no evidence that this machine has ever turned off.
TL;DR bored security guard terrified at prospect of computer uptime longer than his own.
beyondoutsidethebox@reddit
IIRC, there was a story a while ago in the news about an old Apple computer running a High-schools HVAC system for nearly 30 years without fail. So when it was and was replaced, the district actually did the intelligent thing and kept the old computer on standby as a backup.
purple_elephant1997@reddit
My job's client records database was new in 1995- IT sellotaped together a solution for it to run on Windows 11. The biggest a single document attachment can be is 10MB, so using Adobe Pro to compress the living daylights out of documents to fit within the limit.
honeyfixit@reddit
Ok guys on three: one...two...three....SQUEEZE! PUT YOUR BACKS INTO IT!
Hey boss, what idiot thought we could compress and 250MB file into 10mb?
The idiot who signs your checka, thats who. Now keep pushing we are nearly there! Go get the security guard, you know the one that eats donuts all the time. Tell him we need him for a special project hes uniquely qualified for.
purple_elephant1997@reddit
Thank you, that image has made me cackle like a loon :)
honeyfixit@reddit
Glad i could help
BillWilberforce@reddit
There's no way that Windows 95 hadn't crashed a million times in the last 30 years.
max_peck@reddit
It would crash after 49.7 days when a counter rolled over — unless a patch released in 1999 was applied
That’s assuming it made it to 49.7 days, which was uncommon
Fixes_Computers@reddit
I remember those days. I had an older box doing some single task and would have to reboot it every so often because of this.
T_Noctambulist@reddit
That's when you just get a vacation timer and cut power for a minute at midnight everyday.
BillWilberforce@reddit
On a computer, particularly one with an HDD and no battery back up.
T_Noctambulist@reddit
If it's something that boots itself. Works great on routers
Stellapacifica@reddit (OP)
If I had to guess, someone probably manually unplugs it every so often, but it's air gapped so no patches anymore.
I've been told that the one (automotive) maintenance fella advised "don't touch that, it gets angry sometimes" which is. Um. Alarming
__wildwing__@reddit
We had a computer hooked up to one of the measuring devices (manufacturing) and it was so old, we didn’t dare try to clean it. Last time we did, the plastic had become so brittle the case started cracking. Pretty much put a vented box over it and a “do not touch” sign.
It’s not like we were manufacturing anything important… just aerospace stuff…
ozzie286@reddit
I used to have to work on a printer like that. LaserJet 8000 or 8100, I don't remember which, and this would have been around 2016-2018, so around 20 years old. Every time a plastic panel had to come off, it would break, usually around the screw holes. Covers were held on with duct tape or gravity. Eventually it lost a power supply, and the 2 "refurb" power supplies both failed, one so bad it tripped a breaker and caught fire. After that we finally told the customer we wouldn't be fixing that dinosaur any more.
Stellapacifica@reddit (OP)
Yikes.jpg
nondescriptzombie@reddit
The cost to recertify a production line after switching out a control module can exceed a quarter million dollars, not including lost productivity from downtime.
What doesn't make sense is that they haven't worked on porting to Linux.
easylikerain@reddit
"But that would cost moneeeey. Just get it working for now." - every single one of them
Stryker_One@reddit
And "for now" is decade after decade after decade.....
BillWilberforce@reddit
There's nothing more permanent than a temporary fix.
Most-Jacket8207@reddit
DisgustedJesus.jpg
Agret@reddit
I think Cloudflare must run this on their edge servers
dreaminginteal@reddit
We used to joke about it being the “ultimate irreproducible bug” due to that.
Cmd_Line_Commando@reddit
Its possible.
I had an installation of Windows NT4 that crashed immediately after installation was complete. I never touched it again.
Various issues with other MS OSes.
But what stands out for me was Windows Vista. Used RC1 and it was beautiful, fast, responsive, not one crash.
Then the release Vista and it was absolute dogshit.
BillWilberforce@reddit
Win 7 RCs never crashed.
Optimal-Condition803@reddit
"What operating system you running?"
"Erm, Vista"
"WE'RE GONNA DIE!"
that_one_wierd_guy@reddit
and if it somehow hasn't, then don't fuckin touch it!!!
BillWilberforce@reddit
Although I would love to try and clone it for the inevitable death.
denimadept@reddit
Especially on a Friday.
Stellapacifica@reddit (OP)
Realistically you're absolutely right, but I couldn't resist the mental image. Didn't know about the rollover bug folks mentioned, either - the likelihood that a patch was implemented in '99 is... far more up in the air than I want to acknowledge.
marshogas@reddit
I worked in the micro electronic industry. We had multi-million dollar machines that were were the size of a room, and controlled by a computer running unique hardware and software. You do not update these computers because that could crash the device specific drivers. We has DOS, windows 3.1, 95, and others running some of these machines. Repairs were usually just take out the damaged part and replace it with an identical part bought off ebay from another computer of the same vintage. And if the unique i/o board failed, you reserve engineer it or shell out a few hundred thousand to a couple million for a new one.
Gadgetman_1@reddit
I recently had to rebuild a WinXP computer to get an old Brake tester working again. A new one was on order, but these are bespoke machines built into concrete floors, with heavy rollers and shit. There's a 6month delivery time.
On monday, if my health holds(been sick with the flu almost all week now) I have to handle a trailer weight. The weight itself works, but the control PC can't talk to it. It worked fine while the PC ran Win10, and even when it decided to upgrade itself to Win11(not my bl**dy fault. I had nothing to do with setting it up), but this heap of irresponsible silicone 'needs to be on the net for support reasons'. It stopped working when we upgraded our network and everything got new IPs. The scales and some displays are hooked up using IP to Serial servers, and the drivers on the PC worked as long as nobody did anything. The config program that was installed was for Win10, and doesn't work for Win11.
So that's my job on monday; drive 2 hours one way to fix shit, drive back to the office, document the heck out of it. Then scan our entire network for more of this shit(I already know of one other site with the same setup, but they got new network setup before their silicone heap decided to upgrade itself.) And have a discussion with IT Security(I'm in a 'hanger on position' and get to be in on meetings)
jkarovskaya@reddit
High end machine shop, similar issues. They got a new PC, Win11, lots of functions not work, G code won't accept updates, etc.
Had to install a Win10 LTSC IoT on a VM for this kind of shite so old CNC machining center could happily continue running another 5 years with no one paying slightest attention to it.
randomdude21@reddit
https://united-silicones.com/blogs/news/silicone-vs-silicon-whats-the-difference
Gadgetman_1@reddit
It's a nice blog post, but what's its relevance?
Hamudra@reddit
You're writing silicone, but what you're talking about is silicon
Gadgetman_1@reddit
Ah... You're right. My bad. Have a couple of updots.
songbolt@reddit
"Here is a Rube Goldberg machine ... It works. Do not touch it."
marshogas@reddit
It works now. If you touch it, it becomes your responsibility.
Stellapacifica@reddit (OP)
Wew. That's... well, it makes sense, but wow.
SeanBZA@reddit
Hospitals, where most of the CT scanners and MRI machines are running something outdated, and old CNC machines which run on MSDOS.
Ajreil@reddit
LTT has a video on a company that makes Windows XP machines using reasonably modern hardware and open source security updates. Apparently they make bank.
tuscaloser@reddit
Surely you would buy a couple once you had gone as far as contracting out new logic boards.
marshogas@reddit
The machines we bought were often at end of life for the top tier manufacturers, the devices would be as complicated as a modern car. Most or the mechanical parts could be repaired or replaced with modern equivalents. The structure could be repaired and renewed. But the mechanics were driven by custom software that would only run on the version of windows that was out at the time the equipment was commissioned. It would the interface with some type of driver board. I had 20 year old motherboards replaced with similar motherboards to keep some running. You can't download new drivers for new hardware and be assured that the driver will still work. The interface board might have been made by a third party or might be designed inhouse. But due to such low production runs, there wouldn't be more than 100 ever made. So without a design drawing, I had relays replaced, or traces fixed, but we didn't have the resources and time to completely reproduce a board. At that point, we would be looking at the used market for a five year old model to replace the 30 year old version we keep limping on with. Ultimately, down time costs more than replacement costs. And it is very time consuming to reverse engineer a non functioning board.
Ich_mag_Kartoffeln@reddit
One former job had a machine controlled by some ancient and mysterious computer. POST reported "16K RAM OK" (presumably that was true) and very little else.
It booted from a 5 1/4" floppy, and used a custom, full length ISA card to interface with the machine. I have no idea what processor actually lurked in the box.
marshogas@reddit
Looks like that would be mid 1990s, end of 486 era and moving to pentium CPUs.
Polymemnetic@reddit
Nah. Older than that. 16k of ram is the Commodore/Atari/Apple II era.
So probably a 6502, Z80, or a Motorola 68000. Maybe an 8088
marshogas@reddit
You are correct. My brain failed and translated it to 16M. 16k is a decade or so older.
TheRealRamanji@reddit
I used to work for a computer repair shop and we had a guy who had an expensive CAD machine or whatever it was running on windows 95. The computer just crapped out so he asked us to build a new pc with all the latest specs. This was like 2006. We get the machine built and i go set it up and the software won’t run. Specs were too high and it would cost him thousands to get the latest software. So we went into our backroom with all of the old pcs and pieced together a machine for him. Just literally old parts that were probably gonna get trashed. Ended up being all profit since they were just old pcs people brought for repairs and decided to just get a new system.
ChrisXDXL@reddit
I've never used Windows 95 but I know it has a Task Manager, in modern Windows there is a performance tab, in there is the CPU, clicking on that will give you the usage and information on the CPU including it's up time which will tell you how long the computer has been on for.
Hope this helps.
JorgeXMcKie@reddit
I worked on the first gen of on board computers for cars. We pulled diagnostic data from them and uploaded sw patches for testing. By the time I got there 2nd gen had been under development for a while. The original hardware specs for the diagnostic pc's were 286 based along with 1024baud modems for communication. Since they were working on 2nd gen they didn't put any effort into upgrading gen 1 sw or hw. By the time I got into the program no one was making 286's or 1200 baud modems but we still needed to maintain and replace hardware. For about a year they were paying more for a 286 than a new P3 and hundreds per modem.
That old hw can get expensive to maintain
Dunnachius@reddit
My first job I rebuilt a dos 3.1 machine that worked a multimillion dollar industrial doohickey.
This was in 2006. It had a stack of 30 identical floppy disks that it needed to boot with, yes 30 of the same disk so there was never a risk of not having it. 10 were wifh the computer and 10 in storage with 10 in a fire safe.
The industrisl doohicky was for spraying fiberglass into hot tub molds to make hot tubs (not jacazzi brand)
They bought it in 80s and they budgeted for using it 30-40 years.
Their entire factory depended on half a dozen of these old dos 3.1 terminals that never saw the internet and shockingly they actually had a backup computer for each one, plus 2 more for floating backups.
The hardware was a serial port and dos 3.1 software that came bundled on a bootable 3.5 inch floppy.
If the computer went down the factory went down and output would go to zero.
Apparently sometime in the history of the factory they lost the last copy of the software from a floppy failure they had the manufacturer stick someone on a deadhead flight from Asia with 2 copies of the software to personally hand deliver a new copy to get the factory up and running because the factory didn’t have internet or some such nonsense.
honeyfixit@reddit
Ah the optism of the 80s! I mean planned obsolescence was such a big thing back then
Dunnachius@reddit
In 2006 a machine they bought in the 80s was still in service. So when I touched it was already over 20 years in service and going strong.
honeyfixit@reddit
I read your story. Thats why i commented.
What is the point of repeating it?
syntaxerror53@reddit
"If the computer went down the factory went down and output would go to zero."
Now that's a P1 if there ever was one.
Shinhan@reddit
How often was UPS battery replaced? Because I usually find out my UPS battery is dead when there's a power loss event and UPS shuts down after seconds instead of minutes.
SeanBZA@reddit
Must have been a fully patched version of Win95, because the release version, at least till a good number of service packs and updates, had a rather nasty bug in that after around 35 days an internal timer would overflow, and then give a BSOD.
anh86@reddit
Someone should actually check the uptime just for fun
syntaxerror53@reddit
There's a strong chance it might crash computer and might not start again.
Comfortable-Scale132@reddit
I remember doing some contract work in 2019 and look over at a work station and see the black screen with dark red letters...
"It is now safe to turn off your computer."
I looked back at the IT Manager and just pointed. He laughed.
syntaxerror53@reddit
The person who programmed that part of the OS must have been working on the Win11 Jan26 Patches. Only it never quite worked and kept rebooting the PCs.
PercyFlage@reddit
Windows 95 used to crash every 49 days (time counter overflow), so I reckon the reset button had been hit a few times.
Embarrassed-Dot-1794@reddit
Is that home edition or some other? Mine never got turned off for months at a time and never crashed
PercyFlage@reddit
I think the issue was patched at some stage.
https://positiveresults.com/blog/technology/the-49-7-day-bug-when-windows-95-needed-a-time-out/
Maleficent_Ad_8890@reddit
Why can’t you get a new computer and run virtual Pc?
RedFive1976@reddit
Because the machine will have some old interface board that's impossible to install on a new machine (i.e. ISA or VESA slot), needs the physical math co-proc, or some other widget that's impossible to virtualize. I've run into problems trying to pass-thru USB license dongles into a VM, so I can only imagine the problems with some ancient interface board that needs an ancient expansion bus slot.
udsd007@reddit
Or if it -can- be virtualized, but virtualization renders it no longer certified and it’s in a man-rated application of some sort.
RedFive1976@reddit
Could be. That wouldn't keep it from functioning, necessarily.
plaid_rabbit@reddit
A lot of this kind of stuff is interfacing with external hardware. I used to work at a place with a "large format printer" that prints out a 30foot/10meter sized banners. The computer runs XP IIRC. The software/hardware for driving the printer only works on XP. I think it at least uses PCI cards... Virtualization won't work, because you won't have all the custom hardware. You can still get support from the manufacturer if it breaks down, but you can't modify the computer. That printer used costs a couple million, getting a new one that doesn't run on XP costs 10M+.
jpe1969@reddit
So there I was, a recent college graduate in 1992 with a new job on the operations side of a hardware distributor when I discovered to my horror that their entire operation wad was controlled by a punch card computer.
desertdilbert@reddit
That was probably the omost stable system out there!
Every line of code was vetted and well understood.
Many of the financial software applications that hold the world together are running on an IBM/360 Emulator....which is running on a IBM System 36 Emulator....which is running on an AS/400 emulator...which itself is on a AIX!
oloryn@reddit
Back in my Burroughs Medium Systems days, we had a payroll system which had to be run on top of two or more layers of emulators. Back in those days, I'm pretty sure Burroughs made sure that a new line of systems would have an emulator for the previous line of systems, just to handle this type of thing (and to make it easier to sell to management of companies who had custom software running on the older line of systems).
djdaedalus42@reddit
This reminded me of working with PLC's (Programmable Logic Controllers). They've existed for just this task since the 1960s and probably have run car washes etc. long before PC's came along. So why anyone would give this job to a PC is mind boggling. Unless of course some genius put a PLC on a PC board and said "you can program it from inside the PC. Of course, it has to stay there forever and ever....."
kalkan1000@reddit
Q9
lunicorn@reddit
Around 2004 I worked on reports about various computers the client used, and write up any exceptions that did not meet standards. I remember explaining why the PDP10 or PDP11 did not have Norton Antivirus.
djdaedalus42@reddit
I've replaced the batteries in several UPS's so far, and none of them were anything like as old as this one. I'm amazed it wasn't beeping like crazy, unless of course somebody fixed things by disabling the beep.
CreideikiVAX@reddit
I don't work with them, mostly I'm a hobbyist. But I do know someone who does…
But have you ever heard of OpenVMS and the VAX minicomputer?
Well, you could cluster VAX systems, and seamlessly share hardware. You could also piece-wise upgrade your VMS systems and VAX hardware, and have systems in the cluster drop-in or drop-out as needed, with the cluster still running so long as cluster quorum is maintained.
So this one VAXcluster my friend works on? Yeah the cluster hasn't seen down time since before the fall of the Soviet Union.
Stellapacifica@reddit (OP)
Oh, that's fantastic! Hotswap hardware gives me the ibbly jibblies, but I know some stuff's designed for it and is just fine.
Planetx32@reddit
There are companies such as Nixsys which makes new PCs with legacy OS and hardware. If you need a DOS pc with ISA boards, they can do it.
lego_not_legos@reddit
I think it would rebel against being shut down.