A bunch of engineers realized a problem several years earlier and made a bunch of targeted fixes that worked pretty well so it seemed like mostly nothing happened. But it wasn’t necessarily overhyped, just well executed.
We addressed several issues in the system I worked on at the time that would’ve caused problems though nothing catastrophic.
Also, "nothing catastrophic" is generous in the presence of multiple co-occurring unusual failures.
The worst incidents are almost never one big thing going wrong, but several little things going wrong and interacting unusually. That's why Y2K was so dangerous. There was a high risk that a lot of little things would go wrong across a lot of heterogeneous, highly interconnected systems all at once.
Yeah, I spent most of the end part of the 90's doing a lot of edge case testing to make sure we were ready. We spent a lot of money to make sure it wasn't an issue.
It did amaze me how many systems were not Y2K compliant that were super vital. Some had some amazingly bad failure conditions.
Some could do that. Mostly we replaced stuff with a proper 4 digit date.
I do remember a weird ass one that had a very odd date format in some dumb system where I guess 6 bytes for date was too expensive so they used only 2 (16-bits). They had encoded the date as 5 bits for day, 4 bits for month, and 7 bits for year (starting at 1900). I suppose the savings in storage was awesome /s. They sequenced it year, month, date. Luckily, they didn't do sorts on the original 2 bytes.
I had to report we didn't have a year 2k problem, but a 2027 problem. We left it alone and did some testing it made it over 2000 fine (which it did including sorting). No idea what happened after that. I do hope someone replaced it given the current year. I do hate too cute by half solutions, but it saved 4 bytes on millions of dates for decades.
I will admit that their date calculation functions were correct and did fine for the 1900 -> 2000 transition. I once thought it would be a cute assignment in some low level computing class.
I didn't do much y2k stuff directly but I was sorta y2k adjacent in the late 90s in that I was one of the kids (early career not an actual child) given a somewhat irresponsible level of authority and autonomy building early web stuff while the people with any actual institutional knowledge were busy making big money working n y2k issues.
It's sorta a pet peeve of mine - on your behalf - that people seem to think y2k was some kind of social panic rather than a real issue because planes didn't fall out of the sky. It is possible it might have been a little overblown, but it was for sure a real problem. The reason we didn't have catastrophic issues is that we spent a decade fixing it. I mean, I personally didn't but a veritable legion of people like you did. I appreciate your service.
It's like acid rain in the 1980s or the ozone hole in the 1990s. We don't have those problems now, but not because they weren't real problems. They were genuine problems, and we fixed them
Yes, one thing that people may not have realized was even if one system was compliant, systems feeding to that system causes bad data, which goes to the next system, and next system. So you had to make sure your upstream was giving you proper information.
Upstream as in any computer using any software on any OS had to be y2k compliant as well. Not upstream as in customers handing money and first ledgers level.
Supply chains are much less of a "given" than you'd think.
A bunch of engineers worked really hard making targeted fixes behind the scenes and catastrophe was averted. Then they were shat upon for overblowing the situation because they did their jobs quietly and competently.
Soon after, people started laying off their IT departments wholesale and outhousing their infrastructure.
I have no confidence such a problem could be addressed in a timely and competent manner these days.
It's going to be an absolute shit show and I'm so looking forward to it. Y2K was taken seriously before investors got involved. The 2038 problem is way harder for lay people to understand "hah they only stored the year as 2 digits so now it thinks it's 1900" is easy, "they store dates as a what? and that somehow becomes -2 billion and breaks? that's stupid, I don't understand!" is what you get for 2038.
The internet is VASTLY more prevalent now, and stacks are far larger and more complicated, it's not all single applications running on single servers like it was when I did the remediation in the late 90s.
Some of the cornerstone stacks that everything else runs on are poorly understood and remediation is now a far taller prospect than it was in the 90s.
The panic and extent is provably false though. Fire up DOSBox or Bochs or any emulator and discover just for yourself how much software works and doesn't work past 2000. It's surprising how much at worst simply doesn't show enough digits (are we talking 1920 or 2020?) and that's the end of the impact.
This is mostly a reiteration of the comments here, but the whole Y2K, except in a narrow set of circumstances, was a cosmetic bug.
Clearly you didn't have to deal with the actual software that actually runs the banks and other big businesses. Very few were concerned about DOS. We cared about all that fun stuff that runs in batches in the middle of the night.
Yep, I feel like it's one of the great examples of the world working well together. Like with getting rid of CFCs, DDT etc. Or dealing with Heartbleed.
The title of the paper sounds kind of dismissive/conspiracy theorish, but it's not. It's actually a pretty straightforward cataloging of what the problems were, the strategies used to mitigate it, and actual failures that happened.
I had to set the date on a network node to 1990 so it wouldn't crap out in 2000. I left detailed instructions everywhere before I left the company in February of 2000.
A bunch of engineers realized a problem several years earlier and made a bunch of targeted fixes that worked pretty well so it seemed like mostly nothing happened. But it wasn’t necessarily overhyped, just well executed.
It was both. It was easy to get Y2K budget, so I remember for instance the local hospital replacing obsolete electronic equipment without even the concept of a date that they never succeeded in clearing the budget for before.
We will face a similar problem with 32 bit systems in 2038 but I have zero faith the current world has the ability to collaborate together to get past it. So I just hope there’s nothing critical still running on 32 bit systems then
the 2038 problem has been long solved in all major operating systems and you can use 64 bit ints on 32 bit systems, you just need to compile it properly.
the only stuff that could break are forgotten embedded (unixy - your local win xp atm is not susceptible!) devices somewhere but that's not going to cause issues even close to the extent of y2ks potential.
Sure, but so had 4 digit years in 1999. It's nothing to do with your playstations or phones; it's about massive systems noone has money to maintain and that were due for sunset a decade ago but still are keeping business running
Rubbish, the only things that will break are the things nobody bothered to update. There will be no problems with this in 2038 so you don’t have to worry about everyone collaborating together to solve it
Also, it was entirely possible to shut down most of the systems “just in case” on Sec 30th when we left work and turn them on on Jan 2nd when we got back. It was likely not needed but it was equivalent of modern “don’t deploy on Fridays” mindsets.
In today’s era of everything available all the time and half hour downtimes of anything making the news, management would rather risk losing 26 years of financial transactions and customer data than shut down their web shop of a neighbourhood specialty coffee brewer.
There was quite a lot of unnecessary work done as well. I ended up going through the Apache web server in detail even though loads of people had already checked it, because my boss was nervous.
I worked on Y2K abatement at two places. At the first one it would have been business-ending catastrophic. And the other we just had to replace a few pieces of equipment and replace some others as part of the standard replacement cycle.
Oh, I know. I was just poking fun at the AI/LLM cheerleaders.
Yea, it is a very difficult problem, but not insurmountable. Like with any large overwhelming problem, the only way to approach it is to understand the fundamental problem and then break it down into groups and identify the risk and then continue breaking it down until you can start implementing strategies that can translate to policies, incentives/task forces and communication/language.
Although, the thought of having all NTP servers set their date to some date in the future post-2038 once a year for a single day and ask everyone to not do anything important/sensitive that day (ie. An international holiday) just to see what breaks amuses me. But I know that’s unrealistic and it assumes a lot about how these embedded systems do timekeeping. But, on the other hand, if our whole civilisation depends on so many flaky systems, maybe it makes sense to break it on purpose to find where the weak points are.
Y2K was in the data center, and 2038 is going to be in the walls. Its all the forgotten embedded systems that are going to cause trouble.
The second bit of fun was who was on the hook for Y2K. It is amazing how money will be spent and plans will be made when the financials are part of the problem. There are (and were) a lot of regulations on what happens to people who don't get it correct. I doubt your HVAC or elevator has the same level of C-Suite scrutiny.
Wait until that big machine on the plant floor has a bad morning to see how it goes.
Not necessarily. Debian i386 will stay on 32 Bit time_t. And Debian is a basis for quite a lot.
This was explicitly decided for compatibility reasons. While a lot of system will already be on amd64, I suppose there will still be around the odd systems using Debian i386 even in 2038.
Also, armhf has switched to 64 Bit time_t in 2024, so there could still be legacy systems up by then not having been updated.
I work with a proprietary operating system whose last update for their previous major version added 64 Bit time_t support in 2024. I assume the vendor expects systems with that operating system to still run in 2038 as well.
Almost nobody in the programming space thinks that Y2K date problem was a “myth” but that doesn’t mean that the panic over it was warranted. When you face a problem that could render a ton of software (software that makes money) useless, programmers will do what they get paid to do and fix the issue, regardless of what needs to be organized and regardless of how difficult the problem is.
The Y2K problem was completely unprecedented in software engineering until that time. They actively had to recruit people and pay them top of the line salaries, often bringing people out of retirement, in order to actually maintain software. Come to think of it, nothing like it has happened before or since.
Lmao, there we go. Assuming you’re an old head who was around during that time. So you were on the front lines, right? Saving the world? That’s how these Y2K threads always go; they turn into old jackasses “um actually”ing everybody for dismissing whatever you contributed to solving that problem.
Having to disable reprogram an oil refinery's automated emergency shutoff sensors Jan 1st 2000 might cause some disruptions in day to day life for a bit. Especially if you have to do that with every other piece of computerised equipment the same day, and especially if that computerised equipment is still reliant on reading code written with punchards.
You know how every few years AWS has an outage and you lose the ability to book an airline ticket, or log into some of your banks, or stream netflix, or use some of the programs at work that relied on AWS. Picture that, but every one of the apps need to be fixed independently.
You've continued to diminish the efforts without any understanding of the scope and scale. If you honestly can't see that, then you're hopeless. It's hard to recognize valuable impact when you've gone your whole career providing none.
Damn im sorry man i should pay you your dues. Thank you so much for saving the world, you’re a hero. Will you be willing to come out of retirement to help before 2038? We will need you. There are no developers as great as you anymore.
Are you also upset when people tell you that potholes are repaired by roadworkers?
Or do you say something along the lines of; "Oh those brave construction workers, /s they deserve so much praise and a livable wage, /s nobody else would have been able to prevent me from getting a flat tire /s"
It’s kind of a clunky comparison, no? Road workers don’t get paid well, whereas developers do (especially ones who were working on updating software to support 4 digit years). Them not being paid particularly well is a strong argument against anybody taking them for granted. Besides, again I am not taking brilliant developers for granted, I am simply baffled that you can say “the panic for y2k wasn’t warranted” and you get all these defensive replies.
I know you are being sarcastic, but there is no way in hell im getting out of retirement to deal with some BS spaghetti code I wrote 30 years ago, or taking an internship that requires me to use COBOL instead of something useful.
but that doesn’t mean that the panic over it was warranted.
It's a bit of a catch 22. There was no need to panic, because everyone was aware of the issue and could get the funding to fix the problem.
But the so-called "panic" was the reason everyone was aware of the problem and found it incredibly easy to get the required funding to fix it.
Some of the predictions for what could happen if nobody did any mitigations might have been a bit hyperbolic, but not by much. The chance of planes falling out of the sky was more or less zero, but the chance of the ATC system failing (along with the scheduling systems of many airlines) and grounding all air traffic for weeks is quite plausible.
The chance of ATM machines suddenly spitting out money might have made for great advertising visuals, but was basically impossible. But the chance of ATM machines, along with the rest of banking infrastructure failing making it impossible to withdraw/transfer money is very plausible. Just imagine how much chaos and disruption that would cause.
And nationwide blackouts? Actually scarily plausible.
The national grid is actually pretty fragile. If 20% of the generation capacity suddenly goes offline due to a y2k software bug, the resulting drop in line frequency will rapidly trigger the safety cutoffs of other generators (to protect them from damage), which quickly leads to the whole grid cascading to a halt. Restarting the grid from this black start scenario is very tricky, even if there were zero other issues happening at the same time.
At my current employer, there are similarly-bad problems that only ever get fixed when one of the following happens:
Someone cares enough to make themselves a persistent annoyance, bringing it up with their manager and skip for like six months to a year before anyone even starts working on it.
Someone cares enough to just ignore their OKRs and sacrifice any career development to work on this problem instead, basically daring management to PIP them for doing the right thing.
The problem finally blows up and costs the company millions, and then, finally, management understands the problem and scrambles people to fix it.
I've worked in places that actually fixed things preemptively without me personally having to go annoy a VP every month about it. Like, you could raise an issue like this, and you'd hear back from the team that's already working on it! But unfortunately, that's not every company, probably not most companies.
I wasn't doing this in Y2K, so I don't know for sure, but I think the panic probably helped. It's the same reason that when there's a major security vulnerability today, it gets its own cute logo, domain name, and social media campaign, because it's so goddamned hard to get most employers to care about CVEs, but you can get them to care about scary names like Heartbleed and Spectre.
And that's exactly why I worry about 2038. It's easier to fix, at least, but we need to convince these same executives who won't fund so much as a version bump without a "We're gonna be fined $mm for violating these regulations if we don't patch this CVE" stick... that we need to fix this thing that's like Y2K but worse? It'll be 2040 before they act.
People constantly fighting fires don't want to hear about something that's merely emitting an alarming quantity of smoke.
Which is how the next thing catches fire and needs an emergency fix.
The obvious thing would be to "get ahead" of these issues, but it's almost impossible to convince a group of people to change their attitude this way, especially if they're constantly in a panic. They can't sit down and rationally plan out the next year if there's alarms blaring constantly.
Personally, I've "taken over" some IT systems from people that essentially burned out and abandoned the ship. I managed to easily rescue the situation and end up bored because so little work remained to do simply by doing the basic maintenance like you said.
PS: My favourite example of this is Microsoft's method of rolling out breaking changes in their monthly security patches. Month #1 is the "enablement" patch, month #2 is the "enforcement" patch. Orgs that patch monthly don't even notice these breaking changes. Months that are "too busy" and patch only quarterly are constantly running around putting out fires caused by "ohmygerd Microsoft broke everything for us again!"
This is a more interesting perspective than those needlessly pointing out that it was expensive and difficult work: that it is very hard to getting businesses / companies to react early to system breaking issues
I lived through it too and while I do think some of it was warrented, a lot of it was kind of silly hysteria by folks who don't understand computers. They had this massive effort at my job to fix stuff and quit a bit of the stuff they were fixing was not even used any more. I think part of it is that the problem was easy to understand "dates won't work" and it was not something like some race condition that will take everything down they don't understand where they go "well I hope that never happens".
and if those projects hadn't gotten done there would've been a major disaster, like the one people were predicting. So people did their boring jobs and saved the world.
i remember reading about this years ago and it genuinely changed how i think about software maintenance. the amount of invisible work that went into fixing y2k is insane - thousands of engineers quietly patching systems nobody even knew existed. and then because nothing broke, everyone assumed it was a hoax. classic case of success being invisible. the real lesson is that boring preventive work never gets credit.
I had two contracts for fixing Y2K issues. One was a new install of PeopleSoft in 1998 to replace an older system written in-house using only 2 digit years which was going to fail in 2000. So the company took advantage of an install, which delivered Payroll, HR & Benefit updates rather than them interpreting themselves from the legislative changes, and to get past the 2 digit year issue.
The other was a system I wrote in PL/1 in 1984 that was supposed to used for only a few years, but was notified that it was still running in 1999. Same problem, rewrite it to allow for 4 digit years, problem solved.
Most systems were corrected in the several years prior to 2000, but still some primarily embedded systems were forgotten about, and caused minor failures. An example that I saw, was a standby generator that had a monthly maintenance cycle that turned itself on for a period of time, then shut down again. After 2000, it ran its cycle, saw that it hadn't been done for 100 years then shutdown completely, until manual replacement of the control board was done. I believe a few locomotives in Australia had this problem as well.
But imagine if there was an affected megawatt standby generator that started then shut down during a power outage, and caused a ripple affect that caused multiple grids to shut down. It could have been much worse across the world, but people did their jobs and fixed the issues, so that it became a non-issue when 2000 rolled around. That's not to say there weren't charlatans all over the place charging exorbitant amounts to fix systems that didn't need any fixing at all. I saw that happen as well.
I don't believe any of these explanations, I had a buddy who had a digital watch that was believed to not be Y2K compliant, and we sat around the table counting down, and it just kept going. So obviously the only explanation that makes sense is time travel, John Titor was real and accomplished his mission.
The problem was caused by poor software engineering. Peter Neumann observed that systematic use of concepts such as abstraction, encapsulation, information hiding, and object-orientation could have allowed the construction of efficient programs in which the representation of dates could be changed easily when needed.
Peter Neumann can suck it.
Most of the systems seriously affected by these problems were so storage-poor and byte-conscious that relatively modern (1990's) concepts like "abstraction, encapsulation, information hiding, and object-orientation" were just not possible during their design and implementation.
For perspective: in 1984 for the IBM PC 64k (that's kilobytes) of memory was $350.
I worked, nothing happened, I got paged about 15 minutes after I left work and had to go in for about 5 minutes. Ended up being 4 hour minimum at quadruple time because y2k on a stat holiday, 16 hours for 5 minutes work.
I was locked in a separate lab, disconnected from the main networks and in a different building, with all the clocks set to either 2000+ or 1999-12-31 23:50.
I worked on foreign exchange software for what was then the largest foreign exchange shop in the world. I made sure all the stuff was working to feed their system.
Three things at once: Y2K so four digit years, SunOS -> Solaris upgrades as SunOS wasn't Y2K compatible, and new versions of the financial libraries for market data transmission (Teknokon and Tibco at the time).
I added the four digit feature to our software, worked out the Solaris migration and fixed a bug in the messaging system used by banks worldwide - without that fix, a large amount of financial companies would have been in the dark and not knowing why. Turned out they'd shipped libraries (C) that had a struct misdocumented. Header files said the fields were in one order, but if you did a core dumb and examined it carefully you'd see the fields were actually in a different order. Fix header, recompile...all good. Shipped that fix to the vendor.
One serious UK problem was only recognised when a Heath Visitor in Yorkshire noticed an unusual number
of babies were being born with Down’s Syndrome. It transpired that more than 150 pregnant women were
given the wrong results from pathology laboratory tests because the PathLAN computer system that was used in
nine hospitals calculated the women’s date of birth incorrectly from January 2000; it had worked perfectly for
the previous decade. The result was that women who should have been identified as being in a high-risk
category were wrongly told that they did not need further testing
The year 2000 was a leap year, which some programmers had overlooked. Since 1582, a year has been a leap year if it divided by 4 except if it divided by 100 – except that if it divides by 400 it is again a leap year. 2000 was the first century leap year since 1600; there will not be another until the year 2400. So programmers who wrongly thought that there was always a leap year every 4 years got 2000 right, whereas some slightly-better-informed programmers who knew the 100 year rule but overlooked the 400 year rule got it wrong, though they may get their revenge in 2100.
This is the best part of working a whole lot of unpaid overtime to fix y2k bugs in time - at the end of the day, all the folks who didn't understand the issue come out of the woodwork to explain to those of us who saved their butts that clearly the panic wasn't warranted, because nothing bad happened. Duh.
We fixed a bunch of problems in our code base. The only ones we missed were some websites that used "19" + today.getYear() and so started displaying the date as Jan 1. 19100.
I was working at a business that rewrote their core app to handle four digit dates before 1999 - because "99" was used as a marker to say "not valid". They made it with about two months to spare, literally the whole business on the line.
Initially: “find our real Y2K problems in our code. Fix them.”
Effect of local efforts: a number of important changes where we stored year in two digits.
Later: “C-suite wants to pay these consultants $BIGNUM to find Y2K problems in our organization”
Effective of consultants: fancy “Y2K Compliant” stickers on CHAIRS, CABINETS, MONITORS, and other things that didn’t care about the year.
Admittedly some of those stickers (the chairs) were a local joke but the stickers were real and the consultants were spending time certifying equipment that had no notion of what year it was.
So yeah. Big problems that got fixed, plus scammers.
I wrote a tracking system for the title in escrow business back in '93. What I didn't take into account was for your digits on the dates. So by 1999 I had to redesign the database to handle a better date system. It would have made problems if the date clicked over to 01 instead of 2001. I also had to evaluate all of their computer systems to see what would be affected. It was a big deal.
The late 90s were a very hopeful time of fixing this tricky bug and saving the ozone layer, but then we just blew it.
I don’t know if it would have made a difference, but it is too bad we weren’t looking thousands of years ahead to Y10K—not just dealing with that extra digit, but making sure humanity will still be around, e.g. Long Now, but more action-oriented than pretentious.
Calendar dates are messy, and there is no reason we can't keep using Unix time until the heat death of the universe, assuming we standardize the relativistic reference frame.
walong0@reddit
A bunch of engineers realized a problem several years earlier and made a bunch of targeted fixes that worked pretty well so it seemed like mostly nothing happened. But it wasn’t necessarily overhyped, just well executed.
We addressed several issues in the system I worked on at the time that would’ve caused problems though nothing catastrophic.
DigThatData@reddit
Also, "nothing catastrophic" is generous in the presence of multiple co-occurring unusual failures.
The worst incidents are almost never one big thing going wrong, but several little things going wrong and interacting unusually. That's why Y2K was so dangerous. There was a high risk that a lot of little things would go wrong across a lot of heterogeneous, highly interconnected systems all at once.
Nothing happened, but the risk was tremendous.
protomyth@reddit
Yeah, I spent most of the end part of the 90's doing a lot of edge case testing to make sure we were ready. We spent a lot of money to make sure it wasn't an issue.
It did amaze me how many systems were not Y2K compliant that were super vital. Some had some amazingly bad failure conditions.
Plank_With_A_Nail_In@reddit
We just did something along the lines of
"if 2_digital_year < 70 then 2000 + 2_digit_year else 1900 + 2_digit_year"
We replaced all of the companies systems in 2005.
protomyth@reddit
Some could do that. Mostly we replaced stuff with a proper 4 digit date.
I do remember a weird ass one that had a very odd date format in some dumb system where I guess 6 bytes for date was too expensive so they used only 2 (16-bits). They had encoded the date as 5 bits for day, 4 bits for month, and 7 bits for year (starting at 1900). I suppose the savings in storage was awesome /s. They sequenced it year, month, date. Luckily, they didn't do sorts on the original 2 bytes.
I had to report we didn't have a year 2k problem, but a 2027 problem. We left it alone and did some testing it made it over 2000 fine (which it did including sorting). No idea what happened after that. I do hope someone replaced it given the current year. I do hate too cute by half solutions, but it saved 4 bytes on millions of dates for decades.
I will admit that their date calculation functions were correct and did fine for the 1900 -> 2000 transition. I once thought it would be a cute assignment in some low level computing class.
markrages@reddit
Why would a sort have failed in 1964?
falconzord@reddit
A lot of those post-Y2K systems would need to be patched again for Y2K38 if they're not replaced already
rodw@reddit
I didn't do much y2k stuff directly but I was sorta y2k adjacent in the late 90s in that I was one of the kids (early career not an actual child) given a somewhat irresponsible level of authority and autonomy building early web stuff while the people with any actual institutional knowledge were busy making big money working n y2k issues.
It's sorta a pet peeve of mine - on your behalf - that people seem to think y2k was some kind of social panic rather than a real issue because planes didn't fall out of the sky. It is possible it might have been a little overblown, but it was for sure a real problem. The reason we didn't have catastrophic issues is that we spent a decade fixing it. I mean, I personally didn't but a veritable legion of people like you did. I appreciate your service.
It's like acid rain in the 1980s or the ozone hole in the 1990s. We don't have those problems now, but not because they weren't real problems. They were genuine problems, and we fixed them
BlobbyMcBlobber@reddit
Such as??
gopher_space@reddit
Payroll was a big one. Cascading failure states.
knightress_oxhide@reddit
Yes, one thing that people may not have realized was even if one system was compliant, systems feeding to that system causes bad data, which goes to the next system, and next system. So you had to make sure your upstream was giving you proper information.
crazedizzled@reddit
That should kind of be a given though, ya?
picklemanjaro@reddit
Upstream as in any computer using any software on any OS had to be y2k compliant as well. Not upstream as in customers handing money and first ledgers level.
Supply chains are much less of a "given" than you'd think.
knightress_oxhide@reddit
and remember that all these systems were still running while this process was being handled and new data was being generated
ketralnis@reddit
"Should" is doing all of the heavy lifting there
protomyth@reddit
That was the other fun. Code from before I was born was one thing, but transfer data file formats were a special kind of hell.
Gwaptiva@reddit
Air Traffic Control systems running OS/2 2.0 or System/36s (if memory serves)
protomyth@reddit
I seem to remember a PDP-8 that had a conveyor hooked to it and the overflow told it to do a speed run at the wrong time.
Nikoli_Delphinki@reddit
I remember doing y2k tests on it home pc and learning most were bunk to just sell you a new computer.
Humdaak_9000@reddit
A bunch of engineers worked really hard making targeted fixes behind the scenes and catastrophe was averted. Then they were shat upon for overblowing the situation because they did their jobs quietly and competently.
Soon after, people started laying off their IT departments wholesale and outhousing their infrastructure.
I have no confidence such a problem could be addressed in a timely and competent manner these days.
fiah84@reddit
we'll find out in about 12 years
OMGItsCheezWTF@reddit
It's going to be an absolute shit show and I'm so looking forward to it. Y2K was taken seriously before investors got involved. The 2038 problem is way harder for lay people to understand "hah they only stored the year as 2 digits so now it thinks it's 1900" is easy, "they store dates as a what? and that somehow becomes -2 billion and breaks? that's stupid, I don't understand!" is what you get for 2038.
The internet is VASTLY more prevalent now, and stacks are far larger and more complicated, it's not all single applications running on single servers like it was when I did the remediation in the late 90s.
Some of the cornerstone stacks that everything else runs on are poorly understood and remediation is now a far taller prospect than it was in the 90s.
atomic1fire@reddit
Programmers store dates in a magic box that can only contain a number that is really big.
When the box gets filled with too much number, the box empties and starts filling up with magic number again.
This throws systems out of chaos because they built everything around this magic number box working as intended.
The solution is to have an even bigger magic number box.
nicofff@reddit
This is basically my retirement plan.
Humdaak_9000@reddit
Mine is cleaning up after the AI slopocalypse.
Just call me Serena Butler.
maxineasher@reddit
The panic and extent is provably false though. Fire up DOSBox or Bochs or any emulator and discover just for yourself how much software works and doesn't work past 2000. It's surprising how much at worst simply doesn't show enough digits (are we talking 1920 or 2020?) and that's the end of the impact.
This is mostly a reiteration of the comments here, but the whole Y2K, except in a narrow set of circumstances, was a cosmetic bug.
protomyth@reddit
Clearly you didn't have to deal with the actual software that actually runs the banks and other big businesses. Very few were concerned about DOS. We cared about all that fun stuff that runs in batches in the middle of the night.
YoghiThorn@reddit
Yep, I feel like it's one of the great examples of the world working well together. Like with getting rid of CFCs, DDT etc. Or dealing with Heartbleed.
avemg@reddit
The title of the paper sounds kind of dismissive/conspiracy theorish, but it's not. It's actually a pretty straightforward cataloging of what the problems were, the strategies used to mitigate it, and actual failures that happened.
JulesSilverman@reddit
I had to set the date on a network node to 1990 so it wouldn't crap out in 2000. I left detailed instructions everywhere before I left the company in February of 2000.
chat-lu@reddit
It was both. It was easy to get Y2K budget, so I remember for instance the local hospital replacing obsolete electronic equipment without even the concept of a date that they never succeeded in clearing the budget for before.
pa_dvg@reddit
We will face a similar problem with 32 bit systems in 2038 but I have zero faith the current world has the ability to collaborate together to get past it. So I just hope there’s nothing critical still running on 32 bit systems then
hammer-jon@reddit
the 2038 problem has been long solved in all major operating systems and you can use 64 bit ints on 32 bit systems, you just need to compile it properly.
the only stuff that could break are forgotten embedded (unixy - your local win xp atm is not susceptible!) devices somewhere but that's not going to cause issues even close to the extent of y2ks potential.
Gwaptiva@reddit
Sure, but so had 4 digit years in 1999. It's nothing to do with your playstations or phones; it's about massive systems noone has money to maintain and that were due for sunset a decade ago but still are keeping business running
JustSingingAlong@reddit
Rubbish, the only things that will break are the things nobody bothered to update. There will be no problems with this in 2038 so you don’t have to worry about everyone collaborating together to solve it
casastorta@reddit
Also, it was entirely possible to shut down most of the systems “just in case” on Sec 30th when we left work and turn them on on Jan 2nd when we got back. It was likely not needed but it was equivalent of modern “don’t deploy on Fridays” mindsets.
In today’s era of everything available all the time and half hour downtimes of anything making the news, management would rather risk losing 26 years of financial transactions and customer data than shut down their web shop of a neighbourhood specialty coffee brewer.
my_beer@reddit
There was quite a lot of unnecessary work done as well. I ended up going through the Apache web server in detail even though loads of people had already checked it, because my boss was nervous.
shotsallover@reddit
I worked on Y2K abatement at two places. At the first one it would have been business-ending catastrophic. And the other we just had to replace a few pieces of equipment and replace some others as part of the standard replacement cycle.
Dwedit@reddit
2038 gonna make Y2K look like a joke.
idebugthusiexist@reddit
AI is fix all our problems. Don’t worry.
fumei_tokumei@reddit
We need something better than LLMs for that to be the case, and nobody knows what that is.
idebugthusiexist@reddit
Oh, I know. I was just poking fun at the AI/LLM cheerleaders.
Yea, it is a very difficult problem, but not insurmountable. Like with any large overwhelming problem, the only way to approach it is to understand the fundamental problem and then break it down into groups and identify the risk and then continue breaking it down until you can start implementing strategies that can translate to policies, incentives/task forces and communication/language.
Although, the thought of having all NTP servers set their date to some date in the future post-2038 once a year for a single day and ask everyone to not do anything important/sensitive that day (ie. An international holiday) just to see what breaks amuses me. But I know that’s unrealistic and it assumes a lot about how these embedded systems do timekeeping. But, on the other hand, if our whole civilisation depends on so many flaky systems, maybe it makes sense to break it on purpose to find where the weak points are.
fumei_tokumei@reddit
Haha, okay. You really need a /s when you jokingly write stuff that is indistinguishable from people's honest believes on this site.
protomyth@reddit
Y2K was in the data center, and 2038 is going to be in the walls. Its all the forgotten embedded systems that are going to cause trouble.
The second bit of fun was who was on the hook for Y2K. It is amazing how money will be spent and plans will be made when the financials are part of the problem. There are (and were) a lot of regulations on what happens to people who don't get it correct. I doubt your HVAC or elevator has the same level of C-Suite scrutiny.
Wait until that big machine on the plant floor has a bad morning to see how it goes.
stevep98@reddit
2038 seemed such a long time away back in 1999. And now look, it’s almost here.
Maybe AI will be our savior for this problem though…
remy_porter@reddit
It won't. The problem isn't making the fixes- it's deploying the fixes to devices which often don't have any sort of network interface.
goranlepuz@reddit
Even 32-bit systems use 64-bit unix epoch since quite some time.
It should merely be a few irrelevant datasets in some dark corner 😉.
0xLeon@reddit
Not necessarily. Debian i386 will stay on 32 Bit time_t. And Debian is a basis for quite a lot.
This was explicitly decided for compatibility reasons. While a lot of system will already be on amd64, I suppose there will still be around the odd systems using Debian i386 even in 2038.
Also, armhf has switched to 64 Bit time_t in 2024, so there could still be legacy systems up by then not having been updated.
I work with a proprietary operating system whose last update for their previous major version added 64 Bit time_t support in 2024. I assume the vendor expects systems with that operating system to still run in 2038 as well.
TribeWars@reddit
Debian probably will also completely drop 32 bit support in one or two releases
Ruben_NL@reddit
Yeah, but I just found that some AI written code was using a 32 bit field to store a timestamp. That happened today.
This is not to blame AI for this, because humans could also make this mistake.
IanisVasilev@reddit
And possibly some slightly more relevant legacy financial, factory and transport systems.
ImHughAndILovePie@reddit
Almost nobody in the programming space thinks that Y2K date problem was a “myth” but that doesn’t mean that the panic over it was warranted. When you face a problem that could render a ton of software (software that makes money) useless, programmers will do what they get paid to do and fix the issue, regardless of what needs to be organized and regardless of how difficult the problem is.
The world was never going to end.
CherryLongjump1989@reddit
The Y2K problem was completely unprecedented in software engineering until that time. They actively had to recruit people and pay them top of the line salaries, often bringing people out of retirement, in order to actually maintain software. Come to think of it, nothing like it has happened before or since.
ImHughAndILovePie@reddit
Programmers had to program and got paid a lot of money to do it. Wow!
Baxkit@reddit
Wow, you really are clueless. This was a little more time sensitive, wide-spread, and high-impact than your little javascript to-do list app.
ImHughAndILovePie@reddit
Lmao, there we go. Assuming you’re an old head who was around during that time. So you were on the front lines, right? Saving the world? That’s how these Y2K threads always go; they turn into old jackasses “um actually”ing everybody for dismissing whatever you contributed to solving that problem.
Incorrect_Oymoron@reddit
Having to disable reprogram an oil refinery's automated emergency shutoff sensors Jan 1st 2000 might cause some disruptions in day to day life for a bit. Especially if you have to do that with every other piece of computerised equipment the same day, and especially if that computerised equipment is still reliant on reading code written with punchards.
You know how every few years AWS has an outage and you lose the ability to book an airline ticket, or log into some of your banks, or stream netflix, or use some of the programs at work that relied on AWS. Picture that, but every one of the apps need to be fixed independently.
ImHughAndILovePie@reddit
Good thing there were developers around willing to take money to fix a hard problem.
marishtar@reddit
Why does this make you so bitter?
ImHughAndILovePie@reddit
How am I the bitter one? I’m not the one coming in and saying “uhm it was actually very hard and expensive” when nobody has disputed that.
Baxkit@reddit
You've continued to diminish the efforts without any understanding of the scope and scale. If you honestly can't see that, then you're hopeless. It's hard to recognize valuable impact when you've gone your whole career providing none.
ImHughAndILovePie@reddit
Damn im sorry man i should pay you your dues. Thank you so much for saving the world, you’re a hero. Will you be willing to come out of retirement to help before 2038? We will need you. There are no developers as great as you anymore.
Incorrect_Oymoron@reddit
But why so bitter?
ImHughAndILovePie@reddit
Just returning shittiness with shittiness
Incorrect_Oymoron@reddit
Are you also upset when people tell you that potholes are repaired by roadworkers?
Or do you say something along the lines of; "Oh those brave construction workers, /s they deserve so much praise and a livable wage, /s nobody else would have been able to prevent me from getting a flat tire /s"
ImHughAndILovePie@reddit
It’s kind of a clunky comparison, no? Road workers don’t get paid well, whereas developers do (especially ones who were working on updating software to support 4 digit years). Them not being paid particularly well is a strong argument against anybody taking them for granted. Besides, again I am not taking brilliant developers for granted, I am simply baffled that you can say “the panic for y2k wasn’t warranted” and you get all these defensive replies.
lithium@reddit
Shouldn't you be busy asking an AI agent to
npm installsomebody else's code for you?ImHughAndILovePie@reddit
Nah why is that what you do
Incorrect_Oymoron@reddit
I know you are being sarcastic, but there is no way in hell im getting out of retirement to deal with some BS spaghetti code I wrote 30 years ago, or taking an internship that requires me to use COBOL instead of something useful.
rzet@reddit
just wait for the Year 2038 Problem
phire@reddit
It's a bit of a catch 22. There was no need to panic, because everyone was aware of the issue and could get the funding to fix the problem.
But the so-called "panic" was the reason everyone was aware of the problem and found it incredibly easy to get the required funding to fix it.
Some of the predictions for what could happen if nobody did any mitigations might have been a bit hyperbolic, but not by much. The chance of planes falling out of the sky was more or less zero, but the chance of the ATC system failing (along with the scheduling systems of many airlines) and grounding all air traffic for weeks is quite plausible.
The chance of ATM machines suddenly spitting out money might have made for great advertising visuals, but was basically impossible. But the chance of ATM machines, along with the rest of banking infrastructure failing making it impossible to withdraw/transfer money is very plausible. Just imagine how much chaos and disruption that would cause.
And nationwide blackouts? Actually scarily plausible.
The national grid is actually pretty fragile. If 20% of the generation capacity suddenly goes offline due to a y2k software bug, the resulting drop in line frequency will rapidly trigger the safety cutoffs of other generators (to protect them from damage), which quickly leads to the whole grid cascading to a halt. Restarting the grid from this black start scenario is very tricky, even if there were zero other issues happening at the same time.
SanityInAnarchy@reddit
At my current employer, there are similarly-bad problems that only ever get fixed when one of the following happens:
I've worked in places that actually fixed things preemptively without me personally having to go annoy a VP every month about it. Like, you could raise an issue like this, and you'd hear back from the team that's already working on it! But unfortunately, that's not every company, probably not most companies.
I wasn't doing this in Y2K, so I don't know for sure, but I think the panic probably helped. It's the same reason that when there's a major security vulnerability today, it gets its own cute logo, domain name, and social media campaign, because it's so goddamned hard to get most employers to care about CVEs, but you can get them to care about scary names like Heartbleed and Spectre.
And that's exactly why I worry about 2038. It's easier to fix, at least, but we need to convince these same executives who won't fund so much as a version bump without a "We're gonna be fined $mm for violating these regulations if we don't patch this CVE" stick... that we need to fix this thing that's like Y2K but worse? It'll be 2040 before they act.
BigHandLittleSlap@reddit
People constantly fighting fires don't want to hear about something that's merely emitting an alarming quantity of smoke.
Which is how the next thing catches fire and needs an emergency fix.
The obvious thing would be to "get ahead" of these issues, but it's almost impossible to convince a group of people to change their attitude this way, especially if they're constantly in a panic. They can't sit down and rationally plan out the next year if there's alarms blaring constantly.
Personally, I've "taken over" some IT systems from people that essentially burned out and abandoned the ship. I managed to easily rescue the situation and end up bored because so little work remained to do simply by doing the basic maintenance like you said.
PS: My favourite example of this is Microsoft's method of rolling out breaking changes in their monthly security patches. Month #1 is the "enablement" patch, month #2 is the "enforcement" patch. Orgs that patch monthly don't even notice these breaking changes. Months that are "too busy" and patch only quarterly are constantly running around putting out fires caused by "ohmygerd Microsoft broke everything for us again!"
ImHughAndILovePie@reddit
This is a more interesting perspective than those needlessly pointing out that it was expensive and difficult work: that it is very hard to getting businesses / companies to react early to system breaking issues
an_angry_dervish_01@reddit
I lived through it and worked as a developer for a major software house. It was just a bunch of projects to mitigate it, not at all a big deal.
dalittle@reddit
I lived through it too and while I do think some of it was warrented, a lot of it was kind of silly hysteria by folks who don't understand computers. They had this massive effort at my job to fix stuff and quit a bit of the stuff they were fixing was not even used any more. I think part of it is that the problem was easy to understand "dates won't work" and it was not something like some race condition that will take everything down they don't understand where they go "well I hope that never happens".
Thirsteh@reddit
and if those projects hadn't gotten done there would've been a major disaster, like the one people were predicting. So people did their boring jobs and saved the world.
G_Morgan@reddit
I mean $200B was spent on dealing with Y2K. It is an example of successfully warning people about a problem and handling it.
sambull@reddit
And it gave us the best piece of American culture ever made..
Office Space (1999)
rzet@reddit
I love this movie... and after few years as dev in corpo I hate they laugh from me ;)
MC68328@reddit
And the ozone hole was never going to spread, and the atmosphere was never going to warm.
enokeenu@reddit
COBOL programmers got jobs again.
sailing67@reddit
i remember reading about this years ago and it genuinely changed how i think about software maintenance. the amount of invisible work that went into fixing y2k is insane - thousands of engineers quietly patching systems nobody even knew existed. and then because nothing broke, everyone assumed it was a hoax. classic case of success being invisible. the real lesson is that boring preventive work never gets credit.
SpookyTheCat96@reddit
I had two contracts for fixing Y2K issues. One was a new install of PeopleSoft in 1998 to replace an older system written in-house using only 2 digit years which was going to fail in 2000. So the company took advantage of an install, which delivered Payroll, HR & Benefit updates rather than them interpreting themselves from the legislative changes, and to get past the 2 digit year issue.
The other was a system I wrote in PL/1 in 1984 that was supposed to used for only a few years, but was notified that it was still running in 1999. Same problem, rewrite it to allow for 4 digit years, problem solved.
Most systems were corrected in the several years prior to 2000, but still some primarily embedded systems were forgotten about, and caused minor failures. An example that I saw, was a standby generator that had a monthly maintenance cycle that turned itself on for a period of time, then shut down again. After 2000, it ran its cycle, saw that it hadn't been done for 100 years then shutdown completely, until manual replacement of the control board was done. I believe a few locomotives in Australia had this problem as well.
But imagine if there was an affected megawatt standby generator that started then shut down during a power outage, and caused a ripple affect that caused multiple grids to shut down. It could have been much worse across the world, but people did their jobs and fixed the issues, so that it became a non-issue when 2000 rolled around. That's not to say there weren't charlatans all over the place charging exorbitant amounts to fix systems that didn't need any fixing at all. I saw that happen as well.
gullydowny@reddit
I don't believe any of these explanations, I had a buddy who had a digital watch that was believed to not be Y2K compliant, and we sat around the table counting down, and it just kept going. So obviously the only explanation that makes sense is time travel, John Titor was real and accomplished his mission.
clintp@reddit
Peter Neumann can suck it.
Most of the systems seriously affected by these problems were so storage-poor and byte-conscious that relatively modern (1990's) concepts like "abstraction, encapsulation, information hiding, and object-orientation" were just not possible during their design and implementation.
For perspective: in 1984 for the IBM PC 64k (that's kilobytes) of memory was $350.
706union@reddit
I worked, nothing happened, I got paged about 15 minutes after I left work and had to go in for about 5 minutes. Ended up being 4 hour minimum at quadruple time because y2k on a stat holiday, 16 hours for 5 minutes work.
Pzzlrr@reddit
I was in 5th grade and was told all our computers would explode like the pager attack. My disappointment was immeasurable and my day was ruined.
mccalli@reddit
I was locked in a separate lab, disconnected from the main networks and in a different building, with all the clocks set to either 2000+ or 1999-12-31 23:50.
I worked on foreign exchange software for what was then the largest foreign exchange shop in the world. I made sure all the stuff was working to feed their system.
Three things at once: Y2K so four digit years, SunOS -> Solaris upgrades as SunOS wasn't Y2K compatible, and new versions of the financial libraries for market data transmission (Teknokon and Tibco at the time).
I added the four digit feature to our software, worked out the Solaris migration and fixed a bug in the messaging system used by banks worldwide - without that fix, a large amount of financial companies would have been in the dark and not knowing why. Turned out they'd shipped libraries (C) that had a struct misdocumented. Header files said the fields were in one order, but if you did a core dumb and examined it carefully you'd see the fields were actually in a different order. Fix header, recompile...all good. Shipped that fix to the vendor.
We did a lot of work.
sciolizer@reddit
Wow
golgol12@reddit
Damn. Thank you author.
BTW, there's a second Y2k like bug coming.
jessek@reddit
A lot of work was done prior to y2k but that doesn’t make for scary headlines
amroamroamro@reddit
there's always a gotcha 😂
merft@reddit
https://www.youtube.com/watch?v=99z_d-iRnsg
stupid_cat_face@reddit
Office Space
NotMyself@reddit
Grifters learned they could use fear to make money.
ysustistixitxtkxkycy@reddit
This is the best part of working a whole lot of unpaid overtime to fix y2k bugs in time - at the end of the day, all the folks who didn't understand the issue come out of the woodwork to explain to those of us who saved their butts that clearly the panic wasn't warranted, because nothing bad happened. Duh.
NotMyself@reddit
Remember all the survivalist crap that people were buying?
gofl-zimbard-37@reddit
Grifters learned that millennia ago. Y2k was real.
zeekar@reddit
We fixed a bunch of problems in our code base. The only ones we missed were some websites that used
"19" + today.getYear()and so started displaying the date as Jan 1. 19100.Logical-Pea-4135@reddit
Nothing. Everyone with a brain just hung out and drank.
zippy72@reddit
I was working at a business that rewrote their core app to handle four digit dates before 1999 - because "99" was used as a marker to say "not valid". They made it with about two months to spare, literally the whole business on the line.
Farsyte@reddit
Initially: “find our real Y2K problems in our code. Fix them.” Effect of local efforts: a number of important changes where we stored year in two digits.
Later: “C-suite wants to pay these consultants $BIGNUM to find Y2K problems in our organization” Effective of consultants: fancy “Y2K Compliant” stickers on CHAIRS, CABINETS, MONITORS, and other things that didn’t care about the year.
Admittedly some of those stickers (the chairs) were a local joke but the stickers were real and the consultants were spending time certifying equipment that had no notion of what year it was.
So yeah. Big problems that got fixed, plus scammers.
Gwaptiva@reddit
Paid for the white goods in my new house, and I was but a small paper pusher; just wishing I'd learnt COBOL
Infymus@reddit
I wrote a tracking system for the title in escrow business back in '93. What I didn't take into account was for your digits on the dates. So by 1999 I had to redesign the database to handle a better date system. It would have made problems if the date clicked over to 01 instead of 2001. I also had to evaluate all of their computer systems to see what would be affected. It was a big deal.
champs@reddit
The late 90s were a very hopeful time of fixing this tricky bug and saving the ozone layer, but then we just blew it.
I don’t know if it would have made a difference, but it is too bad we weren’t looking thousands of years ahead to Y10K—not just dealing with that extra digit, but making sure humanity will still be around, e.g. Long Now, but more action-oriented than pretentious.
MC68328@reddit
Calendar dates are messy, and there is no reason we can't keep using Unix time until the heat death of the universe, assuming we standardize the relativistic reference frame.
But then that all got fucked by leap seconds.
hitanthrope@reddit
I was on my honeymoon in 2004. Met a guy there who had been a project manager for 15 years, and before that spent 3 years as a cobol guy.
Claimed he made 3 million contracting doing y2k work between 1996-1999.
That’s what really happened.
Boye@reddit
yeah , i have an uncle who's an old school programmer, appearantly he also made a pretty penny in the years leading up to 2000
Multidream@reddit
That was a great read. Thanks for posting it!
farnsworth@reddit
My fifth grade teacher told us to unplug all of our clocks before nye