GitLabs is my recommendation for a corporate solution with all the bells and whistles, minimal manual setup.
But for small business Forgejo + Woodpecker CI is my current go to. I had it up and running on my homelab (k8s) with dynamic scaling runners in like 1 hour.
For me its just so much extra work at all points, from setup to maintenance to every new pipeline you want to add, at least compared to gitlab which was dead simple across the board in comparison
And let's be honest, while some projects absolutely need super complicated CI systems with super flexible container based pipelines, the majority of repos in GitHub would be just as well served by a Makefile.
The demand for that stuff is partly driven by "because it's there" and it has become the path of least resistance for some basic testing and deploying. There are many, many, many projects I've seen where the CI pipeline spends orders of magnitude more time and resources shuffling data and queuing and spawning and reporting, than actually running the two or three small test cases for a Python script that doesb;r even need a build process.
Then Github goes WE MUST OMEGA LEVEL HYPER SCALE and expand to meet the needs of the expanding bureaucracy that most people never actually needed. If you self-host your CI, you can make it work however is convenient, and you don't necessarily need to shape everything into something as flexible as the public runner models. Often times literally a few hundred milliseconds on a server accomplished more than five minutes of pipelines on a public service.
I totally agree with you. I personally hate all the pipeline tools. I unfortunately made my career out of DevOps and Platform Engineering so I've had to evaluate every tool ever.
My personal apps are a huge mono repo and a single orchestrator script generates (templates not LLM) build/test/publish/deploy methods at runtime for each project.
Windows runners are still a pain. 90% of OSS CI/CD pipelines are linux only because they use containers for the sandboxing. Hence I have a tiny Jenkins instance running for my projects I want to support Windows.
On the other hand I'm considering just dropping Windows support
And perhaps ironically, a little bit of elbow grease + some agents running can get your team a plenty good enough code review tool that can integrate with your source control. Obviously harder to scale for a massive organization, but for most shops they could likely build and operate most of what they need.
I mean, it starts out by saying that it’s unacceptable and then they go on to explain all of the reasons why you should accept it as opposed to making some kind of change or taking action in response, which would be what it looks like to not accept it.
Not really, doesn't sound unrealistic to me given what I've seen on my own open source project. The barrier to entry was lowered by orders of magnitude, I see a lot more spot contributors and a lot more slop PRs.
For GitHub it doesn't matter if the traffic is useful, all traffic must be handled, so they are the ones suffering here (albeit contributing to the mess themselves via Copilot)
More details are available in the incident root cause analysis.
This post is intended to communicate a high level overview of what they’ve seen fail and how they are addressing those failures.
It not intended to be a full post-mortem, nor would I want it to be. I just want to know what they’ve learned from their failures and how they are architecting a solution. That’s exactly what this post does at a high level.
The details you think are missing are in their incident root cause analysis, which is exactly what they stated before your quote.
lol nope. I’m an SRE on a team with six others. We “self host” GitLab in AWS, and GitHub functionality pales in comparison to the CI/CD and organizational management of GitLab.
Our team of seven supports infrastructure across roughly 3-4 dozen AWS accounts costing us tens of millions of dollars a month. And we support hundreds of developers and engineers running CI/CD workflows 24/7 across six continents.
I self host Forgejo, a Gitea fork, in my homelab for most of my own repos.
I also have about 30 repos on GitHub for various projects.
I just have a thing for identifying and calling out bad faith actors ;)
Except everyone agreed with me, because this post is devoid of any information to the point that it’s insulting. There’s not even a hyperlink to these allegedly more detailed post mortems, and even if those were good, they could still provide any amount of color in this post.
Instead of saying “we had a bad process and now it’s fixed, don’t worry.” There’s no circumstance, ever, where I’m going to read that and not be annoyed.
Yes. I read the numbers that say the upvotes I got far outweigh the upvotes for dissenting posts. So it truly feels that I know how to read better than you.
Yes, a lot of people in the subreddit that is notorious for being negatively polarized to LLMs are, indeed, negatively polarized towards anything related to the topic, such as a post by a CTO saying they're getting hammered by an accelerated amount of LLM-generated code.
That you cannot read the post in question, which clearly describes that details of issues are in incident reports, when posting how angry and offended you are that this post doesn't contain those details, is a skill issue of your own right.
That there are others here who causally upvote anything negatively polarized towards LLMs simply because of the polarization does not take away from the fact that you, specifically, are unable to read.
But I read the post and was unsatisfied with its content. I think I understood it at least. Could you maybe summarize the parts that you thought were good so I could understand? Maybe I actually misread it
Github has just really taken a quality nosedive since the MS acquisition. Seems like there is a lot of pressure to deliver and things are being accelerated despite not being ready.
Microslop lying about things and also making them worse, well I never. But for real, their track record of doing *anything* good for the software industry has been eroding over the last decade or two
It seems increasingly evident to me that public services like github are going to be unusable and unreliable, and that on an enterprise level, the path forward is with tightly controlled inhouse or onprem instances. Something tells me that ops/devops is going to be eating good as public services continue to degrade.
There is a certain irony that LLMs and in particular agentic coding workflows have highlighted a lot of the problems that have always been there with "Agile" development but it wasn't popular for developers to point them out. It turns out that moving faster but in semi-random and constantly changing directions because your product and leadership people are incapable of ever making a decision or committing to anything that takes more than five minutes doesn't really build actual business value faster. Who knew?
And now the high costs of those agentic workflows - both financial and otherwise - are becoming clearer and they're highlighting the problems of trying to define everything in great detail up front and not emphasising an adaptable coding style and software design that is easy to maintain and extend as requirements change or new requirements are added. If only there had been anyone around with the knowledge and experience to warn about this problem!
I'm buying popcorn for the moment in a year or two's time when business leaders finally realise that a lot of getting good results comes down to having good judgement about how to balance a load of competing factors and that judgement is most likely to be found in highly experienced developers who have worked their way up building a range of systems over a long period of time - in other words exactly the people they won't have any of left in a few years because they broke the talent pipeline at every level in their haste to replace skill and knowledge with AI. It's going to be hilarious watching the excuses flow. Though naturally most of the executives involved will fail upwards anyway so that part will probably be a bit annoying to watch.
We already went through this with outsourcing. It's basically the same action and the same result but outsourcing to an LLM instead of an underpaid Indian junior dev. It all feels like dejavu.
It's not quite the same because at least some underpaid Indian junior devs actually learn from their experience and then become better devs. The only way the same is going to happen with LLMs is when they decide they are going to train on your source code and prompts now. I'll save the second bag of popcorn for when the lawyers find out that their organisation is now full of shadow IT as staff upload sensitive information and company IP to organisations that explicitly said they were going to share that information with the entire world.
You think they aren't already training on our source code and prompts? I feel like that's how these LLM for coding tools got crazy good in this last year.
Some of them openly are - they changed their terms to explicitly allow for this a while back. But I'd be very surprised if any of the enterprise-level packages was doing it when they publicly guarantee not to. The liability if they were caught would be off the charts.
It seems increasingly evident to me that public services like github are going to be unusable and unreliable
I don't think that's a given at all. I think it's clear that the world needs more compute, for AI and everything else, and we just can't keep up on the hardware side right now. But no problem lasts forever if there is money to be made in solving that problem, and that's pretty clearly the case here. It will take some time to ramp up as it's complicated, but I think it will be addressed.
And also I agree with the other thread. No matter what you do, software always has points of failure. Insourcing doesn't save you from shit hitting the fan, it just changes how you deal with it. Unless github and the like totally degrade I don't see any reason to move that shitsplosion in house. I'd rather them stress over it than me.
I seriously doubt that inhouse cures those problems. We are hosting a lot of stuff on our own, but that still breaks infrequently and then we have to scramble the people with the domain knowledge together to try to figure out what went south and how to fix it.
I think the big difference is: if someone elses system (you rely on) breaks, you sit there cursing at them. If your own system breaks you don't have time to curse; you are in panic to get it back online. And if you have multiple departments, then the other departments will be the ones who sit there cursing your IT.
Cures? Absolutely not. But it provides a layer of insulation against what is inevitably going to be a continual degradation of publicly available services that are swarmed with low quality and high volume usage.
Unless the software you use is an inhouse solution, you are still at the mercy of your supplier. If new versions become worse and worse, you still have a problem you can't fix on your own. Staying at old versions is typically nothing you can do too long either, since the security issues pile up fast.
Not a fantastic point at all. The argument is that because you rely on any software that’s not built within your org you are taking on risk. That’s not strategic, that’s idiocy. How far down does that logic go? Are you writing your network drivers too because you don’t want to rely on a third party? Running your own source control and build pipelines has been done for decades and is easier than ever now with Ansible, etc.
I think it's a valid point because it highlights the multiple points of failure. If the codebase behind a product is solid, but it's unable to keep up with the realities of being hammerred by automation and public usage, then an internal deployment of that product would be fine, but if the issue comes with the software degrading as opposed to the software being unable to keep up with usage, then it's a valid point.
Source control is simple, but when it comes to other cloud services that are more complex than git, but still have a service degradation from public usage, there's a very real concern of the code itself or updates to the code being a failure point. It just so happens to be that source control and CI/CD is comparatively simple and option-laden.
I think another aspect of it is that people (and orgs) overestimate their abilities. They can look at outages by a third party and say “we can do this more reliably”. Someone will make the decision and brag about how much of an improvement they made.
When they can’t that can justify “why” their own outages occurred.
At the end of the day orgs will move stuff on prem and call it a win. In a few years, someone new will move into the position, move it back, and call it a win again.
It is the same song and dance we have been doing and will continue to do.
Especially with coding agents that risk has significantly increased, since it looks a lot simpler to develop something inhouse now. To a certain degree that may also be right and could work out. But as you said, many problems look a lot simpler on the surface than they turn out to be once you get more familiar with them.
The only time we’ve ever had an outage with our gitlab cluster was when we had to forcibly take it down to apply security updates for a 0 day.
Occasionally we have run into issues with runners but that is only because we use the AWS spot instances and occasionally the bid/demand would spike. Now we have a blend of spot and reserved instances for runners.
I've been having these exact thoughts recently. At some point the pain of using GitHub is going to far exceed the value it provides to us. Although GitHub Enterprise is still a thing, so GitHub will profit either way...
"An update on" was used so often at another tech company to mean "we're killing this thing" that it became a meme. Thanks for reminding me, GitHub blog post about how availability is getting worse.
We all know it's because GitHub has a management and priorities problem. They're no longer a platform for hosting gir repos. Their priority is AI and using code for training.
Yes and they don’t have a CEO. The C-suite is running around on Twitter placating tech influencers.
Their leadership team should be completely replaced, if they were competent they would have made a CEO happen and taken adult decisions to stop the impact of their existing customers over the many years these issues have been happening for.
The charts are there to gaslight yo, their problems started before the AI boom. It’s just a convenient disaster they are choosing to opt into so garner sympathy.
Those charts might be incredibly misleading if they've fucked with the scale, but, given the absolute values they give (90M PRs merged, 1.4B commits, 20M new repos/month), it's pretty reasonable to assume linear scale starting at zero.
it's pretty reasonable to assume linear scale starting at zero.
If so, those graphs make it appear that even though Github was started in 2008, there was almost no traffic at all until 2022, when these graphs start.
Is this really true?
I found one graph with actual numbers: https://pslmodels.github.io/Git-Tutorial/content/background/GitHubHistory.html - while it's measuring something different, it does not tell the same story at all.
Indeed, I would take 100% the reverse "reasonable assumption". When I see a graph with no axes and no scales, I think it's "reasonable to assume" that the person creating these doesn't give a flying fuck about axes, accuracy, or being able to read data off the graphs.
assuming it's a linear scale and the bottom of the graph is 0. I'm pretty sure of the former and I'd like to assume the latter, but without a scale, I can't actually know
how the heck does the cto of github have a nearly completely empty github account?
two badges -- not even any of the employee ones and a ZERO on the contribution graph (which can show work in private repos).
How does one lead the technical direction of a platform and know how to fix it's issues when they're obviously not even a user?
Additionally, the made-for-the-job github username which should be an alarm - if they're a cto that understands technical work why aren't they demonstrating that.
People do keep their personal and professional lives separate. You know. I always create a forward work and per project accounts, but only for projects which have accrued a significant history one.
One never knows when when a employer may decide to come after you or any goddamn reason.
So I'm sorry to say you're a red flag is kind of sort of garbage
You can't advertise and encourage AI and your agents everywhere in your products and then go 'whoops we actually can't handle AI doing that much stuff'
I was wondering why the issues page was just giving me an error the other day! Interesting that it's their search-related backend, maybe that's a bit of a downside of treating the issues, pull requests, etc as a search even when we're just trying to list all of them.
Do they acknowledge at all that the sudden growth in pull requests being merged that drove the need to suddenly 30x the capacity was 100% driven by AI/LLM code generation that Microsoft has been hyper aggressive in pushing, and it is a 100% self-imposed problem that arises from having reorganized GitHub from sort sort of pure developer services role to being a component of their AI department?
Or are they just sorta implying the faceplant happened by magic, and that the spike is resource consumption is some sort of organic phenomenon because the platform is just so gosh darned popular with actual human developers? Because if the AI they are pushing on everybody was so smart, maybe it could have told them they need to be able to scale resource availability in coordination with pushing out the stuff that demands it, rather than shoving it down everybody's throats and then doing a Surprised Pikachu to try to scale availability in a 100% reactive way that was obviously going to have more risk.
How does this comport with Cursor and Claude Code, tools built by entirely different companies, being the tools of choice by developers building significantly more code and pushing to a public repo? If GitHub had invested literally nothing in AI tools they could face this exact same problem.
I’m defending it because it’s really not a crazy sentence.
“I’m trying to minimize X by minimizing Y” is just such a common phraseology when you’re talking about min/maxing anything lol.
Maybe you’re thrown off because the phrase “blast radius” was used? But that’s also very common in this industry … so I think maybe you’ve got a bit too much tinfoil on your hat here.
Seriously, though, if you start interacting with generative AI across the different models, you'll start picking up on their patterns of writing. What's going to be annoying, though, is when humans start unironically and unintentionally mimicking their patterns of writing as result of our tendency to code switch
"At high scale, small inefficiencies compound: queues deepen, cache misses become database load, indexes fall behind, retries amplify traffic, and one slow dependency can affect several product experiences."
Maybe not the whole article but it's one "that's a testament to" from being no doubt AI.
This is the end of public reliability. Everyone should move to their own self hosted enterprises. The whole "but we guarantee xyz compliance" was a differentiator .. 3 months ago. The internet was ever designed to handle this much traffic, even before AI, regular crawler bots were such a strain.
Before reading this, I will tell you that 2 PRs just completely disappeared from the PR tab on my repo today. And I am waiting minutes to reload a page.
I've moved off Github and onto Gitlabs hosted on my own VPS. I don't want to deal with a bunch of vibecoded bullshit sitting around forever slowly eroding the platform I'm dependent on.
lol, well I guess you bring up a good point. I've no idea. I can at least control the environment a lot better and don't have to worry about some AI bullshit scanning through all my projects.
I wonder how long it will be until GitHub (and maybe other large SaaS companies) start charging for API usage. AI agents crawling through GitHub commits and PRs looks to be contributing to a massive growth in API usage
Most of this seems fine but these two statements caught my eye:
We also leveraged our migration to Azure to stand up a lot more compute.
And
While we were already in progress of migrating out of our smaller custom data centers into public cloud, we started working on path to multi cloud.
Do these not seem like two significant steps backwards? I know GitHub is owned by Microsoft so they would have some impetus to use Azure resources, but Azure is barely hanging on at this point. Why would they not just improve the data center they already have?
I read it as "no Git operations, and no Git APIs we're affected". Pull requests aren't actually a part of Git, they're an abstraction built by GitHub. The PRs themselves still exist, it's just the ability to search for them that is broken
tobakist@reddit
I congratulated my team today since we’ve effortlessly managed a higher availability than GitHub! Well done, us!
urbrainonnuggs@reddit
It's never been easier to self host git based source control and pipeline+runners. So many solutions
LGXerxes@reddit
Discussing with others at my parent company, other services like coderabbit, aikido, some others only work on github, sometimes gitlab/bitbucket.
So moving to something is possible but often unsure about the time invested.
Especially runners etc
urbrainonnuggs@reddit
Avoid bitbucket at all cost
GitLabs is my recommendation for a corporate solution with all the bells and whistles, minimal manual setup.
But for small business Forgejo + Woodpecker CI is my current go to. I had it up and running on my homelab (k8s) with dynamic scaling runners in like 1 hour.
-Jersh@reddit
I miss our self-hosted GitLab setup. 😢Best software I’ve ever worked with
slaymaker1907@reddit
In terms of reliability, self-hosted GitLab can easily be worse than cloud platforms for large enterprises.
MysteriousEmployee54@reddit
Why would you suggest avoiding Bitbucket?
urbrainonnuggs@reddit
It's just.. bad. I can't explain it other than its like unseasoned dry chicken..
SiegeAe@reddit
For me its just so much extra work at all points, from setup to maintenance to every new pipeline you want to add, at least compared to gitlab which was dead simple across the board in comparison
wrosecrans@reddit
And let's be honest, while some projects absolutely need super complicated CI systems with super flexible container based pipelines, the majority of repos in GitHub would be just as well served by a Makefile.
The demand for that stuff is partly driven by "because it's there" and it has become the path of least resistance for some basic testing and deploying. There are many, many, many projects I've seen where the CI pipeline spends orders of magnitude more time and resources shuffling data and queuing and spawning and reporting, than actually running the two or three small test cases for a Python script that doesb;r even need a build process.
Then Github goes WE MUST OMEGA LEVEL HYPER SCALE and expand to meet the needs of the expanding bureaucracy that most people never actually needed. If you self-host your CI, you can make it work however is convenient, and you don't necessarily need to shape everything into something as flexible as the public runner models. Often times literally a few hundred milliseconds on a server accomplished more than five minutes of pipelines on a public service.
urbrainonnuggs@reddit
I totally agree with you. I personally hate all the pipeline tools. I unfortunately made my career out of DevOps and Platform Engineering so I've had to evaluate every tool ever.
My personal apps are a huge mono repo and a single orchestrator script generates (templates not LLM) build/test/publish/deploy methods at runtime for each project.
QCKS1@reddit
Windows runners are still a pain. 90% of OSS CI/CD pipelines are linux only because they use containers for the sandboxing. Hence I have a tiny Jenkins instance running for my projects I want to support Windows.
On the other hand I'm considering just dropping Windows support
urbrainonnuggs@reddit
Do it. Drop windows. I changed careers to escape Windows and Jenkins lol. I'm so happy
phillipcarter2@reddit
And perhaps ironically, a little bit of elbow grease + some agents running can get your team a plenty good enough code review tool that can integrate with your source control. Obviously harder to scale for a massive organization, but for most shops they could likely build and operate most of what they need.
the_ai_wizard@reddit
care to make a blog post or something?
jug6ernaut@reddit
what is the recommended stack these days? Last I checked (and its been years) there werent any really good open-source options for pipeline/runners.
ImNotABotScoutsHonor@reddit
Forgejo across the board, pretty much.
Forgejo for git hosting, PRs, issues, and basic project hosting, Forgejo Actions for CI/CD definitions, and Forgejo Runner for executing jobs.
There's also ole reliable Jenkins, I guess. But I think Jenkins has too much maintenance burden, personally.
urbrainonnuggs@reddit
From someone who built and maintained Jenkins servers for a decade, I can't recommend it anymore.
Forgejo is the best option for small businesses but GitLabs self host is still pretty cool too.
For runners Codeburg and others are using Woodpecker CI which is a great project
urbrainonnuggs@reddit
https://www.reddit.com/r/programming/s/03rSDnuvZn
boiledbarnacle@reddit
Jupp. After the bare repos are created (easily done with a single ssh command) you can just git push with ssh.
lordjmann@reddit
Yawn
campbellm@reddit
"Our uptime might have a '9' in it."
bring_back_the_v10s@reddit
I didn't read it. Is it a rage coping article?
Ilirian@reddit
It was triple 9 when uptime was 89.99%
jpfed@reddit
Cmon I gotta have at least one 9
water_bottle_goggles@reddit
09.0
editor_of_the_beast@reddit
What a totally empty post.
Wow, thanks for the overwhelming detail here.
ComplianceAuditor@reddit
I mean, it starts out by saying that it’s unacceptable and then they go on to explain all of the reasons why you should accept it as opposed to making some kind of change or taking action in response, which would be what it looks like to not accept it.
mikeblas@reddit
They don't owe you anything.
ric2b@reddit
No one said they did.
mikeblas@reddit
Tune in next week when ric2b learns about "implication".
ric2b@reddit
All I see is a criticism of the post, which is perfectly fair.
mikeblas@reddit
The criticism is about detail, and the company is under no obligation to report any detail.
Maybe they should, but they don't owe that to anybody. And there are lots of reasons for them not to do so.
ric2b@reddit
No one said they were.
I can say someone drew a shit painting without it being an implication that they owe me a good painting.
mikeblas@reddit
Sure. But that's not what editor_of_the_beast said. Read their post in context, and in its entirety.
I'm not sure whether you are simply being a jerk for the sake of it, or really just that obstinate and obtuse.
editor_of_the_beast@reddit
Then they don’t get my money
watabby@reddit
corporate speak, I hate it
oridb@reddit
Probably an LLM.
Sigmatics@reddit
My main takeaway: AI agents are causing a rapid increase in traffic, which they are struggling to handle
IanisVasilev@reddit
Microsoft is promoting the use of agents that are overloading their own infrastructure.
Truly marvelous times.
Sigmatics@reddit
Pretty much this.
ouchmythumbs@reddit
The dog that caught the car.
mikeblas@reddit
The PM that caught the cloud.
zergotron9000@reddit
Sounds like bullshit frankly. I think this is an excuse and a marketing post in one
Sigmatics@reddit
Not really, doesn't sound unrealistic to me given what I've seen on my own open source project. The barrier to entry was lowered by orders of magnitude, I see a lot more spot contributors and a lot more slop PRs.
For GitHub it doesn't matter if the traffic is useful, all traffic must be handled, so they are the ones suffering here (albeit contributing to the mess themselves via Copilot)
anarchist1312161@reddit
Someone is responsible. Somebody is always responsible.
MooseBoys@reddit
Almost as useless as typical patch notes: "bug fixes and performance improvements".
pragmojo@reddit
"Fixed a rare bug which caused some users nudes to be posted to their work slack"
Scream_Tech7661@reddit
Literally the sentence before your quote is:
This post is intended to communicate a high level overview of what they’ve seen fail and how they are addressing those failures.
It not intended to be a full post-mortem, nor would I want it to be. I just want to know what they’ve learned from their failures and how they are architecting a solution. That’s exactly what this post does at a high level.
The details you think are missing are in their incident root cause analysis, which is exactly what they stated before your quote.
editor_of_the_beast@reddit
Found the GitHub engineer
Scream_Tech7661@reddit
lol nope. I’m an SRE on a team with six others. We “self host” GitLab in AWS, and GitHub functionality pales in comparison to the CI/CD and organizational management of GitLab.
Our team of seven supports infrastructure across roughly 3-4 dozen AWS accounts costing us tens of millions of dollars a month. And we support hundreds of developers and engineers running CI/CD workflows 24/7 across six continents.
I self host Forgejo, a Gitea fork, in my homelab for most of my own repos.
I also have about 30 repos on GitHub for various projects.
I just have a thing for identifying and calling out bad faith actors ;)
editor_of_the_beast@reddit
No one cares what you do? What does that have to do with the lack of information in the post?
phillipcarter2@reddit
If you could read, you'd read that the post describes how there's more detail in incident reports.
editor_of_the_beast@reddit
Why are you defending them? The incident reports also suck. They suck at making software.
phillipcarter2@reddit
I'm pointing out that you make bad posts, not defending GitHub.
editor_of_the_beast@reddit
Except everyone agreed with me, because this post is devoid of any information to the point that it’s insulting. There’s not even a hyperlink to these allegedly more detailed post mortems, and even if those were good, they could still provide any amount of color in this post.
Instead of saying “we had a bad process and now it’s fixed, don’t worry.” There’s no circumstance, ever, where I’m going to read that and not be annoyed.
phillipcarter2@reddit
Not everyone is agreeing with you, and again, you are just making bad posts.
editor_of_the_beast@reddit
Why does the initial comment have so many upvotes then?
phillipcarter2@reddit
Why do other comments that disagree with you have upvotes?
editor_of_the_beast@reddit
Are you aware of what numbers are? Why does the top post have over 100 upvotes, but these later comments about defending GitHub all have under 10?
phillipcarter2@reddit
Are you aware of how to read?
editor_of_the_beast@reddit
Yes. I read the numbers that say the upvotes I got far outweigh the upvotes for dissenting posts. So it truly feels that I know how to read better than you.
phillipcarter2@reddit
Yes, a lot of people in the subreddit that is notorious for being negatively polarized to LLMs are, indeed, negatively polarized towards anything related to the topic, such as a post by a CTO saying they're getting hammered by an accelerated amount of LLM-generated code.
That you cannot read the post in question, which clearly describes that details of issues are in incident reports, when posting how angry and offended you are that this post doesn't contain those details, is a skill issue of your own right.
That there are others here who causally upvote anything negatively polarized towards LLMs simply because of the polarization does not take away from the fact that you, specifically, are unable to read.
editor_of_the_beast@reddit
But I read the post and was unsatisfied with its content. I think I understood it at least. Could you maybe summarize the parts that you thought were good so I could understand? Maybe I actually misread it
mikeblas@reddit
The list of people who agreed with you is very much shorter than you think.
Scream_Tech7661@reddit
Just correcting misinformation. You stated what you assumed I do. I responded with what I actually do. Seems like you seem to care a lot?
emdeka87@reddit
AI slop
KeytarVillain@reddit
They must have gotten Nintendo to write the patch notes
Augunrik@reddit
It baffles me how such a distributed system like git has been so underwhelmingly designed that it doesn’t scale (in GitHub). I blame system design.
xSaviorself@reddit
Github has just really taken a quality nosedive since the MS acquisition. Seems like there is a lot of pressure to deliver and things are being accelerated despite not being ready.
Who would have thought?
foramperandi@reddit
"Just"? Microsoft bought GitHub eight years ago. You've got a weird sense of time.
xSaviorself@reddit
2018 was yesterday and you won't tell me otherwise.
yon_@reddit
Microslop lying about things and also making them worse, well I never. But for real, their track record of doing *anything* good for the software industry has been eroding over the last decade or two
dom_ding_dong@reddit
You must be new here.
araujoms@reddit
Yep: https://damrnelson.github.io/github-historical-uptime/
light24bulbs@reddit
As predicted. MS is such ass, it's wild
b34gl4@reddit
The push to AI all the things at Microsoft has certainly reduced the quality of windows updates, not surprising if its affecting other products.
brokenja@reddit
Management Takeaway: more developer resources for Copilot!
mrfixij@reddit
It seems increasingly evident to me that public services like github are going to be unusable and unreliable, and that on an enterprise level, the path forward is with tightly controlled inhouse or onprem instances. Something tells me that ops/devops is going to be eating good as public services continue to degrade.
want_to_want@reddit
I wouldn't make such a blanket statement. There's no fundamental reason github couldn't have as many nines as gmail. It's just a skill issue.
3MU6quo0pC7du5YPBGBI@reddit
And so the pendulum swings...
mrfixij@reddit
Everything old is new again. "spec driven development" is just waterfall all over again and will run into the same issues.
Silhouette@reddit
There is a certain irony that LLMs and in particular agentic coding workflows have highlighted a lot of the problems that have always been there with "Agile" development but it wasn't popular for developers to point them out. It turns out that moving faster but in semi-random and constantly changing directions because your product and leadership people are incapable of ever making a decision or committing to anything that takes more than five minutes doesn't really build actual business value faster. Who knew?
And now the high costs of those agentic workflows - both financial and otherwise - are becoming clearer and they're highlighting the problems of trying to define everything in great detail up front and not emphasising an adaptable coding style and software design that is easy to maintain and extend as requirements change or new requirements are added. If only there had been anyone around with the knowledge and experience to warn about this problem!
I'm buying popcorn for the moment in a year or two's time when business leaders finally realise that a lot of getting good results comes down to having good judgement about how to balance a load of competing factors and that judgement is most likely to be found in highly experienced developers who have worked their way up building a range of systems over a long period of time - in other words exactly the people they won't have any of left in a few years because they broke the talent pipeline at every level in their haste to replace skill and knowledge with AI. It's going to be hilarious watching the excuses flow. Though naturally most of the executives involved will fail upwards anyway so that part will probably be a bit annoying to watch.
FionaSarah@reddit
We already went through this with outsourcing. It's basically the same action and the same result but outsourcing to an LLM instead of an underpaid Indian junior dev. It all feels like dejavu.
Silhouette@reddit
It's not quite the same because at least some underpaid Indian junior devs actually learn from their experience and then become better devs. The only way the same is going to happen with LLMs is when they decide they are going to train on your source code and prompts now. I'll save the second bag of popcorn for when the lawyers find out that their organisation is now full of shadow IT as staff upload sensitive information and company IP to organisations that explicitly said they were going to share that information with the entire world.
canihavethatfire@reddit
You think they aren't already training on our source code and prompts? I feel like that's how these LLM for coding tools got crazy good in this last year.
Shadow IT cat is out of the bag
Silhouette@reddit
Some of them openly are - they changed their terms to explicitly allow for this a while back. But I'd be very surprised if any of the enterprise-level packages was doing it when they publicly guarantee not to. The liability if they were caught would be off the charts.
MiniGiantSpaceHams@reddit
I don't think that's a given at all. I think it's clear that the world needs more compute, for AI and everything else, and we just can't keep up on the hardware side right now. But no problem lasts forever if there is money to be made in solving that problem, and that's pretty clearly the case here. It will take some time to ramp up as it's complicated, but I think it will be addressed.
And also I agree with the other thread. No matter what you do, software always has points of failure. Insourcing doesn't save you from shit hitting the fan, it just changes how you deal with it. Unless github and the like totally degrade I don't see any reason to move that shitsplosion in house. I'd rather them stress over it than me.
aksdb@reddit
I seriously doubt that inhouse cures those problems. We are hosting a lot of stuff on our own, but that still breaks infrequently and then we have to scramble the people with the domain knowledge together to try to figure out what went south and how to fix it.
I think the big difference is: if someone elses system (you rely on) breaks, you sit there cursing at them. If your own system breaks you don't have time to curse; you are in panic to get it back online. And if you have multiple departments, then the other departments will be the ones who sit there cursing your IT.
mrfixij@reddit
Cures? Absolutely not. But it provides a layer of insulation against what is inevitably going to be a continual degradation of publicly available services that are swarmed with low quality and high volume usage.
aksdb@reddit
Unless the software you use is an inhouse solution, you are still at the mercy of your supplier. If new versions become worse and worse, you still have a problem you can't fix on your own. Staying at old versions is typically nothing you can do too long either, since the security issues pile up fast.
mrfixij@reddit
Fantastic point. This is why I don't work at the strategic level.
IM_A_MUFFIN@reddit
Not a fantastic point at all. The argument is that because you rely on any software that’s not built within your org you are taking on risk. That’s not strategic, that’s idiocy. How far down does that logic go? Are you writing your network drivers too because you don’t want to rely on a third party? Running your own source control and build pipelines has been done for decades and is easier than ever now with Ansible, etc.
mrfixij@reddit
I think it's a valid point because it highlights the multiple points of failure. If the codebase behind a product is solid, but it's unable to keep up with the realities of being hammerred by automation and public usage, then an internal deployment of that product would be fine, but if the issue comes with the software degrading as opposed to the software being unable to keep up with usage, then it's a valid point.
Source control is simple, but when it comes to other cloud services that are more complex than git, but still have a service degradation from public usage, there's a very real concern of the code itself or updates to the code being a failure point. It just so happens to be that source control and CI/CD is comparatively simple and option-laden.
laStrangiato@reddit
I think another aspect of it is that people (and orgs) overestimate their abilities. They can look at outages by a third party and say “we can do this more reliably”. Someone will make the decision and brag about how much of an improvement they made.
When they can’t that can justify “why” their own outages occurred.
At the end of the day orgs will move stuff on prem and call it a win. In a few years, someone new will move into the position, move it back, and call it a win again.
It is the same song and dance we have been doing and will continue to do.
aksdb@reddit
Especially with coding agents that risk has significantly increased, since it looks a lot simpler to develop something inhouse now. To a certain degree that may also be right and could work out. But as you said, many problems look a lot simpler on the surface than they turn out to be once you get more familiar with them.
A1oso@reddit
Exactly. When my company was still self-hosting a Bitbucket instance, we also had a few outages per year.
SpaceToaster@reddit
The only time we’ve ever had an outage with our gitlab cluster was when we had to forcibly take it down to apply security updates for a 0 day.
Occasionally we have run into issues with runners but that is only because we use the AWS spot instances and occasionally the bid/demand would spike. Now we have a blend of spot and reserved instances for runners.
exodusTay@reddit
tailscale shares going up
pimp-bangin@reddit
I've been having these exact thoughts recently. At some point the pain of using GitHub is going to far exceed the value it provides to us. Although GitHub Enterprise is still a thing, so GitHub will profit either way...
n-space@reddit
"An update on" was used so often at another tech company to mean "we're killing this thing" that it became a meme. Thanks for reminding me, GitHub blog post about how availability is getting worse.
Dirty_Socrates@reddit
So when are they getting rid of the free tier to lower bot usage?
shgysk8zer0@reddit
We all know it's because GitHub has a management and priorities problem. They're no longer a platform for hosting gir repos. Their priority is AI and using code for training.
kitsunde@reddit
Yes and they don’t have a CEO. The C-suite is running around on Twitter placating tech influencers.
Their leadership team should be completely replaced, if they were competent they would have made a CEO happen and taken adult decisions to stop the impact of their existing customers over the many years these issues have been happening for.
foramperandi@reddit
Funny, GitHub also didn't have a CEO for two years before MS bought them either, but apparently those were the halcyon days.
Lulzagna@reddit
They fucking post this while their search is still broken
Sigmatics@reddit
I'll be the first to let you know the CTO doesn't work on this problem and never will
Lulzagna@reddit
That doesn't make it not tone-deaf
R2_SWE2@reddit
Wow those charts
kitsunde@reddit
The charts are there to gaslight yo, their problems started before the AI boom. It’s just a convenient disaster they are choosing to opt into so garner sympathy.
SupaSlide@reddit
I didn’t do exact math but I’m highly skeptical that they were having effectively zero PRs merge before 2023.
occio@reddit
Here are some others https://mrshu.github.io/github-statuses/#about
stuross@reddit
Are incredibly misleading without a y-axis
pdpi@reddit
Those charts might be incredibly misleading if they've fucked with the scale, but, given the absolute values they give (90M PRs merged, 1.4B commits, 20M new repos/month), it's pretty reasonable to assume linear scale starting at zero.
dodeca_negative@reddit
You think those numbers were 0 three years ago? Really?
pdpi@reddit
Those numbers don't start at zero, and there's other posts of theirs you can refer to for reference.
E.g. their October 2025 report quotes 43M monthly PRs, versus 90M on this new report, and the chart lines up relatively well with a 2.3x increase.
dodeca_negative@reddit
So when you said “ it’s pretty reasonable to assume linear scale starting at 0” you meant…?
inigohr@reddit
That the axis starts at zero. The lines do not touch the x axis which does not mean "those numbers were 0".
crunchmuncher@reddit
I understood it as the y-axis starts at 0 (the line, however, does not start at the lowest end of the y-axis).
djsz@reddit
They're saying the y-axis starts at 0.
The 2023 numbers are not touching the bottom of the y-axis, so the numbers were not 0 three years ago.
HommeMusical@reddit
I just measured the ratio between the lowest and highest points on those three graphs on the screen.
The ratio is very roughly 100 to 1.
Norci@reddit
How is it reasonable to assume, given the scale only dates 3 years ago while GitHub been out for more than 15.
Ddog78@reddit
I think the parent commenter is saying that it represents the d/dx ie. the rate of increase - assuming the chart itself is accurate, ofc.
HommeMusical@reddit
If so, those graphs make it appear that even though Github was started in 2008, there was almost no traffic at all until 2022, when these graphs start.
Is this really true?
I found one graph with actual numbers: https://pslmodels.github.io/Git-Tutorial/content/background/GitHubHistory.html - while it's measuring something different, it does not tell the same story at all.
Indeed, I would take 100% the reverse "reasonable assumption". When I see a graph with no axes and no scales, I think it's "reasonable to assume" that the person creating these doesn't give a flying fuck about axes, accuracy, or being able to read data off the graphs.
kintar1900@reddit
It would have been before we invented "marketing".
JustOneAvailableName@reddit
I think they actually start at 0 (and end up with the highlighted number), but right now it’s a bit of a useless chart.
McHoff@reddit
You think they were getting almost 0 pull requests and 0 committs in early 2023?
JustOneAvailableName@reddit
Yes, I think they had ~2M monthly PRs and ~30M monthly commits in early 2023.
HommeMusical@reddit
Why would any skeptical person give any credence to a graph with no axes?
Just baffled.
nicholashairs@reddit
I mean if you know the total value you can work backwards from there
frymaster@reddit
assuming it's a linear scale and the bottom of the graph is 0. I'm pretty sure of the former and I'd like to assume the latter, but without a scale, I can't actually know
nicholashairs@reddit
Oh yeah good point, forgot about the bottom part which may or may not be zero 🤦♂️🤦♂️🤦♂️🤦♂️🤦♂️
Sykaro@reddit
made by a 10x engineer, maybe 100x even
CherryLongjump1989@reddit
Those are not charts, they are pictures. No meaningful data can be obtained from them.
IanisVasilev@reddit
"Sky is the limit" doesn't sound so grandiose for the cloud.
gene_wood@reddit
Here's a data visualization for context : https://damrnelson.github.io/github-historical-uptime/
little_cat8992@reddit
how the heck does the cto of github have a nearly completely empty github account?
two badges -- not even any of the employee ones and a ZERO on the contribution graph (which can show work in private repos).
How does one lead the technical direction of a platform and know how to fix it's issues when they're obviously not even a user?
Additionally, the made-for-the-job github username which should be an alarm - if they're a cto that understands technical work why aren't they demonstrating that.
I smell a rat
dom_ding_dong@reddit
People do keep their personal and professional lives separate. You know. I always create a forward work and per project accounts, but only for projects which have accrued a significant history one.
One never knows when when a employer may decide to come after you or any goddamn reason.
So I'm sorry to say you're a red flag is kind of sort of garbage
little_cat8992@reddit
you're not the cto of a coding platform that is designed around social coding.
also as a former github employee myself, this is a red flag.
PerkyPangolin@reddit
Same as the new Xbox head who faked a gamertag with month worth of playtime.
dom_ding_dong@reddit
Assuming he gamed perhaps because he didn't want to share his personal tag or be harassed.
It's not like the gaming community has not had its issues
DDFoster96@reddit
Irony is it broke again today (having broken yesterday as well). None of my Actions runs actually started.
needmoresynths@reddit
You can't advertise and encourage AI and your agents everywhere in your products and then go 'whoops we actually can't handle AI doing that much stuff'
thegreatpotatogod@reddit
I was wondering why the issues page was just giving me an error the other day! Interesting that it's their search-related backend, maybe that's a bit of a downside of treating the issues, pull requests, etc as a search even when we're just trying to list all of them.
wrosecrans@reddit
Do they acknowledge at all that the sudden growth in pull requests being merged that drove the need to suddenly 30x the capacity was 100% driven by AI/LLM code generation that Microsoft has been hyper aggressive in pushing, and it is a 100% self-imposed problem that arises from having reorganized GitHub from sort sort of pure developer services role to being a component of their AI department?
Or are they just sorta implying the faceplant happened by magic, and that the spike is resource consumption is some sort of organic phenomenon because the platform is just so gosh darned popular with actual human developers? Because if the AI they are pushing on everybody was so smart, maybe it could have told them they need to be able to scale resource availability in coordination with pushing out the stuff that demands it, rather than shoving it down everybody's throats and then doing a Surprised Pikachu to try to scale availability in a 100% reactive way that was obviously going to have more risk.
phillipcarter2@reddit
How does this comport with Cursor and Claude Code, tools built by entirely different companies, being the tools of choice by developers building significantly more code and pushing to a public repo? If GitHub had invested literally nothing in AI tools they could face this exact same problem.
wrosecrans@reddit
Ban bot accounts from spamming pull requests.
Shoox@reddit
Yes, please, give me more arguments so I can finally convince everyone at my job that we should self-host forgejo and tekton.
miramboseko@reddit
This fool used an LLM to write this smh
thomas_m_k@reddit
I didn't notice anything. What stood out to you?
miramboseko@reddit
The lack of any real content combined with the LLMisms and general tone.
CookingAppleBear@reddit
"minimizing the blast radius by minimizing single points of failure", for me
GrayLiterature@reddit
That’s not slop at all, that’s just a sentence.
How would you write it?
JennySlopez@reddit
Why are you defending this? Are you the author?
GrayLiterature@reddit
I’m defending it because it’s really not a crazy sentence.
“I’m trying to minimize X by minimizing Y” is just such a common phraseology when you’re talking about min/maxing anything lol.
Maybe you’re thrown off because the phrase “blast radius” was used? But that’s also very common in this industry … so I think maybe you’ve got a bit too much tinfoil on your hat here.
grauenwolf@reddit
Answer the question. How would you write it?
If I wrote that in my own words, it would be "minimizing the blast radius by reducing the number of single points of failure".
Malnilion@reddit
There it is, that's the smoking gun!
Seriously, though, if you start interacting with generative AI across the different models, you'll start picking up on their patterns of writing. What's going to be annoying, though, is when humans start unironically and unintentionally mimicking their patterns of writing as result of our tendency to code switch
grauenwolf@reddit
Yes and no. Phrases like "blast radius" and "single point of failure" have been used for decades to describe outages.
karmiccloud@reddit
That is unfortunately already happening.
MettaWorldWarTwo@reddit
"At high scale, small inefficiencies compound: queues deepen, cache misses become database load, indexes fall behind, retries amplify traffic, and one slow dependency can affect several product experiences."
Maybe not the whole article but it's one "that's a testament to" from being no doubt AI.
grauenwolf@reddit
That sounds exactly like something I would have written.
What you are failing to understand is that AI is trained on how real people write.
IanisVasilev@reddit
There is no single line of text that can be shared on the internet without somebody screaming "AI!" in the comments.
awj@reddit
Isn’t the future awesome? So glad we’re burning mountains of cash for this. /s
scottsman88@reddit
Sounds like something an AI would say :p
GrayLiterature@reddit
Man lol that paragraph is not AI slop.
Tight-Requirement-15@reddit
This is the end of public reliability. Everyone should move to their own self hosted enterprises. The whole "but we guarantee xyz compliance" was a differentiator .. 3 months ago. The internet was ever designed to handle this much traffic, even before AI, regular crawler bots were such a strain.
PerkyPangolin@reddit
I'm sure it's totally unrelated to Azure issues described in these series of posts:
D3PyroGS@reddit
Microsoft, as dysfunctional as ever
demchaav@reddit
Corporate speak for "we're not sure what went wrong but trust us". Every outage post-mortem sounds the same these days.
scorcher24@reddit
For a moment I thought they implemented IPv6. Soon.
boiledbarnacle@reddit
Oh noooo!
~~Git~~Hub has to get more compute to handle AI generated commits that ate said compute.
no_brains101@reddit
Before reading this, I will tell you that 2 PRs just completely disappeared from the PR tab on my repo today. And I am waiting minutes to reload a page.
krileon@reddit
I've moved off Github and onto Gitlabs hosted on my own VPS. I don't want to deal with a bunch of vibecoded bullshit sitting around forever slowly eroding the platform I'm dependent on.
PerkyPangolin@reddit
Any indication GitLab is not vibecoded at this point?
krileon@reddit
lol, well I guess you bring up a good point. I've no idea. I can at least control the environment a lot better and don't have to worry about some AI bullshit scanning through all my projects.
twerktle@reddit
I wonder how long it will be until GitHub (and maybe other large SaaS companies) start charging for API usage. AI agents crawling through GitHub commits and PRs looks to be contributing to a massive growth in API usage
PerkyPangolin@reddit
Next: charging per clone/pull/push.
flanger001@reddit
Most of this seems fine but these two statements caught my eye:
And
Do these not seem like two significant steps backwards? I know GitHub is owned by Microsoft so they would have some impetus to use Azure resources, but Azure is barely hanging on at this point. Why would they not just improve the data center they already have?
CyberneticWerewolf@reddit
I haven't read the article, but I assume it's the same deal as seeing an official Google blog post with a title like "An update on Google Reader"?
cesarbiods@reddit
And the culprit here is, wait for it, AI slop. So much AI slop that their servers and infra can’t keep up. But go ahead keep pushing for more AI slop.
travelinzac@reddit
Dear Vlad, your platform is in the toilet. Please do better.
Stinkman982@reddit
Oh their system is having trouble with all the requests? Maybe they should strip all that AI search bullshit off the homepage then
stipo42@reddit
Bullshit yesterday's incident didn't affect APIs, the pull request API is still saying my Connors have no PRs when they clearly do on the UI
coldblade2000@reddit
I read it as "no Git operations, and no Git APIs we're affected". Pull requests aren't actually a part of Git, they're an abstraction built by GitHub. The PRs themselves still exist, it's just the ability to search for them that is broken
stayoungodancing@reddit
“While we were already in progress of migrating out of our smaller custom data centers into public cloud, we started working on path to multi cloud.”
Does anyone else feel that this would be much less secure?
blackwhattack@reddit
> This incident exposed multiple process failures, and we are changing those processes to prevent this class of issue from recurring
**Document type**
This document contains contents that you would expect from contents of that document.
Ok but why?
jtonl@reddit
I read nothing good.