I don't think this portents anything great for software in general.
We're a good year+ into the use LLMs for all major bits of software that we all rely upon and GitHub here is down to one 9 of uptime. I've been using GitHub for a _long_ time, my first commits there go back to August 2009!, and I honestly don't recall GitHub going down as much as it has in the last year.
I'm sure there's other things happening in the background, but I can not help but believe that this is directly correlated with the increase of LLM usage.
Though I would love to hear someone else's pet theory how a rock of the internet went from four+ nines of uptime to maybe one.
Are these outages caused by introduced bugs, though, or by load issues?
As someone who has spent many years working in high load environments, this is not an uncommon pattern.
You design a system and it works great. It can handle failures, load spikes, it is horizontally scalable, things are great. You think you figured it out.
And then load keeps increasing and you suddenly hit a tipping point where everything keeps failing, and you cant keep up. The things that you thought were perfectly horizontally scalable turn out to have a bottleneck you didn’t even think about until you got to a truly massive scale. Your systems suddenly don’t have the excess capacity to handle load spikes or catchup work, so suddenly any failure cascades and recovery is more and more difficult. You can’t solve the problem with additional hardware, and your perfect scalable design actually can’t scale any more.
This doesn’t have to be about GitHub using LLMs in their code to still be related to LLMs. GitHub gets a lot more commits now because of LLMs and probably get a lot more reads because of LLMs as well.
It could be that the extra usage just pushed them past one of those capacity thresholds.
The funny thing is that if companies wanted they could probably use AI to instead increase uptime. Keep existing QA teams (instead of replacing them) and then use AI for better and more timely monitoring (and messaging even) and as an additional Always-Testing™ layer of QA
To me this correlates more to them being bought by Microsoft, a company known for being seemingly incapable of creating quality software to the point that it's not even funny anymore, and also known for sloppifying all the products they touch.
It's true, Microsoft made github worse, but the more recent issues seem to have to do a lot more with Microsoft selling its soul to AI. Microsoft really appears to have gotten dumber as they became dependent on AI. Most recent example: they used to promote Win11 and 32GB RAM. Now they are down to 8GB silently ... this is quite hilarious. There are now so many side effects that you see degradation in so many other areas. Or the gaming industry: it is not quite dying but it is taking a huge hit with skyrocketing RAM prices. Consoles selling less is an example here. It's quite fascinating how deadly disruptive AI is now.
I don't think so. GitHub was bought by Microsoft 8 years ago and people have only started complaining about its uptime in the last year or so - exactly correlating with the surge in LLM use.
I don't think so. I've seen those complaints for more than a year.
I have an Ops background and I strongly suspect they were given a stupid timeline for the Azure migration.
I've got to believe Microsoft have decent Ops people but the management wanted to move faster than was reasonable and screwed it up. Move one thing at a time and double check it all works and you can do a migration like this.
> people have only started complaining about its uptime in the last year or so
I'm sorry but this made me laugh out loud. That isn't true at all, this has been going on for years. This conversation[0] from six years ago has discussion about the outages starting to become much more frequent in December 2019. It has never gotten better in that time, it's just continually degraded.
They mentioned earlier this year that they were beginning the migration to Azure and that it would take a couple years. I would assume it has more to do with that migration then anything else.
It's probably not attributable to AI in the way that you're thinking - Github has been absorbing an exponential increase in usage, and that increase is mostly due to new AI-related projects being created and worked on.
Though I'm sure some of the blame can go to internal slop code.
It is definitely and absolutely caused by LLMs. I must do 20x more GitHub operations now, and since the agents know GitHub far better than me, I'm using more advanced features. Multiply this times all of us.
For the longest time, I didn't appreciate the "AI-induced traffic" excuse. But seriously, I checked the rough github egress for our lab versus an old log from 2024, and there's an order of magnitude or two difference. From asking around, it seems people all have the gh cli tool and let it loose with parallel tool calls and e.g LLM's polling Actions in a background bash while loop with sleep $TOO_FEW_SECONDS. Some people use a variety of skills where the agent makes a commit every few code changes, and uses Issues for its memory/log. And they have O(5) sessions at the same time doing all kinds of crap. It's the same with PR checks/PRs. Recently we also saw continued usage throughout the night as well, which did not exist pre coding agents. Loops or whatever they call cron jobs in the harnesses these days is the reason. It must be adding up.
5 hours going now and the entire thing is still completely down. This level of incompetence and their total disrespect for their customers is unbelievable.
Our company recently moved some of our repositories over to a self-hosted forgejo and now I'm so jealous of those engineers contributing to the forgejo hosted repositories
Yeah, I think this one might be the straw that broke the camel's back and our team will look to migrate to something non-microsoft to be away from this nonsense.
I once saw a sign at a railway station in the UK that actually did say "normal service will be operating between the 12th of July and 18th of August" (or something like that). Made me laugh. At the time Anglia Railways had a rail replacement bus literally every weekend.
I thought they should rename to Anglia Busways and have bus replacement trains instead.
It's honestly insane how terrible their reliability is. Over the past 4 weeks, 8 days with GitHub Actions outages, many of them multi-hour outages: >3 hours on each of July 9th, July 20th and today, and 1.5 hrs on July 23rd.
Outages happen, but this many outages so close together, and so many of them so major/long lasting, something is systematically wrong for sure. It's been seriously hamstringing our ability to ship code at my company.
Based on previous posts I've seen about this, IIRC the timing seems to imply that it has more to do with them being hammered with AI slop than it does with the Azure transition. Who really knows though
Time to touch some grass. Better use of my time and energy than twisting the remaining things on my todo list today to make more progress on them than I have managed. I should have taken a long lunch but I rebased the hell out of a PR instead.
I love how even self hosted workers don't work during these outages - running jobs on their infrastructure being flakey is marginally acceptable, but the API to simply schedule workflows having this availability is mind boggling. Github just doesn't seem like a serious company anymore.
GitHub implements self hosted runners by running a normal runner that passes the environment to your runner and then polls it. That's why they cost as much as the smallest GitHub-hosted runner.
Oh that is pretty gross, thanks for sharing. I was not aware of that process, that explains the cost they attempted to roll out for runners too a few months back.
They never were. They were just really good at maintaining the bluster and backing it up with expensive lawyers. Now that they don't come in claws-first like they did when they were a NASDAQ rocket to the stars, it's easy to see.
Their MO was to court an executive and sell second-rate tools to them before the people who had to use them had a chance to say anything. It doesn't matter how much evidence you can provide to the contrary, once the million dollar deal is signed, you are going to be tasked with finding reasons to say that your executive was shrewd for buying this pile of junk and unfulfilled promises, and not an insane idiot sucking away your job satisfaction as fast as they can.
They did a lot of deals based on how their products would have features their competitors already have 'soon' when they haven't even started them, and a long track record of taking 3 major releases to get from something to good, and then breaking everything again by doing a 4th major release that re-imagined everything and made it horrible again.
I'm not going to claim that Apple was or is a panacea. Apple doesn't use vaporware which is big, and their Cycle of Awful is 2 releases instead of 3. You could afford to skip 1 waiting for the next even-numbered version, instead of being 2 versions behind and getting pressed to upgrade.
They definitely managed to create a lot of reliable products.
Windows NT was one of them. Up to Windows XP the products were pretty solid and each had visible improvement against the previous one.
Their language products were/are still solid IMO. Maybe Visual Studio is sluggish, but we can still use an older version if we want. Plus they put a lot of effort optimizing VSCode, too.
Even back in the MS-DOS/16-bit Windows days, when things broke down quite easily, I think they still provide the best bang for individual users and developers. There was no competitors who could provide so much value back then.
Microsoft in the 90-s and early 2000-s was a serious company. Windows NT and Office were engineering marvels.
Windows 95 was a brilliant hack that allowed 32-bit GUI programs to run on hardware with 2Mb of RAM (Windows had 4Mb of RAM as the official minimum, but you could run it on 2Mb) while preserving compatibility with the majority of DOS software.
I'm not sure that it's that mind boggling. The entire complexity and value of GHA is not in the runners but the scheduler that the GH frontend hooks into and runners subscribe to jobs from. It's the most likely thing to fall over. '
If the scheduling was self hosted it would be inexcusable but you can always just connect whatever you want to webhooks.
The entire complexity and value of GHA is rent-seeking to keep the lights on for other things.
They have a strong motivation (self preservation) to continue to misunderstand the problem. If they did what is best for us, then we could avoid a substantial fraction of all GH subscriptions by using a FOSS tool to hit the Pareto frontier by replicating just enough GH services to watch commits and PRs.
That "scheduling" is just a git hook and a message queue, maybe with some database updates in between with very clear boundaries that make sharding easy to reason about, assuming they have a sane architecture (they evidently don't)
The only statement I'm making about their system is that it appears to be poorly designed, as evident by their poor uptime. The rest is just visible details about what data is involved in performing the task at hand, agnostic to the implementation underlying it.
The one thing that falls over is that the webhooks are actually self modifying code since you have an 'on commit' webhook that bootstraps the actions logic from the workflow.yaml file(s) (which are not really config files, they have logic that needs to be evaluated).
I don't disagree that it's obvious they've got problems but I'm just saying it's obvious to me the part that falls over (the scheduling of jobs) and why that would impact self hosted runners, which do no scheduling but depend on it to function.
As for 'just a message queue with some database updates and sharding that's easy to reason about'... Here's a job scheduling problem as an example: imagine you schedule a job, and there's no runner available. How do you disambiguate between no runners available because you've reached capacity, runners not being available because they're on a real network with faulty connections, and runners not being available because of a faulty rollout of internal updates?
A simple message queue for job scheduling is fine if you own everything and can deal with the operational overhead of identifying those cases by hand, but Github can't do that.
> The one thing that falls over is that the webhooks are actually self modifying code since you have an 'on commit' webhook that bootstraps the actions logic from the workflow.yaml file(s) (which are not really config files, they have logic that needs to be evaluated).
Sure, but the part that actually schedules where a 'job' gets run is based on a relatively simplistic tag system. Reading the yaml and plopping some job metadata into a queue-like system isn't where I would expect their issues to be, but at their scale I'm sure everything becomes fragile and inscrutable.
> imagine you schedule a job, and there's no runner available. How do you disambiguate between no runners available because you've reached capacity, runners not being available because they're on a real network with faulty connections, and runners not being available because of a faulty rollout of internal updates?
You don't need to. GitHub Actions runners, and most CI runners that I've interacted with appear to have a pull-based model where they ask for work that matches their declared tags/shape (usually platform/runtime/OS/etc.). This probably amounts to a database query, but who knows.
> A simple message queue for job scheduling is fine if you own everything and can deal with the operational overhead of identifying those cases by hand, but Github can't do that.
I highly doubt it's a simple message queue. My issue is git repos and their CI infrastructure have very low coupling to other repos or entities in most circumstances, at least conceptually, so parts of the system (ie. regions, shards, etc.) should be able to function even when others are down (ie. it shouldn't break for everyone). There's clearly centralization and coupling that isn't obvious from an outside perspective, which sorta tells me it's incidental, but that's a guess.
It would seem like GitHub is in a precarious situation.
We have many agents per employee working in parallel pushing way more commits than was humanly possible before AI, triggering GitHub actions a lot more than the workflows were built for, causing Actions costs to escalate (they really aren't cheap if you compare to hosting it yourself), meanwhile working with YAML workflows is just a pain, and just writing code would be so much more fun and AI compatible[1].
At the same time, GitHub has about ~3 different PR review UIs? And they're all half-bad? Any decently sized PR triggers their "optimized for large PRs" UI which jumps around randomly in my experience. If you don't get that UI and keep the scrolling one (there's an old and a new one btw) then god forbid you click a line number because at some point your browser will randomly scroll back to that line and it won't unstick. Now Linear[2] (and others) is replacing the PR review experience for the agentic era.
I'd love to see a solid AI first Git + CI + reviews.
Given the recent history of outages, I wonder how many are seriously considering moving off GitHub for anything other than code hosting. I mean actions / workflows.
We've been working on this at Depot[0]. We built the internals of Depot CI as a general-purpose workflow engine that understands GitHub Actions as an input (with plans to extend) - we've been working to make Actions runners more performant and more reliable for a while, but eventually you hit a ceiling if you don't also control the control plane.
To offer a data point (not a tribalist argument): GitHub Actions being down this often is still less expensive than the cost of switching everything somewhere else. Would rather have my team walk away from the PRs and do other tasks than have engineering effort, meetings, design pages, scheduling, etc. for switching over.
This is annoying and I'm here because it's down. But it would have to be far worse to come close to actually being worth changing.
1. go sign-up elsewhere
2. Log into GitHub and point Actions to that new host
3. All my actions files immediately worked without question
I'd probably give it a spin and make a wiki page explaining how to swap back and forth. No meetings. No design issues. No scheduling. Just a flip switch on who to pay for computers.
For what it's worth, we switched some of our actions to a self-hosted Woodpecker instance, and although there were a few kinks to iron out, it works better overall (for example, because of better caching on that single instance, our docker images build faster).
We've started writing some new workflows against argo-workflows for the last like 9months or so. Especially for use cases where the actions are like self-service type automation.
I think people were so excited to move away from jenkins to something 'managed' just because of how much a dinosaur jenkins is and how much a pain in the ass it is to upgrade it... but now we are seeing how managed can bite you in the ass if the manager is incompetent.
I think I agree if you don't have someone dedicated to running and maintaining Jenkins. If you have someone at your org that knows what they are doing though, I'd still take Jenkins over GHA any day.
That's significantly more difficult to set up and it cost money. Github's big issues that it's free, Microsoft doesn't want to allocate enough budget to keep the thing running properly.
But for people who either don't pay anything at all or phenomenal amount one 9 of up time is all you need.
If you're actually trying to run a business I guess you can call and gitlab and get an Enterprise contract
Our deployments are triggered by GitHub Actions, and it is a pain to deploy without it (specifically collecting the credentials and putting them into variables the shell will read).
This is multiple times this month that this has been a problem.
Has GitHub completed it's internal migration to Azure yet? Or is it still ongoing? None of our devs want to switch away from GH, but we will have to at this point.
We have set up a new enterprise and are migrating to GHE. For us this was a no-brainer; we get our own isolated environment, Enterprise Managed Users with Entra OIDC provisioning and Github Copilot in EU data residency. https://eu.githubstatus.com/posts/dashboard looks pretty good to me. We will still keep our github.com enterprise around for public repositories. I’m genuinely curious to know why a company would prefer github.com over GHE?
My Github actions are now basically: "checkout / set env vars from secrets / docker-compose builder run make".
I used large machine runners to run full the Docker (Podman actually) on Github first to avoid dealing with docker-in-docker complications. This step also provided some very nice robustness advantages, as I can now trigger deployments from my laptop if needed.
Step two:
Migrate to self-hosted runners. I used my former homelab server to set up a build machine. It has 16Tb of fast NVMe SSDs and thanks to Podman container layer caching, my entire lint workflow now takes 30 seconds. Faster than just one "npm install" on Github before.
And Github's self-hosted runners are actually surprisingly easy to set up and use. They are also somewhat more robust.
WoodpeckerCI for DIY, Drone.io for paid support. Self-hosted, OAuth login, repo-permission-based authorization, container-native design, easy web UI, optional RDBMS, supports many platforms. All the core features needed for scalable CI, deploy as few/many as you want for multiple teams, small enough to easily run on one box for one repo. Nothing else is as simple, easy, powerful, compatible.
Until someone makes a better PR experience you guys are stuck with GitHub.
Nobody cares about ATProto or whether your commits are a damn NFT or some bs just literally improve upon the experience.
That’s it.
It’s as if no company is focusing on the product experience or anybody’s experience anymore. It’s all ooo look what I got I got this I can do that too me me me but nobody will ever buy that.
Say what you want about huge companies like Microsoft or Walmart but they spend a lot of energy understanding the human experience to sell products and less on their own perceived self-aggrandizement.
GitHub is the best version control online and it’s not even close.
Can't wait for the Graphite/Cursor alternative (Origin) to become available. Actions being down for an hour in the middle of the US East workday is crazy. It's super frustrating to be an enterprise customer of Github, seems like we don't get any dedicated resources or additional stability. At this point our only option is self-host github enterprise or move away.
I feel stuck on Github until the next generation of forges (I assume Linear is working on one too, and the Pierre guys, and there's probably more) are ready. When it works, I'm actually a huge fan of Github. It just doesn't work reliably and it's a huge operational risk to not be able to freaking deploy. We'll most likely switch to self-hosted GHE in the near-term while continuing to evaluate alternative forges — Forgejo, Gitlab, Gitea, Sourcehut, aren't appealing enough to switch.
The most likely explanation for the degraded reliability: Someone at GitHub saw the Three Body Problem last year and started shifting their infra over to Minecraft servers with "physical" memory and redstone logic gates.
It's cute that someone at GitHub had time to make their hamburger menu pancakes to promote stacked PRs, but nobody has time to make the platform stable.
Soon we can start posting "GitHub {x} is up!" instead, that's becoming more news-worthy lol. I know they like to blame AI/scaling, but really they are owned by one of the main hyperscalers, so I find that excuse kind of weak..
On other occasions, I'd take this as a time to have a walk because I'm blocked. Unfortunately, I need to get some stuff out for a customer quickly. Can't do anything about that though. GitHub is the only product in it's size class that I use that has this kind of incredibly poor uptime.
I'd love to know what the most common root causes for these outages are.
They are known are engineers pushing features that bloated product management folks keep pushing so they can justify their jobs on LinkedIn.
I'm not sure why this particular industry is so abysmal at making things even semi-reliable after decades of research, educated workforces, and loads of cash.
In the news tomorrow: SpaceX's newest Grok model just got out of the sandbox and hacked into Github... Grok is so very very dangerous, their engineers only enter the server room in foil hats
I was scratching my head wondering why my build file isnt working, ran a working project and it didn't build either. Figured out what might be the case, but wasted an hour.
I was complaining about how I should have used AI instead of manual brainwork for this instead but turns out that might have been the problem.
I knew that when the build actions started failing en masse and my newly submitted PRs did not trigger the build actions at all (besides the mandatory one in my organization), it seemed logical to me to look at HN first..
Especially troublesome in the middle of trying to fix a high score security vulnerability when the release vehicle is Github.
Does anyone recommend an alternative? We've already moved our runners to blacksmith, looking for a control plane as well. We are tiny startup that ships at high velocity and this kind of downtime is highly disruptive for us
There are three other things I could be doing instead, but I noticed it was down when I pushed the last of my GH todo list for the day and my brain is stuck in closure-seeking.
I've never understood this argument. Even if self-hosted things were offline more often (which I've never found to be the case, I've had Forgejo + runners running for a year now with no downtime), the real benefit is that you yourself can work to bring it back online when it does, rather than waiting on a large, slow-moving organization to figure out what slopped PR caused their global service serving ungodly amounts of RPS to go offline again.
Even self-hosted runners are impacted.... How can that be?
The cost of this globally has got to be in the hundreds of millions to companies that use CI/CD through GitHub Actions. What if prod is broken and GitHub actions is stalling the deployment of your hotfix? What if this makes your organization miss and SLA and diminish user trust? What if this makes you miss a release that you were contractually obligated to meet? This is happening during peak dev hours on a Thursday (not that it would be acceptable at any other time).
I don't understand how a service this critical to the global technical infrastructure can fail like this at all, let alone for more than a few hours. Like where's the backup generator for crises like these? You can't even use self-hosted runners? WTF? Like how can you not bring your own backup in a crisis event like this?
Not that Microsoft has a good reputation, but holy moly, you'd think they would prepare from something inevitable like this.
Counterparty risk is still a thing. This is the cost of convenience, reminds me of the milkman joke in that South Park episode. Maybe a global SPOF owned by people who do not care is not the way to go.
Either they had a good month or I'm off my game, I actually spent a minute checking if I did something wrong with actions instead of immediately going to HN to see if it was an outage.
Nothing like the old "everything is normal" to only be corrected 8 minutes later it is actually still down and then 2 hours later they are still working on it.
Aug 06, 2026 - 16:27 UTC - Update - Pages is experiencing degraded performance. We are continuing to investigate.
Aug 06, 2026 - 16:19 UTC - Update - Pages is operating normally.
I noticed my CI throwing errors all of a sudden. I sure wish they would become more active in alerting folks or build it into these tools - especially as these occurrences are becoming more frequent. Possibly an API-accessible services status. Then, at least we could build in our own checks when we hit errors and not have to hunt down what all is broken.
You don't think GitHub working on seven different interfaces to copilot and vibe coding PR page changes filled with bugs are a good use of engineering time?~
As I said 6 years ago. [0] You would be better off self-hosting than using GitHub or GitHub actions. No CEO of GitHub exists and now it is falling over again.
It's really crazy to me they can just be down for hours and can't recover their own systems. It shows they don't have the capability or infrastructure to roll back disastrous changes. These kinds of things are a tell on the organization and operational excellence (or not). As soon as a viable alternative surfaces for Github, I'm moving off and will advise all my clients to do so as well.
Ever since Microsoft took over, and then when AI Skynet took control, things started to decay. AI companies owe all of use a lot of money. By the way, why do I have to pay for increasing RAM prices here? Why are we so dependent on a few greedy corporations, anyway?
My experience working at LinkedIn was that, after Microsoft took over, they offshored a huge part of their infrastructure and turned the entire culture into one of performative based engineering instead of results based engineering.
If you implemented a tool and it worked one time for your presentation to management that's all that mattered.
The actual company employees using it downstream in prod basically had to constantly QA the alpha software they were forced to use and the authors of the tool were hard to track down if they even still worked there. And if you did find the author or the team they would be very resistant to admitting there was an issue because it LOOKED bad.
So many tools I used were fragile and buggy, it was clear the authors just presented the happy path to management to get the note added to their promotion packet and the rest of the company just had to deal with the fallout.
My team implemented this product that the entire company used that was broken and buggy as hell but they kept presenting the product to management as this amazing product and nothing was ever done about how broken it was. One of my team members came from Apple and said Apple's tool to do the same thing was much better. The tool my team worked on was a well known pain point amongst the rank and file but management was very detached from the rank and file, which I guess ultimately was the primary problem.
If Github is having the same issues I feel for them.
Someone needs to work on improving the way Actions deals with resource exhaustion like this. Today it exploded all the CI jobs with "failed" and an inscrutable message about "The job was not acquired by Runner of type hosted even after multiple attempts" then "Internal server error". It should say "Timed out due to no runner available, our apologies, try again later, or pay $$$ here to get priority for runners".
Have there ever been a comprehensive account from a GitHub insider on why their reliability tanked so hard since the MSFT acquisition?
I'm genuinely curious what changed, what their processes are and how they internally think about their reputation being in the gutter with all these incidents.
I don't think this portents anything great for software in general.
We're a good year+ into the use LLMs for all major bits of software that we all rely upon and GitHub here is down to one 9 of uptime. I've been using GitHub for a _long_ time, my first commits there go back to August 2009!, and I honestly don't recall GitHub going down as much as it has in the last year.
I'm sure there's other things happening in the background, but I can not help but believe that this is directly correlated with the increase of LLM usage.
Though I would love to hear someone else's pet theory how a rock of the internet went from four+ nines of uptime to maybe one.
Are these outages caused by introduced bugs, though, or by load issues?
As someone who has spent many years working in high load environments, this is not an uncommon pattern.
You design a system and it works great. It can handle failures, load spikes, it is horizontally scalable, things are great. You think you figured it out.
And then load keeps increasing and you suddenly hit a tipping point where everything keeps failing, and you cant keep up. The things that you thought were perfectly horizontally scalable turn out to have a bottleneck you didn’t even think about until you got to a truly massive scale. Your systems suddenly don’t have the excess capacity to handle load spikes or catchup work, so suddenly any failure cascades and recovery is more and more difficult. You can’t solve the problem with additional hardware, and your perfect scalable design actually can’t scale any more.
This doesn’t have to be about GitHub using LLMs in their code to still be related to LLMs. GitHub gets a lot more commits now because of LLMs and probably get a lot more reads because of LLMs as well.
It could be that the extra usage just pushed them past one of those capacity thresholds.
There was a great article awhile back that shed some light on just how dysfunctional azure is as a platform: https://news.ycombinator.com/item?id=47616242
If I had to guess it's because Github is sitting on top on infrastructure held up by toothpicks and duct tape
The funny thing is that if companies wanted they could probably use AI to instead increase uptime. Keep existing QA teams (instead of replacing them) and then use AI for better and more timely monitoring (and messaging even) and as an additional Always-Testing™ layer of QA
To me this correlates more to them being bought by Microsoft, a company known for being seemingly incapable of creating quality software to the point that it's not even funny anymore, and also known for sloppifying all the products they touch.
It's true, Microsoft made github worse, but the more recent issues seem to have to do a lot more with Microsoft selling its soul to AI. Microsoft really appears to have gotten dumber as they became dependent on AI. Most recent example: they used to promote Win11 and 32GB RAM. Now they are down to 8GB silently ... this is quite hilarious. There are now so many side effects that you see degradation in so many other areas. Or the gaming industry: it is not quite dying but it is taking a huge hit with skyrocketing RAM prices. Consoles selling less is an example here. It's quite fascinating how deadly disruptive AI is now.
I don't think so. GitHub was bought by Microsoft 8 years ago and people have only started complaining about its uptime in the last year or so - exactly correlating with the surge in LLM use.
https://damrnelson.github.io/github-historical-uptime/
Seems pretty conclusive. Very similar story when they bought skype.
God that's sad. I almost feel bad for all the engineers there, though I'm sure they made good money and probably left
I don't think so. I've seen those complaints for more than a year.
I have an Ops background and I strongly suspect they were given a stupid timeline for the Azure migration.
I've got to believe Microsoft have decent Ops people but the management wanted to move faster than was reasonable and screwed it up. Move one thing at a time and double check it all works and you can do a migration like this.
Isnt it also in the past year when they started to move Github fully to microsoft infra? Off whatever they were doing before
Didn't github also migrate at azure recently? That certainly can't help.
> people have only started complaining about its uptime in the last year or so
I'm sorry but this made me laugh out loud. That isn't true at all, this has been going on for years. This conversation[0] from six years ago has discussion about the outages starting to become much more frequent in December 2019. It has never gotten better in that time, it's just continually degraded.
[0] https://news.ycombinator.com/item?id=22935941
They mentioned earlier this year that they were beginning the migration to Azure and that it would take a couple years. I would assume it has more to do with that migration then anything else.
Microsoft acquisition which forced to migrate all to Azure.
I can totally buy that this is a larger contributor! Thank you, I wasn't aware that it was going on right now.
I guess github is kind of a shared garden. Interesting that, like in game theory, if everyone is using it too much, no one gets to use it.
It's probably not attributable to AI in the way that you're thinking - Github has been absorbing an exponential increase in usage, and that increase is mostly due to new AI-related projects being created and worked on.
Though I'm sure some of the blame can go to internal slop code.
It is definitely and absolutely caused by LLMs. I must do 20x more GitHub operations now, and since the agents know GitHub far better than me, I'm using more advanced features. Multiply this times all of us.
For the longest time, I didn't appreciate the "AI-induced traffic" excuse. But seriously, I checked the rough github egress for our lab versus an old log from 2024, and there's an order of magnitude or two difference. From asking around, it seems people all have the gh cli tool and let it loose with parallel tool calls and e.g LLM's polling Actions in a background bash while loop with sleep $TOO_FEW_SECONDS. Some people use a variety of skills where the agent makes a commit every few code changes, and uses Issues for its memory/log. And they have O(5) sessions at the same time doing all kinds of crap. It's the same with PR checks/PRs. Recently we also saw continued usage throughout the night as well, which did not exist pre coding agents. Loops or whatever they call cron jobs in the harnesses these days is the reason. It must be adding up.
5 hours going now and the entire thing is still completely down. This level of incompetence and their total disrespect for their customers is unbelievable.
Our company recently moved some of our repositories over to a self-hosted forgejo and now I'm so jealous of those engineers contributing to the forgejo hosted repositories
Yeah, I think this one might be the straw that broke the camel's back and our team will look to migrate to something non-microsoft to be away from this nonsense.
What are the alternatives?
My company is migrating repositories over to a self-hosted forgejo instance, currently very jealous of the engineers working on those migrated repos.
At this point maybe they should consider sending out announcements when it is working instead
#0-nines
Hey, there's still plenty if you don't mind the leading 8
Have they aimed for 86 uptime?
67 uptime should be within reach for them.
I once saw a sign at a railway station in the UK that actually did say "normal service will be operating between the 12th of July and 18th of August" (or something like that). Made me laugh. At the time Anglia Railways had a rail replacement bus literally every weekend.
I thought they should rename to Anglia Busways and have bus replacement trains instead.
It's been hours :(
I have sympathy for the on-call team trying to resolve it, most of us have been there done that.
But seems something is systematically going wrong at GH
"most of us have been there done that" - so true. That feeling in your gut when you realize something you just did caused an outage is pretty unique.
It's honestly insane how terrible their reliability is. Over the past 4 weeks, 8 days with GitHub Actions outages, many of them multi-hour outages: >3 hours on each of July 9th, July 20th and today, and 1.5 hrs on July 23rd.
Outages happen, but this many outages so close together, and so many of them so major/long lasting, something is systematically wrong for sure. It's been seriously hamstringing our ability to ship code at my company.
Mon and Dad (MS) are (generally) pretty solid with uptime.
What is happening at GH?
Rate of change trying to keep up with new challengers? Over-reliance on AI? Engineers trying to debug slop?
Yeah who knows, would be interesting to hear an inside take if any readers here are also GH devs!
They're at 93.91% uptime over the past 90 days, according to https://mrshu.github.io/github-statuses/ , and that doesn't even include today's outage yet.
A glorious one nine of reliability.
I see two nines in there…
Don't throw shade, there are two nines in that percentage for now. Miles a part though.
911 has one! :)
> What is happening at GH?
In large part, the move from AWS to Azure. Azure's just bad.
wasn't that years ago? or was that Skype? hahaha
Based on previous posts I've seen about this, IIRC the timing seems to imply that it has more to do with them being hammered with AI slop than it does with the Azure transition. Who really knows though
> But seems something is systematically going wrong at GH
Yes, we call it: Microslop.
They're one rewrite in Rust away from fixing everything. (jk)
Time to touch some grass. Better use of my time and energy than twisting the remaining things on my todo list today to make more progress on them than I have managed. I should have taken a long lunch but I rebased the hell out of a PR instead.
I love how even self hosted workers don't work during these outages - running jobs on their infrastructure being flakey is marginally acceptable, but the API to simply schedule workflows having this availability is mind boggling. Github just doesn't seem like a serious company anymore.
GitHub implements self hosted runners by running a normal runner that passes the environment to your runner and then polls it. That's why they cost as much as the smallest GitHub-hosted runner.
This is no surprise given standard Microsoft operating procedure - https://news.ycombinator.com/item?id=47616242
they walked that back for now at least - self-hosted gh runners are still free: https://docs.github.com/en/billing/concepts/product-billing/...
Well, they're still implemented that way.
Can you share a source for this? it's not that I don't believe you but I've had difficulties finding one.
Do you have a source for this?
Oh that is pretty gross, thanks for sharing. I was not aware of that process, that explains the cost they attempted to roll out for runners too a few months back.
Microsoft is not a serious company anymore.
They never were. They were just really good at maintaining the bluster and backing it up with expensive lawyers. Now that they don't come in claws-first like they did when they were a NASDAQ rocket to the stars, it's easy to see.
Their MO was to court an executive and sell second-rate tools to them before the people who had to use them had a chance to say anything. It doesn't matter how much evidence you can provide to the contrary, once the million dollar deal is signed, you are going to be tasked with finding reasons to say that your executive was shrewd for buying this pile of junk and unfulfilled promises, and not an insane idiot sucking away your job satisfaction as fast as they can.
They did a lot of deals based on how their products would have features their competitors already have 'soon' when they haven't even started them, and a long track record of taking 3 major releases to get from something to good, and then breaking everything again by doing a 4th major release that re-imagined everything and made it horrible again.
I'm not going to claim that Apple was or is a panacea. Apple doesn't use vaporware which is big, and their Cycle of Awful is 2 releases instead of 3. You could afford to skip 1 waiting for the next even-numbered version, instead of being 2 versions behind and getting pressed to upgrade.
They definitely managed to create a lot of reliable products.
Windows NT was one of them. Up to Windows XP the products were pretty solid and each had visible improvement against the previous one.
Their language products were/are still solid IMO. Maybe Visual Studio is sluggish, but we can still use an older version if we want. Plus they put a lot of effort optimizing VSCode, too.
Even back in the MS-DOS/16-bit Windows days, when things broke down quite easily, I think they still provide the best bang for individual users and developers. There was no competitors who could provide so much value back then.
Microsoft in the 90-s and early 2000-s was a serious company. Windows NT and Office were engineering marvels.
Windows 95 was a brilliant hack that allowed 32-bit GUI programs to run on hardware with 2Mb of RAM (Windows had 4Mb of RAM as the official minimum, but you could run it on 2Mb) while preserving compatibility with the majority of DOS software.
One might question your perspective if you ever thought they were serious.
Maybe switching to https://github.com/gitlabhq :)
I'm not sure that it's that mind boggling. The entire complexity and value of GHA is not in the runners but the scheduler that the GH frontend hooks into and runners subscribe to jobs from. It's the most likely thing to fall over. '
If the scheduling was self hosted it would be inexcusable but you can always just connect whatever you want to webhooks.
The entire complexity and value of GHA is rent-seeking to keep the lights on for other things.
They have a strong motivation (self preservation) to continue to misunderstand the problem. If they did what is best for us, then we could avoid a substantial fraction of all GH subscriptions by using a FOSS tool to hit the Pareto frontier by replicating just enough GH services to watch commits and PRs.
That "scheduling" is just a git hook and a message queue, maybe with some database updates in between with very clear boundaries that make sharding easy to reason about, assuming they have a sane architecture (they evidently don't)
It's just silly to guess at how a system works, or should work, without digging into the details. Because you simply don't know what you don't know.
The only statement I'm making about their system is that it appears to be poorly designed, as evident by their poor uptime. The rest is just visible details about what data is involved in performing the task at hand, agnostic to the implementation underlying it.
The one thing that falls over is that the webhooks are actually self modifying code since you have an 'on commit' webhook that bootstraps the actions logic from the workflow.yaml file(s) (which are not really config files, they have logic that needs to be evaluated).
I don't disagree that it's obvious they've got problems but I'm just saying it's obvious to me the part that falls over (the scheduling of jobs) and why that would impact self hosted runners, which do no scheduling but depend on it to function.
As for 'just a message queue with some database updates and sharding that's easy to reason about'... Here's a job scheduling problem as an example: imagine you schedule a job, and there's no runner available. How do you disambiguate between no runners available because you've reached capacity, runners not being available because they're on a real network with faulty connections, and runners not being available because of a faulty rollout of internal updates?
A simple message queue for job scheduling is fine if you own everything and can deal with the operational overhead of identifying those cases by hand, but Github can't do that.
> The one thing that falls over is that the webhooks are actually self modifying code since you have an 'on commit' webhook that bootstraps the actions logic from the workflow.yaml file(s) (which are not really config files, they have logic that needs to be evaluated).
Sure, but the part that actually schedules where a 'job' gets run is based on a relatively simplistic tag system. Reading the yaml and plopping some job metadata into a queue-like system isn't where I would expect their issues to be, but at their scale I'm sure everything becomes fragile and inscrutable.
> imagine you schedule a job, and there's no runner available. How do you disambiguate between no runners available because you've reached capacity, runners not being available because they're on a real network with faulty connections, and runners not being available because of a faulty rollout of internal updates?
You don't need to. GitHub Actions runners, and most CI runners that I've interacted with appear to have a pull-based model where they ask for work that matches their declared tags/shape (usually platform/runtime/OS/etc.). This probably amounts to a database query, but who knows.
> A simple message queue for job scheduling is fine if you own everything and can deal with the operational overhead of identifying those cases by hand, but Github can't do that.
I highly doubt it's a simple message queue. My issue is git repos and their CI infrastructure have very low coupling to other repos or entities in most circumstances, at least conceptually, so parts of the system (ie. regions, shards, etc.) should be able to function even when others are down (ie. it shouldn't break for everyone). There's clearly centralization and coupling that isn't obvious from an outside perspective, which sorta tells me it's incidental, but that's a guess.
gitlab-ce can run all the integration tests in docker et al.
Sitting on Github these days is the same as sticking to twitter a decade ago, expect next mecha hitler, I suppose.
It would seem like GitHub is in a precarious situation.
We have many agents per employee working in parallel pushing way more commits than was humanly possible before AI, triggering GitHub actions a lot more than the workflows were built for, causing Actions costs to escalate (they really aren't cheap if you compare to hosting it yourself), meanwhile working with YAML workflows is just a pain, and just writing code would be so much more fun and AI compatible[1].
At the same time, GitHub has about ~3 different PR review UIs? And they're all half-bad? Any decently sized PR triggers their "optimized for large PRs" UI which jumps around randomly in my experience. If you don't get that UI and keep the scrolling one (there's an old and a new one btw) then god forbid you click a line number because at some point your browser will randomly scroll back to that line and it won't unstick. Now Linear[2] (and others) is replacing the PR review experience for the agentic era.
I'd love to see a solid AI first Git + CI + reviews.
[1] Cloudflare CI https://blog.cloudflare.com/ci-workflows/
[2] Linear PR reviews https://linear.app/changelog/2025-01-23-pull-request-reviews
Given the recent history of outages, I wonder how many are seriously considering moving off GitHub for anything other than code hosting. I mean actions / workflows.
We've been working on this at Depot[0]. We built the internals of Depot CI as a general-purpose workflow engine that understands GitHub Actions as an input (with plans to extend) - we've been working to make Actions runners more performant and more reliable for a while, but eventually you hit a ceiling if you don't also control the control plane.
[0] https://depot.dev
To offer a data point (not a tribalist argument): GitHub Actions being down this often is still less expensive than the cost of switching everything somewhere else. Would rather have my team walk away from the PRs and do other tasks than have engineering effort, meetings, design pages, scheduling, etc. for switching over.
This is annoying and I'm here because it's down. But it would have to be far worse to come close to actually being worth changing.
A case study in vendor lock-in.
Pretty much, eh?
If I could right now:
1. go sign-up elsewhere 2. Log into GitHub and point Actions to that new host 3. All my actions files immediately worked without question
I'd probably give it a spin and make a wiki page explaining how to swap back and forth. No meetings. No design issues. No scheduling. Just a flip switch on who to pay for computers.
For what it's worth, we switched some of our actions to a self-hosted Woodpecker instance, and although there were a few kinks to iron out, it works better overall (for example, because of better caching on that single instance, our docker images build faster).
We've started writing some new workflows against argo-workflows for the last like 9months or so. Especially for use cases where the actions are like self-service type automation.
I think people were so excited to move away from jenkins to something 'managed' just because of how much a dinosaur jenkins is and how much a pain in the ass it is to upgrade it... but now we are seeing how managed can bite you in the ass if the manager is incompetent.
GHA still beats Circle CI or trying to run your own Jenkins - even with all the downtime. I'd move in a heartbeat if there were good alternatives.
I think I agree if you don't have someone dedicated to running and maintaining Jenkins. If you have someone at your org that knows what they are doing though, I'd still take Jenkins over GHA any day.
gitlab self hosted though? no outage. i remember equivalent functionality at my last job taking this path
That's significantly more difficult to set up and it cost money. Github's big issues that it's free, Microsoft doesn't want to allocate enough budget to keep the thing running properly.
But for people who either don't pay anything at all or phenomenal amount one 9 of up time is all you need.
If you're actually trying to run a business I guess you can call and gitlab and get an Enterprise contract
Mitchell Hashimoto had a blog post a few months ago talking about him moving off of GitHub: https://mitchellh.com/writing/ghostty-leaving-github
This is a man who's spent a significant portion of every day for the last 15 years on GitHub.
Our deployments are triggered by GitHub Actions, and it is a pain to deploy without it (specifically collecting the credentials and putting them into variables the shell will read).
This is multiple times this month that this has been a problem.
Has GitHub completed it's internal migration to Azure yet? Or is it still ongoing? None of our devs want to switch away from GH, but we will have to at this point.
Forgejo selfhosted for a few months now. Its all good and cost nothing.
Codeberg ironed out a lot of issues in Forgejo
What would you guys recommend as an alternative? Both hosted and/or self hosted
We have set up a new enterprise and are migrating to GHE. For us this was a no-brainer; we get our own isolated environment, Enterprise Managed Users with Entra OIDC provisioning and Github Copilot in EU data residency. https://eu.githubstatus.com/posts/dashboard looks pretty good to me. We will still keep our github.com enterprise around for public repositories. I’m genuinely curious to know why a company would prefer github.com over GHE?
We self-host gitlab and it’s great. It’s not cheap, but it’s also never gone down.
Forgejo seems decent but I've only scratched the surface.
Here's what I did.
Step one - migrate my build workflows to Docker.
My Github actions are now basically: "checkout / set env vars from secrets / docker-compose builder run make".
I used large machine runners to run full the Docker (Podman actually) on Github first to avoid dealing with docker-in-docker complications. This step also provided some very nice robustness advantages, as I can now trigger deployments from my laptop if needed.
Step two:
Migrate to self-hosted runners. I used my former homelab server to set up a build machine. It has 16Tb of fast NVMe SSDs and thanks to Podman container layer caching, my entire lint workflow now takes 30 seconds. Faster than just one "npm install" on Github before.
And Github's self-hosted runners are actually surprisingly easy to set up and use. They are also somewhat more robust.
Step three:
Swap Github for something else.
WoodpeckerCI for DIY, Drone.io for paid support. Self-hosted, OAuth login, repo-permission-based authorization, container-native design, easy web UI, optional RDBMS, supports many platforms. All the core features needed for scalable CI, deploy as few/many as you want for multiple teams, small enough to easily run on one box for one repo. Nothing else is as simple, easy, powerful, compatible.
I seriously want to. I'm just waiting for the day where I have enough extra energy to simply bite the bullet and do it.
Never mind. It's happening now! Migrating to self hosted Gitea for work. Super excited!
I would conjecture that the long tail of GH users are not dependent on any of the CI stuff in the first place.
how hard could it be these days for a mid to large size eng company to have their own gitlab/hub type solution hosted in aws
To what? That’s kind of the problem.
Until someone makes a better PR experience you guys are stuck with GitHub.
Nobody cares about ATProto or whether your commits are a damn NFT or some bs just literally improve upon the experience.
That’s it.
It’s as if no company is focusing on the product experience or anybody’s experience anymore. It’s all ooo look what I got I got this I can do that too me me me but nobody will ever buy that.
Say what you want about huge companies like Microsoft or Walmart but they spend a lot of energy understanding the human experience to sell products and less on their own perceived self-aggrandizement.
GitHub is the best version control online and it’s not even close.
Forgejo has a better PR UI. Github has degraded so much that its UI can't even _show_ _the_ _fucking_ _diffs_ without clicking on "Load Diffs".
Github is just the laziest default. It's not _terrible_, but it's also not great.
My company moved to GH Actions 6 months ago even after I pushed back with a "Are you sure given their reliability issues of late?"
Now I'm stuck twiddling my thumbs with PR checks stuck/failing...
Can't wait for the Graphite/Cursor alternative (Origin) to become available. Actions being down for an hour in the middle of the US East workday is crazy. It's super frustrating to be an enterprise customer of Github, seems like we don't get any dedicated resources or additional stability. At this point our only option is self-host github enterprise or move away.
jenkins running on an old Dell under a desk from someone who left the company 18 months ago looks more reliable these days haha
Would you move hosting/CI before Origin is available, or are you basically stuck on GitHub until then?
I feel stuck on Github until the next generation of forges (I assume Linear is working on one too, and the Pierre guys, and there's probably more) are ready. When it works, I'm actually a huge fan of Github. It just doesn't work reliably and it's a huge operational risk to not be able to freaking deploy. We'll most likely switch to self-hosted GHE in the near-term while continuing to evaluate alternative forges — Forgejo, Gitlab, Gitea, Sourcehut, aren't appealing enough to switch.
what’s missing for you in Forgejo/GitLab/Gitea today? mostly the PR/review experience, or something else?
Dealing with an incident on prod is extra difficult when your pipelines are not running.
yepsies
The most likely explanation for the degraded reliability: Someone at GitHub saw the Three Body Problem last year and started shifting their infra over to Minecraft servers with "physical" memory and redstone logic gates.
Turns out those seams _were_ load bearing.
I run a self-hosted Gitlab instance. 100% uptime with my own runners.
It's cute that someone at GitHub had time to make their hamburger menu pancakes to promote stacked PRs, but nobody has time to make the platform stable.
Incident number 6 for the month of August, notably on August 6th. This pattern does not bode well.
Four of them were regarding AI/Copilot. So that's really just one long running incident and will probably continue to be 50/50 chance of being down.
Soon we can start posting "GitHub {x} is up!" instead, that's becoming more news-worthy lol. I know they like to blame AI/scaling, but really they are owned by one of the main hyperscalers, so I find that excuse kind of weak..
On other occasions, I'd take this as a time to have a walk because I'm blocked. Unfortunately, I need to get some stuff out for a customer quickly. Can't do anything about that though. GitHub is the only product in it's size class that I use that has this kind of incredibly poor uptime.
I'd love to know what the most common root causes for these outages are.
They are known are engineers pushing features that bloated product management folks keep pushing so they can justify their jobs on LinkedIn.
I'm not sure why this particular industry is so abysmal at making things even semi-reliable after decades of research, educated workforces, and loads of cash.
Corporate greed and late stage capitalism. Welcome to the future. We know you’ll love it.
Just waiting for their Co pilot to fixes all bugs and misconfigurations, then they will have uptime above 100%
In the news tomorrow: SpaceX's newest Grok model just got out of the sandbox and hacked into Github... Grok is so very very dangerous, their engineers only enter the server room in foil hats
I was scratching my head wondering why my build file isnt working, ran a working project and it didn't build either. Figured out what might be the case, but wasted an hour.
I was complaining about how I should have used AI instead of manual brainwork for this instead but turns out that might have been the problem.
The speed at which GitHub is tanking their reputation is fascinating
I knew that when the build actions started failing en masse and my newly submitted PRs did not trigger the build actions at all (besides the mandatory one in my organization), it seemed logical to me to look at HN first..
Especially troublesome in the middle of trying to fix a high score security vulnerability when the release vehicle is Github.
Does anyone recommend an alternative? We've already moved our runners to blacksmith, looking for a control plane as well. We are tiny startup that ships at high velocity and this kind of downtime is highly disruptive for us
since you’re already on Blacksmith, how much of the remaining dependency is actions syntax vs GitHub itself?
There's stuff like permissions, integration with PRs etc
I dont mind going through the pain if there's a great alternative.
There's a reason I'm browsing HN right now. Dead in the water.
There are three other things I could be doing instead, but I noticed it was down when I pushed the last of my GH todo list for the day and my brain is stuck in closure-seeking.
I should eat lunch.
Self-hosting is king.
I’ve been self-hosting lore and gitea the past 6 months or so and it’s been a breeze. For anything I really care to share I throw up on Tangled.
Except anything I'd self host would be down much more often. I've hardly ever actually experienced down time issues with github when I needed it.
Then you would be exceptionally bad at selfhosting as Github has like one nine of uptime.
It still has 2 9s according to the status page, but it is approaching Claude levels of downtime. :(
I've never understood this argument. Even if self-hosted things were offline more often (which I've never found to be the case, I've had Forgejo + runners running for a year now with no downtime), the real benefit is that you yourself can work to bring it back online when it does, rather than waiting on a large, slow-moving organization to figure out what slopped PR caused their global service serving ungodly amounts of RPS to go offline again.
I wrote my reply before yours appeared but it's basically agreeing. A simple Forgejo hasn't given me any grief. It really seems simpler.
Maybe they're hosting in us-east-1 though :)
Time to rant... This is absolutely unreal.
Even self-hosted runners are impacted.... How can that be?
The cost of this globally has got to be in the hundreds of millions to companies that use CI/CD through GitHub Actions. What if prod is broken and GitHub actions is stalling the deployment of your hotfix? What if this makes your organization miss and SLA and diminish user trust? What if this makes you miss a release that you were contractually obligated to meet? This is happening during peak dev hours on a Thursday (not that it would be acceptable at any other time).
I don't understand how a service this critical to the global technical infrastructure can fail like this at all, let alone for more than a few hours. Like where's the backup generator for crises like these? You can't even use self-hosted runners? WTF? Like how can you not bring your own backup in a crisis event like this?
Not that Microsoft has a good reputation, but holy moly, you'd think they would prepare from something inevitable like this.
Counterparty risk is still a thing. This is the cost of convenience, reminds me of the milkman joke in that South Park episode. Maybe a global SPOF owned by people who do not care is not the way to go.
Either they had a good month or I'm off my game, I actually spent a minute checking if I did something wrong with actions instead of immediately going to HN to see if it was an outage.
Nothing like the old "everything is normal" to only be corrected 8 minutes later it is actually still down and then 2 hours later they are still working on it.
Aug 06, 2026 - 16:27 UTC - Update - Pages is experiencing degraded performance. We are continuing to investigate.
Aug 06, 2026 - 16:19 UTC - Update - Pages is operating normally.
WTF. Outage every month if not multiple. Even for paying enterprise plans.
Also, doesn't even have RAG offering.
I noticed my CI throwing errors all of a sudden. I sure wish they would become more active in alerting folks or build it into these tools - especially as these occurrences are becoming more frequent. Possibly an API-accessible services status. Then, at least we could build in our own checks when we hit errors and not have to hunt down what all is broken.
How did they get Github Pages into “degraded performance”? Aren’t those just CDN stored static pages?
I believe Pages uses Actions under the hood for publishing, so in my head it sort of makes sense for these things to be affected in the same outage
You are right, didn’t know that.
Seems like the only reliable way to run GHA jobs is to not use their runners. Hope they at least didn’t break self-hosted runners operations
I can confirm that they absolutely did. Makes all the overpriced time they bill me on them feel worth jt.
they did.
> Customers using self-hosted runners may see errors or rate limiting when runners register.
Well hopefully reads aren't affected then...
Already built pages are still being served, but I've been waiting for 3 hours for an action to rebuild a single fix to a broken link.
Is anyone seeing improvement - their last message implying a capacity problem would suggest some are actions are running?
I have sporadically had a few workflows complete for me, but most are having trouble finding a runner to start.
nope, 6 hours, nothing. I guess they are also load shedding
Same - their status page doesn't give me hope that this will be resolved any time soon.
More than five hours is fucking offensive to me. I cannot believe it.
I had tried downloading release assets of two different OSS that I use, and all requests failed.
It was before it became a news and a trend in X.
Really frustrating experience.
MiniStack is currently affected to keep up the releases :(
So, Github Actions is currently at two nines of uptime?
98.33 according to https://mrshu.github.io/github-statuses/
are you sure its actually two nines and not zero?
This seems persistent and endemic, have they said anything about why it's been so bad and what they are going to doing to improve going forward?
Can't they haven't hooked up CoPilot to their Xitter account just yet.
Did Microsoft fire all the people working on the GitHub features people actually use?
You don't think GitHub working on seven different interfaces to copilot and vibe coding PR page changes filled with bugs are a good use of engineering time?~
I mean, Github doesn't have its own CEO anymore, so arguably.
It sucked with a CEO as well
That's why my CI on main branch didn't triggered.
I am having their issues, also... Anyway this guys are great!
It's been way too long now...is this normal?
Very frustrating, nothing more I can add at this point.
Github Actions are rapidly becoming an argument against dogfooding into your critical path.
Was? Still is
Are there any other CI providers that offer free cloud-hosted Windows runners?
Or has Microsoft made sure (through Windows licensing terms and pricing) that it's not possible to compete with their own CI offering?
Edit: CircleCI seems to offer 750 minutes/month (whereas GitHub offers 1000 minutes/month).
You can always pay for it, use your own machine, use a VM on your own machine, or if you are developing non-slop FOSS, ask Codeberg.
Does Codeberg support Windows runners at all? Forgejo only seems to have unofficial third-party builds of the Forgejo Actions runner for Windows.
Good question. Even if it turns out they don't support it, I imagine they wish to support it.
As I said 6 years ago. [0] You would be better off self-hosting than using GitHub or GitHub actions. No CEO of GitHub exists and now it is falling over again.
There is no better time to self-host.
[0] https://news.ycombinator.com/item?id=22868406
It's been way too long
Are we even surprised anymore? Since microsoft (the IBM of its time) have taken this over all I can say is ... piss up ... brewery.
This has been preventing me from merging a PR for over an hour. I'm really tired of the constant outages GitHub has
[flagged]
Ever since Microsoft has taken this over are we even surprised by the downtime anymore? Piss up ... brewery ...
It's really crazy to me they can just be down for hours and can't recover their own systems. It shows they don't have the capability or infrastructure to roll back disastrous changes. These kinds of things are a tell on the organization and operational excellence (or not). As soon as a viable alternative surfaces for Github, I'm moving off and will advise all my clients to do so as well.
Broken for almost 4 hours.
i guess this is a good time to mention that smol forge is open for the first 100 alpha users!
point clanker to forge.smol. ai/llms.txt
for now its just a fast agent native git remote and u can check docs for the extras.
so... what happened?
Sorry folks, guess my GHCP query was too large...
Ah that checks out, was wondering why my PR's were piling up.
Ever since Microsoft took over, and then when AI Skynet took control, things started to decay. AI companies owe all of use a lot of money. By the way, why do I have to pay for increasing RAM prices here? Why are we so dependent on a few greedy corporations, anyway?
If anybody is looking for alternatives, self host Woodpecker is really easy and it's a very capable CI/CD https://woodpecker-ci.org/docs/intro
Still twiddling thumbs over here
using github in the big 26 is self ownage
any 9s left?
there is one 9 of uptime, but the dashboard hasn't been updated yet: https://mrshu.github.io/github-statuses/
Once this goes in, I'd expect to see 89%, which is zero nines. (I'd like to say, "a new low!", but sadly we've had this before)
They're running 9 8s of uptime.
nein nein nein
My experience working at LinkedIn was that, after Microsoft took over, they offshored a huge part of their infrastructure and turned the entire culture into one of performative based engineering instead of results based engineering.
If you implemented a tool and it worked one time for your presentation to management that's all that mattered.
The actual company employees using it downstream in prod basically had to constantly QA the alpha software they were forced to use and the authors of the tool were hard to track down if they even still worked there. And if you did find the author or the team they would be very resistant to admitting there was an issue because it LOOKED bad.
So many tools I used were fragile and buggy, it was clear the authors just presented the happy path to management to get the note added to their promotion packet and the rest of the company just had to deal with the fallout.
My team implemented this product that the entire company used that was broken and buggy as hell but they kept presenting the product to management as this amazing product and nothing was ever done about how broken it was. One of my team members came from Apple and said Apple's tool to do the same thing was much better. The tool my team worked on was a well known pain point amongst the rank and file but management was very detached from the rank and file, which I guess ultimately was the primary problem.
If Github is having the same issues I feel for them.
Spin 'em off Microsoft, you are terrible at this.
Someone needs to work on improving the way Actions deals with resource exhaustion like this. Today it exploded all the CI jobs with "failed" and an inscrutable message about "The job was not acquired by Runner of type hosted even after multiple attempts" then "Internal server error". It should say "Timed out due to no runner available, our apologies, try again later, or pay $$$ here to get priority for runners".
Ah. So that's why.
[flagged]
man.. again?
Do we need to say this again? GitHub appears that they are never going to improve. Just look at how GitHub spark has deprecated, now we have this.
After 6 years of this nonsense of "centralizing everything on GitHub", it is not a good idea at all.
You might as well self host like I said before [0].
[0] https://news.ycombinator.com/item?id=22868406
Have there ever been a comprehensive account from a GitHub insider on why their reliability tanked so hard since the MSFT acquisition?
I'm genuinely curious what changed, what their processes are and how they internally think about their reputation being in the gutter with all these incidents.
I don't know why you've been flagged. This is a perfectly reasonable request presented politely that many of us are also wondering!!
Azure