Experienced teams gone with layoffs. AI based attacks smarter. Going forward its going to be part of life and cost of doing business. When these disruptions hit analog things (water, power, transportation) its going to be even crazier
Are they legally required to be honest? In the past companies have hidden attacks because they fear it makes them a repeat target and it shakers the confidence of their clients.
I don’t believe that’s the case. That only applies if the breach might be materially relevant and that’s the same wording used in the SEC 4-day reporting rule. There’s even been pushback against the SEC trying to pierce the corporate veil (successful with Joe Sullivan, not with SolarWinds). I’m not so sure it’s as clear cut as you claim.
The evidence should now point to the contrary, i.e. that the code was not AI-generated. There are many people on forums like this one saying that at their (big, although not-named) big companies they haven't touched actual code when programming for the last 6 months or so, and judging by what I encounter on my search for a job right now this is what the general "spirit" of the industry is.
The problem is nobody in leadership ever gets punished for any of this. So you have quarter after quarter of cost-cutting for share buybacks and then eventually it's too much and the entire thing blows up. And then you just blame it on someone or something else.
And then things need to be fixed, often manually, and that's impossible because all the customer service staff were fired, offshored to India and/or replaced with AI chatbots. Nobody knows what they're doing because there's no training, high turnover and 3 hour wait times anyway.
And then some big firm comes in and replaces the system at the cost of hundreds of millions if not billions and it barely works at all. But it's convenient because the leadership can blame McKinsey (or whoever) while everybody is making money all the way down and that CEO, if they ever get ousted, will get another job anyway.
The desperate desire to displace, deskill and offshore every ounce of labor involved in all of this is going to be a huge problem for society at some point too.
This is mostly why I don't believe any of these outages were cyberattacks. Why cyberattack something that'll simply fall apart all on its own soon enough?
And while these all are separate incidents with different reasons, I don't think the public is going to make that distinction. I don't know what the long term consequences of this but I get the feeling that public sentiment is going to significantly change in the coming years as these issues will (IMHO) become more common.
The reckoning is not here quiet yet but it is for sure coming soon. AI is the accelerant that will tip most people over the line and into things falling apart.
I think on the long-term we will have another quality revolution but it will take time. Likely it will have to get pretty ugly first unfortunately.
Microsoft, FAA, the banking system, now this- I'm usually not one to believe in conspiracy theories, but this is starting to look really odd. I know no one has said any of these were cyberattacks, but could it be a really covert attack that is unlike something we've seen before? Some unreleased internal model going haywire?
This. At $PREVIOUS_JOB people would call in the day of a release (a saas product) and insist something broke in a new way for them only for us to dig in and see it was an existing bug that people only noticed because they knew there was a release.
It's not odd in the brave new world where any idiot with an LLM can pwn your company, and where any internal idiot with an LLM can push out a fix for that exploit that will break your company.
Part of it is recency bias. For instance after a plane crash, the media tends to report on every single aviation incident following it that otherwise wouldn’t have made the news. We’re seeing the same thing play out here in the IT space.
It seems patently obvious there are a series of cyber attacks going on. In the UK we have been told to prep food and water, which is obviously for a cyber attack, so the govt clearly knows more than they’re letting on.
They don’t want people to panic, they want to decide how and when to retaliate without it being all over the press.
It’s going to get weirder and it’s going to take a while to get through this.
> In the UK we have been told to prep food and water, which is obviously for a cyber attack,
Agreed - and we've seen cyberattacks against e.g. supermarket fridges so it's clear supply chain infrastructure is at risk.
> so the govt clearly knows more than they’re letting on.
Do they? It's no great secret that many internet-of-things/SCADA/PLC systems have very poor security, and that a lot of it gets connected to the internet anyway. It's also well known that any organisation of thousands of employees will have someone fooled by a phishing e-mail from time to time.
We also live in an age with a lot of sabre-rattling, and a lot of war - Ukraine/Russia, Israel/Palestine, America/Iran.
My pet theory: LLMs make software development go a lot faster. Their code also has fewer defects per unit shipped than human-written code does, on average and when LLMs are used properly. But since the volume is up, the absolute defect count is up too, even though the defect rate per unit of code shipped is lower.
Any thoughts on why the changes, now being made more quickly, aren't taking the systems to a more robust state? Is it just the same old "we're not paying off tech debt" complaint, now at hyperspeed, or is something about the speed creating new kinds of debt?
Most software systems aren't being made more robust because robustness in most orgs is still prioritized the same as it has always been, they're just shipping product faster. You could ship product with fewer defects at a slower pace, but that's not Moving Fast and Breaking Things™. Market forces dictate the speed at which product needs to ship, and you have to balance that with the defect rate to be competitive.
> Their code also has fewer defects per unit shipped than human-written code does
Nope, and here's why:
> when LLMs are used properly
Yeah, and when they (humans) keep failing to use something "properly" there is always "No True Scotsman" argument. Nothing new in this world, we had it before. FP, strong type systems, memory safety etc. Bugs keep coming though.
Im genuinely 50/50 on what the boring Occam's razor explanation is. Because obviously it should be "software be buggy sometimes" like duh. However, hypothetically:
- let there country A and country C
- A and C are caught in technological arms race involving technology that enables cyberattcks
- A has a much more powerful version of the technology than C so C probably does not want a direct escalatory cyberwar
- A large event occurs in which Cs apparent access to an important enabling technology for this arms race is revealed and the consequences likely involve A cutting them off
- The leaders of A and C are also scheduled to meet at a summit within days of the large event.
So we are in a period between the large event and a summit. A period where C may feel compelled to make a point/insinuate consequences before a pivotal meeting. But also if they do anything that's obviously a cyber attack and traceable it could escalate and they might not win.
If suddenly there are a series of high profile embarrassing disruptive cyber occurrences with no clear shared cause during that interval impacting As government and key industry players in their tech banking and defense sectors, well, look. Occam's razor is still probably "software be buggy." But also like they're on their way to a summit literally this week which sounds like a really obvious time to amp up the pressure a bit no?
Anthropic later today: apologiez everyone, that was our model. Just hyping our IPO in this grim post-Huang environment. Please forgive us and don't forget to invest.
The boring explanation is no one actually gives any fucks at all about reliability. They sometimes think they do and they sometimes say they do, but when the KPIs meet the roadmap, reliability reliably gets prioritized behind shipping more features faster. Fail fast. Agile. Ship the MVP and iterate. Etc. If you've read fight club or watched the movie and remember the bit about whether Ford will do a vehicle recall, they're essentially doing that but for making the system reliable. There's a very strong argument that they're actually right, because as painful and expensive as these outages are, they pale in comparison to the money that gets raked in from being the first-mover and establishing a moat.
That's not a fun thing to arrive at for people who want to build reliable and secure systems, but if that's what you actually want, get a job coding something that is actually EAL (or something similar) certified even to just level 1, because everything else is just lying to themselves and everyone else when it says it cares about reliability. No, an SLA isn't equivalent.
While we're on the subject, there was also the Jensen leak and immediate follow up of other victims.... all seemed quite odd on timing with everything else.
gutting teams with experienced people in favor of cheaper talent with AI will produce these results. It's nothing new. We saw this in the past with heavy outsourcing pushes too, the velocity of the change is just unlike we've ever seen before.
Combine that with offensive cybersecurity research now leveraging AI and you have a recipe for disaster.
I wonder if all these outages are caused by very bad security bugs (AI) being found, rushing out a fix, and there being issues in the fix or the updates surrounding it.
A lot of these systems I suspect are not built to be updated at the velocity current security patches require.
This is the tension I'm seeing at work. One side of the business saying, "update everything NOW!!!!", while the other side is saying, "don't you dare skip the 15 levels of approval and testing required to roll that patch out everywhere. You need to coordinate 300 teams to test if you want to roll this out."
Action and inaction both result in job risk, and it's also a race against the clock hoping nothing is exploited while sorting out the politics.
Yeah it's total victory for the CI devops always-be-deploying people tbh, because everything on the internet needs to be more or less instantly updatable from here on out. Traditional change control is going to seem comically quaint in a few years.
Anthropic later today: apologiez everyone, that was our model. Just hyping our IPO in this grim post-Huang environment. Please forgive us and don't forget to invest.
A friend of mine years ago said that civilization would end before GTA VI came out. I'm wondering now if Rockstar needs to push out the date indefinitely.
A lot of security researchers/tinkerers, who would occasionally stumble upon vulnerabilities in websites and report them responsibly, threw the towel long ago. They'd rather not get sucked into lawsuits for being samaritans. It left a gap wide open for ruthless LLMs/agents. I guess you can't subpoena an LLM / agent yet
so the ramp up to the holiday change freeze issues are hitting hard this year.
I wouldn't want to be travelling much this year. Southwest had their issue because, what floppies? They were quick to pivot to make a solution for their crew manifests be cloud based and well...
One airline crew issue will cascade to the others quickly.
I'm starting to think that the AI companies might've been lying about what the singularity looks like and it'll actually be a world where 90% uptime is the new norm. Thanks, I hate it.
Yes near term instability in systems that undergo large changes never improves. Cars are still as dangerous as they were in the 1950s and power plants never got scrubbers. OSes today crash as often as MS-DOS did in the 1980s.
Cars are killing fewer people than in the 1950s in America; this is not actually debatable. Even if you single out only pedestrians killed by cars, it's down from the 1950's peak.
Are they killing too many people still? Yup! Should Americans buy smaller cars? Yup! Etc.
Power plants are not being deregulated as a whole. The current US regime is hostile to any sort of regulation but that regime is on the way out starting in January.
Windows and macOS are fine. macOS remains exceptionally stable. I moved to Linux on my PC so I can't vouch for Windows as of right now, but I used 11 up until the AI screenshot cruft went in.
Experienced teams gone with layoffs. AI based attacks smarter. Going forward its going to be part of life and cost of doing business. When these disruptions hit analog things (water, power, transportation) its going to be even crazier
They're saying it's not a cyber attack, AI-based or not. It's just a good old fashioned production bug, like the last several.
Are they legally required to be honest? In the past companies have hidden attacks because they fear it makes them a repeat target and it shakers the confidence of their clients.
Yes, because of the threat of securities fraud lawsuits.
I don’t believe that’s the case. That only applies if the breach might be materially relevant and that’s the same wording used in the SEC 4-day reporting rule. There’s even been pushback against the SEC trying to pierce the corporate veil (successful with Joe Sullivan, not with SolarWinds). I’m not so sure it’s as clear cut as you claim.
They can certainly lie if they are directed to by the government. And some governments are opposed to AI regulation as a matter of national security.
Such as someone using genAI for mission-critical production code.
If you have evidence to back up your claim, please share it. Or is this baseless speculation?
The evidence should now point to the contrary, i.e. that the code was not AI-generated. There are many people on forums like this one saying that at their (big, although not-named) big companies they haven't touched actual code when programming for the last 6 months or so, and judging by what I encounter on my search for a job right now this is what the general "spirit" of the industry is.
The problem is nobody in leadership ever gets punished for any of this. So you have quarter after quarter of cost-cutting for share buybacks and then eventually it's too much and the entire thing blows up. And then you just blame it on someone or something else.
And then things need to be fixed, often manually, and that's impossible because all the customer service staff were fired, offshored to India and/or replaced with AI chatbots. Nobody knows what they're doing because there's no training, high turnover and 3 hour wait times anyway.
And then some big firm comes in and replaces the system at the cost of hundreds of millions if not billions and it barely works at all. But it's convenient because the leadership can blame McKinsey (or whoever) while everybody is making money all the way down and that CEO, if they ever get ousted, will get another job anyway.
The desperate desire to displace, deskill and offshore every ounce of labor involved in all of this is going to be a huge problem for society at some point too.
This is mostly why I don't believe any of these outages were cyberattacks. Why cyberattack something that'll simply fall apart all on its own soon enough?
And while these all are separate incidents with different reasons, I don't think the public is going to make that distinction. I don't know what the long term consequences of this but I get the feeling that public sentiment is going to significantly change in the coming years as these issues will (IMHO) become more common.
> The problem is nobody in leadership ever gets punished for any of this.
It only matters to leadership if it affects the stock price. If it affects the stock price, there's definitely punishment.
The reckoning is not here quiet yet but it is for sure coming soon. AI is the accelerant that will tip most people over the line and into things falling apart.
I think on the long-term we will have another quality revolution but it will take time. Likely it will have to get pretty ugly first unfortunately.
It's Monday evening and this week already feels like Bedlam.
Microsoft, FAA, the banking system, now this- I'm usually not one to believe in conspiracy theories, but this is starting to look really odd. I know no one has said any of these were cyberattacks, but could it be a really covert attack that is unlike something we've seen before? Some unreleased internal model going haywire?
Please, someone give the boring explanation...
I don't know the cause, but if you step back -- 13% of a single airline's departures are delayed. This isn't all that rare.
I suspect it's a good example of media bandwagoning and availability cascade.
This the stance I'd usually take if it was just 2 or 3 incidents, but with 4 major systems being completely disrupted within 4 days is really odd.
The United issue affected 13% of flights and was resolved in under 45 minutes.
This would not even be news if people weren't jumpy after last week's chaos.
This. At $PREVIOUS_JOB people would call in the day of a release (a saas product) and insist something broke in a new way for them only for us to dig in and see it was an existing bug that people only noticed because they knew there was a release.
It's not odd in the brave new world where any idiot with an LLM can pwn your company, and where any internal idiot with an LLM can push out a fix for that exploit that will break your company.
Part of it is recency bias. For instance after a plane crash, the media tends to report on every single aviation incident following it that otherwise wouldn’t have made the news. We’re seeing the same thing play out here in the IT space.
It seems patently obvious there are a series of cyber attacks going on. In the UK we have been told to prep food and water, which is obviously for a cyber attack, so the govt clearly knows more than they’re letting on.
They don’t want people to panic, they want to decide how and when to retaliate without it being all over the press.
It’s going to get weirder and it’s going to take a while to get through this.
> In the UK we have been told to prep food and water, which is obviously for a cyber attack,
Agreed - and we've seen cyberattacks against e.g. supermarket fridges so it's clear supply chain infrastructure is at risk.
> so the govt clearly knows more than they’re letting on.
Do they? It's no great secret that many internet-of-things/SCADA/PLC systems have very poor security, and that a lot of it gets connected to the internet anyway. It's also well known that any organisation of thousands of employees will have someone fooled by a phishing e-mail from time to time.
We also live in an age with a lot of sabre-rattling, and a lot of war - Ukraine/Russia, Israel/Palestine, America/Iran.
My pet theory: LLMs make software development go a lot faster. Their code also has fewer defects per unit shipped than human-written code does, on average and when LLMs are used properly. But since the volume is up, the absolute defect count is up too, even though the defect rate per unit of code shipped is lower.
Any thoughts on why the changes, now being made more quickly, aren't taking the systems to a more robust state? Is it just the same old "we're not paying off tech debt" complaint, now at hyperspeed, or is something about the speed creating new kinds of debt?
Most software systems aren't being made more robust because robustness in most orgs is still prioritized the same as it has always been, they're just shipping product faster. You could ship product with fewer defects at a slower pace, but that's not Moving Fast and Breaking Things™. Market forces dictate the speed at which product needs to ship, and you have to balance that with the defect rate to be competitive.
It’s almost code freeze time for many organisations, so people are pushing things out to close the year with something for performance review.
Also, if you flip a fair coin enough times, you’ll eventually get a streak that feels rigged or strange.
> Their code also has fewer defects per unit shipped than human-written code does
Nope, and here's why:
> when LLMs are used properly
Yeah, and when they (humans) keep failing to use something "properly" there is always "No True Scotsman" argument. Nothing new in this world, we had it before. FP, strong type systems, memory safety etc. Bugs keep coming though.
The cost and special abilities required to hack almost any system have plummeted because of AI.
As such, cyberattacks are no longer limited to people and organizations with unusual capabilities.
Airlines have major outages frequently, its a pretty fragile system
Oddly enough they didn’t used to and used to have rock solid computer systems with paper backup procedures. So did banks
Don't worry, it's gonna get worse.
Im genuinely 50/50 on what the boring Occam's razor explanation is. Because obviously it should be "software be buggy sometimes" like duh. However, hypothetically: - let there country A and country C - A and C are caught in technological arms race involving technology that enables cyberattcks - A has a much more powerful version of the technology than C so C probably does not want a direct escalatory cyberwar - A large event occurs in which Cs apparent access to an important enabling technology for this arms race is revealed and the consequences likely involve A cutting them off - The leaders of A and C are also scheduled to meet at a summit within days of the large event.
So we are in a period between the large event and a summit. A period where C may feel compelled to make a point/insinuate consequences before a pivotal meeting. But also if they do anything that's obviously a cyber attack and traceable it could escalate and they might not win.
If suddenly there are a series of high profile embarrassing disruptive cyber occurrences with no clear shared cause during that interval impacting As government and key industry players in their tech banking and defense sectors, well, look. Occam's razor is still probably "software be buggy." But also like they're on their way to a summit literally this week which sounds like a really obvious time to amp up the pressure a bit no?
Anthropic later today: apologiez everyone, that was our model. Just hyping our IPO in this grim post-Huang environment. Please forgive us and don't forget to invest.
The boring explanation is no one actually gives any fucks at all about reliability. They sometimes think they do and they sometimes say they do, but when the KPIs meet the roadmap, reliability reliably gets prioritized behind shipping more features faster. Fail fast. Agile. Ship the MVP and iterate. Etc. If you've read fight club or watched the movie and remember the bit about whether Ford will do a vehicle recall, they're essentially doing that but for making the system reliable. There's a very strong argument that they're actually right, because as painful and expensive as these outages are, they pale in comparison to the money that gets raked in from being the first-mover and establishing a moat.
That's not a fun thing to arrive at for people who want to build reliable and secure systems, but if that's what you actually want, get a job coding something that is actually EAL (or something similar) certified even to just level 1, because everything else is just lying to themselves and everyone else when it says it cares about reliability. No, an SLA isn't equivalent.
While we're on the subject, there was also the Jensen leak and immediate follow up of other victims.... all seemed quite odd on timing with everything else.
gutting teams with experienced people in favor of cheaper talent with AI will produce these results. It's nothing new. We saw this in the past with heavy outsourcing pushes too, the velocity of the change is just unlike we've ever seen before.
Combine that with offensive cybersecurity research now leveraging AI and you have a recipe for disaster.
I wonder if all these outages are caused by very bad security bugs (AI) being found, rushing out a fix, and there being issues in the fix or the updates surrounding it.
A lot of these systems I suspect are not built to be updated at the velocity current security patches require.
This is the tension I'm seeing at work. One side of the business saying, "update everything NOW!!!!", while the other side is saying, "don't you dare skip the 15 levels of approval and testing required to roll that patch out everywhere. You need to coordinate 300 teams to test if you want to roll this out."
Action and inaction both result in job risk, and it's also a race against the clock hoping nothing is exploited while sorting out the politics.
Yeah it's total victory for the CI devops always-be-deploying people tbh, because everything on the internet needs to be more or less instantly updatable from here on out. Traditional change control is going to seem comically quaint in a few years.
I've even seen issues there where the security team wants action today, but the CI process requires waiting for packages to be properly released.
Anthropic later today: apologiez everyone, that was our model. Just hyping our IPO in this grim post-Huang environment. Please forgive us and don't forget to invest.
this following the Microsoft, Claude, FAA / NWS, and Banking Big B outages
with Anthropic's IPO and GTA VI hitting the same day this week, what could possibly go wrong?
A friend of mine years ago said that civilization would end before GTA VI came out. I'm wondering now if Rockstar needs to push out the date indefinitely.
Reading this suddenly made me want to become amish
never fear, "coding is solved"
A lot of security researchers/tinkerers, who would occasionally stumble upon vulnerabilities in websites and report them responsibly, threw the towel long ago. They'd rather not get sucked into lawsuits for being samaritans. It left a gap wide open for ruthless LLMs/agents. I guess you can't subpoena an LLM / agent yet
so the ramp up to the holiday change freeze issues are hitting hard this year.
I wouldn't want to be travelling much this year. Southwest had their issue because, what floppies? They were quick to pivot to make a solution for their crew manifests be cloud based and well...
One airline crew issue will cascade to the others quickly.
All the people in the other thread saying it's all just a coincidence.
Anyone have any explanation that is not just a wild guess rooted in trend du jour?
I'm starting to think that the AI companies might've been lying about what the singularity looks like and it'll actually be a world where 90% uptime is the new norm. Thanks, I hate it.
Yes near term instability in systems that undergo large changes never improves. Cars are still as dangerous as they were in the 1950s and power plants never got scrubbers. OSes today crash as often as MS-DOS did in the 1980s.
Not sure if sarcasm?
Cars are killing more people than in the 1950s, particularly as of late in America thanks to the vehicle size arms race.
Power plants are being deregulated
Windows has completely gone to shit, MacOS following suit. Not as crash happy as DOS was but not great
Cars are killing fewer people than in the 1950s in America; this is not actually debatable. Even if you single out only pedestrians killed by cars, it's down from the 1950's peak.
https://en.wikipedia.org/wiki/Motor_vehicle_fatality_rate_in...
Are they killing too many people still? Yup! Should Americans buy smaller cars? Yup! Etc.
Power plants are not being deregulated as a whole. The current US regime is hostile to any sort of regulation but that regime is on the way out starting in January.
Windows and macOS are fine. macOS remains exceptionally stable. I moved to Linux on my PC so I can't vouch for Windows as of right now, but I used 11 up until the AI screenshot cruft went in.
All these places put out statements really quickly that it's not a cyberattack to the point that it makes me suspicious that these _are_ cyberattacks.
Saying it is a cyberattack could lead to a greater loss of people trusting your platform, so probably.
It is a cyberattack in a sense. The cyberattack of low-regulation capitalism on complex systems.