There are two aspects: first that there is currently a much higher pace of critical vulnerabilities and exploits being discovered, and second that there is a much shorter window between discovery and large-scale exploitation.
People are saying the pace will settle down once LLMs find and fix all the bugs. Okay, sure, if you want to believe that sort of thing, but that still leaves the much shorter time window, and that's not going to go away anytime soon.
If you're on the side of the industry where your job or hobby involves breaking things, this is all very exciting. But if you're on the side of the industry where your job is to prevent things from being broken, then it really sucks.
I lost a full day yesterday because some big players discovered a thing, told other big players about it, "oopsed" enough public information to let everyone else know about it, and then left everyone else to figure out what to do with almost no information to act upon.
This cycle has happened several times this year now. If you're on the short list of big hosting providers, you get advance notice of what's coming and early fixes; if you're not, sucks to be you, hope you didn't have any plans today.
For people like me, this is just yet one more way LLMs are making my life materially worse.
> But if you're on the side of the industry where your job is to prevent things from being broken, then it really sucks
Is it? In using it at an opportunity to improve our devops pipelines, software supply chains, etc. Perhaps I'm weird and enjoy strange things? Not to mention that all these security incidents will require plenty of people to fix things = more paid work for us humans going forward.
The coordinated disclosure thing will have to stop soon. Ideally, if you found a vuln from the defensive side, you'd tell all the "good guys" about it and none of the bad guys. The world doesn't work that way, so instead, the compromise was to tell a small number of important big actors who were unlikely to leak it, so the most good could be done before everyone found out.
Now the simple knowledge that there might be an exploitable memory issue in some given piece of software is enough for bad actors to point LLMs at it and find the issue in hours. That will only keep getting worse.
At some point, it's going to become a speed game. If you find a vuln, you might as well disclose to everyone immediately. Anyone who runs internet-exposed software will need an agent watching for disclosures around the clock and triaging them automatically (either by upgrading, improving filtering with some protective shell, or taking the whole service offline until humans can help) to keep up with the offensive agents watching around the clock and attacking automatically.
I'm wondering how long till we see the first exploit that goes like this:
* A bad actor finds a real, theoretically severe but not easy-to-actually-exploit vulnerability in some popular software. E.g. it's an exploit that can get you pwned 100% with RCE, but only if you are using it in configuration X on platform Y with kernel version Z on a Tuesday.
* They design a patch that closes that vulnerability but subtly opens a more dangerous one known only to them. They privately test their patch against LLM reviewers, tuning the approach until the introduced vulnerability typically escapes detection by automated review.
* They "responsibly disclose" the original exploit and helpfully share the patch. In the hurry to fix the original exploit, the patch itself gets only cursory (claudy, codexy) review. By now this has happened so many times that just applying whatever code changes ClodSed were GTG is the status quo.
* When the new vulnerability is eventually discovered, they can plausibly say "oh my agent didn't catch that", which is true. In the meantime, they use the new exploit that only they know about.
One can defend by quarantining any submitted fix, developing your own without reading it. Then only after that compare them.
In any case you don't even need malice --- sincere reporter can vibe up a fix that accidentally introduced a vulnerability, owing to their own lack of domain expertise and model errors... and then this bad fix makes it in because in the thrall of confirmation bias you fail to notice its shortcomings. Reporter also presumably tested it against their own LLM, so that will pass too.
This happened recently in a cryptocurrency project and contributed to a significant exploit.
> Now the simple knowledge that there might be an exploitable memory issue in some given piece of software is enough for bad actors to point LLMs at it and find the issue in hours.
It is all but certain there are exploitable memory issues in Chrome, Firefox, Node.js, Linux kernel, Nginx, etc.
Can we expect CVEs in a few hours? I'm sorry I ruined everyone's weekend.
You may joke, but I know someone who told Kimi to "go wild and fuzz filesystems in the Linux kernel", woke up the next morning to "26 bugs found and fixes prepared".
I understand several have been accepted since, and he stopped the fuzzers because his backlog of submissions to make was getting out of hand.
> If you find a vuln, you might as well disclose to everyone immediately.
This still doesn't work, since attackers are much more motivated and able to pay attention to such things.
If your goal is finding vulnerabilities / stealing money / whatever, you focus all your energy looking for new exploits.
The rest of the world has other things to do (listen to users, run services, whatever) and by definition cannot spend as much of their time watching every social media in the world for the chance of an announcement.
That's why anyone who has a lot to lose from a data breach had better either harden the shell of their system such that it's not vulnerable (very hard to do correctly) or automate the process of listening for and reacting to vulns.
> People are saying the pace will settle down once LLMs find and fix all the bugs
Yeah I don't buy it either. Look at github right now, it is a total slopfest with people vibing projects they don't know the first thing about. Plenty of LLM use will be delivering code full of new exploits. It will be interesting to see the frequency of these exploits being found correlated to LLM friendly/unfriendly contribution policies once the dust has settled to tell for certain though.
I have the opposite view: these vibe-coded projects are MUCH higher security than even top-talent human.
Ask a top agent today to create a to-do app and watch it deploy enterprise grade security: anti CSRF, XSS, CSP, secure cookies, HMACs, rate-limiting of logins, and all kinds of things you never heard of.
I've asked it to implement a simple OAuth and saw it classify inbound IPs for private/public subnets, and put in workarounds for very theoretical DNS re-bindings attacks.
We all know LLMs love to over-engineer stuff, they are having a field day turning your static 3 page blog into crypto-exchange grade security.
> For people like me, this is just yet one more way LLMs are making my life materially worse.
I wonder if we start to hit a point where the pace that exploits are found and deployed increases faster than they get repaired, because there is a natural higher risk appetite for the bad actors, whereas the good actors still require careful thought and prudence. As a result of this, would the internet/digital economy largely seize up and crash? That would be bad, especially for a lot of people on this site. Maybe this leads to a regression in convenience, but requires more in-person interaction in order to function in society. You can already see this pattern today emerging in higher education. Maybe, in a deeper way, this improves humanity?
> People are saying the pace will settle down once LLMs find and fix all the bugs.
Only for projects that are in a maintenance state. New features will produce new bugs, unless everyone pays for the tokens to scan every release with the top-of-the-line agents or otherwise spends more time than many do ATM to ensure that changes and new code don't introduce significant exploit paths. Things will slow down once the low-hanging fruit from the past is mostly detected and pulled, but the level of vigilance will always have to be at least a little higher than it has been until recently.
Time between updates releasing and people installing them has been an increasing problem for some time, we are just going to have to be a lot more disciplined about that. This will be a problem for places that have strict availability requirements as they might have to compromise on the testing time they spend on new patches before updating their installations, and we might have to move to having more flexible maintenance windows (the old “an hour or two once a month after Patch Tuesday” hasn't cut it for some time and certainly won't going forward) to allow faster updates.
The bigger problem, compared to the patch-availability-to-general-use time which can be reduced somewhat via automation (though that might accelerate bad-patch based attacks…), is greater constraints for the time between bug discovery and patch release: as soon as there is a rumour of an issue many will start scanning for it (as that is easier now) so the time to wide exploitation is going to be much shorter. This might lead to buggy patches that add new issues of their own.
Perhaps we'll see a lot more projects having explicit LTS releases that only get security updates or signifiant bug fixes, which will reduce the attack surface area (and fault surface area) most users, letting those with less stability concerns deal with the bleeding edge.
> For people like me, this is just yet one more way LLMs are making my life materially worse.
> New features will produce new bugs, unless everyone pays for the tokens to scan every release with the top-of-the-line agents
Insufficient. Models available to you are already cyber-gimped today and seems likely that they will be moreso in the future (see the recent K3.1 announcement for example). There will be attackers that lack such handicap.
That’s job security in the same way that a building maintenance guy gets job security from somebody filling the toilets with garbage everyday and flooding the bathrooms
Like sure it’s true that you’re more necessary now, but when you took the job you weren’t expecting to need to be elbow deep in shit every day, and you’d probably prefer to go back to the days of changing lightbulbs and vacuuming hallways
The web as we know it was never securable. LLMs made it obvious. We made a wrong turn somewhere back around JavaScript. Eventually we'll get fed up with the quick fixes and design something better. I don't blame the LLMs for this. I blame whoever made that wrong turn back in the 90's (and us, for not turning around sooner).
You're blaming JavaScript? The language that succeeded at delivering fully sandboxed rich client-side applications better than any other tech? With the worst incidents generally being 'just' various forms of fingerprinting?
Or, if you're thinking server-side JavaScript: would the web have been more secure running PHP and Python?
The language that made it normal to blindly run code that you found lying around on some page? Elsewhere we'd call that a remote code execution exploit. JavaScript just calls it another day at the office.
And it's not fully sandboxed. It betrays its user all the time by abusing unrestricted access to the network, or activating browser functionality that most of us would disable if we could.
I'm not saying it's THE bad decision, but it is the product of a generation of thinkers who were not thinking about threats the way we do now. If we were to go back in time and have the opportunity to do it differently, I think we would take that opportunity.
But we don't actually need a time machine. We can just accept some sunk costs and move forward anyhow.
Blindly running client side code (within a very carefully designed sandbox) is necessary for the modern web to operate, and people like the modern web.
We aren't going back to a page refresh every time you want to update something any time soon
Are you sure people like the modern web? It's increasingly used as a weapon against them. I don't know anybody under 20 who likes the modern web. As for the others, I think its more about nostalgia.
People will often claim to dislike something, even though they would dislike the alternative even more (and they might even be actively choosing against the alternative). Pay more attention to what people’s actions reveal than what they claim.
The tech that made the old web still exists. Everything is a GET, a form POST, and needs a page reload. You can still use tables for layout like it's 1996.
1. I'm not proposing the users go back and live in the past. I'm proposing that we, the technologists, go back to the drawing board as it existed at that time and build a different future.
2. I'm not going to feel safe if I go make a web site that works that way. I'm going to feel safe if my bank does.
> I mean, Facebook, my bank, Amazon… these can all be created with no client side code. It would be a lot of refreshes, but it can be done.
HN could too - perhaps the biggest use on HN is not refreshing when up (or down) voting something. YouTube could do something similar - refreshing when 'liking' a video or getting more comments. But would you really find those to be a better experience for you? Even on actions that HN does refresh on, such as 'Next page', or submitting a comment, HN doesn't freeze sorts, so posts and/or comments you didn't see will often invisibly end up before the new location HN sent you to (or posts/comments you already saw earlier in the list now showing up "again" later in the list).
A declarative approach would go a long way. Like, I'm not evaluating restaurants on the basis of their website quality, so why not have a restaurant browser and they can all just expose data according to some common schema. Same this with banks. Same thing with most things.
If you want a wildly interactive art experience, just load the appropriate browser for that.
I have to wonder, what is the real substantive number difference in exploits. Its in the hype train's interest to pump the CVEs in both quantity and severity so I do wonder how much of that is happening.
The time to exploit is also somewhat hypothetical. We're assuming infinite attacker resources which hasn't really been the assumption.
We see a lot more news and fear but are we seeing more actual "large scale exploitation"?
You say this like it's their fault. The only way not to be sure you won't end up hosting an application vulnerable to a 0-day is not to host anything at all.
Everything I've read indicates that yes, this is the new normal.
When Mythos was first announced, pretty much every security expert I talk to freaked directly out. A significant portion of my job now revolves around mitigating the impact of AI-generated exploits.
The rate at which we're churning out AI-generated code is going to make this worse, too, because AI-generated code is kind of notoriously terrible at security.
> because AI-generated code is kind of notoriously terrible at security.
This is simply not true, perhaps it was in the past but recent models are very capable... To the point that I trust them more than humans to write secure code. It's all about the prompting. Have a step in your development pipeline whereby an LLM scans, checks, audits your code. Make it a different model/session from the LLM that wrote the code in the first place. Acts like an independent auditor.
It very much depends on how you generate it, from prompting to test suites, but most importantly, your attitude. The extra security costs you time and money.
There are still folks, many of them, who think that time to market is everything, and anything beside it should be ignored. "It compiles? Ship it!" is a phrase from 1990s; LLMs just make this approach much easier. We will have plenty of machine-generated software with the security posture of a colander hit by a buckshot.
Its not entirely a prompt issue, but also what it was trained on.
Nowadays, the frontier is pretty decent all around, but it used to be generated php/javascript was questionable, ts was a bit better, and Rust was consistently decent.
Even just setting the max width of your README to 80 chars vs using full markdown could get you very different C code.
My take is that its pretty easy to write _much_ more secure code; use fewer dependencies and the ones you have should only do what you need them to do with no superfluous unused code compiled in. Configuring that used to be too much work, now its a single prompt.
Most software is written without giving a thought to security. Because security is a source of cost. That was the case before LLMs, I don't think it will change with LLMs.
Now I wonder: given that the LLMs are steered by the companies who own them, doesn't it mean that they can somehow make security better? It cuts both ways obviously: that a company can read and generate a large part of the code in the world is a risk.
As an open source maintainer, I can most definitely tell you that the quality of the average PR is now better. The problem I am facing is more that some people tend to "flood" and misuse AI in different ways. But their AI writes better code than they do.
In our somewhat “developer assisted” AI development it planned to use a library. We looked the library up and the GitHub page indicated it was “no longer maintained use other library Z”. When told AI agreed and compared the original to the new library (helpful), but it seemed to heavily weight GitHub stars more than active development.
This is where a human with a discerning taste and wide knowledge of the field should notice it during a review, or at least instruct their agents to check for that. If such a human is still employed in this capacity, of course.
Tell an LLM to break into its own system as part of a feed back loop, but with no context of the initial creation? It should iteratively get to 'safe' shouldn't it (subject to the tokens you want to spend)?
It is not just code, it is also deploying Software. I constantly have to correct or steer the 'ai' in conversations towards safer directions or better following even the most basic security principles. Here I feel it is still more the overambitious intern that needs to be carefully overseen, but has a lot of time and energy at hand.
Yeah they are pretty bad at it still, latest models tried to ship a basic SQL injection flaw.
Was also happy to build out an xss flawed module end-to-end.
Definitely have to keep a close eye on it.
My vibe code friends have pretty much infinite security issues.
Don't even have basic data modeling at client-server boundaries or anything.
Also had Google analytics embed that read and saved everyone's credit card details as clear text in analytics.
Seems to make pretty much infinite mistakes if you don't take care and tell it what not to do or how to build certain features.
It's been shown to be bad at secure coding _unless_ you specifically prompt models to watch out for common vulnerabilities, at which point they're actually quite good at containing the issues, though the studies here are somewhat older (eg https://arxiv.org/abs/2211.03622).
The issue then is that most people aren't actually good at keeping an eye out for security issues in code proactively, so things slip through the cracks (whether human-written or not), just at a much accelerated pace.
That study is from 2022, with at most a last update 2023? (if I read it right)
I'm not at all saying that "oh 3 weeks ago, discard", but three years ago absolutely has a "before and after".
Three years ago AI generated code could basically not be used at all, and I had to rewrite just about every line. Today I have to watch out for generated code making steps in the wrong strategic and architectural direction, and over engineering, no longer hallucinating nonexisting APIs and forgetting to check the auth token.
I still look for those kinds of mistakes, but I think citing a 2022 paper for "can AI do X" is not useful.
It just means that the safeguard we put in place over decades of software engineering with humans also need to remain in place when engineering with agents.
Having properly validated libs with separate concerns and a lot of eyeballs (and agent sensors) on the code so that it can be integrated cleanly in other products. Static code analysis for our apps. System prompts that require output to be secure, etc.
All of these practices that have been touted to be obsolete because agents can now just generate tailor made code just when and where you need, are anything but obsolete. On the contrary, they are paramount to the future of software engineering.
> Overall, we find that participants who had access to an AI assistant based on OpenAI's codex-davinci-002 model wrote significantly less secure code than those without access.
That model came out in March 2022, it's GPT-3.5 !!!
I'm sorry but it's absurd to use that as evidence for anything related to AI coding today.
You can't just hedge by saying "it's somewhat older". It's like discussing dunno a new train connection between cities and citing a paper measure steam locomotives.
Hence why I said the studies here are quite old, and I'd like to see something more recent, though I guess it'll be some time before that's done.
I think most people writing code with AI nowadays will find it's still lacking in some areas. Not because the model isn't good (definitely miles ahead than what we had access to in 2022-2023), just because it wasn't actively mentioned and so the model skipped over it. The same will apply to security.
I'd also love to see recent studies. It's frustrating how even new studies end up using models which are behind the frontier at the time of publishing.
AI is advancing faster than academia can keep up. Which in general shouldn't matter. But you have this pattern where a paper comes out with a conclusion that just doesn't hold anymore by the time it's read, but people use it to form/support opinions.
I agree that we can’t compare older models to newer ones, but how much code was generated during those months/years when the models weren’t good at secure code? How much is still running publicly today? It’s still worth considering the impact to the overall ecosystem.
Luckily we have newer models going back and fixing these things.
In terms of the set of all bugs, the AIs from 2022-2023 pale in comparison with the amount of bugs written by hand in previous half century, even if 2022 AIs were worse than the average programmer.
I know you're joking - but I think it will remain a challenge.
Quality and Security are emergent properties of software - especially when considering software-as-deployed in the real world.
And the problems with "good taste" and good architecture and data modelling - will probably leave ai code open to a lot of issues (Claude code cli issues is a great example of how bad this can get).
Having good architects in charge can help - but it requires a lot of discipline to keep the feature set small so that maintaining quality and security become feasible.
It _is_ wonderful. You're never required to adopt software engineering best practices, but eventually everyone does because either it makes our lives easier or code better.
Having tests / CI pipelines, etc are the same thing? Now you can choose to add a cheap step in your pipeline checking if e.g. some engineer did something stupid like allow SQL injections.
The floor of software quality once it gets to review is permanently raised.
Dystopian days, where you will be forced to write some syntax, that is then handed off to a an Ai tester. The tester will return it to you ready for handing the code to three frontier Ai's. The judges of code.
They approve your code, gold, if not; back to the coding machine and minus 1000 human tokens. This will be done via the implants of brain chips
Gabe Newell's Valve brain chip hits market and everyone gets one because Valve consistently gets left out of the "evil big tech" stack for some reason.
Can you explain the asymmetry you are claiming that exists where some models would be excellent at finding faults in code/systems and exploiting them often through coding the required tools to exploit them... but at the same time they would be bad at avoiding this kind of faults when coding the system itself?
Feels like you should be able to prompt the kind of exploits you want to prevent and the model should be able to make the system secured against them, no?
When you know you're looking for an exploit, it's easily verifiable and you can iterate fairly rapidly and get fast feedback, you can just have a test that you don't get a 403 back or something. LLMs, in general, seem good at problems like this.
When you're developing, there's a ton of degrees of freedom, and security encompasses only some of them. To be sure, if you set up a quickly verifiable test for every security flaw, you could probably get LLMs to satisfy them somehow, but there's a lot of those, and it's not necessarily practical.
> The rate at which we're churning out AI-generated code is going to make this worse, too, because AI-generated code is kind of notoriously terrible at security.
I think this was true in the past but AI is helping us finding exploits which have been in our code for many years. All out there by humans. We’re seeing fewer customer reported issues, and despite the pen tests getting far more rigorous, we’re passing all of them. Obviously this is one data point, but I’m hearing similar things from my friends in security and ops.
Yea most the people I know working in cyber security weren't worried about mythos in a serious way. All the people selling cyber products spread fear over it because profit.Everyone knows there are exploits laying in wait. Always has been always will be.
Most the cyber security people I know were upset once llms came out that could do basic os int and author their own emails, and mimic speech/video. That's how the majority of real losses occur. No one even talks about it. It's mind numbing hearing about the zebras and unicorns when there are 10 billion ants entering houses a year. Exotic stuff sells...
Now lots of sane people are genuinely and correctly worried that turning all engineers into 24/7 sloppers is and will continue to make the world far more exploitable. That's the real risk as far as I can tell
> The rate at which we're churning out AI-generated code is going to make this worse, too, because AI-generated code is kind of notoriously terrible at security.
This seems incongruent with what we're witnessing. Right now we're witnessing vulnerabilities that are 6 years old be exposed. That code wasn't generated by AI slop.
Security has always been about attacker/defender symmetry. So if the attacker has an LLM so does the defender. Reality is, the tax is getting higher as security tooling extracts more perceived value.
Let’s not forget the AI industry developed the technology and infrastructure for that security nightmare to exists. And no, none of that was inevitable. At every single step the AI leadership took decisions that makes it more likely that we face the current threats
> The rate at which we're churning out AI-generated code is going to make this worse
Not only this, a significant number of developers have stopped looking at code, trusting the bots. I've seen major security regressions, bot rolled auth implementations, and falsified benchmarks... just this last week, all in projects that get hyped.
> Not only this, a significant number of developers have stopped looking at code
This is what I find frustrating about the whole AI/slop debate. Developers stopped looking at code a long time ago... Well before LLMs. Take for example a typical nodejs project with its 100s of dependencies. Is a developer going to look through the layers upon layers of code running their app? Of course not, there's to much of it. We've seen supply chain attacks even this week.
Actually what I think will happen is that all those abstractions, dependencies will become irrelevant now that an LLM can just write custom code to solve the business problem. This will massively reduce the supply chain problem.
Then have LLMs dedicated to scanning codebases for security issues.
“Authentication” is extremely far from being the only relevant characteristic of security. That in fact is the whole problem with security: it’s a pervasive concern that has to be considered throughout a system, at every level.
The way the bots fundamentally work means that we can never trust their output. We have ample cases at work where bots catch human mistakes and humans catch bot mistakes. The important aspect here is not completely handing things over to the bots.
I suspect you would disregard or find fault in any benchmark I or others provide you.
Alternatively, if you want to see a benchmark so badly, you can create such a benchmark for your guess. I would imagine you would have more faith in the results of your own work.
If someone were to genuinely tell me that with the two SOTA models with the review step, severe security issues were left in the code, I would believe it.
That's not what anyone tells me though. I only hear theorizing and vague references to how AI failed people.
You're spot on that the thing I trust most is my own experience. I'm mostly working on mobile apps, my project does not have user accounts, so I couldn't tell you about authentication.
However, it is my experience that these models got very good at catching high-priority issues if you use the review step as I described.
Thus I theorize that SOTA models are likely to produce secure implementations.
"secure implementation" is vague, for example, here is an OpenCode issue (links to related PRs/follow ups) that I worked with their team through to improve security.
Our hunch is that the Ai told them this was ok to do, but it broke invariants users relied on. Perhaps it was not "hackable" in the sense I think you are getting at, but it did open a very big security hole through choices and design. The critical thinking was lacking (imo).
The bots make mistakes all the time, even through multiple rounds of reviews. Thus I have zero faith that they will produce secure code reliably 100% of the time. More Ai does not magically fix this, they go down the wrong path and double down, or work around incorrect assumptions.
Also worth noting that OpenCode Console was hacked days after the configuration regression and two PRs to change it back. Notable because it shows a pattern.
A poor design decision in OpenCode configuration handling is interesting, but not directly relevant to the question.
Assuming the question is whether not reviewing generated code is likely to create vulnerabilities despite SOTA models having audited the implementation.
Here it seems to have been a design choice, rather than an implementation mistake. I do not know what models were used and whether a review or audit has been done.
We are basically back at a vague the AI failed.
And sure, this happens, but there are some strong claims I'm reading here that are imo not evident:
- not reviewing implementation is foolish
- doing so will result in a flood of insecure software
- more AI cannot fix this
--
My arguments on each point:
- depends on risk tolerance, attack surface, error rate
- depends on actual performance of the proper setup I described on typical tasks, compared to the average engineer
- more AI can fix this, provided the audits become accurate enough to identify vulnerabilities with eg. 99.9% reliability
--
Are you sure that such accuracy for auditing is impossible, and will remain impossible in the future?
> Are you sure that such accuracy for auditing is impossible, and will remain impossible in the future?
No one can be sure either way, but I have high confidence perfectly secure code will remain out of the reach for man and/or machine, especially while they remain based on the transformer architecture.
> Was not your concern that the generated code would be significantly less secure than what an average engineer produces?
No, generally I think it possible to improve code quality with agents. The main problem I see to this is eyes free coding and compounding errors therein. With regards to major security issues, I do not think transformers can reliably find them, even when stacked, looped, or teamed. I've seen this setup still miss things or make mistakes.
> The rate at which we're churning out AI-generated code is going to make this worse, too, because AI-generated code is kind of notoriously terrible at security.
So you're saying it's good at finding exploits but bad at generating code? I can tell you don't know what you're talking about because pretty much all the exploits that are being found were exploits that were enabled by human written code. So let me correct your statement for you:
There are exploits everywhere and we humans have always been bad at writing code without exploits.
I have often whined about the quality (or lack, thereof) of today's software, but I certainly have no room to be smug. I recently turned an LLM loose on a number of mature packages that I wrote, and would have sworn were "perfect," and it found issues (thankfully, none, serious, but issues, nonetheless).
I think LLMs are a gamechanger. I was just talking to a friend of mine, last night, and he pointed out that it would have been quite difficult for a human (on their own) to find the PHP2Shell issue, but he thinks that multiple LLMs probably found it (it seems that some people knew about it before it was initially "discovered").
I think that we, as programmers, need to run LLMs on our own code -no matter what we think of them, for authoring the code- and do everything we can, to batten down the hatches.
There is tons of old software still around you can use your LLM to poke at. You'll see that it was crap too. Software has always sucked, very little of it has been formally verified or built.
I remember accidentally finding buffer overflows in Netware in school but I had no idea what that actually was at the time.
As someone relatively anti-LLM, I fully agree we should run them on our own code for a bug scan. I think the only two scenarios that make sense to avoid it is if you are ethically opposed to them AND releasing something just for fun, or if you are worried about IP issues from them training on your data AND haven't released the product yet. I have something in that second category, my plan is to run an LLM on it immediately before releasing.
Walk down the street in your local neighborhood. Most modern subdivisions have a single builder, but imagine it's an older street with a variety of homes. They all stand, they all hold off the elements well enough. They are heated, they are wired, and they are certainly comfortable enough to live in.
Now imagine you are an inspector. You enter these homes one by one and discover suboptimal electrical wiring, misaligned trimming, nonsensical handle placement, lazy tile jobs, etc.
Is it a problem? Only once it becomes one. There might be 10 things in your house right now that need fixing, but you will probably not become aware of them until they reach a critical point: a leak, a fire, a collapse.
The human body is similar, no? How many ticking health time bombs are you sitting on?
Maybe this is a reminder for you to go get that exam, haha.
My experience exactly, and also Linus Torvalds experience apparently, leading to a controversial and I think misunderstood take on AI-assisted coding.
Way too many people do the parallel LLM assistance = vibe-coding, opponents call it slop, proponents claim increased productivity. But vibe coding is just one aspect, but to me, analysis is where it shines, at least when quality matters.
We have static analyzers and sanitizers, sometimes fuzzers, which I think are essential to serious development work. Now, I think we can add LLMs to the list.
We blame AI for many evils, like a lack of understanding, code ownership, poor quality output, etc... This is the complete opposite, here the AI is not there to do the thinking for you, instead it says "here look, have your thought about this?", what to do next is up to you, you are in full control of the situation. It also makes development slower instead of faster, again, the opposite of what people claim AI does. But to me, it is the good kind of slow, as in "slow and steady wins the race".
I’d like to push back on the idea that this bug would have been quite difficult for a human to discover. I’m not much of a C programmer these days, but I’ve known longjmp to be a code smell for decades now. The reasons why were broadly covered in a 1968 CACM letter to the editor by Dijkstra that we’ve all certainly heard about. Though possibly not everyone makes the connection because if all you know of the article is its title then you can perhaps be forgiven for thinking it was only talking about one language keyword in particular.
What I’m getting at is that this PHP bug was adorned with marquee lights and sirens and a giant sign saying, “Hey, look here!”
That said, when that code was written 20 years ago they could perhaps be forgiven for this approach to handling OOM. The alternatives apparently come with a pretty decent performance penalty, and culture around memory safety wasn’t as strong then as it is now. But in 2026 there is an argument to be made that exploits are so common and damage potential is so high that we need to take the same mindset that flight control software developers have in their coding standards: some ways of doing things are dangerous enough that simply using them is a defect, even when they work exactly as intended, because they are just too hard to reason about.
The security debt is temporal and going to decline because old software is going to be patched or thrown away and new one is going to be security-scanned and tested.
I'm very positive about the change. If we don't look at it purely technical, vermin from ransomware gangs and Russian/Israeli/USA spy agencies already knew a subset of these things. That shows how vulnerable we were, actually. What's a stupid bug for one, might be life and death question for another. Before, from business side of things, there was no urgency to maintain proper security and fix things, now we have a more equalized grounds and everyone has to invest both in security assessment, sandboxing/better isolation on system level and formal verification.
That will literally improve small countries and individual sovereignty against the spies and improve the humanity in the long term.
While it's a pity for many maintainers during the transition, after some time we will have much better world where you can trust your computer regardless of who you are.
> because old software is going to be patched or thrown away and new one is going to be security-scanned and tested.
I highly disagree with this, I've worked in many environment and I can tell you some medium/big companies do not adapt fast enough, some have multiple projects, and when someone critical leaves, many of these projects remain unmaintained but "working". If you don't have deep monitoring, you won't even detect that you've been "hacked" or "infiltrated". Companies accepted that risk because the pace from CVE -> exploitation was big enough, but now with AI, exploitation at scale is easier than ever.
> Companies accepted that risk because the pace from CVE -> exploitation was big enough, but now with AI, exploitation at scale is easier than ever.
So that's what I'm saying, the business incentives are changed, there is no more room for the ignorance of the security. You either maintain it or you are out of business. Brutal period, but now everybody is forced to update, fix and improve the isolation because every script-kiddy can deploy a 0day every week. That helps to fix things from actually dangerous actors who had resources for 0days before the AI boom and used this negligence to harm people worldwide.
The US military had a big influence on it. It comes from the same people who brought you the pronunciation noo-kew-la. To quote Gene Wilder, “You know… morons.”
> Maybe we just need to accept that if you expose something to the wider internet, it will get hacked, and plan accordingly. Maybe it’s our expectations that are at fault.
This is the new playing field, but this game can't last forever. Most software is not churning through a software factory of endless patches and releases. Most software, once mature, is mostly done.
I see a world in which we all get some generous Dependabot 2.0 on large forges because leaving so much software insecure otherwise is untenable, but once the software is not iterating a billion times a week, you're not introducing new classes of bugs just sitting there.
You can have a frontier security model pentesting your program every night while you sleep and you can wake up knowing that whatever happens, you did the best you could.
Alas, our brave new LLM world is making iterating one billion releases per week all too easy.
And on the positive side, LLMs make remediation much easier too. It took me around 15 minutes and a few dimes worth of DeepSeek tokens to upgrade some gnarly, janky, exploitable PHP 7.x code to the latest fully patched version of 8.3 today. Without LLMs, I would probably have been having a serious discussion about whether we should ditch the app entirely instead.
I think this is a dated mindset with AI in the mix. Not only is AI making it easier to do big-bang rewrites which wouldnt have been done before, it is making working on brownfield codebases more feasible. It is lowering the barrier to entry for non-devs to contribute to codebases. Most software did not stagnate because there wasnt a need to grow it more, it stagnated because the friction and/or cost of change hit a critical mass.
I think we are much too early into the "new era" of software to understand what it looks like tomorrow, especially in terms of what we assume to be true today.
The caveat being that most software never matures. Maybe it's a cynical take, but virtually all software I use regularly is updated at least once a month. Right now, I'm using Chrome, and I see the "Relaunch to update" button waiting for me to press it. I see the arrow on the Ollama icon telling me I have to relaunch it to apply updates. I have Visual Studio Code running next to my browser, and two extensions need to be restarted to apply updates.
But at least somebody is maintaining that software.
That’s three examples of software that are way more complex than they should have been for the general use cases.
Yes, a lot of software do continuous development, but in a more stable manner, where they introduce some features, takes time to iron bugs out, and then release a new version. Meanwhile, the users only have to care about some small security fixes every now and then. When there’s no new features (especially when you’re not creating a kitchen sink), things can be calm for months and years.
Personally - I think most software is way more complex than it should be.
Depends what you're working on, but from what I have seen software always gravitates towards "unmaintainable mess of complexity" - and our job is to steer the system AWAY from that mess
It's complacency and having consciously or naively relied on security through obscurity for so long. Now that LLMs are a thing, such design flaws are trivial to expose even for the lay person, leading to a pretty massive awakening of how bad it all really was all this time, or what kind of stupid decisions have been made (i.e. the supermarket cooling incident).
I suspect the problem will solve itself though eventually, as the threat level of bad security practices has risen now, so companies have incentive to throw money at the problem for risk management, and LLMs may just as well assist with detecting potential exploits during the design phase. Thinking of it this way, I believe that security will actually improve a lot in the near future.
The article cites this commercial blog patting themselves on the back how they detect the exploit, yet after reading it I don't see any actual exploiting? Is scanning/probing the same as active exploitation all of a sudden?
> Who knew a regex engine was such a security issue
Anyone who has ever implemented a regex engine, I would imagine. Regex has always been a security nightmare, especially due to persistent attempts to use regexes to validate untrusted input
It already is a thing since I believe April as I follow cybersec.
Last year May, June, July counts of sec bulbs was order of magnitude lower than this year.
But idea is that it will not last forever. New code will be vuln checked by AI as well. So now we are in transition period that is awful but it won’t last.
Security folks have understood this math for years: there have always been virtually infinite vulnerabilities in production software. Better researchers find them in any repo rather rapidly. The only thing keeping folks safe has been the limit to the research against any given repo that any highly-capable researcher would devote.
As the cost of research approaches 0 via AI tooling that approaches highly capable (and now tireless) researchers, the rate of findings will increase, as will the rate of subsequent incidents.
What remains an open question is whether in-production code quality can improve sufficiently over time to return to a state of relative security.
I'm not in a position to know. But Jason Clinton seemed to approximate "18 months" as an upper bound for the crisis period:
>as bad as things have gotten in the past 4 days, they are going to get a lot worse before they get better. security leaders, never let a crisis go to waste: please go to your boards and ask for help right now. not for ai, for the basics. (we need to hold on for ~18 months.)
Yes, but to be pedantic, no, this is a short term rise in CVEs.
This is a shift in the equilibrium of difficulty to find these exploits, but there are increased defensive tools and new awareness of these issues as important. There will be a large glut of AI detected bugs, there are decades of code surface to scan and clean up, so this will last a while.
Eventually net new software will end up being written in a more secure way, and important old software will be patched or replaced, or taken offline.
People running old versions and not applying patches rigorously is also normal. Its been normal for decades, nobody likes doing maintenance so it gets forgotten or ignored.
More like the old normal. For those who are not old enough to remember 25+ years ago, we used to have weekly stories about worms etc. and companies often ignored vulnerability reports. It was a wild time, and in many ways we are better prepared to deal with it today than we were back then.
I can't help but wonder if we're going to see the OS and browser maintainers begin streamlining code and reducing complexity. Software these days almost seems like a fractal of insecurity, the only way to make the problem more tractable is to reduce attack surface.
Have there been any signs this will occur? I would have guessed the opposite would occur, that AI would mean code would instead grow even faster than before as AI makes it far faster and cheaper than before to implement new functionality.
My guess is that browsers (such as Chrome) have already been pursuing for years an approach similar to what you recommended, such as limiting flexibility in Manifest v3, so I'm not sure how much room remains?
On the other hand, I would love if Windows implemented optional sandboxing functionality for GUI apps where users could easily control what folders apps can read/write to, etc.
Big embarrassing network-exploitable bugs in major open-source projects happen sometimes, and network-scanning attack botnets have been around forever. Small shady CAs mess things up sometimes. People upload malware to PyPI and NPM all the time, and "move fast and break things" software-deployment startups get breached all the time, to the point where it doesn't even really feel like news to me when those things happen again.
It's unfortunate that a few of those events all happened at around the same time. But, like, even if that's not a coincidence and these events are actually happening more frequently, you should have been preparing for them anyway!
Yes, this is a new normal. No amount of what we historically would have thought of as "normal" preparation can prepare you for being instantly 0-dayed across one or sometimes multiple simultaneous fronts. Merely mentioning something tangentially related to a 0-day is generally enough for any random joe to "clean room" recreate the 0-day and be off to the races.
The historic embargo system is likely not sustainable and we need a shift in both approach and security posture (ie fully trustless architectures by default). Not only that, but because of the insanely amplified frequency and risk of the 0-days, you generally have to throw the baby out with the bathwater and leave yourself completely vulnerable to supply chain attacks. Because not short-circuiting taking upstream updates leaves you open to attack.
Theres shades of the old normal in this stuff, but absolutely this is the dial being turned up to 11 and I dont know if theres any right answers for small teams out there.
Yes, the PHP risks have been known, and partly assumed. I watched a group from MIT ignore sage advice on avoiding PHP but built their administration interface with it anyways. Now this device is deployed in heavy industry, in mining, in transportation, in healthcare, it's deployed everywhere and in multiple continents and in multiple huge cities. It's concerning that the recommendations we made were eshewed and they went ahead with this anyways and now we are in this situation. This group from MIT had formed a company, and they were bought by a large network device manufacturer. It's deeply deeply concerning.
That's not sage advice. That's just biased, reactionary rhetoric. PHP is just as good or bad as anything else, and it's often the best tool for the job. As with any other language, it can be implemented correctly or incorrectly.
> Two different security teams, one from California security company Calif (as part of OpenAI’s “Patch the Planet” program) and another from Israeli-U.S. cloud security company Wiz (they’re part of Google Cloud, apparently), both found these two Remote Code Execution (RCE) bugs in PHP.
Am I the only surprised that Google and OpenAI do not care to allow time to patch this before releasing it in the wild?
It is obviously true that there can only be a finite number of serious security bugs in any given software. For a given level of LLM capability, there is a finite subset of these bugs that is actually discoverable and exploitable. Once this finite subset is exhausted, the tide of new vulnerabilities will stop.
Perhaps we will see a few more tides like this, as LLM capabilities advance further. But there will be an end.
Is it obviously true? We are talking about Turing machines here and i am absolutely not an expert but i wouldn’t be surprised if there is an infinite number of potential vulnerabilities in most software.
I know there is some kind of mathematical proof that can be done to show the correctness of some software, but is that universally applicable or only in certain situations?
Would love to hear opinions on this from some one more knowledgeable in this area, as the “finite number of bugs” is a statement I’ve heard a lot lately and I’m not sure if it’s actually justified or not
Umm, what? Did we just enforce a global codefreeze in perpetuity? Or, even easier, did we just enforce a global codefreeze for the linux operating system in perpetuity? No? Ok, then how do we end up with a finite number of serious security bugs?
Yes, new bugs could be introduced when software is modified. But any modifications should be safety-tested by frontier LLMs, so the likelyhood of them going into production is low. There will be occasional security bugs slipping by, but no longer a tide like today.
For now yes. When LLMs can iterate and test at speeds humans cannot, it is going to continue to find holes.
An interesting side effect will be whether we accelerate moving towards memory safe languages (and development of more of those languages) in more sectors than just 'low-level'.
Not forgetting that this is only one mitigation of cyber-attacks and exploits and potentially removes a whole category of attacks developed by analyzing *source code*
My concern is that while we could absolutely eliminate most security bugs through LLM-assisted scanning of source code written in deterministic languages, we are shifting away from deterministic languages because of LLMs themselves. Instead of having purpose-built programs to carry out computational tasks that have a tree of logic that you can trace out and formally prove, we're expecting a black box to guess what we want done and we can't prove what it will do.
It really depends on your gut feeling about how deep the rabbit holes in terms of number of game changer vulnerabilities.
How many RCEs are in PHP of this kind? Many people would've said zero before. After the fixes, it may be zero again. It could be thousands, it could be eighty-six, it could be four, it could be zero. But even if you don't think it's zero again, most people will find themselves that there's probably not many left.
Extend this to other world ecosystem - how many high impact devastating security bugs are there? Dozens, hundreds, thousands, millions? At least when it comes to open source security, there has to be some low upper bound given how countable the number of projects there are where impact could be high and most have taken some care for their quality (despite obvious hardships). And while closed source software is harder to reason about (also on the ecosystem level), if there's a low upper bound for open source software surely the bound for closed source software can't be many orders of magnitude higher.
We now have new tools that allows us to discover these. Both the discovery as well as changes in the development process most are taken should bring these numbers closer to zero again. The main question is - are we working through a small backlog of issues, or a very deep one with no practical end? I'm still inclined to believe it's a manageable backlog, but I'm an optimist and everybody's intuitions differ on this.
There was a literal PoC video that started the panic, then multiple writeups. It's not hard (for people who know what they're doing) to exploit the PHP vuln. It's just not one exploit to rule every PHP installation anywhere, since this is a low-level exploit that needs to bypass ASLR, instead of a logic bug - so you need to tweak your exploit for every architecture, system, configuration, etc. And you need to know a fair bit about your target.
The response was overblown, but your dismissive reaction is too far in the other direction.
Hi, I wrote the blog post and wish I'd just written your exact words instead.
A lot of the lab PoCs still use contrivances (ignoring the ones that are disabling ASLR or just inserting `vulnerable.php` files on the server) that you don't really get in the real world - even the requirement of having to heap groom the same PHP worker repeatedly and sequentially is quite hard to exploit if the website has traffic. (this is why some of the slop PoCs reconfigure apache to have one worker, or pin WORKERS-1 with infinitely long requests).
This is in stark contrast to other, more severe vulns that the internet has seen (and will see in the near future) where there's effectively a blank cheque to fuck anything that is unpatched. This is not a blank cheque. It is still bad.
This doesnt matter though - luck and statistics are not serious security walls. Patch your servers.
What confuses me the most is that nobody talks about the nature on how to exploit this vulnerability.
This has been a well known vulnerability for years but considered too complicated to exploit. It is the type of work that would have taken an expert weeks to hack into a single system and the idea that an AI can speed this is up is not unfounded but currently without evidence.
Yes, the attack surface is huge but when compared to Log4J this is materially more difficult to execute and the countless PoC that have been seen were AI slop.
As a security expert I see why you would not want to downplay this, but I think the involved actors have all the incentive to use this for their typical messaging.
As a tech community we have a greater responsibility to not be driven by hype but understand what is going on and let make ourselves some tools and be swayed by the messaging and instead talk about this in a more nuanced way. Perhaps I am wrong because I am not a cyber person myself but this all strikes me as similar to the whole Specter outrage.
Bold of you to assume that AI-written and -reviewed code doesn't introduce new vulnerabilities. I think after we're done with this stretch of finding human-made vulnerabilities, we'll have a long road ahead to clean up the mess that's introduced by AI.
> Bold of you to assume that AI-written and -reviewed code doesn't introduce new vulnerabilities.
If AI can find a vulnerability in a given piece of code, it can also do so during testing, not only when used by a malicious actor. This means that the whole thing becomes an issue only if someone is writing a safety-critical program and not doing proper LLM safety testing.
Everyone should use LLMs to test the safety of their code today.
"Should" is doing a lot of heavy lifting there. Most shops are a long way off from secure coding, and from my experience in the last 10 years most devs lack basic awareness of security. I still see people being surprised by OWASP Top 10 or simple things like trust boundaries.
There's no guarantee AI will find existing vulnerabilities proactively during coding, or that it won't introduce new vulnerabilities in the process of shipping features. You're right that shops _should_ be critical and test code, but I think due to the lack of awareness people will end up implicitly trusting any first pass written by AI and not bother to double check it.
The bugs will only disappear until the model that was used to security test the generated code becomes obsoleted. So, yes, this is the new cyber normal. Unless we replace the tech stack with a formally verified one. Which will need to happen, one way or another, seems to me. But cheap & sloppy generated code won't be feasible in this paradigm, so I believe we'll need a much more thoughtful approach.
> The bugs will only disappear until the model that was used to security test the generated code becomes obsoleted.
I think you're ignoring the distinct possibility that even if the code is not formally verified, LLMs could simply stop writing bugs, or this kind of bugs. No matter how smart an attacker AI, it cannot exploit bugs that are simply not there.
But hell, even formally verified ones can only verify back to the spec. There are plenty of examples of bugs in formally verified software.
I place a higher likelihood on LLMs winning (producing fewer bugs than formally verified) over the long run. Mostly because verified software can, IIUC, only verify back to the spec. Whereas LLMs can go "uh, your spec itself has a bug, you really want me to codify that?".
> you're ignoring the distinct possibility that [...] LLMs could simply stop writing bugs
That just doesn't seem like a reasonable possibility to me, given what we're seeing from and learning about these models. They have no capacity to be aware and adhere to an overarching vision, which is why they inevitably corrode architectural coherence over time [1]. The only reason these agents are able to still create usable code is because harnesses take advantage of model's ability to find bugs, giving us the famous loop, adding duct tape on top of duct tape, ratcheting up the complexity, until the adversarial model isn't able to find any more issues. Thus, the bugs are pushed out into higher complexity dimensions, temporarily invisible until a stronger model comes along which can detect them.
> There are plenty of examples of bugs in formally verified software.
Of course. Which is why humans will always need to be involved in the software creation process. We have to specify what we want. It's true that we ourselves often don't know what we want, in fact this is a big part of what software development is about: requirements gathering and continuously recondisering and reworking them. But it would be beyond naive to imagine that software might one day know better than us humans what it is we actually want, no?
It could lead to a mess, sure. But my favourite prompt on an LLM-untouched codebase is along the lines of "find UB and other bugs. If the fix is less than 5 lines, fix it and commit it. One change per commit". Then I of course check the commit history before pushing.
My experience over the last couple of years with this type of prompt is that eventually you run out of bugs to find (meaning you reach "I see what you mean, but I disagree").
And those fixes don't cause messes.
And a new model may find something new, but not even remotely the volume seen the first time. This indicates to me that the supply of bugs is finite, and after this period of house cleaning, it should go down.
But anyway, a mess (decoherence) doesn't necessarily mean an exploitable mess.
Reading that good article, I find myself thinking that we did not need LLMs to get decoherence in complex software or systems. At least LLMs don't have personal fiefdoms, egos, and company politics adding to encouraging decoherence.
Allegedly AIs sometimes even forget to implement authentication for APIs, in the way anyone can just call them. So no, bugs will continue to exist, we've just magically got tons more supply of economical labor.
Somewhat. I take heart in the fact that people actually took the "website down now" messaging seriously and that there was a quick patch to address the issue. It'll still be an arms race between red team and blue team, just faster paced.
No. We're in a "brief" [0] transition period where a scaleable superhuman developer enters the scene and discovers a whole heap of bugs. Once the majority of code is written by similar systems then the tide of critical bugs being found will slow down.
The article does sorta say this but lets not bury the lede to deeply. It isn't realistic to expect new security vulnerabilities to be found at this rate, we're clearing out years of backlog and the maintainers haven't had time to deploy AI to do the software right the first time. Once the situation stabilises I'd personally expect very few public vulnerabilities because the expertise in how to assess code security is moving from a handful of experts to everyone.
> The NHS approach: close source everything.
Yeah good luck. I've seen how these systems work at decompiling and I expect they're probably going to be really good at fuzzing too.
> Once the majority of code is written by similar systems then the tide of critical bugs being found will slow down.
I don’t know if this necessarily follows. I repeatedly see that AI will write code, then you have it audit the code for security vulnerabilities, and it still finds them. Just because AI can find bugs doesn’t mean it will write bug-free code. AI’s know how to take shortcuts to achieve their goals, and do it all the time.
Yes, you can have another AI audit the first AI’s code, but in practice this doesn’t always happen, especially because once the human gets their hands on the finished product, they don’t always take the extra step to ensure it’s audited. This is particularly true because people seem to think “well the AI wrote it in the first place, why would it find its own bugs?”, ignoring that context is everything, and an AI being asked to find bugs is going to approach the code a lot differently than an AI tasked with adding features.
That’s not to say it’s not possible to set things up properly into multi-agent stages with QA and security as first-class phases of development… I only mean to say that it’s not automatically true that “AI can find bugs” implies “AI doesn’t produce bugs”.
> AI’s know how to take shortcuts to achieve their goals, and do it all the time.
1. Is AI code more defensive than code you'd previously write? Like maybe I'd write one line to call a function and a quick check for exit code, but AI instead writes a dozen lines to gracefully handle different possible failure cases. Is this not others' experience as well?
2. You say AI can and does take shortcuts - I guess it really comes down to whether humans or AI is more likely to take 'shortcuts'?
Is most NHS code client-side or server-side (/are they still making binaries available)? If binaries aren't being distributed, server-side code still can't be decompiled, so the exploitation likelihood may be a lot smaller, especially if they have basic rate limiting or similar in place. (Regardless, I still prefer open.)
> Even though this was found by “responsible security researchers” at “respectable companies”, by the time the security patch actually got released, it was already being exploited in the wild.
I am really not pleased with the "let's delay" strategy. IMO, you should without any delay, let everyone know of security-related issues. This "try to keep it hidden so that malicious actors can not exploit" is incredibly unfair to everyone else. Why would I want to support that? I want to know the exploit, I want it explained and then, in that order, I also expect that it can be fixed eventually. This closed-source thinking is very annoying. If one LLM can find an exploit, so can others, so what's with the delay?
The idea is that there is still an advantage even in times of LLM and the delay helps with an orchestrated fix. (E.g. not only php needed the fix, but also several docker image etc.)
But I agree it is doubtful that there is still such an advantage...
I'm a bit of an AI doomer but I'm going to disagree. I don't think this is the new mormal but there is going to be a dark transitional period.
What we're seeing is a bit of a "gold rush" where LLm advancements are coming thick and fast and they're being pointed out code bases that simply haven't had the level of scrutiny where relatively easy bugs to find have managed to survive. That's temporary. Here are my predictions:
1. Correctness and, in particular, memory safety is going to become increasingly important. I was bullish on Rust before this AI hype wave. LLMs finding and exploiting bugs has reinforced just important it is that Rust prevents as certain class of bugs that are the most prone to rmote exploitation ie certain memory safety issues.
I've long said that modern CPU architectures, out-of-order execution, hyperthreading (does that still exist? or did Intel fully kill it?), different CPU cores and compiler optimizations that it is essentially impossible for a human to write correct multithreaded code, particularly in userspace with "leaf node" code. LLMs just make that more apparent; and
2. Mature code bases for core functionality are going to become increasingly important because they've been "proven" in the wild (as much as you can prove a negative).
My sincerest hope, which is probably more wishful thinking than anything else, is that LLMs will be another nail in the coffin for C++, which is simply too complex and dragged down with legacy features and backwards compatibility to be salvageable.
> At the time of writing (23rd Oct), eMudhra has announced that it will be retiring from the certificate authority space. In unrelated news, a farmer has announced that he has shut his stable door, following reports of an escaped horse.
It will be until we (hopefully) change the way we work.
Security has been an afterthought for most of the past 30-40 years, and there are virtually no regulations requiring security, audits, or inspections before releasing software.
Now that attacks are cheap and easy, the only way to prevent these constant compromises will be to secure things before releasing them - not an afterthought, but a prerequisite. If we do finally as a society demand this, then the compromises will go away, because we won't release any new versions of software without securing it first.
This depends on having open weights with strong security scanning. If nations keep banning powerful open weight models "because they're dangerous", those nations' companies will be at more risk, because they won't have the tools to secure their own software. Countries with more permissive models will allow their own businesses to secure their stacks earlier, more people will use those businesses, they will thrive. The invisible hand of the market will choose whoever can improve first, not whoever bans things first.
There are two aspects: first that there is currently a much higher pace of critical vulnerabilities and exploits being discovered, and second that there is a much shorter window between discovery and large-scale exploitation.
People are saying the pace will settle down once LLMs find and fix all the bugs. Okay, sure, if you want to believe that sort of thing, but that still leaves the much shorter time window, and that's not going to go away anytime soon.
If you're on the side of the industry where your job or hobby involves breaking things, this is all very exciting. But if you're on the side of the industry where your job is to prevent things from being broken, then it really sucks.
I lost a full day yesterday because some big players discovered a thing, told other big players about it, "oopsed" enough public information to let everyone else know about it, and then left everyone else to figure out what to do with almost no information to act upon.
This cycle has happened several times this year now. If you're on the short list of big hosting providers, you get advance notice of what's coming and early fixes; if you're not, sucks to be you, hope you didn't have any plans today.
For people like me, this is just yet one more way LLMs are making my life materially worse.
> But if you're on the side of the industry where your job is to prevent things from being broken, then it really sucks
Is it? In using it at an opportunity to improve our devops pipelines, software supply chains, etc. Perhaps I'm weird and enjoy strange things? Not to mention that all these security incidents will require plenty of people to fix things = more paid work for us humans going forward.
The coordinated disclosure thing will have to stop soon. Ideally, if you found a vuln from the defensive side, you'd tell all the "good guys" about it and none of the bad guys. The world doesn't work that way, so instead, the compromise was to tell a small number of important big actors who were unlikely to leak it, so the most good could be done before everyone found out.
Now the simple knowledge that there might be an exploitable memory issue in some given piece of software is enough for bad actors to point LLMs at it and find the issue in hours. That will only keep getting worse.
At some point, it's going to become a speed game. If you find a vuln, you might as well disclose to everyone immediately. Anyone who runs internet-exposed software will need an agent watching for disclosures around the clock and triaging them automatically (either by upgrading, improving filtering with some protective shell, or taking the whole service offline until humans can help) to keep up with the offensive agents watching around the clock and attacking automatically.
I'm wondering how long till we see the first exploit that goes like this:
* A bad actor finds a real, theoretically severe but not easy-to-actually-exploit vulnerability in some popular software. E.g. it's an exploit that can get you pwned 100% with RCE, but only if you are using it in configuration X on platform Y with kernel version Z on a Tuesday.
* They design a patch that closes that vulnerability but subtly opens a more dangerous one known only to them. They privately test their patch against LLM reviewers, tuning the approach until the introduced vulnerability typically escapes detection by automated review.
* They "responsibly disclose" the original exploit and helpfully share the patch. In the hurry to fix the original exploit, the patch itself gets only cursory (claudy, codexy) review. By now this has happened so many times that just applying whatever code changes ClodSed were GTG is the status quo.
* When the new vulnerability is eventually discovered, they can plausibly say "oh my agent didn't catch that", which is true. In the meantime, they use the new exploit that only they know about.
One can defend by quarantining any submitted fix, developing your own without reading it. Then only after that compare them.
In any case you don't even need malice --- sincere reporter can vibe up a fix that accidentally introduced a vulnerability, owing to their own lack of domain expertise and model errors... and then this bad fix makes it in because in the thrall of confirmation bias you fail to notice its shortcomings. Reporter also presumably tested it against their own LLM, so that will pass too.
This happened recently in a cryptocurrency project and contributed to a significant exploit.
This was already a thing before AI, the term is "variant analysis". But it's been way juiced up by LLMs and is 100% a major cause for concern.
> the patch itself gets only cursory (claudy, codexy) review
FWIW Claude is actually quite good at discovering variants of existing bugs.
> Now the simple knowledge that there might be an exploitable memory issue in some given piece of software is enough for bad actors to point LLMs at it and find the issue in hours.
It is all but certain there are exploitable memory issues in Chrome, Firefox, Node.js, Linux kernel, Nginx, etc.
Can we expect CVEs in a few hours? I'm sorry I ruined everyone's weekend.
You may joke, but I know someone who told Kimi to "go wild and fuzz filesystems in the Linux kernel", woke up the next morning to "26 bugs found and fixes prepared".
I understand several have been accepted since, and he stopped the fuzzers because his backlog of submissions to make was getting out of hand.
> If you find a vuln, you might as well disclose to everyone immediately.
This still doesn't work, since attackers are much more motivated and able to pay attention to such things.
If your goal is finding vulnerabilities / stealing money / whatever, you focus all your energy looking for new exploits.
The rest of the world has other things to do (listen to users, run services, whatever) and by definition cannot spend as much of their time watching every social media in the world for the chance of an announcement.
That's why anyone who has a lot to lose from a data breach had better either harden the shell of their system such that it's not vulnerable (very hard to do correctly) or automate the process of listening for and reacting to vulns.
> or automate the process of listening for and reacting to vulns
1. This is also very hard to do correctly.
2. What automated steps would you take without any patches available to apply?
3. Potentially makes you more vulnerable to supply chain attacks
> People are saying the pace will settle down once LLMs find and fix all the bugs
Yeah I don't buy it either. Look at github right now, it is a total slopfest with people vibing projects they don't know the first thing about. Plenty of LLM use will be delivering code full of new exploits. It will be interesting to see the frequency of these exploits being found correlated to LLM friendly/unfriendly contribution policies once the dust has settled to tell for certain though.
I have the opposite view: these vibe-coded projects are MUCH higher security than even top-talent human.
Ask a top agent today to create a to-do app and watch it deploy enterprise grade security: anti CSRF, XSS, CSP, secure cookies, HMACs, rate-limiting of logins, and all kinds of things you never heard of.
I've asked it to implement a simple OAuth and saw it classify inbound IPs for private/public subnets, and put in workarounds for very theoretical DNS re-bindings attacks.
We all know LLMs love to over-engineer stuff, they are having a field day turning your static 3 page blog into crypto-exchange grade security.
That may be true, but we're talking about core library vulnerabilities that major projects use, not one-off's that nobody else cares about.
> For people like me, this is just yet one more way LLMs are making my life materially worse.
I wonder if we start to hit a point where the pace that exploits are found and deployed increases faster than they get repaired, because there is a natural higher risk appetite for the bad actors, whereas the good actors still require careful thought and prudence. As a result of this, would the internet/digital economy largely seize up and crash? That would be bad, especially for a lot of people on this site. Maybe this leads to a regression in convenience, but requires more in-person interaction in order to function in society. You can already see this pattern today emerging in higher education. Maybe, in a deeper way, this improves humanity?
> People are saying the pace will settle down once LLMs find and fix all the bugs.
Only for projects that are in a maintenance state. New features will produce new bugs, unless everyone pays for the tokens to scan every release with the top-of-the-line agents or otherwise spends more time than many do ATM to ensure that changes and new code don't introduce significant exploit paths. Things will slow down once the low-hanging fruit from the past is mostly detected and pulled, but the level of vigilance will always have to be at least a little higher than it has been until recently.
Time between updates releasing and people installing them has been an increasing problem for some time, we are just going to have to be a lot more disciplined about that. This will be a problem for places that have strict availability requirements as they might have to compromise on the testing time they spend on new patches before updating their installations, and we might have to move to having more flexible maintenance windows (the old “an hour or two once a month after Patch Tuesday” hasn't cut it for some time and certainly won't going forward) to allow faster updates.
The bigger problem, compared to the patch-availability-to-general-use time which can be reduced somewhat via automation (though that might accelerate bad-patch based attacks…), is greater constraints for the time between bug discovery and patch release: as soon as there is a rumour of an issue many will start scanning for it (as that is easier now) so the time to wide exploitation is going to be much shorter. This might lead to buggy patches that add new issues of their own.
Perhaps we'll see a lot more projects having explicit LTS releases that only get security updates or signifiant bug fixes, which will reduce the attack surface area (and fault surface area) most users, letting those with less stability concerns deal with the bleeding edge.
> For people like me, this is just yet one more way LLMs are making my life materially worse.
I'm with you there…
> New features will produce new bugs, unless everyone pays for the tokens to scan every release with the top-of-the-line agents
Insufficient. Models available to you are already cyber-gimped today and seems likely that they will be moreso in the future (see the recent K3.1 announcement for example). There will be attackers that lack such handicap.
As an industry, we also know two more aspects
1. many have underinvested in security for a long time
2. many are checking out of the development process
I generally think we can improve software with Ai tools, but I'm skeptical the current incentives will lead to this.
> But if you're on the side of the industry where your job is to prevent things from being broken, then it really sucks.
It's called job security :)
That’s job security in the same way that a building maintenance guy gets job security from somebody filling the toilets with garbage everyday and flooding the bathrooms
Like sure it’s true that you’re more necessary now, but when you took the job you weren’t expecting to need to be elbow deep in shit every day, and you’d probably prefer to go back to the days of changing lightbulbs and vacuuming hallways
I've been on software projects where more time was spent shoveling shit like this than doing real work. It's only going to get worse.
The web as we know it was never securable. LLMs made it obvious. We made a wrong turn somewhere back around JavaScript. Eventually we'll get fed up with the quick fixes and design something better. I don't blame the LLMs for this. I blame whoever made that wrong turn back in the 90's (and us, for not turning around sooner).
You're blaming JavaScript? The language that succeeded at delivering fully sandboxed rich client-side applications better than any other tech? With the worst incidents generally being 'just' various forms of fingerprinting?
Or, if you're thinking server-side JavaScript: would the web have been more secure running PHP and Python?
The language that made it normal to blindly run code that you found lying around on some page? Elsewhere we'd call that a remote code execution exploit. JavaScript just calls it another day at the office.
And it's not fully sandboxed. It betrays its user all the time by abusing unrestricted access to the network, or activating browser functionality that most of us would disable if we could.
I'm not saying it's THE bad decision, but it is the product of a generation of thinkers who were not thinking about threats the way we do now. If we were to go back in time and have the opportunity to do it differently, I think we would take that opportunity.
But we don't actually need a time machine. We can just accept some sunk costs and move forward anyhow.
To be fair, Java applets and the Flash plugin socialized that before Javascript tried to walk us back from the edge just a bit.
And before that, whatever Microsoft "macro" or embedded code variant was popular that week in all those document formats...
Blindly running client side code (within a very carefully designed sandbox) is necessary for the modern web to operate, and people like the modern web.
We aren't going back to a page refresh every time you want to update something any time soon
I would take a page refresh that takes two seconds over waiting half a minute for a page to "hydrate"
Are you sure people like the modern web? It's increasingly used as a weapon against them. I don't know anybody under 20 who likes the modern web. As for the others, I think its more about nostalgia.
> Are you sure people like the modern web?
People will often claim to dislike something, even though they would dislike the alternative even more (and they might even be actively choosing against the alternative). Pay more attention to what people’s actions reveal than what they claim.
(This happens not just in tech.)
In other news, anyone who has ever claimed to want to quit smoking is lying.
The tech that made the old web still exists. Everything is a GET, a form POST, and needs a page reload. You can still use tables for layout like it's 1996.
1. I'm not proposing the users go back and live in the past. I'm proposing that we, the technologists, go back to the drawing board as it existed at that time and build a different future.
2. I'm not going to feel safe if I go make a web site that works that way. I'm going to feel safe if my bank does.
People do? I loathe the modern web.
Give me a https://motherfuckingwebsite.com/ instead.
Perhaps the web was more secure when it wasn't running anything—just serving up static web pages?
Sure. It was also a lot less functional.
Kind of, but in practice not really. Sure, something like a photo editor would never be possible in Web 1.0.
But 99% of web applications are forms and displaying stuff. All JS accomplished is making it look prettier and enhanced tracking.
I mean, Facebook, my bank, Amazon… these can all be created with no client side code. It would be a lot of refreshes, but it can be done.
> I mean, Facebook, my bank, Amazon… these can all be created with no client side code. It would be a lot of refreshes, but it can be done.
HN could too - perhaps the biggest use on HN is not refreshing when up (or down) voting something. YouTube could do something similar - refreshing when 'liking' a video or getting more comments. But would you really find those to be a better experience for you? Even on actions that HN does refresh on, such as 'Next page', or submitting a comment, HN doesn't freeze sorts, so posts and/or comments you didn't see will often invisibly end up before the new location HN sent you to (or posts/comments you already saw earlier in the list now showing up "again" later in the list).
You can have the buttons in iframes and have that POST and refresh.
Not saying that's exactly what should be done, but an example that there may be another way than full-on generic code execution wrapping XHR.
Ah, so I guess the idea is that this page (with its ~200 comments) would have ~200 iframes added to it?
A declarative approach would go a long way. Like, I'm not evaluating restaurants on the basis of their website quality, so why not have a restaurant browser and they can all just expose data according to some common schema. Same this with banks. Same thing with most things.
If you want a wildly interactive art experience, just load the appropriate browser for that.
Because restaurants don't want businesses like DoorFash to scrape it.
I have to wonder, what is the real substantive number difference in exploits. Its in the hype train's interest to pump the CVEs in both quantity and severity so I do wonder how much of that is happening.
The time to exploit is also somewhat hypothetical. We're assuming infinite attacker resources which hasn't really been the assumption.
We see a lot more news and fear but are we seeing more actual "large scale exploitation"?
I'd love to see some numbers in that regard.
> For people like me, this is just yet one more way LLMs are making my life materially worse.
AI skynet is killing humans. We know that already though.
You were the one hosting an exploitable application.
You say this like it's their fault. The only way not to be sure you won't end up hosting an application vulnerable to a 0-day is not to host anything at all.
> first that there is currently a much higher pace of critical vulnerabilities and exploits being discovered
The slowest that will ever be.
> second that there is a much shorter window between discovery and large-scale exploitation.
The longest that will ever be.
Aren’t exponentials beautiful?
That would also be true if change was linear, or logarithmic.
Everything I've read indicates that yes, this is the new normal.
When Mythos was first announced, pretty much every security expert I talk to freaked directly out. A significant portion of my job now revolves around mitigating the impact of AI-generated exploits.
The rate at which we're churning out AI-generated code is going to make this worse, too, because AI-generated code is kind of notoriously terrible at security.
> because AI-generated code is kind of notoriously terrible at security.
This is simply not true, perhaps it was in the past but recent models are very capable... To the point that I trust them more than humans to write secure code. It's all about the prompting. Have a step in your development pipeline whereby an LLM scans, checks, audits your code. Make it a different model/session from the LLM that wrote the code in the first place. Acts like an independent auditor.
But if AI produces code with 90% fewer vulnerabilities per line, but generates 10 times as much code, the total security exposure remains the same .
_Is_ AI generated code notoriously terrible at security?
My experience with AI-code is that it's extremely defensive and good at minimizing 'obvious' mistakes.
Are we comparing to old models generating code? Or mythos-class models?
It very much depends on how you generate it, from prompting to test suites, but most importantly, your attitude. The extra security costs you time and money.
There are still folks, many of them, who think that time to market is everything, and anything beside it should be ignored. "It compiles? Ship it!" is a phrase from 1990s; LLMs just make this approach much easier. We will have plenty of machine-generated software with the security posture of a colander hit by a buckshot.
Its not entirely a prompt issue, but also what it was trained on.
Nowadays, the frontier is pretty decent all around, but it used to be generated php/javascript was questionable, ts was a bit better, and Rust was consistently decent.
Even just setting the max width of your README to 80 chars vs using full markdown could get you very different C code.
My take is that its pretty easy to write _much_ more secure code; use fewer dependencies and the ones you have should only do what you need them to do with no superfluous unused code compiled in. Configuring that used to be too much work, now its a single prompt.
Most software is written without giving a thought to security. Because security is a source of cost. That was the case before LLMs, I don't think it will change with LLMs.
Now I wonder: given that the LLMs are steered by the companies who own them, doesn't it mean that they can somehow make security better? It cuts both ways obviously: that a company can read and generate a large part of the code in the world is a risk.
As an open source maintainer, I can most definitely tell you that the quality of the average PR is now better. The problem I am facing is more that some people tend to "flood" and misuse AI in different ways. But their AI writes better code than they do.
I think that even worse is that AI might choose just to write something, and skip using a library that is well maintained
Or use an out of date library.
In our somewhat “developer assisted” AI development it planned to use a library. We looked the library up and the GitHub page indicated it was “no longer maintained use other library Z”. When told AI agreed and compared the original to the new library (helpful), but it seemed to heavily weight GitHub stars more than active development.
Tools like SCA and SBOM are still needed to ensure your stuff isn't out of date. That's not exactly a problem an LLM solves.
When a project talks about how many stars they have, I immediately mistrust them.
Usually download counters and stars are signals for VC, which indirectly serve also as negative signals to me.
This is where a human with a discerning taste and wide knowledge of the field should notice it during a review, or at least instruct their agents to check for that. If such a human is still employed in this capacity, of course.
that can sometimes be a plus considering how dependencies used by a lot of projects are themselves targets
it costs time until the new Opus7 is trained on hardened code.
Physician heal thyself?
Tell an LLM to break into its own system as part of a feed back loop, but with no context of the initial creation? It should iteratively get to 'safe' shouldn't it (subject to the tokens you want to spend)?
I’ve found they won’t always comply with adversarial review.
No doubt because us unwashed masses aren’t blessed with the super special “security” ones.
We might be looking for exploits for bad reasons, I guess.
Depends where it is blind at.
Adversarial fixing. I like it
It is not just code, it is also deploying Software. I constantly have to correct or steer the 'ai' in conversations towards safer directions or better following even the most basic security principles. Here I feel it is still more the overambitious intern that needs to be carefully overseen, but has a lot of time and energy at hand.
Yeah they are pretty bad at it still, latest models tried to ship a basic SQL injection flaw. Was also happy to build out an xss flawed module end-to-end. Definitely have to keep a close eye on it.
My vibe code friends have pretty much infinite security issues. Don't even have basic data modeling at client-server boundaries or anything. Also had Google analytics embed that read and saved everyone's credit card details as clear text in analytics.
Seems to make pretty much infinite mistakes if you don't take care and tell it what not to do or how to build certain features.
What models are people actually using to pump out code? That is the relevant comparison, not the absolute best in class models.
It's been shown to be bad at secure coding _unless_ you specifically prompt models to watch out for common vulnerabilities, at which point they're actually quite good at containing the issues, though the studies here are somewhat older (eg https://arxiv.org/abs/2211.03622).
The issue then is that most people aren't actually good at keeping an eye out for security issues in code proactively, so things slip through the cracks (whether human-written or not), just at a much accelerated pace.
That study is from 2022, with at most a last update 2023? (if I read it right)
I'm not at all saying that "oh 3 weeks ago, discard", but three years ago absolutely has a "before and after".
Three years ago AI generated code could basically not be used at all, and I had to rewrite just about every line. Today I have to watch out for generated code making steps in the wrong strategic and architectural direction, and over engineering, no longer hallucinating nonexisting APIs and forgetting to check the auth token.
I still look for those kinds of mistakes, but I think citing a 2022 paper for "can AI do X" is not useful.
It just means that the safeguard we put in place over decades of software engineering with humans also need to remain in place when engineering with agents.
Having properly validated libs with separate concerns and a lot of eyeballs (and agent sensors) on the code so that it can be integrated cleanly in other products. Static code analysis for our apps. System prompts that require output to be secure, etc.
All of these practices that have been touted to be obsolete because agents can now just generate tailor made code just when and where you need, are anything but obsolete. On the contrary, they are paramount to the future of software engineering.
from your paper:
> Overall, we find that participants who had access to an AI assistant based on OpenAI's codex-davinci-002 model wrote significantly less secure code than those without access.
That model came out in March 2022, it's GPT-3.5 !!!
I'm sorry but it's absurd to use that as evidence for anything related to AI coding today.
You can't just hedge by saying "it's somewhat older". It's like discussing dunno a new train connection between cities and citing a paper measure steam locomotives.
Hence why I said the studies here are quite old, and I'd like to see something more recent, though I guess it'll be some time before that's done.
I think most people writing code with AI nowadays will find it's still lacking in some areas. Not because the model isn't good (definitely miles ahead than what we had access to in 2022-2023), just because it wasn't actively mentioned and so the model skipped over it. The same will apply to security.
I'd also love to see recent studies. It's frustrating how even new studies end up using models which are behind the frontier at the time of publishing.
AI is advancing faster than academia can keep up. Which in general shouldn't matter. But you have this pattern where a paper comes out with a conclusion that just doesn't hold anymore by the time it's read, but people use it to form/support opinions.
I agree that we can’t compare older models to newer ones, but how much code was generated during those months/years when the models weren’t good at secure code? How much is still running publicly today? It’s still worth considering the impact to the overall ecosystem.
Luckily we have newer models going back and fixing these things.
In terms of the set of all bugs, the AIs from 2022-2023 pale in comparison with the amount of bugs written by hand in previous half century, even if 2022 AIs were worse than the average programmer.
like most software, do you assume people are driving lamborghinis, or the cheapest make and model they can find...
You just need to use AI to test your AI code.
(What a wonderful new world we live in)
I know you're joking - but I think it will remain a challenge.
Quality and Security are emergent properties of software - especially when considering software-as-deployed in the real world.
And the problems with "good taste" and good architecture and data modelling - will probably leave ai code open to a lot of issues (Claude code cli issues is a great example of how bad this can get).
Having good architects in charge can help - but it requires a lot of discipline to keep the feature set small so that maintaining quality and security become feasible.
It _is_ wonderful. You're never required to adopt software engineering best practices, but eventually everyone does because either it makes our lives easier or code better.
Having tests / CI pipelines, etc are the same thing? Now you can choose to add a cheap step in your pipeline checking if e.g. some engineer did something stupid like allow SQL injections.
The floor of software quality once it gets to review is permanently raised.
Dystopian days, where you will be forced to write some syntax, that is then handed off to a an Ai tester. The tester will return it to you ready for handing the code to three frontier Ai's. The judges of code.
They approve your code, gold, if not; back to the coding machine and minus 1000 human tokens. This will be done via the implants of brain chips
Wait, when did brain chips enter the picture?
Gabe Newell's Valve brain chip hits market and everyone gets one because Valve consistently gets left out of the "evil big tech" stack for some reason.
Elon chipped all of us when we weren't paying attention.
According to some independent thinkers, we all got them from Bill Gates’ COVID vaccines.
I really don't get the issue here. Is the rust borrow checker dystopian?
You're forced to write syntax that appeases the tester. If they are not happy you need to return and rewrite.
As software engineering matures, we're adding more linting and static checkers in codebases. This is generally considered good.
> You're forced to write syntax that appeases the tester. If they are not happy you need to return and rewrite.
that's called the write/compile/check loop
but if suddenly it's ai, people freakout at having to do this?
It was never the skilled way of doing things.
Also an AI is not a compiler, stop saying that.
Compilers transform source code in one language to another target language.
Transformer models can transform natural language to another target language.
That’s the commonality. What’s the distinction you want to make?
Can you explain the asymmetry you are claiming that exists where some models would be excellent at finding faults in code/systems and exploiting them often through coding the required tools to exploit them... but at the same time they would be bad at avoiding this kind of faults when coding the system itself?
Feels like you should be able to prompt the kind of exploits you want to prevent and the model should be able to make the system secured against them, no?
When you know you're looking for an exploit, it's easily verifiable and you can iterate fairly rapidly and get fast feedback, you can just have a test that you don't get a 403 back or something. LLMs, in general, seem good at problems like this.
When you're developing, there's a ton of degrees of freedom, and security encompasses only some of them. To be sure, if you set up a quickly verifiable test for every security flaw, you could probably get LLMs to satisfy them somehow, but there's a lot of those, and it's not necessarily practical.
> The rate at which we're churning out AI-generated code is going to make this worse, too, because AI-generated code is kind of notoriously terrible at security.
I think this was true in the past but AI is helping us finding exploits which have been in our code for many years. All out there by humans. We’re seeing fewer customer reported issues, and despite the pen tests getting far more rigorous, we’re passing all of them. Obviously this is one data point, but I’m hearing similar things from my friends in security and ops.
Yea most the people I know working in cyber security weren't worried about mythos in a serious way. All the people selling cyber products spread fear over it because profit.Everyone knows there are exploits laying in wait. Always has been always will be.
Most the cyber security people I know were upset once llms came out that could do basic os int and author their own emails, and mimic speech/video. That's how the majority of real losses occur. No one even talks about it. It's mind numbing hearing about the zebras and unicorns when there are 10 billion ants entering houses a year. Exotic stuff sells...
Now lots of sane people are genuinely and correctly worried that turning all engineers into 24/7 sloppers is and will continue to make the world far more exploitable. That's the real risk as far as I can tell
I have the same experience. Most of articles I read was reducing it to "just a tool, nothing more"
Agreed. Shouldn't continuous pen testing be a thing by now?
> The rate at which we're churning out AI-generated code is going to make this worse, too, because AI-generated code is kind of notoriously terrible at security.
This seems incongruent with what we're witnessing. Right now we're witnessing vulnerabilities that are 6 years old be exposed. That code wasn't generated by AI slop.
Security has always been about attacker/defender symmetry. So if the attacker has an LLM so does the defender. Reality is, the tax is getting higher as security tooling extracts more perceived value.
Let’s not forget the AI industry developed the technology and infrastructure for that security nightmare to exists. And no, none of that was inevitable. At every single step the AI leadership took decisions that makes it more likely that we face the current threats
The AI just accelerated the discovery. The issues have existed for years and many major versions.
> The rate at which we're churning out AI-generated code is going to make this worse
Not only this, a significant number of developers have stopped looking at code, trusting the bots. I've seen major security regressions, bot rolled auth implementations, and falsified benchmarks... just this last week, all in projects that get hyped.
Rule #2 - do not blindly trust Ai output
https://susam.net/inverse-laws-of-robotics.html
> Not only this, a significant number of developers have stopped looking at code
This is what I find frustrating about the whole AI/slop debate. Developers stopped looking at code a long time ago... Well before LLMs. Take for example a typical nodejs project with its 100s of dependencies. Is a developer going to look through the layers upon layers of code running their app? Of course not, there's to much of it. We've seen supply chain attacks even this week.
Actually what I think will happen is that all those abstractions, dependencies will become irrelevant now that an LLM can just write custom code to solve the business problem. This will massively reduce the supply chain problem.
Then have LLMs dedicated to scanning codebases for security issues.
It's not stupid if it works.
Concretely, I would like to see it benchmarked:
Have Opus 5.5 implement the authentication, with a 6.1 Sol reviewer. Out of 100 times, how often is the result insecure?
I wouldn't be surprised if it's zero times.
There is also the possibility of specifically prompting for a security audit on top, which would perhaps further lower the risk of overlooked issues.
Now one can say that due to the non-deterministic nature of the tools, there is always a non-zero chance of producing insecure implementation.
That's not really important here - obviously manual review isn't deterministic either, the risk will never be zero.
> Have Opus 5.5 implement the authentication
“Authentication” is extremely far from being the only relevant characteristic of security. That in fact is the whole problem with security: it’s a pervasive concern that has to be considered throughout a system, at every level.
The way the bots fundamentally work means that we can never trust their output. We have ample cases at work where bots catch human mistakes and humans catch bot mistakes. The important aspect here is not completely handing things over to the bots.
> I wouldn't be surprised if it's zero times.
Statements like this concern me.
If you're so concerned you can create such a benchmark and show us all how wrong my guess is.
Instead I mostly read such opinions based on active imagination and no concrete experience with using AI for such work in a setup as described.
I suspect you would disregard or find fault in any benchmark I or others provide you.
Alternatively, if you want to see a benchmark so badly, you can create such a benchmark for your guess. I would imagine you would have more faith in the results of your own work.
If someone were to genuinely tell me that with the two SOTA models with the review step, severe security issues were left in the code, I would believe it.
That's not what anyone tells me though. I only hear theorizing and vague references to how AI failed people.
You're spot on that the thing I trust most is my own experience. I'm mostly working on mobile apps, my project does not have user accounts, so I couldn't tell you about authentication.
However, it is my experience that these models got very good at catching high-priority issues if you use the review step as I described.
Thus I theorize that SOTA models are likely to produce secure implementations.
"secure implementation" is vague, for example, here is an OpenCode issue (links to related PRs/follow ups) that I worked with their team through to improve security.
Our hunch is that the Ai told them this was ok to do, but it broke invariants users relied on. Perhaps it was not "hackable" in the sense I think you are getting at, but it did open a very big security hole through choices and design. The critical thinking was lacking (imo).
https://github.com/anomalyco/opencode/pull/51337
The bots make mistakes all the time, even through multiple rounds of reviews. Thus I have zero faith that they will produce secure code reliably 100% of the time. More Ai does not magically fix this, they go down the wrong path and double down, or work around incorrect assumptions.
This is a good talk by a member of the Linux security team: https://www.youtube.com/watch?v=NnV_cWeoo5Q
Also worth noting that OpenCode Console was hacked days after the configuration regression and two PRs to change it back. Notable because it shows a pattern.
https://news.ycombinator.com/item?id=50133401
A poor design decision in OpenCode configuration handling is interesting, but not directly relevant to the question.
Assuming the question is whether not reviewing generated code is likely to create vulnerabilities despite SOTA models having audited the implementation.
Here it seems to have been a design choice, rather than an implementation mistake. I do not know what models were used and whether a review or audit has been done.
We are basically back at a vague the AI failed.
And sure, this happens, but there are some strong claims I'm reading here that are imo not evident:
- not reviewing implementation is foolish
- doing so will result in a flood of insecure software
- more AI cannot fix this
--
My arguments on each point:
- depends on risk tolerance, attack surface, error rate
- depends on actual performance of the proper setup I described on typical tasks, compared to the average engineer
- more AI can fix this, provided the audits become accurate enough to identify vulnerabilities with eg. 99.9% reliability
--
Are you sure that such accuracy for auditing is impossible, and will remain impossible in the future?
> Are you sure that such accuracy for auditing is impossible, and will remain impossible in the future?
No one can be sure either way, but I have high confidence perfectly secure code will remain out of the reach for man and/or machine, especially while they remain based on the transformer architecture.
The benchmark is hardly perfectly secure code.
Was not your concern that the generated code would be significantly less secure than what an average engineer produces?
The goal posts here basically moved toward infinity.
> Was not your concern that the generated code would be significantly less secure than what an average engineer produces?
No, generally I think it possible to improve code quality with agents. The main problem I see to this is eyes free coding and compounding errors therein. With regards to major security issues, I do not think transformers can reliably find them, even when stacked, looped, or teamed. I've seen this setup still miss things or make mistakes.
> because AI-generated code is kind of notoriously terrible at security.
and vaccines notoriously cause autism. But you might investigate why that belief is so widespread and you might not believe it that much anymore.
Can you point at something specifically that indicates this statement is true?
> The rate at which we're churning out AI-generated code is going to make this worse, too, because AI-generated code is kind of notoriously terrible at security.
So you're saying it's good at finding exploits but bad at generating code? I can tell you don't know what you're talking about because pretty much all the exploits that are being found were exploits that were enabled by human written code. So let me correct your statement for you:
There are exploits everywhere and we humans have always been bad at writing code without exploits.
"New normal"? Is that phrase back again? It was heavily pushed in 2020.
Talking about the new normal is the new normal.
I have often whined about the quality (or lack, thereof) of today's software, but I certainly have no room to be smug. I recently turned an LLM loose on a number of mature packages that I wrote, and would have sworn were "perfect," and it found issues (thankfully, none, serious, but issues, nonetheless).
I think LLMs are a gamechanger. I was just talking to a friend of mine, last night, and he pointed out that it would have been quite difficult for a human (on their own) to find the PHP2Shell issue, but he thinks that multiple LLMs probably found it (it seems that some people knew about it before it was initially "discovered").
I think that we, as programmers, need to run LLMs on our own code -no matter what we think of them, for authoring the code- and do everything we can, to batten down the hatches.
>of today's software
There is tons of old software still around you can use your LLM to poke at. You'll see that it was crap too. Software has always sucked, very little of it has been formally verified or built.
I remember accidentally finding buffer overflows in Netware in school but I had no idea what that actually was at the time.
As someone relatively anti-LLM, I fully agree we should run them on our own code for a bug scan. I think the only two scenarios that make sense to avoid it is if you are ethically opposed to them AND releasing something just for fun, or if you are worried about IP issues from them training on your data AND haven't released the product yet. I have something in that second category, my plan is to run an LLM on it immediately before releasing.
Walk down the street in your local neighborhood. Most modern subdivisions have a single builder, but imagine it's an older street with a variety of homes. They all stand, they all hold off the elements well enough. They are heated, they are wired, and they are certainly comfortable enough to live in.
Now imagine you are an inspector. You enter these homes one by one and discover suboptimal electrical wiring, misaligned trimming, nonsensical handle placement, lazy tile jobs, etc.
Is it a problem? Only once it becomes one. There might be 10 things in your house right now that need fixing, but you will probably not become aware of them until they reach a critical point: a leak, a fire, a collapse.
The human body is similar, no? How many ticking health time bombs are you sitting on?
Maybe this is a reminder for you to go get that exam, haha.
My experience exactly, and also Linus Torvalds experience apparently, leading to a controversial and I think misunderstood take on AI-assisted coding.
Way too many people do the parallel LLM assistance = vibe-coding, opponents call it slop, proponents claim increased productivity. But vibe coding is just one aspect, but to me, analysis is where it shines, at least when quality matters.
We have static analyzers and sanitizers, sometimes fuzzers, which I think are essential to serious development work. Now, I think we can add LLMs to the list.
We blame AI for many evils, like a lack of understanding, code ownership, poor quality output, etc... This is the complete opposite, here the AI is not there to do the thinking for you, instead it says "here look, have your thought about this?", what to do next is up to you, you are in full control of the situation. It also makes development slower instead of faster, again, the opposite of what people claim AI does. But to me, it is the good kind of slow, as in "slow and steady wins the race".
I’d like to push back on the idea that this bug would have been quite difficult for a human to discover. I’m not much of a C programmer these days, but I’ve known longjmp to be a code smell for decades now. The reasons why were broadly covered in a 1968 CACM letter to the editor by Dijkstra that we’ve all certainly heard about. Though possibly not everyone makes the connection because if all you know of the article is its title then you can perhaps be forgiven for thinking it was only talking about one language keyword in particular.
What I’m getting at is that this PHP bug was adorned with marquee lights and sirens and a giant sign saying, “Hey, look here!”
That said, when that code was written 20 years ago they could perhaps be forgiven for this approach to handling OOM. The alternatives apparently come with a pretty decent performance penalty, and culture around memory safety wasn’t as strong then as it is now. But in 2026 there is an argument to be made that exploits are so common and damage potential is so high that we need to take the same mindset that flight control software developers have in their coding standards: some ways of doing things are dangerous enough that simply using them is a defect, even when they work exactly as intended, because they are just too hard to reason about.
The security debt is temporal and going to decline because old software is going to be patched or thrown away and new one is going to be security-scanned and tested.
I'm very positive about the change. If we don't look at it purely technical, vermin from ransomware gangs and Russian/Israeli/USA spy agencies already knew a subset of these things. That shows how vulnerable we were, actually. What's a stupid bug for one, might be life and death question for another. Before, from business side of things, there was no urgency to maintain proper security and fix things, now we have a more equalized grounds and everyone has to invest both in security assessment, sandboxing/better isolation on system level and formal verification.
That will literally improve small countries and individual sovereignty against the spies and improve the humanity in the long term.
While it's a pity for many maintainers during the transition, after some time we will have much better world where you can trust your computer regardless of who you are.
> because old software is going to be patched or thrown away and new one is going to be security-scanned and tested.
I highly disagree with this, I've worked in many environment and I can tell you some medium/big companies do not adapt fast enough, some have multiple projects, and when someone critical leaves, many of these projects remain unmaintained but "working". If you don't have deep monitoring, you won't even detect that you've been "hacked" or "infiltrated". Companies accepted that risk because the pace from CVE -> exploitation was big enough, but now with AI, exploitation at scale is easier than ever.
> Companies accepted that risk because the pace from CVE -> exploitation was big enough, but now with AI, exploitation at scale is easier than ever.
So that's what I'm saying, the business incentives are changed, there is no more room for the ignorance of the security. You either maintain it or you are out of business. Brutal period, but now everybody is forced to update, fix and improve the isolation because every script-kiddy can deploy a 0day every week. That helps to fix things from actually dangerous actors who had resources for 0days before the AI boom and used this negligence to harm people worldwide.
When did we start to shorten cyber security with just cyber? I see this quite often now and I think it's just stupid, it doesn't make sense.
The observed shift was noted by Ed Felton in 2008 at Princeton [0]:
> Over time, “cybersecurity” somehow became “cyber security” and then just “cyber”.
[0] https://blog.citp.princeton.edu/2008/07/24/whats-cyber-cyber...
Make cyber not cyber.
The US military had a big influence on it. It comes from the same people who brought you the pronunciation noo-kew-la. To quote Gene Wilder, “You know… morons.”
> Maybe we just need to accept that if you expose something to the wider internet, it will get hacked, and plan accordingly. Maybe it’s our expectations that are at fault.
It's this.
This is the new playing field, but this game can't last forever. Most software is not churning through a software factory of endless patches and releases. Most software, once mature, is mostly done.
I see a world in which we all get some generous Dependabot 2.0 on large forges because leaving so much software insecure otherwise is untenable, but once the software is not iterating a billion times a week, you're not introducing new classes of bugs just sitting there.
You can have a frontier security model pentesting your program every night while you sleep and you can wake up knowing that whatever happens, you did the best you could.
Alas, our brave new LLM world is making iterating one billion releases per week all too easy.
And on the positive side, LLMs make remediation much easier too. It took me around 15 minutes and a few dimes worth of DeepSeek tokens to upgrade some gnarly, janky, exploitable PHP 7.x code to the latest fully patched version of 8.3 today. Without LLMs, I would probably have been having a serious discussion about whether we should ditch the app entirely instead.
> Most software, once mature, is mostly done.
I think this is a dated mindset with AI in the mix. Not only is AI making it easier to do big-bang rewrites which wouldnt have been done before, it is making working on brownfield codebases more feasible. It is lowering the barrier to entry for non-devs to contribute to codebases. Most software did not stagnate because there wasnt a need to grow it more, it stagnated because the friction and/or cost of change hit a critical mass.
I think we are much too early into the "new era" of software to understand what it looks like tomorrow, especially in terms of what we assume to be true today.
> Most software, once mature, is mostly done.
The caveat being that most software never matures. Maybe it's a cynical take, but virtually all software I use regularly is updated at least once a month. Right now, I'm using Chrome, and I see the "Relaunch to update" button waiting for me to press it. I see the arrow on the Ollama icon telling me I have to relaunch it to apply updates. I have Visual Studio Code running next to my browser, and two extensions need to be restarted to apply updates.
But at least somebody is maintaining that software.
That’s three examples of software that are way more complex than they should have been for the general use cases.
Yes, a lot of software do continuous development, but in a more stable manner, where they introduce some features, takes time to iron bugs out, and then release a new version. Meanwhile, the users only have to care about some small security fixes every now and then. When there’s no new features (especially when you’re not creating a kitchen sink), things can be calm for months and years.
Personally - I think most software is way more complex than it should be.
Depends what you're working on, but from what I have seen software always gravitates towards "unmaintainable mess of complexity" - and our job is to steer the system AWAY from that mess
It's complacency and having consciously or naively relied on security through obscurity for so long. Now that LLMs are a thing, such design flaws are trivial to expose even for the lay person, leading to a pretty massive awakening of how bad it all really was all this time, or what kind of stupid decisions have been made (i.e. the supermarket cooling incident).
I suspect the problem will solve itself though eventually, as the threat level of bad security practices has risen now, so companies have incentive to throw money at the problem for risk management, and LLMs may just as well assist with detecting potential exploits during the design phase. Thinking of it this way, I believe that security will actually improve a lot in the near future.
> already being exploited in the wild [0]
The article cites this commercial blog patting themselves on the back how they detect the exploit, yet after reading it I don't see any actual exploiting? Is scanning/probing the same as active exploitation all of a sudden?
[0]: https://ellio.tech/en/blog/php2shell-what-we-observed-and-wh...
> Who knew a regex engine was such a security issue
Anyone who has ever implemented a regex engine, I would imagine. Regex has always been a security nightmare, especially due to persistent attempts to use regexes to validate untrusted input
> especially due to persistent attempts to use regexes to validate untrusted input
This is not the case here, it's an untrusted regex which can be the issue.
Meanwhile, your average enterprise app team has like a 30 day SLA for remediating critical vulns, after which they become overdue.
Unfortunately speaking from experience in AppSec
Not new normal.
It already is a thing since I believe April as I follow cybersec.
Last year May, June, July counts of sec bulbs was order of magnitude lower than this year.
But idea is that it will not last forever. New code will be vuln checked by AI as well. So now we are in transition period that is awful but it won’t last.
We have 30 years of sins to pay for, it will sure feel like forever
Security folks have understood this math for years: there have always been virtually infinite vulnerabilities in production software. Better researchers find them in any repo rather rapidly. The only thing keeping folks safe has been the limit to the research against any given repo that any highly-capable researcher would devote.
As the cost of research approaches 0 via AI tooling that approaches highly capable (and now tireless) researchers, the rate of findings will increase, as will the rate of subsequent incidents.
What remains an open question is whether in-production code quality can improve sufficiently over time to return to a state of relative security.
I'm not in a position to know. But Jason Clinton seemed to approximate "18 months" as an upper bound for the crisis period:
>as bad as things have gotten in the past 4 days, they are going to get a lot worse before they get better. security leaders, never let a crisis go to waste: please go to your boards and ask for help right now. not for ai, for the basics. (we need to hold on for ~18 months.)
https://x.com/JasonDClinton/status/2113325263157395456
Yes, but to be pedantic, no, this is a short term rise in CVEs.
This is a shift in the equilibrium of difficulty to find these exploits, but there are increased defensive tools and new awareness of these issues as important. There will be a large glut of AI detected bugs, there are decades of code surface to scan and clean up, so this will last a while.
Eventually net new software will end up being written in a more secure way, and important old software will be patched or replaced, or taken offline.
People running old versions and not applying patches rigorously is also normal. Its been normal for decades, nobody likes doing maintenance so it gets forgotten or ignored.
I can tell you, if it is, I'm going to be radically changing the software architectures and patterns that I am recommending to people.
Data privacy and security must not be sacrificed.
A shift is needed towards simplification and minimization of attack surfaces.
I've entirely removed several dependencies which provide some pretty nice features, but unfortunately make things almost unmaintainable.
I want to go much further though to simplify architectures themselves even if it makes far worse ux.
yes.
Not profit.Hey, but all of us who've been complaining for years about unfixed security issues, here we go!
More like the old normal. For those who are not old enough to remember 25+ years ago, we used to have weekly stories about worms etc. and companies often ignored vulnerability reports. It was a wild time, and in many ways we are better prepared to deal with it today than we were back then.
Missing from the article: Node2Shell https://www.imperva.com/blog/node2shell-from-brotli-data-to-...
I can't help but wonder if we're going to see the OS and browser maintainers begin streamlining code and reducing complexity. Software these days almost seems like a fractal of insecurity, the only way to make the problem more tractable is to reduce attack surface.
Have there been any signs this will occur? I would have guessed the opposite would occur, that AI would mean code would instead grow even faster than before as AI makes it far faster and cheaper than before to implement new functionality.
My guess is that browsers (such as Chrome) have already been pursuing for years an approach similar to what you recommended, such as limiting flexibility in Manifest v3, so I'm not sure how much room remains?
On the other hand, I would love if Windows implemented optional sandboxing functionality for GUI apps where users could easily control what folders apps can read/write to, etc.
Is this the new normal, or just, uh, normal?
Big embarrassing network-exploitable bugs in major open-source projects happen sometimes, and network-scanning attack botnets have been around forever. Small shady CAs mess things up sometimes. People upload malware to PyPI and NPM all the time, and "move fast and break things" software-deployment startups get breached all the time, to the point where it doesn't even really feel like news to me when those things happen again.
It's unfortunate that a few of those events all happened at around the same time. But, like, even if that's not a coincidence and these events are actually happening more frequently, you should have been preparing for them anyway!
Yes, this is a new normal. No amount of what we historically would have thought of as "normal" preparation can prepare you for being instantly 0-dayed across one or sometimes multiple simultaneous fronts. Merely mentioning something tangentially related to a 0-day is generally enough for any random joe to "clean room" recreate the 0-day and be off to the races.
The historic embargo system is likely not sustainable and we need a shift in both approach and security posture (ie fully trustless architectures by default). Not only that, but because of the insanely amplified frequency and risk of the 0-days, you generally have to throw the baby out with the bathwater and leave yourself completely vulnerable to supply chain attacks. Because not short-circuiting taking upstream updates leaves you open to attack.
Theres shades of the old normal in this stuff, but absolutely this is the dial being turned up to 11 and I dont know if theres any right answers for small teams out there.
Yes, the PHP risks have been known, and partly assumed. I watched a group from MIT ignore sage advice on avoiding PHP but built their administration interface with it anyways. Now this device is deployed in heavy industry, in mining, in transportation, in healthcare, it's deployed everywhere and in multiple continents and in multiple huge cities. It's concerning that the recommendations we made were eshewed and they went ahead with this anyways and now we are in this situation. This group from MIT had formed a company, and they were bought by a large network device manufacturer. It's deeply deeply concerning.
> sage advice on avoiding PHP
That's not sage advice. That's just biased, reactionary rhetoric. PHP is just as good or bad as anything else, and it's often the best tool for the job. As with any other language, it can be implemented correctly or incorrectly.
PHP is a perfectly fine language. It’s memory-safe, garbage collected, and high-level. It’s just as reasonable of a choice as C# or Java.
The PHP runtime has had very little vulnerabilities. Much less than Node.
> Two different security teams, one from California security company Calif (as part of OpenAI’s “Patch the Planet” program) and another from Israeli-U.S. cloud security company Wiz (they’re part of Google Cloud, apparently), both found these two Remote Code Execution (RCE) bugs in PHP.
Am I the only surprised that Google and OpenAI do not care to allow time to patch this before releasing it in the wild?
This situation is temporary.
It is obviously true that there can only be a finite number of serious security bugs in any given software. For a given level of LLM capability, there is a finite subset of these bugs that is actually discoverable and exploitable. Once this finite subset is exhausted, the tide of new vulnerabilities will stop.
Perhaps we will see a few more tides like this, as LLM capabilities advance further. But there will be an end.
Is it obviously true? We are talking about Turing machines here and i am absolutely not an expert but i wouldn’t be surprised if there is an infinite number of potential vulnerabilities in most software.
I know there is some kind of mathematical proof that can be done to show the correctness of some software, but is that universally applicable or only in certain situations?
Would love to hear opinions on this from some one more knowledgeable in this area, as the “finite number of bugs” is a statement I’ve heard a lot lately and I’m not sure if it’s actually justified or not
Umm, what? Did we just enforce a global codefreeze in perpetuity? Or, even easier, did we just enforce a global codefreeze for the linux operating system in perpetuity? No? Ok, then how do we end up with a finite number of serious security bugs?
Yes, new bugs could be introduced when software is modified. But any modifications should be safety-tested by frontier LLMs, so the likelyhood of them going into production is low. There will be occasional security bugs slipping by, but no longer a tide like today.
For now yes. When LLMs can iterate and test at speeds humans cannot, it is going to continue to find holes.
An interesting side effect will be whether we accelerate moving towards memory safe languages (and development of more of those languages) in more sectors than just 'low-level'.
Not forgetting that this is only one mitigation of cyber-attacks and exploits and potentially removes a whole category of attacks developed by analyzing *source code*
My concern is that while we could absolutely eliminate most security bugs through LLM-assisted scanning of source code written in deterministic languages, we are shifting away from deterministic languages because of LLMs themselves. Instead of having purpose-built programs to carry out computational tasks that have a tree of logic that you can trace out and formally prove, we're expecting a black box to guess what we want done and we can't prove what it will do.
It really depends on your gut feeling about how deep the rabbit holes in terms of number of game changer vulnerabilities.
How many RCEs are in PHP of this kind? Many people would've said zero before. After the fixes, it may be zero again. It could be thousands, it could be eighty-six, it could be four, it could be zero. But even if you don't think it's zero again, most people will find themselves that there's probably not many left.
Extend this to other world ecosystem - how many high impact devastating security bugs are there? Dozens, hundreds, thousands, millions? At least when it comes to open source security, there has to be some low upper bound given how countable the number of projects there are where impact could be high and most have taken some care for their quality (despite obvious hardships). And while closed source software is harder to reason about (also on the ecosystem level), if there's a low upper bound for open source software surely the bound for closed source software can't be many orders of magnitude higher.
We now have new tools that allows us to discover these. Both the discovery as well as changes in the development process most are taken should bring these numbers closer to zero again. The main question is - are we working through a small backlog of issues, or a very deep one with no practical end? I'm still inclined to believe it's a manageable backlog, but I'm an optimist and everybody's intuitions differ on this.
No. This is just the start. It's going to get way worse before it gets better.
It’s mostly hype. Days in now and nobody can exploit the PHP vuln. Yawn.
There was a literal PoC video that started the panic, then multiple writeups. It's not hard (for people who know what they're doing) to exploit the PHP vuln. It's just not one exploit to rule every PHP installation anywhere, since this is a low-level exploit that needs to bypass ASLR, instead of a logic bug - so you need to tweak your exploit for every architecture, system, configuration, etc. And you need to know a fair bit about your target.
The response was overblown, but your dismissive reaction is too far in the other direction.
It had an anime dancing girl on it, must be serious.
Hi, I wrote the blog post and wish I'd just written your exact words instead.
A lot of the lab PoCs still use contrivances (ignoring the ones that are disabling ASLR or just inserting `vulnerable.php` files on the server) that you don't really get in the real world - even the requirement of having to heap groom the same PHP worker repeatedly and sequentially is quite hard to exploit if the website has traffic. (this is why some of the slop PoCs reconfigure apache to have one worker, or pin WORKERS-1 with infinitely long requests).
This is in stark contrast to other, more severe vulns that the internet has seen (and will see in the near future) where there's effectively a blank cheque to fuck anything that is unpatched. This is not a blank cheque. It is still bad.
This doesnt matter though - luck and statistics are not serious security walls. Patch your servers.
That's not true. The vuln was exploited and utilized within minutes of the disclosure.
LiveOverflow has exploited the PHP vulnerability, it’s even linked in the article.
What confuses me the most is that nobody talks about the nature on how to exploit this vulnerability.
This has been a well known vulnerability for years but considered too complicated to exploit. It is the type of work that would have taken an expert weeks to hack into a single system and the idea that an AI can speed this is up is not unfounded but currently without evidence.
Yes, the attack surface is huge but when compared to Log4J this is materially more difficult to execute and the countless PoC that have been seen were AI slop.
As a security expert I see why you would not want to downplay this, but I think the involved actors have all the incentive to use this for their typical messaging.
As a tech community we have a greater responsibility to not be driven by hype but understand what is going on and let make ourselves some tools and be swayed by the messaging and instead talk about this in a more nuanced way. Perhaps I am wrong because I am not a cyber person myself but this all strikes me as similar to the whole Specter outrage.
These are all human-generated software bugs. At some point when AI writes all code, the bugs should all but disappear. Almost.
Bold of you to assume that AI-written and -reviewed code doesn't introduce new vulnerabilities. I think after we're done with this stretch of finding human-made vulnerabilities, we'll have a long road ahead to clean up the mess that's introduced by AI.
> Bold of you to assume that AI-written and -reviewed code doesn't introduce new vulnerabilities.
If AI can find a vulnerability in a given piece of code, it can also do so during testing, not only when used by a malicious actor. This means that the whole thing becomes an issue only if someone is writing a safety-critical program and not doing proper LLM safety testing.
Everyone should use LLMs to test the safety of their code today.
"Should" is doing a lot of heavy lifting there. Most shops are a long way off from secure coding, and from my experience in the last 10 years most devs lack basic awareness of security. I still see people being surprised by OWASP Top 10 or simple things like trust boundaries.
There's no guarantee AI will find existing vulnerabilities proactively during coding, or that it won't introduce new vulnerabilities in the process of shipping features. You're right that shops _should_ be critical and test code, but I think due to the lack of awareness people will end up implicitly trusting any first pass written by AI and not bother to double check it.
The bugs will only disappear until the model that was used to security test the generated code becomes obsoleted. So, yes, this is the new cyber normal. Unless we replace the tech stack with a formally verified one. Which will need to happen, one way or another, seems to me. But cheap & sloppy generated code won't be feasible in this paradigm, so I believe we'll need a much more thoughtful approach.
> The bugs will only disappear until the model that was used to security test the generated code becomes obsoleted.
I think you're ignoring the distinct possibility that even if the code is not formally verified, LLMs could simply stop writing bugs, or this kind of bugs. No matter how smart an attacker AI, it cannot exploit bugs that are simply not there.
But hell, even formally verified ones can only verify back to the spec. There are plenty of examples of bugs in formally verified software.
I place a higher likelihood on LLMs winning (producing fewer bugs than formally verified) over the long run. Mostly because verified software can, IIUC, only verify back to the spec. Whereas LLMs can go "uh, your spec itself has a bug, you really want me to codify that?".
> you're ignoring the distinct possibility that [...] LLMs could simply stop writing bugs
That just doesn't seem like a reasonable possibility to me, given what we're seeing from and learning about these models. They have no capacity to be aware and adhere to an overarching vision, which is why they inevitably corrode architectural coherence over time [1]. The only reason these agents are able to still create usable code is because harnesses take advantage of model's ability to find bugs, giving us the famous loop, adding duct tape on top of duct tape, ratcheting up the complexity, until the adversarial model isn't able to find any more issues. Thus, the bugs are pushed out into higher complexity dimensions, temporarily invisible until a stronger model comes along which can detect them.
[1] see for example the "Decoherence Trap" article discussed here yesterday: https://maevyn.codes/decoherence
> There are plenty of examples of bugs in formally verified software.
Of course. Which is why humans will always need to be involved in the software creation process. We have to specify what we want. It's true that we ourselves often don't know what we want, in fact this is a big part of what software development is about: requirements gathering and continuously recondisering and reworking them. But it would be beyond naive to imagine that software might one day know better than us humans what it is we actually want, no?
It could lead to a mess, sure. But my favourite prompt on an LLM-untouched codebase is along the lines of "find UB and other bugs. If the fix is less than 5 lines, fix it and commit it. One change per commit". Then I of course check the commit history before pushing.
My experience over the last couple of years with this type of prompt is that eventually you run out of bugs to find (meaning you reach "I see what you mean, but I disagree").
And those fixes don't cause messes.
And a new model may find something new, but not even remotely the volume seen the first time. This indicates to me that the supply of bugs is finite, and after this period of house cleaning, it should go down.
But anyway, a mess (decoherence) doesn't necessarily mean an exploitable mess.
Reading that good article, I find myself thinking that we did not need LLMs to get decoherence in complex software or systems. At least LLMs don't have personal fiefdoms, egos, and company politics adding to encouraging decoherence.
Allegedly AIs sometimes even forget to implement authentication for APIs, in the way anyone can just call them. So no, bugs will continue to exist, we've just magically got tons more supply of economical labor.
I think that’s a fallacy, see https://news.ycombinator.com/item?id=50207540
1. The average human software-generator sucks at security. 2. LLMs are trained on human-generated software. 3. Therefore, LLMs suck at security.
> Makes a few very wealthy tech CEOs very happy.
why is this both pros and cons?
Somewhat. I take heart in the fact that people actually took the "website down now" messaging seriously and that there was a quick patch to address the issue. It'll still be an arms race between red team and blue team, just faster paced.
By generating the entire website in AI, yes, you are contributing to the new normal
That's new Poisson
> Is this the new cyber normal?
No. We're in a "brief" [0] transition period where a scaleable superhuman developer enters the scene and discovers a whole heap of bugs. Once the majority of code is written by similar systems then the tide of critical bugs being found will slow down.
The article does sorta say this but lets not bury the lede to deeply. It isn't realistic to expect new security vulnerabilities to be found at this rate, we're clearing out years of backlog and the maintainers haven't had time to deploy AI to do the software right the first time. Once the situation stabilises I'd personally expect very few public vulnerabilities because the expertise in how to assess code security is moving from a handful of experts to everyone.
> The NHS approach: close source everything.
Yeah good luck. I've seen how these systems work at decompiling and I expect they're probably going to be really good at fuzzing too.
[0] Could take a while.
> Once the majority of code is written by similar systems then the tide of critical bugs being found will slow down.
I don’t know if this necessarily follows. I repeatedly see that AI will write code, then you have it audit the code for security vulnerabilities, and it still finds them. Just because AI can find bugs doesn’t mean it will write bug-free code. AI’s know how to take shortcuts to achieve their goals, and do it all the time.
Yes, you can have another AI audit the first AI’s code, but in practice this doesn’t always happen, especially because once the human gets their hands on the finished product, they don’t always take the extra step to ensure it’s audited. This is particularly true because people seem to think “well the AI wrote it in the first place, why would it find its own bugs?”, ignoring that context is everything, and an AI being asked to find bugs is going to approach the code a lot differently than an AI tasked with adding features.
That’s not to say it’s not possible to set things up properly into multi-agent stages with QA and security as first-class phases of development… I only mean to say that it’s not automatically true that “AI can find bugs” implies “AI doesn’t produce bugs”.
> AI’s know how to take shortcuts to achieve their goals, and do it all the time.
1. Is AI code more defensive than code you'd previously write? Like maybe I'd write one line to call a function and a quick check for exit code, but AI instead writes a dozen lines to gracefully handle different possible failure cases. Is this not others' experience as well?
2. You say AI can and does take shortcuts - I guess it really comes down to whether humans or AI is more likely to take 'shortcuts'?
>> The NHS approach: close source everything.
> Yeah good luck.
Is most NHS code client-side or server-side (/are they still making binaries available)? If binaries aren't being distributed, server-side code still can't be decompiled, so the exploitation likelihood may be a lot smaller, especially if they have basic rate limiting or similar in place. (Regardless, I still prefer open.)
>It isn't realistic to expect new security vulnerabilities to be found at this rate
I wouldn't challenge fate like this.
+1. There aren't an infinite amount of serious security vulnerabilities out there.
> > The NHS approach: close source everything.
> Yeah good luck. I've seen how these systems work at decompiling and I expect they're probably going to be really good at fuzzing too.
Me too. The agents are much faster if it has access to source. NHS would have been easy hunting grounds. Now at least they don't stand out.
"a system should be secure, even if everything about the system—except the key—is public knowledge"
Should indeed, but we've all been caught with our pants down.
> Even though this was found by “responsible security researchers” at “respectable companies”, by the time the security patch actually got released, it was already being exploited in the wild.
I am really not pleased with the "let's delay" strategy. IMO, you should without any delay, let everyone know of security-related issues. This "try to keep it hidden so that malicious actors can not exploit" is incredibly unfair to everyone else. Why would I want to support that? I want to know the exploit, I want it explained and then, in that order, I also expect that it can be fixed eventually. This closed-source thinking is very annoying. If one LLM can find an exploit, so can others, so what's with the delay?
The idea is that there is still an advantage even in times of LLM and the delay helps with an orchestrated fix. (E.g. not only php needed the fix, but also several docker image etc.)
But I agree it is doubtful that there is still such an advantage...
I'm a bit of an AI doomer but I'm going to disagree. I don't think this is the new mormal but there is going to be a dark transitional period.
What we're seeing is a bit of a "gold rush" where LLm advancements are coming thick and fast and they're being pointed out code bases that simply haven't had the level of scrutiny where relatively easy bugs to find have managed to survive. That's temporary. Here are my predictions:
1. Correctness and, in particular, memory safety is going to become increasingly important. I was bullish on Rust before this AI hype wave. LLMs finding and exploiting bugs has reinforced just important it is that Rust prevents as certain class of bugs that are the most prone to rmote exploitation ie certain memory safety issues.
I've long said that modern CPU architectures, out-of-order execution, hyperthreading (does that still exist? or did Intel fully kill it?), different CPU cores and compiler optimizations that it is essentially impossible for a human to write correct multithreaded code, particularly in userspace with "leaf node" code. LLMs just make that more apparent; and
2. Mature code bases for core functionality are going to become increasingly important because they've been "proven" in the wild (as much as you can prove a negative).
My sincerest hope, which is probably more wishful thinking than anything else, is that LLMs will be another nail in the coffin for C++, which is simply too complex and dragged down with legacy features and backwards compatibility to be salvageable.
> At the time of writing (23rd Oct), eMudhra has announced that it will be retiring from the certificate authority space. In unrelated news, a farmer has announced that he has shut his stable door, following reports of an escaped horse.
Ah, wit is alive and strong!
LLMs seem to be very good at detecting vulnerabilities caused by using C and other forms of human stupidity.
So yes.
Not for me. I’m taking my shit offline. Out of harms way.
Am I the only one that hates seeing "cyber" when refering to cybersecurity?
The same with "crypto" to refer to cryptocoins instead of cryptography.
It will be until we (hopefully) change the way we work.
Security has been an afterthought for most of the past 30-40 years, and there are virtually no regulations requiring security, audits, or inspections before releasing software.
Now that attacks are cheap and easy, the only way to prevent these constant compromises will be to secure things before releasing them - not an afterthought, but a prerequisite. If we do finally as a society demand this, then the compromises will go away, because we won't release any new versions of software without securing it first.
This depends on having open weights with strong security scanning. If nations keep banning powerful open weight models "because they're dangerous", those nations' companies will be at more risk, because they won't have the tools to secure their own software. Countries with more permissive models will allow their own businesses to secure their stacks earlier, more people will use those businesses, they will thrive. The invisible hand of the market will choose whoever can improve first, not whoever bans things first.
I hope not. A lot of LLM assisted vulnerability stuff has been noise, slop, fluff, or fake/false data. Drowns out the real issues.