As someone who has been programming for about 25 years now and has always been frustrated by the tendency of software devs, and in particular web devs, to not address the root issues but just scaffold on top of them and try to hide the ugly bits under the carpet, I feel a bit vindicated.
Everybody in the industry knows that we've been building on quicksand. Moore's law meant that we could just sacrifice performance and add a dozen layer on top of technology from the 90s (if not older) and keep the thing chugging along.
Look at the PHP situation:
- HTTP is an old and quirky protocol with a lot of surface open for attackers that are rarely useful nowadays and might be better left off to higher layers. But of course, we can't break compat, so everybody still implements all the naughty parts, just in case.
- PHP was a joke language from the start, we were making fun of it in the early 2000s, the fact that over 20 years later it still powers a huge chunk of the web is enough to demonstrate that we're not serious people. Cue the PHP fans telling me that "it's good now". Maybe, I don't really care, the effort that went into making it good should have gone into ditching PHP for a better language 20 years ago. The bandaid mentality is what got us in this situation.
- And of course to add insult to injury, PHP, this security-critical language, is still implemented in garbo C using things like setjmp/longjmp. Just to make sure to absolutely maximize the damage if a vuln is discovered, I suppose. Brilliant.
- Oh and all of this is still running overwhelmingly on top of x86 which is its own bag of legacy garbage. "No but you don't get it, modern CPUs are really more like RISC CPUs, the RISC ISA is translated internally" yeah exactly that's my point let's stop doing things like that.
We're a clown industry. We only made it this far because, usually, buggy code doesn't kill people and we can iterate and patch things quickly. We never had to do things correctly, we could flourish with abysmal levels of quality acceptance. This may not be true anymore.
Of course the reason things are done this way is generally to maintain backward compatibility and make transition easier. At least now that we have LLMs this excuse will hopefully vanish: porting even huge codebases to a new language/framework, something that was a tremendous undertaking mere months ago, is now vastly easier and faster. We can leave the 1990s behind us.
I have actually gotten a tremendous amount of value from PHP and WordPress: the via negativa! They showed me, in remarkable detail, how things should not be designed.
(I'm not kidding. If you take each element of their designs and invert it, you end up with a very solid design.)
On that note, I misspoke somewhat: I assert that PHP and JavaScript were not "badly designed", because that would imply that someone sat down, carefully considered how the different parts of the system interact, second order consequences etc. — and then somehow came to all the incorrect decisions. Rather, it's quite clear a serious design process has simply not occurred to begin with.
Once I recognized this phenomenon, I began to see it in all areas of life. Things are much worse than they could be, simply because nobody has designed them to be good. They just ended up the way they are now, by accident of history.
--
Polemic aside, I am fond of JS and PHP in many respects. I just think we should clean them up! "This mess has always been here, so it's too late to clean it now" would be laughed at in any other context.
> We're a clown industry. We only made it this far because, usually, buggy code doesn't kill people and we can iterate and patch things quickly
I think you have things backwards. Devs aren't clowns who make it this far. But in industries where buggy code doesn't kill people, devs are expected to ship fast and iterate, because they can and more importantly, it serves the business.
Personally, I'm all about good software engineering principles, unfortunately I don't think you can last long in most companies if you're too opinionated about this and can't compromise (and it's getting even worse with AI, reading the code is no longer a requirement, go explain your teammate that there's an abstraction leak in their code...).
It's coordination problems and large scale distributed action, not some moral character of how serious the people are. Effort is not fungible. To make php good, you only have to act at one point, in a localized way, at the php foundation. To ditch php, you have to act in a coordinated way across any resource strapped companies and individuals. Just see how hard it was to ditch Python 2 or how ipv4 is still not ditched.
Trying to understand underlying reasons and incentive structures is much more productive and useful than simply venting about the worse moral character and seriousness of people in one industry vs the next. Curiously, the grass seems always greener - when you ask people in other industries, you'll often hear endless complaints about the state of things there too.
> PHP was a joke language from the start.... Cue the PHP fans telling me that "it's good now".
First, I am not a PHP fan, but I will absolutely stick up for the "its good now".
Yes, absolutely no debate the old PHP 4.x/5.x was shocking. However to give PHP their due, they absolutely did make serious efforts in 7.x/8.x. They deserve due recognition for that.
Unfortunately lots of people – I'm guessing you're one of them – still have battle-scars from 4.x/5.x and they think (incorrectly) that 7.x/8.x is "more of the same".
Second "joke language", well, cast your mind back to what else was available at the time... ColdFusion, CGI, ASPX ... it was not exactly a beautiful world back then.
Sure, to borrow your terminology, the Java fans will be along soon to tell us all how awesome Java is for the web. But many of us still bear battle-scars of the horrors of Java stacktraces, messing around with incomprehensible XML config files, Java's penchant for eating RAM for breakfast, and generally trying to figure out why the hell Tomcat is not working (again !).
Wait a second, in very recent years the major Java projects I touched were still messing around with incomprehensible XML config files. Has that actually changed in some part of the industry?
Yes, when I started getting into Spring(Boot?) in 2013-14 I was complaining about the annotation magic that you needed to sprinkle around the code, so that you could have a Java project take a long time to start up and fall over. The response at the time was that it was so much better than the old XML way that you "had" to do. So that kind of XML hasn't been needed for 10+ years.
Unless you count the XML dependency management with Maven, which isn't too bad.
> Has that actually changed in some part of the industry?
I suspect the only thing that's changed is that Oracle now charge you for the pleasure. ;)
More seriously though, its been a long time since I've touched Java, but I suspect the only thing that's changed is that its all hidden in magical kubernetes containers these days.
>We're a clown industry. We only made it this far because, usually, buggy code doesn't kill people
Another industry is does even better: the medical/health industry. It's both a clown industry AND kills people AND they have made it way beyond what software can only dream.
I'll generally agree, and I include myself in this industry, but we've simply have been bound by compatibility. I don't think anyone chose intentionally to keep the crap around; it was simply never worth it to pay the "break compatibility" cost, until now.
Absolutely. The industry could have moved towards safer languages, better coding methodology and formal verification years ago (see for example MISRA C as at least a partial movement in this direction). It didn't, because it was cheaper not to.
> Oh and all of this is still running overwhelmingly on top of x86 which is its own bag of legacy garbage. "No but you don't get it, modern CPUs are really more like RISC CPUs, the RISC ISA is translated internally" yeah exactly that's my point let's stop doing things like that.
The interesting thing is that memory bandwidth has become a big issue. CISC instructions take up less space than the equivalent RISC instructions, so you can effectively get more instructions per cache line. Effectively the CISC to RISC conversion acts as a type of decompression. Because if that, stopping the CISC messiness without moving to a compressed instruction stream would actually hurt performance.
Not sure if I disagree with all your points, but this one caught my attention:
> HTTP is an old and quirky protocol with a lot of surface open for attackers that are rarely useful nowadays
HTTP 1.x is that, perhaps. But I was in the trenches when HTTP/2 came out, brand new and running on binary. It was so new that there were very few implementations around, and my team had to code one. The people who worked on the spec had a lot of learning to apply from HTTP 1.1. That resulted in many nice bits in HTTP/2, but HTTP/2 also had serious issues at a spec level. Which is my way of saying that "modern" doesn't necessarily mean "bug free".
The reason PHP and other, "joke languages" are still in use is because business needs power businesses, not engineering wants. Plenty of businesses are still run on pen, paper, and custom spreadsheets because those things just work or work well enough. Could they do a better job or scale up more with a SaaS offering or two? Possibly, but why?
>frustrated by the tendency of software devs, and in particular web devs
Are you sure that's not the typical cost optimisation of management/c-levels rather than a developer problem?
What you wrote reads a bit like ye olde typical "nyeeeh javascript! you're only a dev if you write everything in assembly! kids these days!"
Yup. Another reminder of how far software engineering lags behind other engineering disciplines in rigor and accountability. The chickens are coming home to roost.
I get your drift but... This sounds a bit like advocating "why not ditch all progamming languages and build some super language that makes no exploits possible". While we are at it, ditch the Von Neumann architecture. Or saying "why not ditch all natural languages in favor of Esperanto".
No, we don't need the "perfect language". But we definitely need to switch to better methods - which were available for decades already - more quickly. There are plenty of memory-safe languages. They have been available for decades. And here we are, a huge amount of servers affected by a memory bug, in especially badly written C. There are probably many more fundamental design bugs in PHP.
> the effort that went into making it good should have gone into ditching PHP
What effort? PHP foundation has one employee!
> The bandaid mentality is what got us in this situation.
> We're a clown industry.
We are clown industry becauee nobody would sponsor solid research, infrastructure and fixing all those bugs! PHP is junk literslly written over nights by some volunteers!
It's especially fun when not implementing the naughty bits is its own source of vulns. Your code is simpler and less prone to RCE, but your proxy mishandling or whatever allows for credential smuggling attacks, identity spoofing, request smuggling, cache poisoning, and all the normal issues at the application layer.
>to not address the root issues but just scaffold on top of them and try to hide the ugly bits under the carpet
It's totally the devs fault and not the environments devs have to work in. Deadlines forced down on you until you have to make compromises, 24/7 having to compete with people who actually don't care because you have to meet months end payments somehow. Whole businesses blinded by a tech gold rush.
>We're a clown industry.
I do agree here, but that includes every non dev person.
You don't see the same problems in say, structural engineering as you do in software engineering. I suggest this is due to needing to go through the formal training required of a structural engineer, and membership bodies like chartered institutions, with standards and codes of ethics. We might want to reflect on that
I agree with you but I would extend that observation to include the age of the industries. We have been making structures for thousands of years. Software is an infant compared to that.
I think that has more to do with than some unverifiable sentiment like "software people are simply inherently less serious than other engineers".
competing with others doing the same thing in a zero sum game. We all just have to simultaneously coordinate to agree not to set unrealistic expectations or deadlines! Sounds easy, right?
You forgot NPM. Whereby we download code from unknown authors on GitHub and run in with full access to our filesystem on our machines. No vulnerability, it's been designed like this, we even have designed scripts to autoupdate vulnerabilites, that's the proof that vulns are not common.
I hope some of these backdoors and/or 0-days being kept under wraps by nation states or corporate espionage companies (Pegasus et al) are being found incidentally through all this.
It’s frustrating to have all these 0days and CVEs to patch for now, but in the long run perhaps we’ll leave with more robust software.
Depends on your incentives, most people and companies are measuring Lines of codes and features as the metric so ofc it will increase it, but AI can create small KISS applications. It might have a little difficulties in doing so sometimes because it sometimes likes to overengineer things but I have to wonder how much of it is because it is RLHF'd to solve questions at any costs in terms of lines of code rather than training it in solving it in KISS principles. Also I wonder how AI will perform if we explicitly treat KISS as the only principle and have checks running on it and have it discard all the code which wouldn't follow KISS principle if it doesn't follow the principle.
I may be ignorant, but what do you mean with this and why is it upvoted so much?
To me KISS is basically about how you’d write the code, the architecture, which is something that - at least currently - humans need to oversee when writing new code with LLMs.
But what does it have to do with AI discovering a ton of bugs in a lot of (open-source) software?
Software that does less is easier to understand and reason about, and typically requires less code. Less opportunity for bugs to appear.
Maybe, when they build a pomodoro timer, people will consider the possibility that the entirety of a browser engine as well as 30 000 javascript dependencies from random unvetted sources introduces unnecessary complexity and a practically infinitely fractal attack surface.
It's interesting that no-one talks about writing new systems and base languages that don't inherit anything from the current toolchain. That's the first step, then new architectures / fully open auditable hardware basis. With html you don't need any of this dependency chain. As long as you can write a browser you can be running on anything underneath.
At this point, a browser is a significant part of the way to being an OS virtualised inside the OS it's really running on. Complete with occasional leaks through the virtualisation to the real host OS/machine.
I'm more annoyed by the fact the fixes and exploits themselves are being artificially delayed in misguided attempts to coordinate disclosure, in world where anyone can just point AI at vague direction of the bug and just find it
This is the real reason for the vulnpocalypse, not "technical debt".
NSA spent 20 years stockpiling vulns and pushing the industry towards half-assed solutions like address sanitization and pointer tagging when they could have put the industry on memory-safe languages instead.
They managed to get the IME/PSP/TrustZone into 99% of the CPUs out there by mandating it for government purchases; they could have done the same thing with memory-safe languages. But instead they chose to stockpile vulns. Hope they burn in hell for this.
Why do you think the NSA could have put the industry on memory-safe languages?
Even if they wanted to, I highly doubt they could push the industry to do that. The industry does what it wants to do, and rewriting tons of software was far from its priorities. There were no LLMs back then to help with such a massive migration.
There's also the fact that rewriting software tends to introduce a lot of new bugs.
The U.S. government could have required all new software for any government-procured system to be written memory-safe languages, thus making it ubiquitous. This could have been phased in slowly with a few years of on-ramp time to allow for industry to acquire the relevant skills.
The DoD tried that with Ada and largely failed. That was with defense contractors whose primary customer mandated it. That was despite Ada being proven to work in its first projects.
But even in memory-safe languages, compiler bugs are still an issue. The theory is perfect but the implementation is flawed. In such case once the compiler bugs are found, the software would have to be rewritten to compile. For example an 11 year old memory bug was discovered recently in the rust compiler: https://www.penligent.ai/hackinglabs/rust-borrow-checker-bug....
Memory safe languages are not a silver bullet either.
The U.S. government had pushed Ada for years and it never worked out. The government was forced to accept C++ because industry was rebelling against ADA, even though ADA was memory safe. Well, Ada did have the ability to be memory unsafe. It was much better by default.
The DoD have even created their own language: ADA. Massive contracts power, but different markets to incentivize.
The distance between the usual consumer/b2b tech startup and the kind of company that sells software to the air force is even bigger than the distance between Google and the kind of company who hires Accenture.
The strain from the amount of bugs is visible in bug bounty programs. A run on many targets will turn up exploits, but if you report them on any of the platforms a majority are instantly marked duplicate. It's so overloaded that companies are shutting down their programs from the volume of reports.
Anthropic is arguing for formal verification [0]. This is an excellent idea on the edge of feasibility, but it's hard to see how this can work for the majority of software that has to adapt rapidly and are so open ended that it's difficult to even conceptualize all the input modes. It's an easy prediction that we will see verification on large trunks of fundamental software in the next few years, and a sub-verification confidence scheme for the software on top of it.
Is it forcing us to "pay off" technical debt or is it starting us on a compulsory endless treadmill of feature-neutral fix-work? Collectively, we just won't know for a while.
It totally feels like the former if you watch it find and fix vulnerabilities or other debt-tasks one after another.
But only so many of these "technical debt" issue are little off-by-one errors that get remedied by a surgically narrow patch.
Many enough require some kind of restructuring or redevelopment of their own and it's going to take time to figure out if AI faces the same dilemma as human developers: where each non-trivial change carries the risk of some new issue, ad infinitum.
If that happens to be the case, and AI isn't able to magically produce perfect software in finite time, then what's really happened is that the threat of hostile AI exploiting our technical debt will force us to always be burning and churning code just to remain a moving target.
The original technical debt, which cannot be paid off, was the decision to use a Von Neumann architectures instead of a Harvard architectures for CPUs.
Basically, have an explicit distinction between data and code at the architectural level, to enforce "Write xor Execute" in hardware. In this way, you would prevent e.g. a buffer overflow letting an attacker injecting shellcode that gets executed later by somehow pointing at it. AFAIK the issue would still remain with some programming languages though.
and it's not sufficient anyways. even with NX you could have the attacker control a function pointer you call. in any sufficiently complex program you'll have enough widgets to build anything you want just out of data.
code and data are the same thing. whatever inputs your system receives is "code" that your application interprets, just like your application is the code that the CPU interprets. turtles all the way down.
I was down voted a lot months ago for saying that a lot of open source software had a false sense of being secure with the assumption that someone else would have audited the code. And this AI auditing is going to make the promise of open source software getting audited by others actually real. Nice to see others waking up to this reality.
There was always, in my mind at least, a tension with Linus's Law ("given enough eyeballs, all bugs are shallow").
It's true but the problem is assuming, just because something is open source, it has received sufficient eyeballs. AI now gives us near unlimited eyeballs so, if we survive the bumpy transition, open source should indeed become very secure compared to closed source.
I do not have an example for this, but the most ridiculous argument for open source software (at lease for most people) has always been that if the software has a bug you could simply fix it yourself. Funny enough this has actually become kind of true with new LLMs.
I don't agree that it was an assumption as part of open source, but that it was a possibility with open source. However, there are lots of assumptions made about computer software in general - part of the problem is having cognitive models of software that aren't necessarily reflected in how the hardware runs code.
Also, closed source software can similarly be analysed with AI, but there's no guarantee that the company involved will deal responsibly with disclosures.
The article takes PHP2Shell as the worst offender of the present wave of vulnerabilities but HTTP2Shell/Brotli2Shell was arguably worse. The consequence over at Brotli may very well be the same, but the timeline shows, if anything, that at least part of reality is actually still very much "humans just sitting on vulnerabilities for months or even years". From the timeline: 8 years in the code, 8 months unreleased (but fix in Git) and at least 4 independent researchers finding the problem and pointing it out.
Having worked on both sides of the table, the reality is that it is far too easy to game certs like ISO27001. The risk calculation by companies taking that route is changing right out from under them thanks to Ai tools.
That's so true. Nobody gave a damn about those bugs because they were too hard to exploit, and companies had a mindset of "move fast and break things." Bugs were given low priority, and developers had no time to fix them since they were buried in higher-priority feature work as part of a cost-optimization effort. Almost all of the smart people who could actually exploit them were afraid of criminal charges. Now, lots of script kiddies are having fun with AI, and state-sponsored actors in different countries are using it to accelerate the discovery and exploitation of vulnerabilities. But instead of extending validation campaigns and allocating larger budgets for hiring and funding security audits, software vendors are mostly trying to lobby for limitations on models' capabilities and regulation of the field, like Amazon reportedly did[1], despite even OpenAI and Anthropic offering subsidized[2,3] access to their frontier cybersecurity models, which are significantly better than those malicious actors have.
[2] https://openai.com/index/coordinated-disclosure-of-critical-... - "To help more defenders use these capabilities, we’ve committed $1 billion in subsidized Daybreak access through Daybreak for Frontline Defenders, focused on organizations with limited security resources protecting essential services."
[3] https://www.anthropic.com/glasswing - "Anthropic is committing up to $100M in usage credits for Mythos Preview across these efforts, as well as $4M in direct donations to open-source security organizations."
ironically, AI is also making us create a new whole new level of technical debt - where _no one_ has actual understanding of the code being written and we're just trusting the machine
Indeed, and it's just going to end up as a zero-sum game where the winner is the one who can most adeptly manipulate the machine to cause harm to the other.
Is a nuclear power plant dangerous? Or an airplane? Both are examples of an advanced form of what other entities in their category do.
Of course we should proceed with even more powerful tools. We also need even stronger governance of said tools. We don't want an AI-version of the Chernobyl incident but we do want the AI-version of the benefits that nuclear power gives.
I like to think of it as a new abstraction layer on top of something we've been comfortable with for years. The developers of the past had to trust that human-readable code would compile to the machine-level bytecode they were used to write, and that even though it wasn't as efficient as they could make it, one day the benefits would outweigh the negatives. Today most developers don't understand compilers well enough to even begin questioning if the code doesn't work because the compiler has a bug or because their code has a bug. They have to trust the abstraction layer beneath their current domain, and they have to find ways to prove that their work is correct in that domain (e.g., writing a sufficient number of unit- and integration tests, compiling with different compilers to different targets, etc.).
We, too, have to adjust to this change, this new abstraction layer, because whether we like it or not, the code an LLM will produce in no more than five years will arguably be much, much better than the code any 10x developer could produce even if given months of time and an infinite supply of caffeine.
We "just" need to find a way to confirm that our work in this new abstraction layer - primarily prompt engineering - results in the code functionality that we had in mind, not necessarily the code implementation that we had in mind.
I've spend the past month-or-two on developing a platform scroller game for smartphones using nothing but AI. It has created everything given my instructions, to my liking, from the way the game is played to graphical assets to sound effects. It looks and feels like any other typical studio-developed game you could find in the store, the difference is that I didn't have a studio of professionals working fulltime for months or years with various levels of project management and what not. It's just me. Me and my agentic coding subscription.
Vibe coding originally meant giving an LLM an instruction, having it regurgitate something potentially wholesome, testing it and feeding back the errors for it to fix it. What I've done in the past couple of months is so, so much more than that. I'm a puppet master and my coding agent is my puppet. I don't need to understand the code when I can describe the problem I want solved and it will do it 100x faster than I ever could.
I didn't use code to describe this level of abstration. I used English prose - and a heck of a lot of software engineering theory, mind you. If you know the solution requires a quad-tree and binary search because in this case, that is the best option, then that's an instruction, not a block of code.
Is it a challenge to Agile methodologies? I mean is most software today designed at all? If we want a pm to be able to write half baked feature ideas and expect the devs to sort it out, rather than just paste the ticket into Claude, don't we make this chaos inevitable. Maybe there is a missing formal engineering layer missing between feature request and prompt. Llms seem to do much better when they are told exactly what to do, vs when they are given room to...express themselves
We could rephrase it like this: “AI has removed all the excuses we had not to remove technical debt”
Yes, AI developed code might have vulnerabilities or bugs, but so already has your JSP based web server that has Oracle and God knows what else among dependencies.
I think what's fascinating is that the mentality of "It's too painful to attack" as a security strategy isn't working anymore in the era of AI. A lot of open issues and bugs were kept because folks assumed as long as they were less vulnerable than other folks they would be fine. However I think that paradigm is being phased out since AI can take the pain of sifting through decades of old code for us to find these vulnerabilities.
we're seeing easier PR approvals when AI is involved. Since changing the code in the future is a lot easier.
Instead of a highly structured engineering process it's more like a gardening process of figuring out what works, what doesn't, what the Business actually wants vs what their requirements were
So more debt because there's more code but it's easier and simpler to change that debt. And in 3 years the product is EOL so shrugs
Earlier this year, the cybersecurity firm Calif offered to independently audit PHP, sponsored by OpenAI. Within two weeks, a researcher at Calif, aided by AI tools, found a way to use the out-of-memory bug to run arbitrary code on the host server, allowing a malicious actor to install malware or read critical data. Now, it was no longer a minor bug—it was a critical vulnerability, and the Calif team alerted the PHP foundation to start working on a patch immediately.
So, the headline should read,
"AI is helping us eliminate 40 years of technical debt."
Since AI found the bug, and probably also generated the patch at this point.
An abundance of solutions is a problem for people who sell solutions.
> As far as anyone is aware, disaster has been averted: The hack was discovered by cybersecurity researchers who reported it to the PHP Foundation, and a patch was released before any bad actors got their hands on the exploit.
Aren't there still vulnerable services right now being hacked?
In my opinion, the whole paradigm how we use computers needs to be replaced. It is like the 80s, but now with internet and a thread larger then anything in computer history before. I remember boot sector viruses and stuff.
My approach is https://tix.im/ - not launched yet. But from expedience: modeling everything as workflows of composeable nix flakes (expert systems from programs, skills, agents, workflows and their snippets, DBs blobs, service descriptions, ...) and then run everything in highly isolated environments. It is not only so much more efficient, reduces errors, increases quality, reduces unnecessary cognitive load, ...
I start to move more and more daily tasks into the systems. Even UI programs become interactive workflow steps, which allow me to attach from anywhere back to a task in an environment optimized for the task.
Dependency updates for example, a workflow. A subgraph from a security scanner expert checks each change in the dependency tree.
I don't mean to be that guy, but your website feels inscrutable to me. I know that a lot of that might be because it's written by an LLM, but the page seems like it's full of private language and is the product of a deep context conversation with your agent that I don't have enough context to understand.
What's hillarious is people vibe coding massive AI apps thinking its not creating tech debt and security holes. If anything, AI is discovering existing holes, and creating them faster than ever before in history.
All those "paying down" technical debt with AI, I would advise reading the code _very_ carefully. Sometimes the "solution" which passes some test or adheres to some described functionality is incredibly narrow / brittle, and/or has failure cases that are non-obvious.
Yes I'm using $latest_model on high, its still not there. Unless your threshold for "fixed" is "the 1M line lean proof runs, we don't know how it works", it still requires a lot of attention, hand-holding, and scrutiny, especially when fixing issues / working around failures (where it tends to shortcut or invent workarounds).
A developer on my team introduced new features behind feature flags despite concerns about whether we should have those features and how they are going to be maintained. Changes touched 200+ files which is half of the files.
Turns out nobody wants to use them, they caused many subtle bugs in existing features and is a distraction for many things we do.
The developer left the team, and I am left with purging all of this.
The same can be said with human-written code. LLM code is by far better than the average developer’s code at this point, and there’s not enough top percentile developers to go around, so it’s a wonderful tool.
Also, those very average developers are also the ones doing the code review.
I'm personally not sharing the connotations of negativity towards software development and the term technical debt. I keep reading implications that most of this has always been in some way preventable, but it really is not that simple. How did nobody find these types of exploits 40 years ago? Why are we suddenly shocked and blame the field for this? I think the majority of people severely underestimate the complexity and seriousness of software. Once that perception stops and the economic incentives are right, we can start to look closer.
Just consider how laughable some working conditions in software are and then come to realize how architecture is shaped by the organization communication patterns and not reality. You will see it in a different light then.
And that's the biggest win with AI. With old-fashioned development, it was considered acceptable to defer addressing valid feedback ad infinitum, hiding behind whatever lousy excuse you could come up with. No more.
The amount of e-mails I get of the form "Important information regarding your xxxxx account" (which essentially is just code for "we've been hacked and your data has been stolen") is quite concerning, but not surprising. Security by obscurity was always big in most companies, as long as the bar to carrying out sophisticated attacks was putting a costly IT professional to the task you could be relatively relaxed as no one would bother to hack your stupid little e-shop, but now carrying out such an attack costs a few cents to a few dollars at most, that changes the game entirely and everyone is a target.
This summer I had my Docker registry poisoned with malware artefacts, which again is something that no sane attacker would do before as it required downloading my images, carefully modifying them and then overwriting them, ensuring they work as before. Now it's just a simple prompt away. I am definitely ramping up my defenses, everything that doesn't need to be Internet connected goes behind double VPN and port knocking schemes, essentially making my infrastructure invisible from the outside, all unknown packet types get hard dropped, end-to-end encryption of all internal traffic, web endpoints behind CDN proxies and WAF providers (as much as I hate doing that), double and tripe security mechanisms everywhere, maximum restriction of privileges, ultrafast automated patching and AI based continuous monitoring, audits and interventions. You should also get used to that, this will surely still get much worse as attacker tooling gets better.
I'm not going to lie, I'm having a great time thinking at the endless discussions around unsafe languages and the relentless smugness of many practitioners claiming everything was fine as long as the responsible and the skilled (them) were in charge; and let's not forget the formulaic "with great power comes great responsibility" closers.
Well now, Great Responsibles, it's clear you too were out of your collective depth and were all paying for your smug hubris. Perhaps take a bite out of that humble pie you've obviously never tasted?
It also inflated almost all of that debt away, though. So in a way people who built "just ship it" buggy things in the last few years, made the right bet.
Yep, the takeaway from this should be that we move to leaner applications that can be actually be formally verified. But what I imagine will actually happen is that we'll patch over these vulns with AI that produces 10x the amount of code, just for the next generation of AI to find vulns in that so we can patch that with more code, repeat ad infinitum.
I think we'll look back at this moment as peak C/C++ before they begin a long, slow decline into irrelevance.
I've been bullish on Rust for years because memory safety is simply too important and Rust can essentially prove an implementation correct and solve a certain class of very important bugs. It doesn't solve every problem and there is a complexity cost.
I've also said that writing multithreaded code is simply too complex. In an era of out-of-order execution, different levels of caching, asymmetric cores, hyperthreading (although this is largely dead now?), branch prediction and having to support multiple architectures it's simply too hard for most people to write correct multithreaded code. Period. If you find yourself as a non-library developer creating a mutex or spawning a thread, somebody has screwed up.
This is also why I think the single-threaded async/await model with post-request teardown of PHP is the correct choice for most people.
We're going to go through a dark period of a lot of 0days. As the article notes, this is paying 40 years of technical debt. It will result in a more secure ecosystem however. I believe this will ultimately also hamper black markets, governments and 3 letter agencies who trade in those 0days.
As painful as this all is, it will ultimately be a good thing, one of the few to come out of AI (IMHO).
As someone who has been programming for about 25 years now and has always been frustrated by the tendency of software devs, and in particular web devs, to not address the root issues but just scaffold on top of them and try to hide the ugly bits under the carpet, I feel a bit vindicated.
Everybody in the industry knows that we've been building on quicksand. Moore's law meant that we could just sacrifice performance and add a dozen layer on top of technology from the 90s (if not older) and keep the thing chugging along.
Look at the PHP situation:
- HTTP is an old and quirky protocol with a lot of surface open for attackers that are rarely useful nowadays and might be better left off to higher layers. But of course, we can't break compat, so everybody still implements all the naughty parts, just in case.
- PHP was a joke language from the start, we were making fun of it in the early 2000s, the fact that over 20 years later it still powers a huge chunk of the web is enough to demonstrate that we're not serious people. Cue the PHP fans telling me that "it's good now". Maybe, I don't really care, the effort that went into making it good should have gone into ditching PHP for a better language 20 years ago. The bandaid mentality is what got us in this situation.
- And of course to add insult to injury, PHP, this security-critical language, is still implemented in garbo C using things like setjmp/longjmp. Just to make sure to absolutely maximize the damage if a vuln is discovered, I suppose. Brilliant.
- Oh and all of this is still running overwhelmingly on top of x86 which is its own bag of legacy garbage. "No but you don't get it, modern CPUs are really more like RISC CPUs, the RISC ISA is translated internally" yeah exactly that's my point let's stop doing things like that.
We're a clown industry. We only made it this far because, usually, buggy code doesn't kill people and we can iterate and patch things quickly. We never had to do things correctly, we could flourish with abysmal levels of quality acceptance. This may not be true anymore.
Of course the reason things are done this way is generally to maintain backward compatibility and make transition easier. At least now that we have LLMs this excuse will hopefully vanish: porting even huge codebases to a new language/framework, something that was a tremendous undertaking mere months ago, is now vastly easier and faster. We can leave the 1990s behind us.
I have actually gotten a tremendous amount of value from PHP and WordPress: the via negativa! They showed me, in remarkable detail, how things should not be designed.
(I'm not kidding. If you take each element of their designs and invert it, you end up with a very solid design.)
On that note, I misspoke somewhat: I assert that PHP and JavaScript were not "badly designed", because that would imply that someone sat down, carefully considered how the different parts of the system interact, second order consequences etc. — and then somehow came to all the incorrect decisions. Rather, it's quite clear a serious design process has simply not occurred to begin with.
Once I recognized this phenomenon, I began to see it in all areas of life. Things are much worse than they could be, simply because nobody has designed them to be good. They just ended up the way they are now, by accident of history.
--
Polemic aside, I am fond of JS and PHP in many respects. I just think we should clean them up! "This mess has always been here, so it's too late to clean it now" would be laughed at in any other context.
> We're a clown industry. We only made it this far because, usually, buggy code doesn't kill people and we can iterate and patch things quickly
I think you have things backwards. Devs aren't clowns who make it this far. But in industries where buggy code doesn't kill people, devs are expected to ship fast and iterate, because they can and more importantly, it serves the business.
Personally, I'm all about good software engineering principles, unfortunately I don't think you can last long in most companies if you're too opinionated about this and can't compromise (and it's getting even worse with AI, reading the code is no longer a requirement, go explain your teammate that there's an abstraction leak in their code...).
It's coordination problems and large scale distributed action, not some moral character of how serious the people are. Effort is not fungible. To make php good, you only have to act at one point, in a localized way, at the php foundation. To ditch php, you have to act in a coordinated way across any resource strapped companies and individuals. Just see how hard it was to ditch Python 2 or how ipv4 is still not ditched.
Trying to understand underlying reasons and incentive structures is much more productive and useful than simply venting about the worse moral character and seriousness of people in one industry vs the next. Curiously, the grass seems always greener - when you ask people in other industries, you'll often hear endless complaints about the state of things there too.
> PHP was a joke language from the start.... Cue the PHP fans telling me that "it's good now".
First, I am not a PHP fan, but I will absolutely stick up for the "its good now".
Yes, absolutely no debate the old PHP 4.x/5.x was shocking. However to give PHP their due, they absolutely did make serious efforts in 7.x/8.x. They deserve due recognition for that.
Unfortunately lots of people – I'm guessing you're one of them – still have battle-scars from 4.x/5.x and they think (incorrectly) that 7.x/8.x is "more of the same".
Second "joke language", well, cast your mind back to what else was available at the time... ColdFusion, CGI, ASPX ... it was not exactly a beautiful world back then.
Sure, to borrow your terminology, the Java fans will be along soon to tell us all how awesome Java is for the web. But many of us still bear battle-scars of the horrors of Java stacktraces, messing around with incomprehensible XML config files, Java's penchant for eating RAM for breakfast, and generally trying to figure out why the hell Tomcat is not working (again !).
Wait a second, in very recent years the major Java projects I touched were still messing around with incomprehensible XML config files. Has that actually changed in some part of the industry?
You can use annotations to get rid of a lot of XML in server side Java. See Spring Boot for example.
Yes, when I started getting into Spring(Boot?) in 2013-14 I was complaining about the annotation magic that you needed to sprinkle around the code, so that you could have a Java project take a long time to start up and fall over. The response at the time was that it was so much better than the old XML way that you "had" to do. So that kind of XML hasn't been needed for 10+ years.
Unless you count the XML dependency management with Maven, which isn't too bad.
> Has that actually changed in some part of the industry?
I suspect the only thing that's changed is that Oracle now charge you for the pleasure. ;)
More seriously though, its been a long time since I've touched Java, but I suspect the only thing that's changed is that its all hidden in magical kubernetes containers these days.
I remember when that exact line of argument was made about '3/4 was silly, but they're putting a real effort into 5'
Also "3 was silly, they got 4 done right".
Although that was more connected to the performance improvements through Zend Engine than language sanity or robustness, at the time.
>We're a clown industry. We only made it this far because, usually, buggy code doesn't kill people
Another industry is does even better: the medical/health industry. It's both a clown industry AND kills people AND they have made it way beyond what software can only dream.
> We're a clown industry.
I'll generally agree, and I include myself in this industry, but we've simply have been bound by compatibility. I don't think anyone chose intentionally to keep the crap around; it was simply never worth it to pay the "break compatibility" cost, until now.
> we're not serious people
Absolutely. The industry could have moved towards safer languages, better coding methodology and formal verification years ago (see for example MISRA C as at least a partial movement in this direction). It didn't, because it was cheaper not to.
> Oh and all of this is still running overwhelmingly on top of x86 which is its own bag of legacy garbage. "No but you don't get it, modern CPUs are really more like RISC CPUs, the RISC ISA is translated internally" yeah exactly that's my point let's stop doing things like that.
The interesting thing is that memory bandwidth has become a big issue. CISC instructions take up less space than the equivalent RISC instructions, so you can effectively get more instructions per cache line. Effectively the CISC to RISC conversion acts as a type of decompression. Because if that, stopping the CISC messiness without moving to a compressed instruction stream would actually hurt performance.
This is also a result of consumers putting up with every kind of shit they've been dealt. The application layer has been the biggest resource hog.
Not sure if I disagree with all your points, but this one caught my attention:
> HTTP is an old and quirky protocol with a lot of surface open for attackers that are rarely useful nowadays
HTTP 1.x is that, perhaps. But I was in the trenches when HTTP/2 came out, brand new and running on binary. It was so new that there were very few implementations around, and my team had to code one. The people who worked on the spec had a lot of learning to apply from HTTP 1.1. That resulted in many nice bits in HTTP/2, but HTTP/2 also had serious issues at a spec level. Which is my way of saying that "modern" doesn't necessarily mean "bug free".
The reason PHP and other, "joke languages" are still in use is because business needs power businesses, not engineering wants. Plenty of businesses are still run on pen, paper, and custom spreadsheets because those things just work or work well enough. Could they do a better job or scale up more with a SaaS offering or two? Possibly, but why?
>frustrated by the tendency of software devs, and in particular web devs Are you sure that's not the typical cost optimisation of management/c-levels rather than a developer problem?
What you wrote reads a bit like ye olde typical "nyeeeh javascript! you're only a dev if you write everything in assembly! kids these days!"
Yup. Another reminder of how far software engineering lags behind other engineering disciplines in rigor and accountability. The chickens are coming home to roost.
I get your drift but... This sounds a bit like advocating "why not ditch all progamming languages and build some super language that makes no exploits possible". While we are at it, ditch the Von Neumann architecture. Or saying "why not ditch all natural languages in favor of Esperanto".
No, we don't need the "perfect language". But we definitely need to switch to better methods - which were available for decades already - more quickly. There are plenty of memory-safe languages. They have been available for decades. And here we are, a huge amount of servers affected by a memory bug, in especially badly written C. There are probably many more fundamental design bugs in PHP.
> the effort that went into making it good should have gone into ditching PHP
What effort? PHP foundation has one employee!
> The bandaid mentality is what got us in this situation.
> We're a clown industry.
We are clown industry becauee nobody would sponsor solid research, infrastructure and fixing all those bugs! PHP is junk literslly written over nights by some volunteers!
> HTTP, naughty bits
It's especially fun when not implementing the naughty bits is its own source of vulns. Your code is simpler and less prone to RCE, but your proxy mishandling or whatever allows for credential smuggling attacks, identity spoofing, request smuggling, cache poisoning, and all the normal issues at the application layer.
>to not address the root issues but just scaffold on top of them and try to hide the ugly bits under the carpet
It's totally the devs fault and not the environments devs have to work in. Deadlines forced down on you until you have to make compromises, 24/7 having to compete with people who actually don't care because you have to meet months end payments somehow. Whole businesses blinded by a tech gold rush.
>We're a clown industry.
I do agree here, but that includes every non dev person.
Time for me to drag this old chestnut back up.
You don't see the same problems in say, structural engineering as you do in software engineering. I suggest this is due to needing to go through the formal training required of a structural engineer, and membership bodies like chartered institutions, with standards and codes of ethics. We might want to reflect on that
You do. Structures fail. Less so in parts of the world where serious regulations are in place.
I think being legally liable for any failures that pop up probably also has something to do with their more cautious culture...
I agree with you but I would extend that observation to include the age of the industries. We have been making structures for thousands of years. Software is an infant compared to that.
I think that has more to do with than some unverifiable sentiment like "software people are simply inherently less serious than other engineers".
The people who set the deadlines are trying to stay competitive so there are contracts with clients so your salary can come from somewhere.
competing with others doing the same thing in a zero sum game. We all just have to simultaneously coordinate to agree not to set unrealistic expectations or deadlines! Sounds easy, right?
You forgot NPM. Whereby we download code from unknown authors on GitHub and run in with full access to our filesystem on our machines. No vulnerability, it's been designed like this, we even have designed scripts to autoupdate vulnerabilites, that's the proof that vulns are not common.
I hope some of these backdoors and/or 0-days being kept under wraps by nation states or corporate espionage companies (Pegasus et al) are being found incidentally through all this.
It’s frustrating to have all these 0days and CVEs to patch for now, but in the long run perhaps we’ll leave with more robust software.
Hopefully we'll also see us back to KISS principles too. But who am I kidding
What I have heard from AI it is anything but KISS... More features and more code as fast as machinelly possible.
> ... More features and more code as fast as machinelly possible.
i am going to quote that a lot...
Depends on your incentives, most people and companies are measuring Lines of codes and features as the metric so ofc it will increase it, but AI can create small KISS applications. It might have a little difficulties in doing so sometimes because it sometimes likes to overengineer things but I have to wonder how much of it is because it is RLHF'd to solve questions at any costs in terms of lines of code rather than training it in solving it in KISS principles. Also I wonder how AI will perform if we explicitly treat KISS as the only principle and have checks running on it and have it discard all the code which wouldn't follow KISS principle if it doesn't follow the principle.
Dunno about your company, mine measures in lines of code removed.
I may be ignorant, but what do you mean with this and why is it upvoted so much?
To me KISS is basically about how you’d write the code, the architecture, which is something that - at least currently - humans need to oversee when writing new code with LLMs.
But what does it have to do with AI discovering a ton of bugs in a lot of (open-source) software?
Software that does less is easier to understand and reason about, and typically requires less code. Less opportunity for bugs to appear.
Maybe, when they build a pomodoro timer, people will consider the possibility that the entirety of a browser engine as well as 30 000 javascript dependencies from random unvetted sources introduces unnecessary complexity and a practically infinitely fractal attack surface.
that people will finally start solving issues by simplifying the software rather than generating even more edge cases.
A lot of these bugs are in rarely exercised code paths that probably shouldn't have been written into the software in the first place.
Plain http 1.0 is KISS. Does not really work that well.
Systems should be made as simple as possible, and no simpler.
That doesn't mean every simple system solves the problems we intend for it to solve. It means that all else being equal, we should prefer parsimony.
Could the global economy survive what that would do to the B2B SaaS industry?
It's interesting that no-one talks about writing new systems and base languages that don't inherit anything from the current toolchain. That's the first step, then new architectures / fully open auditable hardware basis. With html you don't need any of this dependency chain. As long as you can write a browser you can be running on anything underneath.
At this point, a browser is a significant part of the way to being an OS virtualised inside the OS it's really running on. Complete with occasional leaks through the virtualisation to the real host OS/machine.
I'm more annoyed by the fact the fixes and exploits themselves are being artificially delayed in misguided attempts to coordinate disclosure, in world where anyone can just point AI at vague direction of the bug and just find it
This is the real reason for the vulnpocalypse, not "technical debt".
NSA spent 20 years stockpiling vulns and pushing the industry towards half-assed solutions like address sanitization and pointer tagging when they could have put the industry on memory-safe languages instead.
They managed to get the IME/PSP/TrustZone into 99% of the CPUs out there by mandating it for government purchases; they could have done the same thing with memory-safe languages. But instead they chose to stockpile vulns. Hope they burn in hell for this.
Why do you think the NSA could have put the industry on memory-safe languages?
Even if they wanted to, I highly doubt they could push the industry to do that. The industry does what it wants to do, and rewriting tons of software was far from its priorities. There were no LLMs back then to help with such a massive migration.
There's also the fact that rewriting software tends to introduce a lot of new bugs.
The U.S. government could have required all new software for any government-procured system to be written memory-safe languages, thus making it ubiquitous. This could have been phased in slowly with a few years of on-ramp time to allow for industry to acquire the relevant skills.
The DoD tried that with Ada and largely failed. That was with defense contractors whose primary customer mandated it. That was despite Ada being proven to work in its first projects.
But even in memory-safe languages, compiler bugs are still an issue. The theory is perfect but the implementation is flawed. In such case once the compiler bugs are found, the software would have to be rewritten to compile. For example an 11 year old memory bug was discovered recently in the rust compiler: https://www.penligent.ai/hackinglabs/rust-borrow-checker-bug....
Memory safe languages are not a silver bullet either.
Definitely not a silver bullet (consider also that memory management isn'the only source for vulns), but still much better than C/C++.
The U.S. government had pushed Ada for years and it never worked out. The government was forced to accept C++ because industry was rebelling against ADA, even though ADA was memory safe. Well, Ada did have the ability to be memory unsafe. It was much better by default.
The DoD have even created their own language: ADA. Massive contracts power, but different markets to incentivize.
The distance between the usual consumer/b2b tech startup and the kind of company that sells software to the air force is even bigger than the distance between Google and the kind of company who hires Accenture.
>IME/PSP/TrustZone into 99% of the CPUs
Yeah, about that. Its the biggest vulnerability in the industry. Nobody wants to confront this fact, though.
Too much profit to be made in subversion and human rights abuses.
The strain from the amount of bugs is visible in bug bounty programs. A run on many targets will turn up exploits, but if you report them on any of the platforms a majority are instantly marked duplicate. It's so overloaded that companies are shutting down their programs from the volume of reports.
Anthropic is arguing for formal verification [0]. This is an excellent idea on the edge of feasibility, but it's hard to see how this can work for the majority of software that has to adapt rapidly and are so open ended that it's difficult to even conceptualize all the input modes. It's an easy prediction that we will see verification on large trunks of fundamental software in the next few years, and a sub-verification confidence scheme for the software on top of it.
[0] https://www.anthropic.com/research/formal-verification
Is it forcing us to "pay off" technical debt or is it starting us on a compulsory endless treadmill of feature-neutral fix-work? Collectively, we just won't know for a while.
It totally feels like the former if you watch it find and fix vulnerabilities or other debt-tasks one after another.
But only so many of these "technical debt" issue are little off-by-one errors that get remedied by a surgically narrow patch.
Many enough require some kind of restructuring or redevelopment of their own and it's going to take time to figure out if AI faces the same dilemma as human developers: where each non-trivial change carries the risk of some new issue, ad infinitum.
If that happens to be the case, and AI isn't able to magically produce perfect software in finite time, then what's really happened is that the threat of hostile AI exploiting our technical debt will force us to always be burning and churning code just to remain a moving target.
It's another "we'll see" thing for now, I think.
The original technical debt, which cannot be paid off, was the decision to use a Von Neumann architectures instead of a Harvard architectures for CPUs.
Explain? (Genuine question because I wanna know)
Basically, have an explicit distinction between data and code at the architectural level, to enforce "Write xor Execute" in hardware. In this way, you would prevent e.g. a buffer overflow letting an attacker injecting shellcode that gets executed later by somehow pointing at it. AFAIK the issue would still remain with some programming languages though.
don't we already have the NX bit?
and it's not sufficient anyways. even with NX you could have the attacker control a function pointer you call. in any sufficiently complex program you'll have enough widgets to build anything you want just out of data.
code and data are the same thing. whatever inputs your system receives is "code" that your application interprets, just like your application is the code that the CPU interprets. turtles all the way down.
and we don't believe a sterilized virtual machine creates a similar situation?
any respective machine doesn't require full Turing completeness?
any ad-hoc machine allows for mutations and flexibility, which I'm sure is an added strength.
2nd edit,
Also, didn't we have a weaker model of this with segmented non-flat memory models on i386-
I was down voted a lot months ago for saying that a lot of open source software had a false sense of being secure with the assumption that someone else would have audited the code. And this AI auditing is going to make the promise of open source software getting audited by others actually real. Nice to see others waking up to this reality.
There was always, in my mind at least, a tension with Linus's Law ("given enough eyeballs, all bugs are shallow").
It's true but the problem is assuming, just because something is open source, it has received sufficient eyeballs. AI now gives us near unlimited eyeballs so, if we survive the bumpy transition, open source should indeed become very secure compared to closed source.
Perhaps it will also challenge the assumption that open source is something you can run a billion dollar business on without re-investing a penny.
a lot of open source software had a false sense of being secure with the assumption that someone else would have audited the code
Do you have examples of people saying this? I don't think I've come across idea being espoused...
I do not have an example for this, but the most ridiculous argument for open source software (at lease for most people) has always been that if the software has a bug you could simply fix it yourself. Funny enough this has actually become kind of true with new LLMs.
I don't agree that it was an assumption as part of open source, but that it was a possibility with open source. However, there are lots of assumptions made about computer software in general - part of the problem is having cognitive models of software that aren't necessarily reflected in how the hardware runs code.
Also, closed source software can similarly be analysed with AI, but there's no guarantee that the company involved will deal responsibly with disclosures.
> ... a lot of open source software had a false sense of being secure with the assumption that someone else would have audited the code ...
I agree on this part, but what's your point here? Are you suggesting closed source is better?
The article takes PHP2Shell as the worst offender of the present wave of vulnerabilities but HTTP2Shell/Brotli2Shell was arguably worse. The consequence over at Brotli may very well be the same, but the timeline shows, if anything, that at least part of reality is actually still very much "humans just sitting on vulnerabilities for months or even years". From the timeline: 8 years in the code, 8 months unreleased (but fix in Git) and at least 4 independent researchers finding the problem and pointing it out.
[0] https://www.bugsink.com/blog/http2shell-timeline/
Having worked on both sides of the table, the reality is that it is far too easy to game certs like ISO27001. The risk calculation by companies taking that route is changing right out from under them thanks to Ai tools.
That's so true. Nobody gave a damn about those bugs because they were too hard to exploit, and companies had a mindset of "move fast and break things." Bugs were given low priority, and developers had no time to fix them since they were buried in higher-priority feature work as part of a cost-optimization effort. Almost all of the smart people who could actually exploit them were afraid of criminal charges. Now, lots of script kiddies are having fun with AI, and state-sponsored actors in different countries are using it to accelerate the discovery and exploitation of vulnerabilities. But instead of extending validation campaigns and allocating larger budgets for hiring and funding security audits, software vendors are mostly trying to lobby for limitations on models' capabilities and regulation of the field, like Amazon reportedly did[1], despite even OpenAI and Anthropic offering subsidized[2,3] access to their frontier cybersecurity models, which are significantly better than those malicious actors have.
[1] https://www.theverge.com/ai-artificial-intelligence/949601/a...
[2] https://openai.com/index/coordinated-disclosure-of-critical-... - "To help more defenders use these capabilities, we’ve committed $1 billion in subsidized Daybreak access through Daybreak for Frontline Defenders, focused on organizations with limited security resources protecting essential services."
[3] https://www.anthropic.com/glasswing - "Anthropic is committing up to $100M in usage credits for Mythos Preview across these efforts, as well as $4M in direct donations to open-source security organizations."
ironically, AI is also making us create a new whole new level of technical debt - where _no one_ has actual understanding of the code being written and we're just trusting the machine
Indeed, and it's just going to end up as a zero-sum game where the winner is the one who can most adeptly manipulate the machine to cause harm to the other.
This is exactly what has happened the past 50+ years, too, your toolbox was just different in the 90's than it is today.
So if it has happened, should we now proceed with even more powerful and dangerous tools, with a much tighter integration with society?
Is a nuclear power plant dangerous? Or an airplane? Both are examples of an advanced form of what other entities in their category do.
Of course we should proceed with even more powerful tools. We also need even stronger governance of said tools. We don't want an AI-version of the Chernobyl incident but we do want the AI-version of the benefits that nuclear power gives.
I like to think of it as a new abstraction layer on top of something we've been comfortable with for years. The developers of the past had to trust that human-readable code would compile to the machine-level bytecode they were used to write, and that even though it wasn't as efficient as they could make it, one day the benefits would outweigh the negatives. Today most developers don't understand compilers well enough to even begin questioning if the code doesn't work because the compiler has a bug or because their code has a bug. They have to trust the abstraction layer beneath their current domain, and they have to find ways to prove that their work is correct in that domain (e.g., writing a sufficient number of unit- and integration tests, compiling with different compilers to different targets, etc.).
We, too, have to adjust to this change, this new abstraction layer, because whether we like it or not, the code an LLM will produce in no more than five years will arguably be much, much better than the code any 10x developer could produce even if given months of time and an infinite supply of caffeine.
We "just" need to find a way to confirm that our work in this new abstraction layer - primarily prompt engineering - results in the code functionality that we had in mind, not necessarily the code implementation that we had in mind.
I'm still unsure if we won't figure out that the right abstraction level to describe what code means is, well, code.
I've spend the past month-or-two on developing a platform scroller game for smartphones using nothing but AI. It has created everything given my instructions, to my liking, from the way the game is played to graphical assets to sound effects. It looks and feels like any other typical studio-developed game you could find in the store, the difference is that I didn't have a studio of professionals working fulltime for months or years with various levels of project management and what not. It's just me. Me and my agentic coding subscription.
Vibe coding originally meant giving an LLM an instruction, having it regurgitate something potentially wholesome, testing it and feeding back the errors for it to fix it. What I've done in the past couple of months is so, so much more than that. I'm a puppet master and my coding agent is my puppet. I don't need to understand the code when I can describe the problem I want solved and it will do it 100x faster than I ever could.
I didn't use code to describe this level of abstration. I used English prose - and a heck of a lot of software engineering theory, mind you. If you know the solution requires a quad-tree and binary search because in this case, that is the best option, then that's an instruction, not a block of code.
Is it a challenge to Agile methodologies? I mean is most software today designed at all? If we want a pm to be able to write half baked feature ideas and expect the devs to sort it out, rather than just paste the ticket into Claude, don't we make this chaos inevitable. Maybe there is a missing formal engineering layer missing between feature request and prompt. Llms seem to do much better when they are told exactly what to do, vs when they are given room to...express themselves
We could rephrase it like this: “AI has removed all the excuses we had not to remove technical debt”
Yes, AI developed code might have vulnerabilities or bugs, but so already has your JSP based web server that has Oracle and God knows what else among dependencies.
I think what's fascinating is that the mentality of "It's too painful to attack" as a security strategy isn't working anymore in the era of AI. A lot of open issues and bugs were kept because folks assumed as long as they were less vulnerable than other folks they would be fine. However I think that paradigm is being phased out since AI can take the pain of sifting through decades of old code for us to find these vulnerabilities.
Everyone’s levered to the tits in tech debt.
You could just as easily headline this as:
AI is enabling us to fix longstanding software security issues that nobody had the time, skill, or desire to find
All a matter of perspective, I suppose.
It's also introducing a new kind of technical debt much faster.
Which can also then be fixed faster?
we're seeing easier PR approvals when AI is involved. Since changing the code in the future is a lot easier.
Instead of a highly structured engineering process it's more like a gardening process of figuring out what works, what doesn't, what the Business actually wants vs what their requirements were
So more debt because there's more code but it's easier and simpler to change that debt. And in 3 years the product is EOL so shrugs
I think I doubled the amount of tempermanent fixes I pushed in my whole career just in the past two weeks.
Earlier this year, the cybersecurity firm Calif offered to independently audit PHP, sponsored by OpenAI. Within two weeks, a researcher at Calif, aided by AI tools, found a way to use the out-of-memory bug to run arbitrary code on the host server, allowing a malicious actor to install malware or read critical data. Now, it was no longer a minor bug—it was a critical vulnerability, and the Calif team alerted the PHP foundation to start working on a patch immediately.
So, the headline should read,
"AI is helping us eliminate 40 years of technical debt."
Since AI found the bug, and probably also generated the patch at this point.
An abundance of solutions is a problem for people who sell solutions.
> As far as anyone is aware, disaster has been averted: The hack was discovered by cybersecurity researchers who reported it to the PHP Foundation, and a patch was released before any bad actors got their hands on the exploit.
Aren't there still vulnerable services right now being hacked?
In my opinion, the whole paradigm how we use computers needs to be replaced. It is like the 80s, but now with internet and a thread larger then anything in computer history before. I remember boot sector viruses and stuff.
My approach is https://tix.im/ - not launched yet. But from expedience: modeling everything as workflows of composeable nix flakes (expert systems from programs, skills, agents, workflows and their snippets, DBs blobs, service descriptions, ...) and then run everything in highly isolated environments. It is not only so much more efficient, reduces errors, increases quality, reduces unnecessary cognitive load, ...
I start to move more and more daily tasks into the systems. Even UI programs become interactive workflow steps, which allow me to attach from anywhere back to a task in an environment optimized for the task.
Dependency updates for example, a workflow. A subgraph from a security scanner expert checks each change in the dependency tree.
I don't mean to be that guy, but your website feels inscrutable to me. I know that a lot of that might be because it's written by an LLM, but the page seems like it's full of private language and is the product of a deep context conversation with your agent that I don't have enough context to understand.
>In my opinion, the whole paradigm how we use computers needs to be replaced.
>But from expedience: modeling everything as workflows
No thanks. I like private computing. I like all the fun stuff you can do with Computers. I'll never want agent based computers.
personally have started going the other way: AI-assisted microkernels, and only the programming that is actually needed for the problem at hand
my webserver never needed a million lines of code just to pass bytes to other people
What's hillarious is people vibe coding massive AI apps thinking its not creating tech debt and security holes. If anything, AI is discovering existing holes, and creating them faster than ever before in history.
All those "paying down" technical debt with AI, I would advise reading the code _very_ carefully. Sometimes the "solution" which passes some test or adheres to some described functionality is incredibly narrow / brittle, and/or has failure cases that are non-obvious.
Yes I'm using $latest_model on high, its still not there. Unless your threshold for "fixed" is "the 1M line lean proof runs, we don't know how it works", it still requires a lot of attention, hand-holding, and scrutiny, especially when fixing issues / working around failures (where it tends to shortcut or invent workarounds).
A developer on my team introduced new features behind feature flags despite concerns about whether we should have those features and how they are going to be maintained. Changes touched 200+ files which is half of the files.
Turns out nobody wants to use them, they caused many subtle bugs in existing features and is a distraction for many things we do.
The developer left the team, and I am left with purging all of this.
The same can be said with human-written code. LLM code is by far better than the average developer’s code at this point, and there’s not enough top percentile developers to go around, so it’s a wonderful tool.
Also, those very average developers are also the ones doing the code review.
I'm personally not sharing the connotations of negativity towards software development and the term technical debt. I keep reading implications that most of this has always been in some way preventable, but it really is not that simple. How did nobody find these types of exploits 40 years ago? Why are we suddenly shocked and blame the field for this? I think the majority of people severely underestimate the complexity and seriousness of software. Once that perception stops and the economic incentives are right, we can start to look closer.
Just consider how laughable some working conditions in software are and then come to realize how architecture is shaped by the organization communication patterns and not reality. You will see it in a different light then.
And that's the biggest win with AI. With old-fashioned development, it was considered acceptable to defer addressing valid feedback ad infinitum, hiding behind whatever lousy excuse you could come up with. No more.
Of course no vulnerability has ever been found by a human or by fuzzing.
AI does find things, but it is a small percentage the overall vulnerabilities.
IEEE is of course and industry owned organization that shills for BigAI, NSA and any other entity with money or influence.
> Of course no vulnerability has ever been found by a human or by fuzzing.
Who do you shill for? because that sounds like someone who want to fuel the flamewar.
The amount of e-mails I get of the form "Important information regarding your xxxxx account" (which essentially is just code for "we've been hacked and your data has been stolen") is quite concerning, but not surprising. Security by obscurity was always big in most companies, as long as the bar to carrying out sophisticated attacks was putting a costly IT professional to the task you could be relatively relaxed as no one would bother to hack your stupid little e-shop, but now carrying out such an attack costs a few cents to a few dollars at most, that changes the game entirely and everyone is a target.
This summer I had my Docker registry poisoned with malware artefacts, which again is something that no sane attacker would do before as it required downloading my images, carefully modifying them and then overwriting them, ensuring they work as before. Now it's just a simple prompt away. I am definitely ramping up my defenses, everything that doesn't need to be Internet connected goes behind double VPN and port knocking schemes, essentially making my infrastructure invisible from the outside, all unknown packet types get hard dropped, end-to-end encryption of all internal traffic, web endpoints behind CDN proxies and WAF providers (as much as I hate doing that), double and tripe security mechanisms everywhere, maximum restriction of privileges, ultrafast automated patching and AI based continuous monitoring, audits and interventions. You should also get used to that, this will surely still get much worse as attacker tooling gets better.
Why are we even discussing this article? Do you know who Dina Genkina is and what are her contributions to the field?
And you know how are those technical debts created in the first place? More than 40 years of negligence in IT infrastructure.
What does that mean exactly?
I'm not going to lie, I'm having a great time thinking at the endless discussions around unsafe languages and the relentless smugness of many practitioners claiming everything was fine as long as the responsible and the skilled (them) were in charge; and let's not forget the formulaic "with great power comes great responsibility" closers.
Well now, Great Responsibles, it's clear you too were out of your collective depth and were all paying for your smug hubris. Perhaps take a bite out of that humble pie you've obviously never tasted?
We've patched more bugs and more vulnerabilities in the last 3 months than entire years at my last job.
I fear it's only going to get worse.
Worse?
Better you mean? Or are you saying you prefer having unfixed bugs?
Worse as in having to stay up late patching more bugs.
I'm sure the NSA has an eye on the groups that are asking AI about security issues in various popular softwares ...
Better sooner than later, to be honest.
luckily it will help us paying it off too
It also inflated almost all of that debt away, though. So in a way people who built "just ship it" buggy things in the last few years, made the right bet.
That's a weird way to say creating 40 years of technical debt per year.
Yep, the takeaway from this should be that we move to leaner applications that can be actually be formally verified. But what I imagine will actually happen is that we'll patch over these vulns with AI that produces 10x the amount of code, just for the next generation of AI to find vulns in that so we can patch that with more code, repeat ad infinitum.
The most pessimistic interpretation is that AI is superhumanly good at exposing technical debt and not that good at avoiding it.
We'd be replacing decades of technical debt with debt we'd have no hope of paying.
While we're at it, I'd appreciate easy secure authentication of a device without having to pull in a third party authentication provider. Thanks.
I think we'll look back at this moment as peak C/C++ before they begin a long, slow decline into irrelevance.
I've been bullish on Rust for years because memory safety is simply too important and Rust can essentially prove an implementation correct and solve a certain class of very important bugs. It doesn't solve every problem and there is a complexity cost.
I've also said that writing multithreaded code is simply too complex. In an era of out-of-order execution, different levels of caching, asymmetric cores, hyperthreading (although this is largely dead now?), branch prediction and having to support multiple architectures it's simply too hard for most people to write correct multithreaded code. Period. If you find yourself as a non-library developer creating a mutex or spawning a thread, somebody has screwed up.
This is also why I think the single-threaded async/await model with post-request teardown of PHP is the correct choice for most people.
We're going to go through a dark period of a lot of 0days. As the article notes, this is paying 40 years of technical debt. It will result in a more secure ecosystem however. I believe this will ultimately also hamper black markets, governments and 3 letter agencies who trade in those 0days.
As painful as this all is, it will ultimately be a good thing, one of the few to come out of AI (IMHO).
The real slop was the human slop we made along the way.
Yeah but AI also helps in lowering the debt ...