A natural evolution of engineers losing touch with the customers and users.
I'm noticing some of the concern play out regarding AI weakening the capabilities of software people.
I gave the team an exact solution on a silver platter and they still failed to identify how to go about it after 3 days slamming it into Claude. The resolution is literally 1 line of code that could be arrived at in about 30 minutes of patient, old school troubleshooting.
I think what's happening is the AI system draws poorly aligned and led engineers into this ego inflation feedback loop where they are completely detached from reality because these tools can simulate a better one.
When I hear this I do wonder to myself how they're using AI.
For me, as long as I'm properly RPI looping it and not blindly pressing "yes" then it will nearly always reach the solution, usually a fair bit quicker, because it effectively becomes an ideation machine that can keep more thoughts and knowledge in it than my brain can.
However, I'm using it through giving it the context, it has access to the repos, access to the sytems, I'm telling it where the logs are, I have docs to show it what each part of the system is for.
Along with that, it does depend on my own instincts/knowledge for me to read its response and for me to say "no, you've over-thought/engineered this and this is actually the better solution", but its very are nowadays for me that it can't find the 1 liner, as long as I've fed in the right context, mostly pre-done because I've spent a bit of time building out the context tree for the repos/systems in a way that allows it to know what it needs to know.
I'm not doubting that people spin their wheels and couldn't find the one liner, but if its as simple as you say, that does seem like people who aren't great at LLMs along with a lack of instinct/experience.
I guess in essence, I think you can use LLMs in an old school troubleshooting way, and I find it still speeds that up the majority of the time. Its basically how I use it most of the time. And like old-school troubleshooting, if you build out the LLMs context over time, it also grows in capability, as long as its being used as a tool and not blindly trusted.
Should I not assume that most senior developers who are serious about LLMs do this?
> 30 minutes of patient, old school troubleshooting
This is also the best way to understand a codebase, and it's quite enjoyable.
I think the models are getting too egotistical. They're so confident of their fixes they won't bother suggesting basic techniques like isolation of the problem by disabling code paths, for example. They'll keep shotgunning less-and-less likely fixes with undiminishing confidence until the code is full of fixes that do nothing. Then they'll double down on why these should be kept.
It's the uncanny valley of AI. It's still not quite good enough that you can trust it blindly on a big codebase, so you still have to read and understand everything - which is often harder than just writing it up yourself.
Pretty much, The one thing I use it for is as a sanity check, pretty much "Look at <SomeFile>, point out issues you see, summarise them tersely" and it'll spot stuff a code review by a human might have spotted (in the mythical land where people actually do code reviews properly and don't just flag a spelling mistake to "show they looked at it").
Beyond that I don't trust it at all and I still write all my code the meat sack way.
Trust is earned not given and it hasn't earned it yet.
If anything, I think this hype cycle is fast exposing just how many people, teams, and companies just don't care about what is correct. They just wanna feel good about themselves and get paid.
I for one welcome the fact that this whole thing has driven me back to books and deeper into the fundamentals. I have never read so much on math, hardware, and history as in the past 3 years or so.
Your assumption is that LLMs will ever leave this uncanny valley.
Maybe unforeseen breakthroughs and different architectures are achieved. Given LLM fundamental shortcomings grounded in mathematics and information theory, I highly doubt they will and we will always need to deal with these issues in some capacity.
I'm seeing this happen in the security space right now. Someone on my team I was helping train and bring along is all of sudden regressing in their understanding of the issues we're working on, and instead focusing on AI tool outputs to do their job for them.
Usually, I take my time to understand each keyword of the code I'm looking at, especially if it is new to me, like terraform.
I work in a team/with one architect, who only did the DevOps/Infra stuff for the past years and I had the expectation he knows what he is doing and talking about.
At around 2 weeks, I noticed how his knowledge has severe gaps and how he takes things at face value or uses terminology interchangably, which confuses me. It sounds plausible, but it does not actually translate into a working system or shared understanding.
Then one day I did some pair programming with him and whenever there was an error or a resource missing, he would type it into the LLM, copy paste it out of it and then brute force error messages. He did not even wait a second to think or reconcile whats happening on the screen or what the exact requirement is. Never taking one step back and questioning any assumption.
Now that the timeline is shifting and everyone starts to be stressed, he continues to vibe code through me and it is so tiring, there is no higher level planning or architecture, its just a reactive type of trial and error to be faster.
It feels like these people are so used to talking to bots, that they treat you like an agent they can chat to or talk through monologs with.
It is quite shocking how people went from being humble (learn the basics or close the gaps in understanding) to full on authority on everything and berating people 24/7...
So right now I'm considering quitting IT for a couple of years until people calm down, but I think its pretty futile
At my place, this is what they want. They want people to smash through things as fast as possible. They don’t want people to sit and craft a solution which takes in to account the whole. They are choosing tools which are low code, and use llm’s to produce what they need. as they say “this is the way things are going”.
> It is quite shocking how people went from being humble (learn the basics or close the gaps in understanding) to full on authority on everything and berating people 24/7...
That was never humility. That was fear. This technology is just enabling people to show their through colors. The unearned confidence is a hell of a drug.
I don't think it's generalizable. The kind of person who copy and pastes from the AI is the kind who did the same from StackOverflow before. It's more compelling, and we probably see more of them because of it, but it's the same general thing.
The kind of person who insists on understanding things and working through the problem has always been rarer. It's not "humble", it's "inquisitive" and "persistent".
I'm seeing people who _used to be_ like that losing that understanding without realizing it's happening - they have a superficial idea of what the code is doing, enough to feel like they understand it, but the change is apparent when watching them handle something unexpected.
You captured this phenomenon very well in this comment. Appreciate you sharing it because it’s hard to describe exactly what makes this sort of behavior so bizarre.
I don't think this is going to change because it helps automate code production. As in, once we reach a point where this method produces acceptable results, we can fire all engineers and automate the process.
So quit pair programming. I never did, never will do that, nor worked at a place that remotely encouraged that. Each to their own, that's how it should be.
I saw this the past year - new employees would put problems into Claude first instead of debugging. A year back I was debug manually first, now I do the same.
The speed AI debugs at is incredible and yes, we lose touch the more we use it like any manager feet up barking orders to their underlings to get things done.
I it usually doesn't get me in this weird state of mind, but I once spent 6 months (all-in) building a thing that I, once finished, just left alone completely (on disk gathering dust). Weird experience. So I'd say AI physchosis is real.
I've felt that AI can figure out and fix 90% issues, but it rarely does minimal, non invasive fixes. That still requires manual effort. But going from a broad to minimal fix is still a different skillset from actual debugging, so in the the end it does lead to skill atrophy.
So how do you build competence in a world where AI is preached to be the most reasonable way to solve problems because it's supposed to be faster than humans?
Have a single human AI chef. Everyone else has to write an engineering statement and submit it to the AI chef. All that interaction is outside the codebase. Engineers will take turns - perhaps 1 month stints - being the AI chef.
I guarantee you'll spend less on tokens, have better documentation, better code, and most importantly more competent engineers.
I think there are many cases where it was the tech workers themselves who argued for isolation from the customer so that they may focus harder on whatever tasks. I used to be one of these workers. I argued very hard for it. I regret that today.
On the surface it seems rational, but it quickly turns into a system of perverse incentives because now the development team must maintain an illusion that they are constantly overwhelmed with tasks and could never hope to spare a microsecond to assist the customer. This misalignment is how you wind up building your own web frameworks and databases from scratch. It turns into a self serving monster that eventually dominates the entire business. From the perspective of the business, many of these development teams look like they're behind some modern day iron curtain.
I think LLMs have some of the same risks and benefits of stimulant drugs. They can make you more productive if used effectively as a tool, but they can also delude you into thinking you are better than you are and create a dependence such that you aren't just less productive without the LLM/drug, you fail to be productive at all because you don't know how to function without it.
If the solution is so simple, why claude did not found it? At this point we can assume, it is better than 90% of engineers (including me).
After three decades of outsourcing to lowest bidder, I do not buy that humans are somehow better!
> patient, old school troubleshooting
I usually see similar arguments around systems with major red flags (no docs, poor CI, decade ago no CVS...). And engineers with private stash of workarounds for job security!
Claude does not do anything special.
Or perhaps claude was misconfigured, it had no access to relevant part of system, and it tryied to work within its limitation. Often it means decompiling binaries in desperate loop...
The funny thing is that if you never understand the codebase then you will keep thinking Claude is doing a great work delivering all this incredible software, when all it has done is created unnecessary tech debt.
> A twenty line for loop. It overengineers most things.
Anecdote I like to tell.. I was working on a financial planning software, intentionally purely vibe coded as an experiment.
I eventually discovered AI had implemented seven duplicate copies of tax calculation functions. All of them different. All of them wrong. All of them giving different answers for same input.
Not even the most junior of newbie junior engineers would do something this crazy. But AI was happy to do it. It will solve the immediate problem, efficiently. Even if the most efficient solution is something ridiculous like this.
They would not do it in the span of a day or a week. But I’ve definitely seen something like that happen over a period of multiple months.
The llm just allows to generate faster.
We can feel smug about that but all it means is that we need to be clearer on our requirements and preferences up front.
State that similar functions should be in one place and there should be only one. Today there has to be compelling reason why that function is different from others. Normalise the function name based on what it does. Why are there different ones?
Then there are all the other guard rails in place.
Better guidance from mentors, reviewers, and automated project tooling helps everyone. Juniors, seniors, and engineers.
I have also a weird story to tell that a human did and it is as crazy as this. It happen in 2019 so no LLMs at all.
A person that was hired as an expert in our startup spent more than one week full time working on implementing his solution to the problem we were having. I checked the code after one week to see the progress and was curious how they are implementing an already crazy sounding idea.
I found that the whole week was spent re-implementing in python, python's built-in "float" function. That was it, the whole code was just that.
Our problem was related to financial services and their implementation of "float" was not even correct.
I have literally been hearing that, over and over and over, since '22.
And whilst it is obvious things are growing... Saying that, sounds almost entirely like the person saying it cannot objectively look at the environment. If everything has changed in the last six months, why has the industry not radically changed to match it?
Everything really did change with the Pentium II. It did with 3dfx. It did with Damerau's taken on Levenshtein. Hell, everything changed with React. The AI leap with seq2seq completely revolutionised the entire industry. But... Its kid, the LLM? Really?
Its different, in that when you teach that engineer, they either leave because now they hate you, or they grow. They change to meet the standards of a project, rather than inventing their own.
We don't get seniors, without juniors. I'd say more than half the job, is just... Learning. People grow.
Exactly. People ask how we get seniors with juniors using llm. The answer is the same. Review the code. Analyse write down what is wrong. What you expect to have been better. Force every change to be documented and explained enough.
I am former java enterprise dev, so yes I often code this way. Unit testing, decomposition... Some projects CI refuse to merge commits with 20 line loop and duplicated code...
But that is not a point. Claude can code tight compact loops, it just needs to be instructed to do so! If it does "enterprise code", it means it had no instructions about code style.
If your documentation, spec, agent.md does not have proper guidance on coding style... yet another red flag!
So after 30 years of outsourcing to the bottom 10%, you think Claude is better than the bottom 90% even though it’s so stupid that it doesn’t even know it should ask for advice or more information when it’s stuck?
Because simplicity is hard and often the result of careful thought. Anybody can keep piling pile of shit on top of pile of shit which is why that sort of code is so common in our industry.
While I actually agree with you (though, outsourcing to lowest bidder would account for much of what you're seeing with humans), I just saw Bug Hunt Bench scores that gave me some pause:
Claude Opus 4.8 ranks near last on this Bug Hunt benchmark, and missed 96% of the deliberately introduced bugs. If you're a developer who has been falling back to Opus 4.8 because of how Opus 5 talks, and Fable 5 being so expensive that it needs to be rationed... well, turns out Opus 4.8 can actually be quite poor for finding bugs.
(Which feels weird to me, because Opus 4.6 fixed a bug that myself and a group of humans had been hunting down for over a decade. Models are spiky.)
Also surprising to me: Luna Max performing better than Fable 5.1 High, at least on this benchmark. But Astra 6 & Fable 5.1 on Max both perform at the top as you would expect.
Maybe, but at least the benchmark provides an objective measurement of the codebase it is tested on. You're also assuming the bugs are newly introduced / regressions.
I can give a concrete example - Fable will not interact with bugs that result in writing to null pointers in C code. That triggers the guardrails and ends the session. If Luna (or GLM Flash, etc) will fix those kinds of memory bugs, that immediately puts it ahead of Fable in some ways, no matter how tiny Luna is. Again, models are spiky.
I still agree with your initial point! It's LLMs all the way down over here. It would need to be a particularly gnarly bug & an exceptionally talented human for me to want to pay another human to work on fixing it now.
> At this point we can assume, it is better than 90% of engineers (including me).
Hard disagree. We absolutely cannot assume that. You can posit it, and we can have an informed debate about it. This is what irks me the most about LLM fans: they constantly try to reframe the debate to have their worldview as the agreed-upon starting point.
Author has a good head on their shoulders, but few if any companies are going to spend time on incident simulations for their SREs.
Why not? Because even pre-AI, very few companies spend time practicing restoring their backups, or disaster recovery, or picking infrequently-used runbooks to practice, or seeing whether they can easily rotate secrets without downtime, or trying to redploy the system onto another vendor's cloud/platform, or, or, or... It is the least-sexy operations work that exists. No executive cares about this. Ops organizations push for flashy work, same as everybody else: new infrastructure for new projects, cool chatbots, new flashy dashboards, make charts go up and to the right, etc.
Airline pilots go through disaster simulation training because the government mandates that training. If it wasn't a condition of holding a pilot's license, no company would pay for it.
Want SREs to spend time training for disasters? Take a step back. Support professional licensure. Make it a condition of holding a license. You won't get industry-wide professional behavior until you professionalize the work. It won't happen without licensing because every corner cut that is not immediately visible to consumers translates to additional profit, and increasing competition eventually requires these corners to be cut in order to keep up with competition and stay in business. Forcing all players to submit to licensing requires all players to pay these costs and thus forbids them from cutting them to become more competitive.
>Why not? Because even pre-AI, very few companies spend time practicing restoring their backups, or disaster recovery, or picking infrequently-used runbooks to practice, or seeing whether they can easily rotate secrets without downtime, or trying to redploy the system onto another vendor's cloud/platform, or, or, or...
Exact. Let insurance cover it, say sorry to your customers twice and shwoop never happened.
The more I use it, the more I have to rely on it to make changes/ fix things in the same system. In the end, I come out feeling empty; no intuitive knowledge of the system "I" built or fixed.
Code review is important but it does not replace the mental model I am able to build when I do all the steps of software development manually without AI.
No worries, at some point your hit the wall with it and the reality will force you to look at the code. It won't be nice, but until that point delulu land is sustainable enough to fall forward
>The more I use it, the more I have to rely on it to make changes/ fix things in the same system. In the end, I come out feeling empty; no intuitive knowledge of the system "I" built or fixed.
Most of us build a system to deliver a product to make money. We don't feel empty for not knowing how the system works, since we don't really care: it's just a means to an end.
It's the uncanny valley of AI. It's still not quite good enough that you can trust it blindly on a big codebase, so you still have to read and understand everything - which is often harder than just writing it up yourself.
I find that it helps a little to maintain an iron grip on the design. Take the time for really detailed change planning. Be pedantic and push back against every little thing that seems wrong or even a little odd in any plan document the LLM spews out. Likewise, watch for the for the moment when the LLM eventually stumbles and starts to make excuses for deviations from the plan. Interrupt immediately and force it back onto the plan (or your vision), potentially with pointers to resolve immediate problems. Completely unattended agents just don't work for important code.
The more code writes autonomously, the less intuition the human owners have about that code. Loss of intuition is a seed of technical debt that grows with time. Over a long enough horizon, it can make looking at your own codebase feel like the first day on the job (sometimes at a company you started).
Luckily, there are ways to mitigate this and essentially translate those human intuition of how the codebase “should” be into guardrails for the agents. But without that, your setting your sails in a stochastic sea where each wave looks nothing like the last.
I've been thinking about this lately - is it like using 3rd party libs to achieve stuff faster? As much as I would lovr to hand craft the datetime logic in my app, I might as well use luxon and invest this time somewhere else. Only now with llms, you get virtually infinite 3rd party libs you can use, you create them on the fly. So if you have strong engineering values, I would say simply it boils down to "contracts over programs", you can still be in touch with the logic that glues it all together and treat some logic as a blackbox the same way we do with 3rd party libs?
Considering how many compromises and problems third party libraries have caused recently, I would say the comparison is apt - if you don't know what you're shipping one day it might explode on you
It's not because with libraries you have a boundary somewhere and can decide to not care what's inside as long as the interface is stable and well designed. The problem of course you need to prioritise building well designed interfaces and decouple components from each other, and that's a skill most developers aren't good at.
I see aviation sentiment raised from time to time. In aviation failure is catastrophic, and systems you operate do not change on the fly (pun intended).
You can probably drill SREs that way but you are only teaching them how to _react_ efficiently, not how to _fix_ unique unicorn root causes of these failures. Aviation analogy would be an airplane engineer that tries to drill himself for in-flight engine failures AND engine debugging / engine redesign both at the same time.
It never made sense in software engineering, and never will. Minute spent on drills is a minute better spent on reworking codebase to reduce changes of future incidents. This post is very SRE centric.
This is why the paradigm for AI use should not be automation but rather the cyborg. Under automation, people are less active and engaged and become mere operators of automated processes. They become slaves of the machines. Under the cyborg model, they arrange the machines in a way to make people masters of a universe that includes the machines helping them be that.
This has already happened in other areas. I'm old enough to remember when customer service people spent all day handling customer problems themselves, which gave them the understanding needed to solve more unusual problems.
These days they do what the computer tells them. Even if they could solve the problem they don't have the agency. They can only select from the options they are given.
In software we still have the agency but we are giving away the understanding. The agency will follow.
I work with programmers and it's not uncommon that they can remember with shocking detail about code they've written in the past.
Someone might mention an issue that has cropped up and they'll stare off into space for a few moments and actually remember where that issue stems from in the code, because they remember writing it like 8 months ago.
This skill will be lost when AI is generating all code, we'll be stuck in a perpetual loop of having AI keep track of the state of the code in order for AI to extend and maintain it.
I mean I think this is just how AI already operates. I’ve seen multiple models go digging in the git blame or past PRs, and even unmerged PRs. When agents work on a project that has well-defined in-repo docs, agents normally update them without prompting. So I doubt this will be a big issue for AI, but agreed we’re losing a skill
I've seen a variation of this where random engineers are pulled into production incident calls and engineers are not expected to be familiar with the system.
They were asked to "just use AI" and understand the component, triage the issue, build a fix etc. The engineer was forced to choose between accepting a potentially mediocre fix AI has suggested or risk being coming across as an incompetent resource who doesn't know how to leverage AI.
You can guess what the engineer chose. The fix wasn't bad but it was suboptimal for some edge cases. We had to later revise it. Have enough of these situations, engineers will eventually definitely give up understanding the system in detail.
I feel a lot of comments here are missing the forest for the trees.
We do not yet have the next generation systems that will manage AI creation and maintenance of systems.
Humans have been making spaghetti code systems and maintaining them poorly for years. Best practices developed… eventually. But certainly not in the 70s and 80s. Spaghetti was the norm for quite some time.
The development paradigm has changed. Forever. You can’t expect yesterday’s tools for managing software development to succeed at this point. We’re still roughly on year one of this transformation.
The new bottlenecks are creating and enforcing boundaries in the code, identifying level of risk within a boundary, subjecting high risk areas to more intense human review and architecture reworks, and so so much more manual testing.
We need a new language for high level development that focuses on architectural constraints. We need analysis software that helps draw boundaries, identify what needs extended human attention, and helps us map and understand a rapidly developing code base. We need to standardize on the use of frameworks and languages like we have on assembly variants.
I’ll bring up a hacker news trope here. NaughtyDog’s GOAL was an amazing system that was a product of an underdeveloped ecosystem for game development at the time. They used lisp and assembly because you still had to write assembly for performance reasons then.
We are in a very similar period. The ecosystem is underdeveloped. We should be looking for new languages and tools to manage this.
CNC machines used to run from punch cards and then hand written NC code. Now we have advanced CAM software. (Which innovative people are actually running with LLMs!)
I’m not sure who will be solving these incidents if 99% of incidents will be “solved” by LLMs. If I’m called once a year my daily rate will be my yearly rate?
If capability increase continues as it has, then an incident that cannot be resolved by AI will stump humans no matter the practice.
I like the plane example from the article,but I think in reality it will be like code. 1.5 years ago engineers would routinely say that they still write code by hand here or there to keep their skills sharp, and that's just not something you hear much if at all.
If an SRE is faced with a situation an AI can't solve, then said SRE will use the AI systems to triage further, point it to different places and so on.
This works for SREs with pre-AI experience and intuition, possibly less so with new recruits coming in post-AI. I don't know what the solution to this is, maybe practice drills is it, but I have a hunch the entire field will be subsumed, same as many other engineering fields.
There is only so much need for taste and judgement, before even that has been incorporated into the models.
It’s tough. The models can at this point very quickly identify issues in a Kubernetes cluster, for example. This because these systems give you a TON of observability, and it can easily see all the different moving parts.
That doesn’t mean the proposed solution is always right, but it is absolutely landing on the root issue faster than most humans would be able to, even pre-AI. Just because it can remember and run through a bunch of commands more quickly that I can.
There are lots of incidents where the symptom doesn’t always clearly point to the issue, so having something that can fairly exhaustively check a lot of different things very quickly is pretty useful!
But I at least partly agree, I think the more complicated and obtuse a system is, the harder it is for AI to do this. But we’ve invested time in making systems much more straightforward to understand and operate from one place (Kubernetes in general), and AI thrives on that.
And yeah, it means your debugging skills wane a bit because, yeah, if the bot can diagnose the issue in 5 minutes, it’d be irresponsible to not use it.
And I’m not really happy about it, and personally I’ve always been able to figure out a tricky bug given enough time. I don’t want to loose that skill. But everyone is under a lot of time pressure these days.
Like the fact that software "engineering" is mostly nothing like real engineering (and it’s further regressing now due to LLM coding!), the general lack of drilling is again one of the things that make software-related stuff look really naive and amateurish from the perspective of those dealing with the real world. Imagine if the military, police, fire service, and so on did not drill and rehearse incident response?
There are drills, tho. It's just that usually they're only done above a certain level. Small companies, "lean" teams and so on don't have (or didn't have) the capacity to implement all those things. Maybe with the exception of netflix and their chaos thing (bring down systems regularly to make sure the whole still works).
But that's also likely to change with AI assistance. Even an "average" system is better than none. So now teams will have the capacity to bring that in to their systems. Backups / recovery drills that are actually tested (either because they're implementing testing or because the AI screws something up and they need to recover). Either way, it'll be included. Same for security ops. And devops.
I still strongly believe that AI assistance is a catalyst / accelerator, and that the "floor" will rise in most domains. So a small team that only had bandwidth to deal with the happy path previously, will now be able to start incorporating processes and procedures that were historically only done at corporate level. And that's a good thing. Even if it won't look like that in the beginning. But we'll get there, eventually.
> If capability increase continues as it has, then an incident that cannot be resolved by AI will stump humans no matter the practice.
I disagree with this. Whatever the AI produces must be embodied in some kind of information system. The moment the output is on disk, it's fish in a barrel for any competent operator.
I've worked in environments that are beyond the pale with regard to complexity. It will take AI another 10 years to product something as complicated and coherent as a semiconductor manufacturing operating system, which is clearly feasible for humans to manage today.
If you are in a situation where you dont know what happened when something goes wrong, the business incentives will not accept “its too complex” as an answer.
Firms aren’t just selling products, they are selling reliability and taking on liability.
Nah, LLM models are already the new compilers. A commodity only engineers know how to use (in the context of software engineering in production environments)
Out of context, but to address "AI will replace engineers".
Recently discussed something about economy/investing with a friend while at work at a slaughterhouse. I really didn't want him to get scammed buying crypto.
So, used ChatGPT to find some sources in Somali, a 3 videos with short description why it's worth watching. Intro into investing, intro about cryptocurrencies and about buying them. Had the text shortened down to 3 pretty short paragraphs, not more than twice this post.
He's a smart guy, but only went to primary Qur'an school. Doesn't read or such, mostly consumes internet in form of video/media. He couldn't read those 3 paragraphs, it was too long. Or rather, it wasn't just 3 paragraphs, it was a lot to read.
Maybe we're already dividing into murlocs and the surface dwellers?
Anyone who's worked in tech in a large company will probably have experienced having an ops team who use RPA tools to do repetitive tasks that tech teams get the blame for when things break. AI will make this so much worse. Things will break, everyone will assume 'tech knows the system', but really it's a new process outside of the tech teams that someone vibe coded but got it wrong.
Audit trails, logs, and tight data governance where things can only be accessed with proper roles is the only possible solution.
If an RPA team ever gets direct access to a production database in your company, look for a new job.
> Robotic process automation is a type of business process automation that automates tasks within business and IT processes using scripts that mimic human interaction with application user interfaces.
For anyone else wondering what RPA means. Never heard that abbreviation before.
This article leads up to a hypothesis presented without evidence.
Yes, it's clear that if AI agents handle routine issues then only the most complex issues will be handled by humans. In no way does it follow that humans will be less prepared to handle those incidents than they are now; by definition, this class of problems have no rote solutions. Each issue of this type already requires deep system knowledge to remediate, and still will. Software is not aviation.
What we are seeing here is an instance of the more general trend where experts are still required to operate complex systems, but AI is destroying the career path that creates those experts. That's what we should be worrying about, not that people are going to be spending less time rebooting boxes. This isn't something you can fix with a "simulator."
I read the article before commenting, as is my habit. A disagreement is not always a failure to process.
To be very explicit, my problem is that the article argues that simulation will be a suitable substitute in the software space, as it is in the aviation space. I see no evidence for that; it's pure hypothesis taken as fact.
I would say that Bainbridge supports my position, if anything. Take a look at section 2.3 of her paper, where she lists all the reasons why simulation can be unsuitable in preparing for complex incidents: https://davidjusth.com/s/Ironies-of-Automation_Bainbridge_19...
When someone else -whether AI agent or a human- solves the recurring minor problems for you, those problems become non-issue, get swept under the rug, just to accumulate more dust.
One day, those may become bigger as they are forgotten, causing havoc. The standard root-cause-analysis depending on systems having certain retention period, which may be expired at that time.
It is important to get real hold of one's systems from end-to-end aspect, which holds true for both AI and human operators...
Meta: The blinking cursor of the "logo" of the blog being sticky in the top left corner makes it impossible for me to read the text. It constantly fires interrupts at me.
Depending on what your goals as the author are, you may or may not want that.
Being able to scroll it out of view might be enough to achieve the aesthetics goal, and the goal of people actually listening to you.
The internet for me (and increasingly non-internet programs) is unusable at this point without excessive modifications. It's hard to imagine what it's like being unbothered by it all.
Isn't there anywhere to "go" from here? In the last decades, introducing new high level abstractions on top of existing paradigms naturally had everyone move up the ladder and work at the next higher level, why should this be different these days? Do we think AI will reach the top of the abstraction ceiling, so there's no where to go from here?
This isn't abstraction though. Outsourcing is a better term.
If things continue moving up that latter, you will see that your agent/agency will pass the buck too. But there should always be some last turtle. Maybe that turtle will be the human that thought he was climbing the latter, who knows.
Coding via LLM is not similar to using an abstraction. Imagine a car. The controls like steering wheel, the pedals, the gear levers. Those are abstractions.
But using LLMs are like driving using a remote control that has probabilistic behavior. You just loss what it feels to be in a car and you fail to improve as a driver because of the erratic remote control.
You’d hope these AI incident responders have very constrained production tools to fix things. You’d hope the humans remain familiar with those tools and they are incredibly well documented.
The article does not get the point. What aircraft companies did was separate training and work, and SRE/IT typically does not.
An AI can handle routine incidents and then present learning cases from that routine work for training, because the skill in SRE is not the mechanical log grepping, grafana dashboard browsing etc but forming the hypothesis. AI incident reports can create training cases that are a much better training for hypothesis forming and testing than the work itself can.
Prediction: this won’t happen. The abstraction will be good enough and people will need to know only as much as they need to know- things will stabilise at the equilibrium.
Ironies of Automation is front and center in this article which is awesome. Many of the conversations on AI automation are describing or rediscovering the insights the paper covered.
Good article and I like the callouts to the aviation industry. For me what's missing is the author should also have touched on CRM and SRM.
Also, that paper "The Ironies of Automation" is one that everyone should read. It's fairly short.
There is a related problem in terms of these situations where the computer system is handing off to the human. It's called "the bumpy transfer of control." Very fascinating concept.
You rather generously assume engineers are in touch with their systems.
Even before layoffs many teams just maintained things org has long lost coherent knowledge of
After layoffs and typical org knowledge churn - you can either rewrite it (but how? Product team responsible for original implement requirements is long gone too) or recoup (reverse document) some of that lost knowledge with AI and actually learn
I am exceedingly tired of poor metaphors that are popping up since AI has taken over writing.
No, operating software is not like operating a plane. Not at all in fact. The people operating the software and resolving incidents are the same people who created the software in the first place, and continue to work on it day to day. Pilots have not and don't.
I'm not sure if I agree with the plane analogy either. Aren't planes supposed to have exceptionally stable hardware and software platform? Does that not significantly reduce failure modes? Most production software by comparison is constantly evolving. Unless you design very simple or basic training examples, most of it's going to be out of date soon.
The former CTO of a large MSP software company once told me on a call the reason their product had so many features removed with price increases was "you can't maintain a plane while it's in the air"
The immediate response was "We don't, your updates bring the on prem RMM down for hours at a time, the plane is grounded for maintenence regularly"
A natural evolution of engineers losing touch with the customers and users.
I'm noticing some of the concern play out regarding AI weakening the capabilities of software people.
I gave the team an exact solution on a silver platter and they still failed to identify how to go about it after 3 days slamming it into Claude. The resolution is literally 1 line of code that could be arrived at in about 30 minutes of patient, old school troubleshooting.
I think what's happening is the AI system draws poorly aligned and led engineers into this ego inflation feedback loop where they are completely detached from reality because these tools can simulate a better one.
When I hear this I do wonder to myself how they're using AI.
For me, as long as I'm properly RPI looping it and not blindly pressing "yes" then it will nearly always reach the solution, usually a fair bit quicker, because it effectively becomes an ideation machine that can keep more thoughts and knowledge in it than my brain can.
However, I'm using it through giving it the context, it has access to the repos, access to the sytems, I'm telling it where the logs are, I have docs to show it what each part of the system is for.
Along with that, it does depend on my own instincts/knowledge for me to read its response and for me to say "no, you've over-thought/engineered this and this is actually the better solution", but its very are nowadays for me that it can't find the 1 liner, as long as I've fed in the right context, mostly pre-done because I've spent a bit of time building out the context tree for the repos/systems in a way that allows it to know what it needs to know.
I'm not doubting that people spin their wheels and couldn't find the one liner, but if its as simple as you say, that does seem like people who aren't great at LLMs along with a lack of instinct/experience.
I guess in essence, I think you can use LLMs in an old school troubleshooting way, and I find it still speeds that up the majority of the time. Its basically how I use it most of the time. And like old-school troubleshooting, if you build out the LLMs context over time, it also grows in capability, as long as its being used as a tool and not blindly trusted.
Should I not assume that most senior developers who are serious about LLMs do this?
> 30 minutes of patient, old school troubleshooting
This is also the best way to understand a codebase, and it's quite enjoyable.
I think the models are getting too egotistical. They're so confident of their fixes they won't bother suggesting basic techniques like isolation of the problem by disabling code paths, for example. They'll keep shotgunning less-and-less likely fixes with undiminishing confidence until the code is full of fixes that do nothing. Then they'll double down on why these should be kept.
It's the uncanny valley of AI. It's still not quite good enough that you can trust it blindly on a big codebase, so you still have to read and understand everything - which is often harder than just writing it up yourself.
I'd say it's more about learning how to organize your work more efficiently.
If you think about a product like marble: it's something that most be chiseled out of time.
Some people can chisel better products: the AI is just a better chisel.
Sometime still has to guide the chisel and judge the art/product.
In our cases, the market judges products.
Pretty much, The one thing I use it for is as a sanity check, pretty much "Look at <SomeFile>, point out issues you see, summarise them tersely" and it'll spot stuff a code review by a human might have spotted (in the mythical land where people actually do code reviews properly and don't just flag a spelling mistake to "show they looked at it").
Beyond that I don't trust it at all and I still write all my code the meat sack way.
Trust is earned not given and it hasn't earned it yet.
If anything, I think this hype cycle is fast exposing just how many people, teams, and companies just don't care about what is correct. They just wanna feel good about themselves and get paid.
I for one welcome the fact that this whole thing has driven me back to books and deeper into the fundamentals. I have never read so much on math, hardware, and history as in the past 3 years or so.
Your assumption is that LLMs will ever leave this uncanny valley.
Maybe unforeseen breakthroughs and different architectures are achieved. Given LLM fundamental shortcomings grounded in mathematics and information theory, I highly doubt they will and we will always need to deal with these issues in some capacity.
I'm seeing this happen in the security space right now. Someone on my team I was helping train and bring along is all of sudden regressing in their understanding of the issues we're working on, and instead focusing on AI tool outputs to do their job for them.
I can share a weird story:
Usually, I take my time to understand each keyword of the code I'm looking at, especially if it is new to me, like terraform.
I work in a team/with one architect, who only did the DevOps/Infra stuff for the past years and I had the expectation he knows what he is doing and talking about.
At around 2 weeks, I noticed how his knowledge has severe gaps and how he takes things at face value or uses terminology interchangably, which confuses me. It sounds plausible, but it does not actually translate into a working system or shared understanding.
Then one day I did some pair programming with him and whenever there was an error or a resource missing, he would type it into the LLM, copy paste it out of it and then brute force error messages. He did not even wait a second to think or reconcile whats happening on the screen or what the exact requirement is. Never taking one step back and questioning any assumption.
Now that the timeline is shifting and everyone starts to be stressed, he continues to vibe code through me and it is so tiring, there is no higher level planning or architecture, its just a reactive type of trial and error to be faster. It feels like these people are so used to talking to bots, that they treat you like an agent they can chat to or talk through monologs with.
It is quite shocking how people went from being humble (learn the basics or close the gaps in understanding) to full on authority on everything and berating people 24/7...
So right now I'm considering quitting IT for a couple of years until people calm down, but I think its pretty futile
Great comment.
At my place, this is what they want. They want people to smash through things as fast as possible. They don’t want people to sit and craft a solution which takes in to account the whole. They are choosing tools which are low code, and use llm’s to produce what they need. as they say “this is the way things are going”.
> It is quite shocking how people went from being humble (learn the basics or close the gaps in understanding) to full on authority on everything and berating people 24/7...
That was never humility. That was fear. This technology is just enabling people to show their through colors. The unearned confidence is a hell of a drug.
I don't think it's generalizable. The kind of person who copy and pastes from the AI is the kind who did the same from StackOverflow before. It's more compelling, and we probably see more of them because of it, but it's the same general thing.
The kind of person who insists on understanding things and working through the problem has always been rarer. It's not "humble", it's "inquisitive" and "persistent".
I'm seeing people who _used to be_ like that losing that understanding without realizing it's happening - they have a superficial idea of what the code is doing, enough to feel like they understand it, but the change is apparent when watching them handle something unexpected.
LLMs reduced interest on tech debt, but it's still there and people who have a tendency to acquire it will go bankrupt eventually.
The difference between the two is that friction is learning.
the less friction the less growth.
So even if they copied from SO they would have better knowledge than an Ai user.
This isn’t speculation either, theres an MIT study which this is based on. https://www.media.mit.edu/publications/your-brain-on-chatgpt...
You captured this phenomenon very well in this comment. Appreciate you sharing it because it’s hard to describe exactly what makes this sort of behavior so bizarre.
I don't think this is going to change because it helps automate code production. As in, once we reach a point where this method produces acceptable results, we can fire all engineers and automate the process.
>> he continues to vibe code through me
So quit pair programming. I never did, never will do that, nor worked at a place that remotely encouraged that. Each to their own, that's how it should be.
I saw this the past year - new employees would put problems into Claude first instead of debugging. A year back I was debug manually first, now I do the same.
The speed AI debugs at is incredible and yes, we lose touch the more we use it like any manager feet up barking orders to their underlings to get things done.
In my mind, this, not copyright or water use, is the best reason to boycott AI. It'll make you incompetent.
At least at my company the OKRs are quite clear and demand heavy AI utilization above all else
I gave the team an exact solution on a silver platter and they still failed to identify how to go about it
I think what's happening is ... poorly aligned and led engineers [in] this ego inflation feedback loop where they are completely detached from reality
A story about a team of humans with some very human problems.
I it usually doesn't get me in this weird state of mind, but I once spent 6 months (all-in) building a thing that I, once finished, just left alone completely (on disk gathering dust). Weird experience. So I'd say AI physchosis is real.
I've felt that AI can figure out and fix 90% issues, but it rarely does minimal, non invasive fixes. That still requires manual effort. But going from a broad to minimal fix is still a different skillset from actual debugging, so in the the end it does lead to skill atrophy.
Sincerely , I think you're blaming the AI incorrectly there. You just got incompetents on your payroll.
So how do you build competence in a world where AI is preached to be the most reasonable way to solve problems because it's supposed to be faster than humans?
Have a single human AI chef. Everyone else has to write an engineering statement and submit it to the AI chef. All that interaction is outside the codebase. Engineers will take turns - perhaps 1 month stints - being the AI chef.
I guarantee you'll spend less on tokens, have better documentation, better code, and most importantly more competent engineers.
That's like asking "how do you build competence in plowing a field with a horse in a world of tractors"
incompetent people, surely?
I don't think it's engineers, it's the rest of the org insulating the tech workers from every side of the business
I think there are many cases where it was the tech workers themselves who argued for isolation from the customer so that they may focus harder on whatever tasks. I used to be one of these workers. I argued very hard for it. I regret that today.
On the surface it seems rational, but it quickly turns into a system of perverse incentives because now the development team must maintain an illusion that they are constantly overwhelmed with tasks and could never hope to spare a microsecond to assist the customer. This misalignment is how you wind up building your own web frameworks and databases from scratch. It turns into a self serving monster that eventually dominates the entire business. From the perspective of the business, many of these development teams look like they're behind some modern day iron curtain.
I think LLMs have some of the same risks and benefits of stimulant drugs. They can make you more productive if used effectively as a tool, but they can also delude you into thinking you are better than you are and create a dependence such that you aren't just less productive without the LLM/drug, you fail to be productive at all because you don't know how to function without it.
That sounds somewhat applicable to many tools. Like Vim/Emacs, for example. Or computers and smart phones in general.
Yeah, like caffeine is to meth
If the solution is so simple, why claude did not found it? At this point we can assume, it is better than 90% of engineers (including me).
After three decades of outsourcing to lowest bidder, I do not buy that humans are somehow better!
> patient, old school troubleshooting
I usually see similar arguments around systems with major red flags (no docs, poor CI, decade ago no CVS...). And engineers with private stash of workarounds for job security!
Claude does not do anything special.
Or perhaps claude was misconfigured, it had no access to relevant part of system, and it tryied to work within its limitation. Often it means decompiling binaries in desperate loop...
Claude regular spits out six helper functions instead of... A twenty line for loop. It overengineers most things.
Overabstracting, deduplicating things that don't need to be. Building metaclasses because it saw a single orchestrator in the whole codebase.
If it is a better engineer than you... You need practice.
The funny thing is that if you never understand the codebase then you will keep thinking Claude is doing a great work delivering all this incredible software, when all it has done is created unnecessary tech debt.
Which at some point the developers who actually still do know how to program are either going to have to clean up or in terminal cases rewrite.
In a way all we've done (currently) is drastically expand the amount of technical debt across the whole industry.
Should be profitable for the ones who can still actually program though and haven't let their skills atrophy by letting Claude do everything.
I don't deny there are use cases for LLM's I'm just don't buy the hype about them either.
As with all tools, you have to understand how to use them to get done what you need to get done without sticking the chisel through your hand.
> A twenty line for loop. It overengineers most things.
Anecdote I like to tell.. I was working on a financial planning software, intentionally purely vibe coded as an experiment.
I eventually discovered AI had implemented seven duplicate copies of tax calculation functions. All of them different. All of them wrong. All of them giving different answers for same input.
Not even the most junior of newbie junior engineers would do something this crazy. But AI was happy to do it. It will solve the immediate problem, efficiently. Even if the most efficient solution is something ridiculous like this.
They would not do it in the span of a day or a week. But I’ve definitely seen something like that happen over a period of multiple months.
The llm just allows to generate faster.
We can feel smug about that but all it means is that we need to be clearer on our requirements and preferences up front.
State that similar functions should be in one place and there should be only one. Today there has to be compelling reason why that function is different from others. Normalise the function name based on what it does. Why are there different ones?
Then there are all the other guard rails in place.
Better guidance from mentors, reviewers, and automated project tooling helps everyone. Juniors, seniors, and engineers.
I have also a weird story to tell that a human did and it is as crazy as this. It happen in 2019 so no LLMs at all.
A person that was hired as an expert in our startup spent more than one week full time working on implementing his solution to the problem we were having. I checked the code after one week to see the progress and was curious how they are implementing an already crazy sounding idea. I found that the whole week was spent re-implementing in python, python's built-in "float" function. That was it, the whole code was just that.
Our problem was related to financial services and their implementation of "float" was not even correct.
Which model though? I have similar anecdotes but all with older models. The jump in capabilities in the last 6 months has been substantial.
I have literally been hearing that, over and over and over, since '22.
And whilst it is obvious things are growing... Saying that, sounds almost entirely like the person saying it cannot objectively look at the environment. If everything has changed in the last six months, why has the industry not radically changed to match it?
Everything really did change with the Pentium II. It did with 3dfx. It did with Damerau's taken on Levenshtein. Hell, everything changed with React. The AI leap with seq2seq completely revolutionised the entire industry. But... Its kid, the LLM? Really?
For some definitions of efficiency.
Oh and how is it any different than most software engineers?
How many times I heard ORM are bad only to recreate the same shit?
How many times I heard ORM had bad performance and see 1+n stuff everywhere?
How many times I have seen tight coupling in the name of DRY?
Its different, in that when you teach that engineer, they either leave because now they hate you, or they grow. They change to meet the standards of a project, rather than inventing their own.
We don't get seniors, without juniors. I'd say more than half the job, is just... Learning. People grow.
Exactly. People ask how we get seniors with juniors using llm. The answer is the same. Review the code. Analyse write down what is wrong. What you expect to have been better. Force every change to be documented and explained enough.
I am former java enterprise dev, so yes I often code this way. Unit testing, decomposition... Some projects CI refuse to merge commits with 20 line loop and duplicated code...
But that is not a point. Claude can code tight compact loops, it just needs to be instructed to do so! If it does "enterprise code", it means it had no instructions about code style.
If your documentation, spec, agent.md does not have proper guidance on coding style... yet another red flag!
> it just needs to be instructed to do so!
Considering how often it overrules, its own rules?
100%
It is my pet peeve with Claude and why I don't prefer it for most stuff
(also the comment spam - but that's a all of them in a way or another)
So after 30 years of outsourcing to the bottom 10%, you think Claude is better than the bottom 90% even though it’s so stupid that it doesn’t even know it should ask for advice or more information when it’s stuck?
It just follows instructions you give it.
Some asian devs will go for weeks without asking for help, all while giving amazing fake status reports. Loosing face etc...
Do you think those devs are in the top 10% of all devs like you said Claude is? Or is the bar suddenly much lower after all?
You're comparing scammers to incompetence
Because simplicity is hard and often the result of careful thought. Anybody can keep piling pile of shit on top of pile of shit which is why that sort of code is so common in our industry.
While I actually agree with you (though, outsourcing to lowest bidder would account for much of what you're seeing with humans), I just saw Bug Hunt Bench scores that gave me some pause:
https://x.com/PawelHuryn/status/2095982259761475945
https://bughunt.productcompass.pm/?preset=all
Claude Opus 4.8 ranks near last on this Bug Hunt benchmark, and missed 96% of the deliberately introduced bugs. If you're a developer who has been falling back to Opus 4.8 because of how Opus 5 talks, and Fable 5 being so expensive that it needs to be rationed... well, turns out Opus 4.8 can actually be quite poor for finding bugs.
(Which feels weird to me, because Opus 4.6 fixed a bug that myself and a group of humans had been hunting down for over a decade. Models are spiky.)
Also surprising to me: Luna Max performing better than Fable 5.1 High, at least on this benchmark. But Astra 6 & Fable 5.1 on Max both perform at the top as you would expect.
Still, basic debuging and trouble shooting is where LLM generally shine. Any model can bisect git history and isolate newly introduced bug.
If model can not automatically reproduce bug, while human manually can... you got a problem in CI.
> Luna Max performing better than Fable 5.1 High
Perhaps you are reading too many benchmarks.
Edit for answer :
I agree Luna is great cheap model.
But if Fable was hitting security limits, yet was still included in benchmarks...
What flies better? Elephant or paper plane. You can make objective benchmark about that. But not much value for logistics company
> Perhaps you are reading too many benchmarks.
Maybe, but at least the benchmark provides an objective measurement of the codebase it is tested on. You're also assuming the bugs are newly introduced / regressions.
I can give a concrete example - Fable will not interact with bugs that result in writing to null pointers in C code. That triggers the guardrails and ends the session. If Luna (or GLM Flash, etc) will fix those kinds of memory bugs, that immediately puts it ahead of Fable in some ways, no matter how tiny Luna is. Again, models are spiky.
I still agree with your initial point! It's LLMs all the way down over here. It would need to be a particularly gnarly bug & an exceptionally talented human for me to want to pay another human to work on fixing it now.
> At this point we can assume, it is better than 90% of engineers (including me).
Hard disagree. We absolutely cannot assume that. You can posit it, and we can have an informed debate about it. This is what irks me the most about LLM fans: they constantly try to reframe the debate to have their worldview as the agreed-upon starting point.
That final 10% is the hard part. 90% is easy.
Author has a good head on their shoulders, but few if any companies are going to spend time on incident simulations for their SREs.
Why not? Because even pre-AI, very few companies spend time practicing restoring their backups, or disaster recovery, or picking infrequently-used runbooks to practice, or seeing whether they can easily rotate secrets without downtime, or trying to redploy the system onto another vendor's cloud/platform, or, or, or... It is the least-sexy operations work that exists. No executive cares about this. Ops organizations push for flashy work, same as everybody else: new infrastructure for new projects, cool chatbots, new flashy dashboards, make charts go up and to the right, etc.
Airline pilots go through disaster simulation training because the government mandates that training. If it wasn't a condition of holding a pilot's license, no company would pay for it.
Want SREs to spend time training for disasters? Take a step back. Support professional licensure. Make it a condition of holding a license. You won't get industry-wide professional behavior until you professionalize the work. It won't happen without licensing because every corner cut that is not immediately visible to consumers translates to additional profit, and increasing competition eventually requires these corners to be cut in order to keep up with competition and stay in business. Forcing all players to submit to licensing requires all players to pay these costs and thus forbids them from cutting them to become more competitive.
>Why not? Because even pre-AI, very few companies spend time practicing restoring their backups, or disaster recovery, or picking infrequently-used runbooks to practice, or seeing whether they can easily rotate secrets without downtime, or trying to redploy the system onto another vendor's cloud/platform, or, or, or...
Exact. Let insurance cover it, say sorry to your customers twice and shwoop never happened.
I find the use of AI like quicksand.
The more I use it, the more I have to rely on it to make changes/ fix things in the same system. In the end, I come out feeling empty; no intuitive knowledge of the system "I" built or fixed.
Code review is important but it does not replace the mental model I am able to build when I do all the steps of software development manually without AI.
No worries, at some point your hit the wall with it and the reality will force you to look at the code. It won't be nice, but until that point delulu land is sustainable enough to fall forward
>The more I use it, the more I have to rely on it to make changes/ fix things in the same system. In the end, I come out feeling empty; no intuitive knowledge of the system "I" built or fixed.
Most of us build a system to deliver a product to make money. We don't feel empty for not knowing how the system works, since we don't really care: it's just a means to an end.
It's the uncanny valley of AI. It's still not quite good enough that you can trust it blindly on a big codebase, so you still have to read and understand everything - which is often harder than just writing it up yourself.
You don’t need to copy-paste the same comment in every reply. It seems like agenda pushing more than contributing.
I find that it helps a little to maintain an iron grip on the design. Take the time for really detailed change planning. Be pedantic and push back against every little thing that seems wrong or even a little odd in any plan document the LLM spews out. Likewise, watch for the for the moment when the LLM eventually stumbles and starts to make excuses for deviations from the plan. Interrupt immediately and force it back onto the plan (or your vision), potentially with pointers to resolve immediate problems. Completely unattended agents just don't work for important code.
The more code writes autonomously, the less intuition the human owners have about that code. Loss of intuition is a seed of technical debt that grows with time. Over a long enough horizon, it can make looking at your own codebase feel like the first day on the job (sometimes at a company you started).
Luckily, there are ways to mitigate this and essentially translate those human intuition of how the codebase “should” be into guardrails for the agents. But without that, your setting your sails in a stochastic sea where each wave looks nothing like the last.
I've been thinking about this lately - is it like using 3rd party libs to achieve stuff faster? As much as I would lovr to hand craft the datetime logic in my app, I might as well use luxon and invest this time somewhere else. Only now with llms, you get virtually infinite 3rd party libs you can use, you create them on the fly. So if you have strong engineering values, I would say simply it boils down to "contracts over programs", you can still be in touch with the logic that glues it all together and treat some logic as a blackbox the same way we do with 3rd party libs?
Considering how many compromises and problems third party libraries have caused recently, I would say the comparison is apt - if you don't know what you're shipping one day it might explode on you
It's not because with libraries you have a boundary somewhere and can decide to not care what's inside as long as the interface is stable and well designed. The problem of course you need to prioritise building well designed interfaces and decouple components from each other, and that's a skill most developers aren't good at.
I see aviation sentiment raised from time to time. In aviation failure is catastrophic, and systems you operate do not change on the fly (pun intended).
You can probably drill SREs that way but you are only teaching them how to _react_ efficiently, not how to _fix_ unique unicorn root causes of these failures. Aviation analogy would be an airplane engineer that tries to drill himself for in-flight engine failures AND engine debugging / engine redesign both at the same time.
It never made sense in software engineering, and never will. Minute spent on drills is a minute better spent on reworking codebase to reduce changes of future incidents. This post is very SRE centric.
This is why the paradigm for AI use should not be automation but rather the cyborg. Under automation, people are less active and engaged and become mere operators of automated processes. They become slaves of the machines. Under the cyborg model, they arrange the machines in a way to make people masters of a universe that includes the machines helping them be that.
For more in this vein, look up "Automation should be like Iron Man, not like Ultron". Sad to see so many people let go of their agency.
without Iron Man's plot armor, Ultron would have won
I still use LLMs, but strictly for sanity checks or quick summaries. Trusting them blindly to debug complex systems often leads to a rabbit hole.
This has already happened in other areas. I'm old enough to remember when customer service people spent all day handling customer problems themselves, which gave them the understanding needed to solve more unusual problems.
These days they do what the computer tells them. Even if they could solve the problem they don't have the agency. They can only select from the options they are given.
In software we still have the agency but we are giving away the understanding. The agency will follow.
Code too.
I work with programmers and it's not uncommon that they can remember with shocking detail about code they've written in the past.
Someone might mention an issue that has cropped up and they'll stare off into space for a few moments and actually remember where that issue stems from in the code, because they remember writing it like 8 months ago.
This skill will be lost when AI is generating all code, we'll be stuck in a perpetual loop of having AI keep track of the state of the code in order for AI to extend and maintain it.
I mean I think this is just how AI already operates. I’ve seen multiple models go digging in the git blame or past PRs, and even unmerged PRs. When agents work on a project that has well-defined in-repo docs, agents normally update them without prompting. So I doubt this will be a big issue for AI, but agreed we’re losing a skill
Will AI remember ? Or rather how will we make AI remember ?
I've seen a variation of this where random engineers are pulled into production incident calls and engineers are not expected to be familiar with the system.
They were asked to "just use AI" and understand the component, triage the issue, build a fix etc. The engineer was forced to choose between accepting a potentially mediocre fix AI has suggested or risk being coming across as an incompetent resource who doesn't know how to leverage AI.
You can guess what the engineer chose. The fix wasn't bad but it was suboptimal for some edge cases. We had to later revise it. Have enough of these situations, engineers will eventually definitely give up understanding the system in detail.
I feel a lot of comments here are missing the forest for the trees.
We do not yet have the next generation systems that will manage AI creation and maintenance of systems.
Humans have been making spaghetti code systems and maintaining them poorly for years. Best practices developed… eventually. But certainly not in the 70s and 80s. Spaghetti was the norm for quite some time.
The development paradigm has changed. Forever. You can’t expect yesterday’s tools for managing software development to succeed at this point. We’re still roughly on year one of this transformation.
The new bottlenecks are creating and enforcing boundaries in the code, identifying level of risk within a boundary, subjecting high risk areas to more intense human review and architecture reworks, and so so much more manual testing.
We need a new language for high level development that focuses on architectural constraints. We need analysis software that helps draw boundaries, identify what needs extended human attention, and helps us map and understand a rapidly developing code base. We need to standardize on the use of frameworks and languages like we have on assembly variants.
I’ll bring up a hacker news trope here. NaughtyDog’s GOAL was an amazing system that was a product of an underdeveloped ecosystem for game development at the time. They used lisp and assembly because you still had to write assembly for performance reasons then.
We are in a very similar period. The ecosystem is underdeveloped. We should be looking for new languages and tools to manage this.
CNC machines used to run from punch cards and then hand written NC code. Now we have advanced CAM software. (Which innovative people are actually running with LLMs!)
I’m not sure who will be solving these incidents if 99% of incidents will be “solved” by LLMs. If I’m called once a year my daily rate will be my yearly rate?
If capability increase continues as it has, then an incident that cannot be resolved by AI will stump humans no matter the practice.
I like the plane example from the article,but I think in reality it will be like code. 1.5 years ago engineers would routinely say that they still write code by hand here or there to keep their skills sharp, and that's just not something you hear much if at all.
If an SRE is faced with a situation an AI can't solve, then said SRE will use the AI systems to triage further, point it to different places and so on.
This works for SREs with pre-AI experience and intuition, possibly less so with new recruits coming in post-AI. I don't know what the solution to this is, maybe practice drills is it, but I have a hunch the entire field will be subsumed, same as many other engineering fields.
There is only so much need for taste and judgement, before even that has been incorporated into the models.
It’s tough. The models can at this point very quickly identify issues in a Kubernetes cluster, for example. This because these systems give you a TON of observability, and it can easily see all the different moving parts.
That doesn’t mean the proposed solution is always right, but it is absolutely landing on the root issue faster than most humans would be able to, even pre-AI. Just because it can remember and run through a bunch of commands more quickly that I can.
There are lots of incidents where the symptom doesn’t always clearly point to the issue, so having something that can fairly exhaustively check a lot of different things very quickly is pretty useful!
But I at least partly agree, I think the more complicated and obtuse a system is, the harder it is for AI to do this. But we’ve invested time in making systems much more straightforward to understand and operate from one place (Kubernetes in general), and AI thrives on that.
And yeah, it means your debugging skills wane a bit because, yeah, if the bot can diagnose the issue in 5 minutes, it’d be irresponsible to not use it.
And I’m not really happy about it, and personally I’ve always been able to figure out a tricky bug given enough time. I don’t want to loose that skill. But everyone is under a lot of time pressure these days.
Like the fact that software "engineering" is mostly nothing like real engineering (and it’s further regressing now due to LLM coding!), the general lack of drilling is again one of the things that make software-related stuff look really naive and amateurish from the perspective of those dealing with the real world. Imagine if the military, police, fire service, and so on did not drill and rehearse incident response?
There are drills, tho. It's just that usually they're only done above a certain level. Small companies, "lean" teams and so on don't have (or didn't have) the capacity to implement all those things. Maybe with the exception of netflix and their chaos thing (bring down systems regularly to make sure the whole still works).
But that's also likely to change with AI assistance. Even an "average" system is better than none. So now teams will have the capacity to bring that in to their systems. Backups / recovery drills that are actually tested (either because they're implementing testing or because the AI screws something up and they need to recover). Either way, it'll be included. Same for security ops. And devops.
I still strongly believe that AI assistance is a catalyst / accelerator, and that the "floor" will rise in most domains. So a small team that only had bandwidth to deal with the happy path previously, will now be able to start incorporating processes and procedures that were historically only done at corporate level. And that's a good thing. Even if it won't look like that in the beginning. But we'll get there, eventually.
Netflix's chaos monkey was this, in a way.
> If capability increase continues as it has, then an incident that cannot be resolved by AI will stump humans no matter the practice.
I disagree with this. Whatever the AI produces must be embodied in some kind of information system. The moment the output is on disk, it's fish in a barrel for any competent operator.
I've worked in environments that are beyond the pale with regard to complexity. It will take AI another 10 years to product something as complicated and coherent as a semiconductor manufacturing operating system, which is clearly feasible for humans to manage today.
If you are in a situation where you dont know what happened when something goes wrong, the business incentives will not accept “its too complex” as an answer.
Firms aren’t just selling products, they are selling reliability and taking on liability.
Nah, LLM models are already the new compilers. A commodity only engineers know how to use (in the context of software engineering in production environments)
Out of context, but to address "AI will replace engineers".
Recently discussed something about economy/investing with a friend while at work at a slaughterhouse. I really didn't want him to get scammed buying crypto. So, used ChatGPT to find some sources in Somali, a 3 videos with short description why it's worth watching. Intro into investing, intro about cryptocurrencies and about buying them. Had the text shortened down to 3 pretty short paragraphs, not more than twice this post.
He's a smart guy, but only went to primary Qur'an school. Doesn't read or such, mostly consumes internet in form of video/media. He couldn't read those 3 paragraphs, it was too long. Or rather, it wasn't just 3 paragraphs, it was a lot to read.
Maybe we're already dividing into murlocs and the surface dwellers?
Anyone who's worked in tech in a large company will probably have experienced having an ops team who use RPA tools to do repetitive tasks that tech teams get the blame for when things break. AI will make this so much worse. Things will break, everyone will assume 'tech knows the system', but really it's a new process outside of the tech teams that someone vibe coded but got it wrong.
Audit trails, logs, and tight data governance where things can only be accessed with proper roles is the only possible solution.
If an RPA team ever gets direct access to a production database in your company, look for a new job.
> Robotic process automation is a type of business process automation that automates tasks within business and IT processes using scripts that mimic human interaction with application user interfaces.
For anyone else wondering what RPA means. Never heard that abbreviation before.
That makes more sense than Rocket Propelled Automation that was my hallucination.
This article leads up to a hypothesis presented without evidence.
Yes, it's clear that if AI agents handle routine issues then only the most complex issues will be handled by humans. In no way does it follow that humans will be less prepared to handle those incidents than they are now; by definition, this class of problems have no rote solutions. Each issue of this type already requires deep system knowledge to remediate, and still will. Software is not aviation.
What we are seeing here is an instance of the more general trend where experts are still required to operate complex systems, but AI is destroying the career path that creates those experts. That's what we should be worrying about, not that people are going to be spending less time rebooting boxes. This isn't something you can fix with a "simulator."
https://en.wikipedia.org/wiki/Ironies_of_Automation
https://www.sciencedirect.com/science/article/abs/pii/000510...
The article is based on evidence, and states it clearly in the body.
I read the article before commenting, as is my habit. A disagreement is not always a failure to process.
To be very explicit, my problem is that the article argues that simulation will be a suitable substitute in the software space, as it is in the aviation space. I see no evidence for that; it's pure hypothesis taken as fact.
The article proposes a resolution based on the challenge and evidence they (correctly) identify at the start.
The simulation solution is backed by their own applied example, and is also the logical intervention based on how the problem is set up.
Perhaps Bainbridge is doing more heavy lifting for me, by filling in blanks and gaps that others are finding.
I would say that Bainbridge supports my position, if anything. Take a look at section 2.3 of her paper, where she lists all the reasons why simulation can be unsuitable in preparing for complex incidents: https://davidjusth.com/s/Ironies-of-Automation_Bainbridge_19...
When someone else -whether AI agent or a human- solves the recurring minor problems for you, those problems become non-issue, get swept under the rug, just to accumulate more dust.
One day, those may become bigger as they are forgotten, causing havoc. The standard root-cause-analysis depending on systems having certain retention period, which may be expired at that time.
It is important to get real hold of one's systems from end-to-end aspect, which holds true for both AI and human operators...
Meta: The blinking cursor of the "logo" of the blog being sticky in the top left corner makes it impossible for me to read the text. It constantly fires interrupts at me.
Depending on what your goals as the author are, you may or may not want that.
Being able to scroll it out of view might be enough to achieve the aesthetics goal, and the goal of people actually listening to you.
If you have something like ublock origin use the block element functionality to target the blinking cursor. It's gone on my end :)
Not just you. Must be nice being fully neurotypical and „not seeing“ all of this kind of stuff.
The internet for me (and increasingly non-internet programs) is unusable at this point without excessive modifications. It's hard to imagine what it's like being unbothered by it all.
I mean seeing this stuff pays the bills and does so quite well, so..
We have been running Agents on infrastructure and letting to create resources, scale up and down, security scans etc.
I agree with premise of thr blog. The question i have been asking internal does knowing your system really matter if you can recreate it in minutes.
We recently had a situation, where in with our internal platform and claude we recreated everything in minutes.
Management in the end cares about the outcome and not how the meat is made.
Isn't there anywhere to "go" from here? In the last decades, introducing new high level abstractions on top of existing paradigms naturally had everyone move up the ladder and work at the next higher level, why should this be different these days? Do we think AI will reach the top of the abstraction ceiling, so there's no where to go from here?
This isn't abstraction though. Outsourcing is a better term. If things continue moving up that latter, you will see that your agent/agency will pass the buck too. But there should always be some last turtle. Maybe that turtle will be the human that thought he was climbing the latter, who knows.
>introducing new high level abstractions...
Coding via LLM is not similar to using an abstraction. Imagine a car. The controls like steering wheel, the pedals, the gear levers. Those are abstractions.
But using LLMs are like driving using a remote control that has probabilistic behavior. You just loss what it feels to be in a car and you fail to improve as a driver because of the erratic remote control.
That’s the big question for sure
You’d hope these AI incident responders have very constrained production tools to fix things. You’d hope the humans remain familiar with those tools and they are incredibly well documented.
The article does not get the point. What aircraft companies did was separate training and work, and SRE/IT typically does not.
An AI can handle routine incidents and then present learning cases from that routine work for training, because the skill in SRE is not the mechanical log grepping, grafana dashboard browsing etc but forming the hypothesis. AI incident reports can create training cases that are a much better training for hypothesis forming and testing than the work itself can.
Prediction: this won’t happen. The abstraction will be good enough and people will need to know only as much as they need to know- things will stabilise at the equilibrium.
We already know this happens though, Bainbridge’s paper came out in 1983.
"Middle devs take care of incidents, senior devs lose touch with their systems"
Ironies of Automation is front and center in this article which is awesome. Many of the conversations on AI automation are describing or rediscovering the insights the paper covered.
Good article and I like the callouts to the aviation industry. For me what's missing is the author should also have touched on CRM and SRM.
Also, that paper "The Ironies of Automation" is one that everyone should read. It's fairly short.
There is a related problem in terms of these situations where the computer system is handing off to the human. It's called "the bumpy transfer of control." Very fascinating concept.
You rather generously assume engineers are in touch with their systems.
Even before layoffs many teams just maintained things org has long lost coherent knowledge of
After layoffs and typical org knowledge churn - you can either rewrite it (but how? Product team responsible for original implement requirements is long gone too) or recoup (reverse document) some of that lost knowledge with AI and actually learn
"THIS IS NOT A DRILL"
It's just a picture of a drill! Thanks, Magritte.
That‘s the goal of the AI tech bro world: get us dependant on their tech, ruin our native/raw skills, lock us into their proprietary skills.
Reminds me of the move from on prem to cloud. Linux sysadmins were killed and replaced by aws focused devops.
I am exceedingly tired of poor metaphors that are popping up since AI has taken over writing.
No, operating software is not like operating a plane. Not at all in fact. The people operating the software and resolving incidents are the same people who created the software in the first place, and continue to work on it day to day. Pilots have not and don't.
I'm not sure if I agree with the plane analogy either. Aren't planes supposed to have exceptionally stable hardware and software platform? Does that not significantly reduce failure modes? Most production software by comparison is constantly evolving. Unless you design very simple or basic training examples, most of it's going to be out of date soon.
> Aren't planes supposed to have exceptionally stable hardware and software platform?
Yes, unless it's Boeing. New system that takes over pitch control? Naw, that can't have bugs, so we don't need to tell pilots how to turn it off.
The former CTO of a large MSP software company once told me on a call the reason their product had so many features removed with price increases was "you can't maintain a plane while it's in the air"
The immediate response was "We don't, your updates bring the on prem RMM down for hours at a time, the plane is grounded for maintenence regularly"