This misses the best use of AI in my opinion, which is to gain understanding. Whst is this bit of code doing? Is there a risk of data leaking here? Are permissions enforced downstream of this function?
Code review can go much deeper now if you use AI to aggressively attack a PR combined with your human insight. Same for planning a feature:
Can I consolidate this logic to a shared function? Does the error surface to the user and are there any gaps? What preexisting functionality is affected by this PR? Can this query be made more efficient?
LLMs are great with focused questions, up and down abstraction layers and across all kinds of concerns. Stack up these focused concerns into a rich understanding of what you are doing or writing.
Understanding is the real output, code is the byproduct.
> LLMs are great with focused questions, up and down abstraction layers and across all kinds of concerns. Stack up these focused concerns into a rich understanding of what you are doing or writing.
I agree. This is much better advice than what the article tries to do, which is to give best practices around how to use or not use AI to code.
Agreed — I also always append to my agents.md — "If there are any technical choices available, please surface them to me so I can understand them and choose"
If your harness has an “AskUserQuestion”-like tool, you can prompt it to use that, or, as I have done, add a stop hook that uses a smaller model to check if there was any similar prose that was output toward the end of the previous response.
Happy to share the implementation or package it up if anyone is interested.
> This misses the best use of AI in my opinion, which is to gain understanding. Whst is this bit of code doing? Is there a risk of data leaking here? Are permissions enforced downstream of this function?
Do I understand when I just let AI regurgitate the function and behavior of the code?
Do you need that analysis every time from an LLM? Those are things I’d expect anyone to grow out of pretty quickly once you get used to understanding code. Especially someone with Senior in their title.
Doing this may give you understanding but won’t really teach you how to understand. For students (as the article is focusing on them), the latter is equally important.
Closely related: one of the biggest AI superpowers I've found is testing. "Please implement a comprehensive unit testing suite for this, making sure to test ..."
You still have to check the work and shouldn't trust it blindly, but in my experience it's pretty damn good.
> Preferably, don’t give the AI access to the codebase. [...] If the AI does have access to the codebase, this access is read-only.
also
> I don’t follow all these religiously.
So if you need an overview of a system, you can and should provide access to the entire codebase. Even the rule itself says "preferably", not "don't ever". If you have different needs, you can always just look at it as a useful guideline. It's just a blog post by some rando on the internet, not the epiphany of AI coding's Ten Commandments.
This only applies to bad code. A formal programmic language beats an informal language anytime. Why on earth would you translate an unambiguous formal language to English?
> Is there a risk of data leaking here?
> Are permissions enforced downstream of this function?
If you trust AI to give you that answer, you are going to run into serious issues.
A formal programmic language beats an informal language anytime. Why on earth would you write a comment in an ambiguous informal language when you could translate it to code?
programmer looks at formal programmic language which has so beaten the informal language that one cannot tell the reason why this stupid bit of code exists in the silly form it does is to prevent an edge condition in a particular browser while still working fine for other browsers.
Luckily genius programmer who understands formal programmic language and does not need weak comments to clarify why the world is not perfect, fixes the stupid looking code to be beautiful, opening company up to some really super neat, seemingly random bugs in one browser that nobody will understand for several months.
tldr: one important reason comments exist is to clarify why code you might expect to look another way looks the way it does, of course if the only reason it looks the way it does is this is the way the AI wrote it I guess I agree the comment is unnecessary.
“But that would be like telling a student in the 70s to pretend that calculators or computers don’t exist.”
If students of the 70s or today pretended they didn’t exist up to a certain point when they needed them to move forward, like bioinformatics or something, they 100% would be better off. There is plenty of research on off loading thinning providing a worse understanding of the material - eg side rules proving a better understanding than calculators.
Exactly. I always think about the ways consumers make terrible decisions with everyday purchases at the grocery store etc. If people could easily estimate unit costs in their head for example, a lot of predatory options would have to go away. But calculators keep people from having to get good with everyday math in their head, while those same calculators sit unused in many everyday scenarios where people don't go to the effort to use them.
Quick estimating (which is a different skill from calculating exact answers using pen and paper) is most useful there. People do carry computers with them these days and they do have calculator apps. Some shops I buy from do have unit prices for many items.
I think AI might be more of a problem, at least for some skills because it is not just doing a small unit of work. its more like the problem of doing more complex calculations. An example I have come across with calculators is students learning statistics without knowing how to calculate a variance, because they just put the numbers into a calculator. There is a failure there to learn the concept - they do not really know what a variance or SD is. I can imagine AI doing a lot of that.
If the challenge is held constant, giving people tools is just teaching a different skill set than teaching them unaugmented. The augmented skill set is more relevant to how people do work in the real world, but working through things unaugmented likely help people build a robust internal schema more quickly. We definitely don't need to spend as much time training unaugmented abilities going forward though, kind of like how the number of assembly related classes in CS departments has gone down over time as we've moved up the stack.
In general, feeding the AI what you write for verification sounds like a good idea. But I think even this should only be done by someone who understands exactly what they're doing, how, and why, because otherwise they won't be able to analyze the accuracy of the AI's responses...
> the term “vibe-coding” suggests a kind of laissez-faire attitude where you don’t really care about the outcome and you’re just having fun. That’s what the phrase meant when it was coined, but the world has moved on. In many companies professional programmers are using AI in such a way that it’s impossible to imagine that they are also reading the resulting code in detail. This is what modern vibe-coding is. Deferring to the AI, not worrying about the individual lines of code, and keeping an eye on whether the code passes its tests and throws up any problems in production.
This way of working is the only one that justifies the trillion-dollar bet on the AI industry. I agree, this method should be called vibe-coding.
It should depend on whether you've taken other steps to constrain the output of the LLM. I've been building some pretty elaborate test harnesses (which I test independently beforehand), to ensure that I get something that does what I ask of it.
I don't care which words we use, but they should separate this sort of empirically-minded work (versus the rationalism of typical development) from just dropping a few sentences into the prompt and crossing your fingers.
Yeah, I understand that this testing-oriented constraint method is what AI coding enthusiasts are pitching. Sure, it's different from pure vibe-coding as originally defined, but nobody is seriously suggesting professionals should do pure vibe-coding anyway. It's a strawman. The word fits.
This article hits very close to what I have come up with myself and its good to see others thinking the same lines.
Core modules: Coded by myself, AI reviews and AI to discover/learn.
Stuff I don't care about Craft: API layer, CLI layer, Smoke tests, Integ tests - Dial AI heavy, and lighter human reviews accordingly
Obviously takes a lot of patience and very easy to sin, but on good days, its doable.
I went from just coding manually, to using LLMs for "surgical edits", to "woah, AGI!", to "haha whoops, not even close", to just coding manually (my brain still works!), to "surgical edits" again.
My current approach is "ask for very small diffs" + "review them very carefully".
I'm not working a job though, I'm working on a multiplayer game.
Main findings: The frontier models can't reliably modify Pong without breaking it, so their skill appears to be quite domain-specific. (OK, to be fair, neither can I half the time!) This is probably because they are "time blind". I had one model try to test a game by running it at 0.1 frames per second and shoving each frame in the vision API...
If you leave any room for a misunderstanding, they will laser in on do it and do the stupidest thing possible. If you're not checking everything carefully, you will discover this later, and you will cry.
Formal proofs, oddly enough, do not improve the situation: they will simply prove mathematically that the absurd and pointless and backwards implementation is completely without defects. (It obviously does help within an implementation, though.)
They can't formally prove what the hell you meant when you told them to build something. That job remains frustratingly human!
Current dissatisfaction: (1) Harnesses are designed for super bloated codebases (i.e. designed to load as little context as possible) which make them pretty clunky for small repos and small edits. (I had a Surgical Edit Tool I need to bring back...), (2) Current LLMs are anal about verifying the most trivial change, even without prompting, even if it's impossible for them to verify it because they're blind so they start measuring pixel data in Python... Both of which eat up Speed and Cost, taking the work even further from Realtime/Interactive to Tedious/Sad.
I think one of the best use cases of AI is as a natural language interface to programming. The syntax of a programming language is an opinionated part of its design that often adds to the complexity of learning the language.
Everything old is new again. The reason natural language never caught on for programming, is that natural language is inherently unsuitable for the task of formulating precise and unambiguous instructions for a machine.
Numerous attempts have been made in past 70 years, but we landed on the compromise that is high level programming languages with their (mostly) well-defined syntax and semantic rules.
If you look outside of the world of programming you'll find that the same is true for every craft and profession that requires precision and unambiguity. Cooking, logistics, military, carpentry, mechanical engineering, law, medicine, etc. all have their own domain specific languages for describing tasks, procedures, and tools. Heck, even sufficiently large organisations have their own internal languages and terms for that reason.
Following a few basic principles - I don't use paid-AI - only free services - so I regularly use ChatGPT to check the code generated by Claude and any others I might have limited access to.
I've recently settled on Deepseek4 for one of my projects, and had it review other code generated by other models, and .. yeah, that was quite eye-opening. Someone in the frontier-models part of the world is definitely paying attention to the AI slop generated by the other models, because having one AI checking the results of another AI has been quite fruitful, lately.
This approach feels just "add friction to your AI usage". It seems the worst of both worlds, both hand-coded and vibe-coded. You paste your code into a chat so it can tell you what to type yourself (the codebase access rule looks optional, but the copy-paste is one way by design). Replace "chat" with "Stack Overflow" and it'll sound familiar. You don't need to paste code into a chatbox if you're going to do an AI review later anyway.
The security argument is the strongest part of the post, and I don't disagree with it, but what it buys you is the review, and a review catches what you missed, whoever typed the characters. None of the ten dogmas follow from that.
My general approach is to design beforehand, do an adversarial review with AI, socialize it with humans (if needed), generate a plan, and start working item by item. Always keeping me, the human, in the loop (not that `/loop`), going through the steps generating code. Finally, a manual review and one AI adversarial review of the feature branch in a clean context, going section by section manually and discussing anything relevant, and off you go.
Writing code by hand feels great, but even local models can generate fine code. Deterministic linters and quality checks are what keep the quality in line. You can always modify things as long as you're in the process, but at the end of the day, you'll review more than you write. We're closer to being the assembly line's inspector than the crafters we once thought we were.
Very interesting. I find this is specially applicable to people learning or perfecting coding skills, not so much for developers already proficient in coding. What do you think?
Developing coding skills is both easier and more difficult than ever at the same time.
50 years ago the main barrier was access to hardware - computers were big, expensive, and not easy to come by for average people.
30 years ago, the biggest hurdle was access to resources: you had to spend a fortune on books that quickly became obsolete or had to wait for your local library to get them for you.
Today, you basically have everything - up-to-date free reference material on pretty much everything in an instant, cheap and accessible hardware (no need for high-end stuff), great online tutorials and interactive courses, message boards, the works.
But then you also have a machine that completely removes the need to put in the effort required to actually learn and hone your skills. Human psychology always seeks shortcuts and more discipline than ever is needed to not give up and have the machine do the thinking for you instead.
Proficient developers are getting pressured into outsourcing their craft and skills to the machine, thus deskilling quickly. Juniors never get the chance to become proficient in the first place - either because they're not getting hired to begin with, or because they succumb to the siren song of the machine...
This misses the best use of AI in my opinion, which is to gain understanding. Whst is this bit of code doing? Is there a risk of data leaking here? Are permissions enforced downstream of this function?
Code review can go much deeper now if you use AI to aggressively attack a PR combined with your human insight. Same for planning a feature:
Can I consolidate this logic to a shared function? Does the error surface to the user and are there any gaps? What preexisting functionality is affected by this PR? Can this query be made more efficient?
LLMs are great with focused questions, up and down abstraction layers and across all kinds of concerns. Stack up these focused concerns into a rich understanding of what you are doing or writing.
Understanding is the real output, code is the byproduct.
> LLMs are great with focused questions, up and down abstraction layers and across all kinds of concerns. Stack up these focused concerns into a rich understanding of what you are doing or writing.
I agree. This is much better advice than what the article tries to do, which is to give best practices around how to use or not use AI to code.
Agreed — I also always append to my agents.md — "If there are any technical choices available, please surface them to me so I can understand them and choose"
If your harness has an “AskUserQuestion”-like tool, you can prompt it to use that, or, as I have done, add a stop hook that uses a smaller model to check if there was any similar prose that was output toward the end of the previous response.
Happy to share the implementation or package it up if anyone is interested.
Wish I could upvote twice or something
> This misses the best use of AI in my opinion, which is to gain understanding. Whst is this bit of code doing? Is there a risk of data leaking here? Are permissions enforced downstream of this function?
Do I understand when I just let AI regurgitate the function and behavior of the code?
Do you need that analysis every time from an LLM? Those are things I’d expect anyone to grow out of pretty quickly once you get used to understanding code. Especially someone with Senior in their title.
Doing this may give you understanding but won’t really teach you how to understand. For students (as the article is focusing on them), the latter is equally important.
Closely related: one of the biggest AI superpowers I've found is testing. "Please implement a comprehensive unit testing suite for this, making sure to test ..."
You still have to check the work and shouldn't trust it blindly, but in my experience it's pretty damn good.
Exactly, i so wish the narratives around LLMs in most fields were around how they can enable us to be better ourselves...
SOMEONE didn't read the article
> You code and you have the AI check your work. Treat it like a reviewer. That’s my advice in a sentence.
Bro you're giving the same advice as the article
>Preferably, don’t give the AI access to the codebase. Copy-paste snippets of your code in the web interface.
Well, I can't ask it for an overview of a system it can't see.
> Preferably, don’t give the AI access to the codebase. [...] If the AI does have access to the codebase, this access is read-only.
also
> I don’t follow all these religiously.
So if you need an overview of a system, you can and should provide access to the entire codebase. Even the rule itself says "preferably", not "don't ever". If you have different needs, you can always just look at it as a useful guideline. It's just a blog post by some rando on the internet, not the epiphany of AI coding's Ten Commandments.
Using AI to review our own comments before posting could be a good idea
> Whst is this bit of code doing?
This only applies to bad code. A formal programmic language beats an informal language anytime. Why on earth would you translate an unambiguous formal language to English?
> Is there a risk of data leaking here?
> Are permissions enforced downstream of this function?
If you trust AI to give you that answer, you are going to run into serious issues.
> Why on earth would you translate an unambiguous formal language to English?
Why on earth are all the books about programming, science and math written in English (and other natural languages) instead of a formal one then?
Why on earth did comments even got invented?
A formal programmic language beats an informal language anytime. Why on earth would you write a comment in an ambiguous informal language when you could translate it to code?
programmer looks at formal programmic language which has so beaten the informal language that one cannot tell the reason why this stupid bit of code exists in the silly form it does is to prevent an edge condition in a particular browser while still working fine for other browsers.
Luckily genius programmer who understands formal programmic language and does not need weak comments to clarify why the world is not perfect, fixes the stupid looking code to be beautiful, opening company up to some really super neat, seemingly random bugs in one browser that nobody will understand for several months.
tldr: one important reason comments exist is to clarify why code you might expect to look another way looks the way it does, of course if the only reason it looks the way it does is this is the way the AI wrote it I guess I agree the comment is unnecessary.
“But that would be like telling a student in the 70s to pretend that calculators or computers don’t exist.”
If students of the 70s or today pretended they didn’t exist up to a certain point when they needed them to move forward, like bioinformatics or something, they 100% would be better off. There is plenty of research on off loading thinning providing a worse understanding of the material - eg side rules proving a better understanding than calculators.
Exactly. I always think about the ways consumers make terrible decisions with everyday purchases at the grocery store etc. If people could easily estimate unit costs in their head for example, a lot of predatory options would have to go away. But calculators keep people from having to get good with everyday math in their head, while those same calculators sit unused in many everyday scenarios where people don't go to the effort to use them.
Quick estimating (which is a different skill from calculating exact answers using pen and paper) is most useful there. People do carry computers with them these days and they do have calculator apps. Some shops I buy from do have unit prices for many items.
I think AI might be more of a problem, at least for some skills because it is not just doing a small unit of work. its more like the problem of doing more complex calculations. An example I have come across with calculators is students learning statistics without knowing how to calculate a variance, because they just put the numbers into a calculator. There is a failure there to learn the concept - they do not really know what a variance or SD is. I can imagine AI doing a lot of that.
If the challenge is held constant, giving people tools is just teaching a different skill set than teaching them unaugmented. The augmented skill set is more relevant to how people do work in the real world, but working through things unaugmented likely help people build a robust internal schema more quickly. We definitely don't need to spend as much time training unaugmented abilities going forward though, kind of like how the number of assembly related classes in CS departments has gone down over time as we've moved up the stack.
“You can use the electronic tuner once you learn to tune by ear” - any decent guitar teacher
I don't know if this was universal, but when I was a student we weren't allowed calculators until I believe 10th grade
You learn stuff by heart, so that when you mess up with the calculator you have a feel that stuff has gone wrong
I had the same thought and came to the exact same name: vibe crafting. Published a skill in the process, steps: https://github.com/scosman/vibe-crafting
In general, feeding the AI what you write for verification sounds like a good idea. But I think even this should only be done by someone who understands exactly what they're doing, how, and why, because otherwise they won't be able to analyze the accuracy of the AI's responses...
Like tests, it should only be used as a safety net and a signal that shows whether the way the program is written is prone to errors
Insightful:
> the term “vibe-coding” suggests a kind of laissez-faire attitude where you don’t really care about the outcome and you’re just having fun. That’s what the phrase meant when it was coined, but the world has moved on. In many companies professional programmers are using AI in such a way that it’s impossible to imagine that they are also reading the resulting code in detail. This is what modern vibe-coding is. Deferring to the AI, not worrying about the individual lines of code, and keeping an eye on whether the code passes its tests and throws up any problems in production.
This way of working is the only one that justifies the trillion-dollar bet on the AI industry. I agree, this method should be called vibe-coding.
It should depend on whether you've taken other steps to constrain the output of the LLM. I've been building some pretty elaborate test harnesses (which I test independently beforehand), to ensure that I get something that does what I ask of it.
I don't care which words we use, but they should separate this sort of empirically-minded work (versus the rationalism of typical development) from just dropping a few sentences into the prompt and crossing your fingers.
> empirically-minded work
Yeah, I understand that this testing-oriented constraint method is what AI coding enthusiasts are pitching. Sure, it's different from pure vibe-coding as originally defined, but nobody is seriously suggesting professionals should do pure vibe-coding anyway. It's a strawman. The word fits.
This article hits very close to what I have come up with myself and its good to see others thinking the same lines.
Obviously takes a lot of patience and very easy to sin, but on good days, its doable.Tests not being core seems suspect
I went from just coding manually, to using LLMs for "surgical edits", to "woah, AGI!", to "haha whoops, not even close", to just coding manually (my brain still works!), to "surgical edits" again.
My current approach is "ask for very small diffs" + "review them very carefully".
I'm not working a job though, I'm working on a multiplayer game.
Main findings: The frontier models can't reliably modify Pong without breaking it, so their skill appears to be quite domain-specific. (OK, to be fair, neither can I half the time!) This is probably because they are "time blind". I had one model try to test a game by running it at 0.1 frames per second and shoving each frame in the vision API...
If you leave any room for a misunderstanding, they will laser in on do it and do the stupidest thing possible. If you're not checking everything carefully, you will discover this later, and you will cry.
Formal proofs, oddly enough, do not improve the situation: they will simply prove mathematically that the absurd and pointless and backwards implementation is completely without defects. (It obviously does help within an implementation, though.)
They can't formally prove what the hell you meant when you told them to build something. That job remains frustratingly human!
Current dissatisfaction: (1) Harnesses are designed for super bloated codebases (i.e. designed to load as little context as possible) which make them pretty clunky for small repos and small edits. (I had a Surgical Edit Tool I need to bring back...), (2) Current LLMs are anal about verifying the most trivial change, even without prompting, even if it's impossible for them to verify it because they're blind so they start measuring pixel data in Python... Both of which eat up Speed and Cost, taking the work even further from Realtime/Interactive to Tedious/Sad.
I think one of the best use cases of AI is as a natural language interface to programming. The syntax of a programming language is an opinionated part of its design that often adds to the complexity of learning the language.
Everything old is new again. The reason natural language never caught on for programming, is that natural language is inherently unsuitable for the task of formulating precise and unambiguous instructions for a machine.
Numerous attempts have been made in past 70 years, but we landed on the compromise that is high level programming languages with their (mostly) well-defined syntax and semantic rules.
If you look outside of the world of programming you'll find that the same is true for every craft and profession that requires precision and unambiguity. Cooking, logistics, military, carpentry, mechanical engineering, law, medicine, etc. all have their own domain specific languages for describing tasks, procedures, and tools. Heck, even sufficiently large organisations have their own internal languages and terms for that reason.
You can let it help with syntax, but you can't rely on it to understand precise semantics.
SQL is the worst offender here: LLMs are very good at remembering its awful syntax, but very bad at writing nontrivial queries.
>It's a verbose coder, it overcomplicates things, and it works fast. Even if you had an inhuman level of attention, you couldn't keep up.
I wonder if that trick about prompting it to write like a 5th grader* would help here. Keep it simple!
*A trick which allows it to pass for human 70% of the time...
Following a few basic principles - I don't use paid-AI - only free services - so I regularly use ChatGPT to check the code generated by Claude and any others I might have limited access to.
I've recently settled on Deepseek4 for one of my projects, and had it review other code generated by other models, and .. yeah, that was quite eye-opening. Someone in the frontier-models part of the world is definitely paying attention to the AI slop generated by the other models, because having one AI checking the results of another AI has been quite fruitful, lately.
This approach feels just "add friction to your AI usage". It seems the worst of both worlds, both hand-coded and vibe-coded. You paste your code into a chat so it can tell you what to type yourself (the codebase access rule looks optional, but the copy-paste is one way by design). Replace "chat" with "Stack Overflow" and it'll sound familiar. You don't need to paste code into a chatbox if you're going to do an AI review later anyway.
The security argument is the strongest part of the post, and I don't disagree with it, but what it buys you is the review, and a review catches what you missed, whoever typed the characters. None of the ten dogmas follow from that.
My general approach is to design beforehand, do an adversarial review with AI, socialize it with humans (if needed), generate a plan, and start working item by item. Always keeping me, the human, in the loop (not that `/loop`), going through the steps generating code. Finally, a manual review and one AI adversarial review of the feature branch in a clean context, going section by section manually and discussing anything relevant, and off you go.
Writing code by hand feels great, but even local models can generate fine code. Deterministic linters and quality checks are what keep the quality in line. You can always modify things as long as you're in the process, but at the end of the day, you'll review more than you write. We're closer to being the assembly line's inspector than the crafters we once thought we were.
Very interesting. I find this is specially applicable to people learning or perfecting coding skills, not so much for developers already proficient in coding. What do you think?
Developing coding skills is both easier and more difficult than ever at the same time.
50 years ago the main barrier was access to hardware - computers were big, expensive, and not easy to come by for average people.
30 years ago, the biggest hurdle was access to resources: you had to spend a fortune on books that quickly became obsolete or had to wait for your local library to get them for you.
Today, you basically have everything - up-to-date free reference material on pretty much everything in an instant, cheap and accessible hardware (no need for high-end stuff), great online tutorials and interactive courses, message boards, the works.
But then you also have a machine that completely removes the need to put in the effort required to actually learn and hone your skills. Human psychology always seeks shortcuts and more discipline than ever is needed to not give up and have the machine do the thinking for you instead.
Proficient developers are getting pressured into outsourcing their craft and skills to the machine, thus deskilling quickly. Juniors never get the chance to become proficient in the first place - either because they're not getting hired to begin with, or because they succumb to the siren song of the machine...