Bitter lesson - anything that can be optimised outside the model or harness will be optimised by the model or harness in the next 2 months.
After trying any such pattern or skill or next big thing, I keep returning to Peter Steinberger’s advice in 2 weeks and follow “Just talk to it” engineering - https://steipete.me/posts/just-talk-to-it
In general, yes, but I'm not sure it can be subsumed so easily, as slowing down the initial human taste part of the process, taking stock of every nuance of scope, design and technical debt, in order to better understand the current feature requirements more clearly, takes a sort of holistic wisdom and patience we're not yet training for, LLM/agent rewards are fine tuned to deliver instant results from partially specified problems, obedient to instruction following. We need to balance Goodhart's law with a Socratic method?
Obviously if the training rewards change too you'll be right soon enough, they'll learn it too.
I'm glad I started this way. Since I started using AI, I've just been telling it what I need to do and letting it tell me what it needs to get it done, instead of trying to tell it how to do things. I know I've avoided so many of the rabbit holes that I'm prone to falling into.
Yes, but people trying new harness improvements like superpowers and this one generate training data which then gets reinforcement learned. It needs to be bootstrapped somehow. If you are an early adopter there's not much to lose by experimenting with harness tweaks.
If your idea gets trained that'd be a huge win. So it's worth proposing these, and if it' s worthwhile, championing it even if it's your own invention. That's a way to voice your opinion to the labs, effective or not.
About 5 weeks ago now, I moved over to Paseo[0] (OSS, Apache 2.0, no affiliation) while trying a few other "meta harnesses".
This turned out to be a very interesting experiment for me.
The agent profiles configuration was the most interesting. It looked way too simple. No tools, no real spec, no configuration of skills. But actually, it somehow turned out way more useful/powerful and subtly different from how the native harness models sub-agents.
I wrote about my setup and why I think it works[1]. The gist of it is that it is effectively a dynamic, multi-agent harness. I know what you're thinking: CC and Codex can do that, too. But Paseo has a few interesting bits: it can use multiple providers, harnesses, and models; the agents can talk to each other or through a coordinator; it has a nice toolset for agents to use and well tuned skills.
So I have a full team set up and the lead orchestrator then automatically decides when it needs a Member of Technical Staff, a DBA, an End User Advocate, a UI Designer to provide input and feedback.
As I wrote in my blog post, the main wins I observe are that: 1) the autonomous output is better, higher quality, more detailed because of internal team critique, 2) I can spread my tokens better across OpenAI, Anthropic, and DeepSeek; it may not be overall more "economical", but I can effectively "stretch" my sessions by distributing tokens.
I recommend checking out https://agentmgmt.dev/ and finding the right meta-harness for you since I think that this setup of specialized agents has worked well for me.
I guess the ship has already sailed on this, but I really wish this project would have at least looked into goldfish before naming their technique after them.
Contrary to popular belief, goldfish have great memories and can even learn tricks and recognize people.
I tried EGM on a few things over the last couple of weeks.
The biggest pitall, which is called out in the docs, is that the goldfish reviewers are never fully satisfied.
The reviewers will always find something, so you need to be able to recognize when the agent is bikeshedding and that it's time to move on.
Otherwise it's a pretty decent approach, and while I don't think it's anywhere near hands-off, it produced decent specs in both cases that didn't require too much additional hand-tuning before I started implementation.
I actually prefer how EGM is more hands on. I find that it gives much better designs and outcomes than yolo oneshotting. It's a good balance between vibe coding vs actually being thoughtful and intentional
This looks pretty much like my standard adversarial review prompt:
"Spin up 3 subagents, each on different models, with fresh contexts, to review this. Two should perform general reviews; one should focus on opportunities to simplify or reduce. Synthesize the results and make the changes you agree with."
I have it set up as a "skill" in my tool of choice (devin for the last few months, ampcode now), though skill is almost making too much of it - it's a small prompt I use often enough not to want to copy paste. A whole plugin feels like massive overkill.
You can tell the idea makes sense because so many people have ended up doing some version of it manually.
If this is the same one that was floating around a few weeks ago from google it's overengineered to the point where I think I probably just don't understand what they actually use it for internally. The intermediate output is very hard to understand and modify along the way and it really grinds through tokens for something that has such limited tools and context.
It may make sense if they're having autonomous agents drive this or something similar. I still have to pay for tokens though so the 20 line skill version I made for myself works better for me. ymmv.
I just since opus 5.5 really dont have the pain anymore that there is a 'coding problem' LIke i only discuss plan and the agents get it right, when I have issues its usually, because I forgot something about the plan or the plan was wrong. When an error happens my agents fix it themselves also.
Generally I agree, but I also see the occasional instance where Claude didn’t follow its Claude.md or where it didn’t remember the business logic of the software (possibly the docs are too big or. It in context or something), but most of the time it works like a dream
I think this risks wasting time reinventing a lot of what you get with frontier reasoning models. If your goal is independence from the frontier providers then this makes sense, but I feel like 6.1 terra on xhigh would probably run circles around any manifestation of this that I could deploy.
I've spent a non-trivial amount of energy attempting to DIY reasoning models and the conclusion I reached is that I have zero hope of competing with the combined engineering resources of OAI, Google and Anthropic.
This is a smart technique to work around context limitations. Is it anything that an user needs to know?
I've avoided any special configurations (skills, etc.) expecting that the system prompt and harness will incorporate the best techniques in short order.
David Rensin wrote about 7,000 words in the (nested) linked article, apparently to waste 27 minutes of my time; I don't think it contains any reason you couldn't reproduce this Goldfish protocol with "spin up review subagent" in Claude Code
Not anymore. Pretty much everyone has converged on treating them like tool calls, maybe with an optional "fork" mode or something that pushes in the context.
Reviewing code in fresh sessions with low to no context avoids the model collapsing over its own slop. I've been doing this manually for a while. This is very promising !
Bitter lesson - anything that can be optimised outside the model or harness will be optimised by the model or harness in the next 2 months.
After trying any such pattern or skill or next big thing, I keep returning to Peter Steinberger’s advice in 2 weeks and follow “Just talk to it” engineering - https://steipete.me/posts/just-talk-to-it
In general, yes, but I'm not sure it can be subsumed so easily, as slowing down the initial human taste part of the process, taking stock of every nuance of scope, design and technical debt, in order to better understand the current feature requirements more clearly, takes a sort of holistic wisdom and patience we're not yet training for, LLM/agent rewards are fine tuned to deliver instant results from partially specified problems, obedient to instruction following. We need to balance Goodhart's law with a Socratic method?
Obviously if the training rewards change too you'll be right soon enough, they'll learn it too.
> Just talk to it
I'm glad I started this way. Since I started using AI, I've just been telling it what I need to do and letting it tell me what it needs to get it done, instead of trying to tell it how to do things. I know I've avoided so many of the rabbit holes that I'm prone to falling into.
Yes, but people trying new harness improvements like superpowers and this one generate training data which then gets reinforcement learned. It needs to be bootstrapped somehow. If you are an early adopter there's not much to lose by experimenting with harness tweaks.
If your idea gets trained that'd be a huge win. So it's worth proposing these, and if it' s worthwhile, championing it even if it's your own invention. That's a way to voice your opinion to the labs, effective or not.
About 5 weeks ago now, I moved over to Paseo[0] (OSS, Apache 2.0, no affiliation) while trying a few other "meta harnesses".
This turned out to be a very interesting experiment for me.
The agent profiles configuration was the most interesting. It looked way too simple. No tools, no real spec, no configuration of skills. But actually, it somehow turned out way more useful/powerful and subtly different from how the native harness models sub-agents.
I wrote about my setup and why I think it works[1]. The gist of it is that it is effectively a dynamic, multi-agent harness. I know what you're thinking: CC and Codex can do that, too. But Paseo has a few interesting bits: it can use multiple providers, harnesses, and models; the agents can talk to each other or through a coordinator; it has a nice toolset for agents to use and well tuned skills.
So I have a full team set up and the lead orchestrator then automatically decides when it needs a Member of Technical Staff, a DBA, an End User Advocate, a UI Designer to provide input and feedback.
As I wrote in my blog post, the main wins I observe are that: 1) the autonomous output is better, higher quality, more detailed because of internal team critique, 2) I can spread my tokens better across OpenAI, Anthropic, and DeepSeek; it may not be overall more "economical", but I can effectively "stretch" my sessions by distributing tokens.
I recommend checking out https://agentmgmt.dev/ and finding the right meta-harness for you since I think that this setup of specialized agents has worked well for me.
[0] https://paseo.sh
[1] https://chrlschn.dev/blog/2026/10/larping-with-agents-and-wh...
I guess the ship has already sailed on this, but I really wish this project would have at least looked into goldfish before naming their technique after them.
Contrary to popular belief, goldfish have great memories and can even learn tricks and recognize people.
I tried EGM on a few things over the last couple of weeks.
The biggest pitall, which is called out in the docs, is that the goldfish reviewers are never fully satisfied.
The reviewers will always find something, so you need to be able to recognize when the agent is bikeshedding and that it's time to move on.
Otherwise it's a pretty decent approach, and while I don't think it's anywhere near hands-off, it produced decent specs in both cases that didn't require too much additional hand-tuning before I started implementation.
I actually prefer how EGM is more hands on. I find that it gives much better designs and outcomes than yolo oneshotting. It's a good balance between vibe coding vs actually being thoughtful and intentional
This looks pretty much like my standard adversarial review prompt:
"Spin up 3 subagents, each on different models, with fresh contexts, to review this. Two should perform general reviews; one should focus on opportunities to simplify or reduce. Synthesize the results and make the changes you agree with."
I have it set up as a "skill" in my tool of choice (devin for the last few months, ampcode now), though skill is almost making too much of it - it's a small prompt I use often enough not to want to copy paste. A whole plugin feels like massive overkill.
Do you find that 'opportunities to simplify or reduce' are often taken, or is that third review often ineffective?
You can tell the idea makes sense because so many people have ended up doing some version of it manually.
If this is the same one that was floating around a few weeks ago from google it's overengineered to the point where I think I probably just don't understand what they actually use it for internally. The intermediate output is very hard to understand and modify along the way and it really grinds through tokens for something that has such limited tools and context.
It may make sense if they're having autonomous agents drive this or something similar. I still have to pay for tokens though so the 20 line skill version I made for myself works better for me. ymmv.
I just since opus 5.5 really dont have the pain anymore that there is a 'coding problem' LIke i only discuss plan and the agents get it right, when I have issues its usually, because I forgot something about the plan or the plan was wrong. When an error happens my agents fix it themselves also.
Generally I agree, but I also see the occasional instance where Claude didn’t follow its Claude.md or where it didn’t remember the business logic of the software (possibly the docs are too big or. It in context or something), but most of the time it works like a dream
I think this risks wasting time reinventing a lot of what you get with frontier reasoning models. If your goal is independence from the frontier providers then this makes sense, but I feel like 6.1 terra on xhigh would probably run circles around any manifestation of this that I could deploy.
I've spent a non-trivial amount of energy attempting to DIY reasoning models and the conclusion I reached is that I have zero hope of competing with the combined engineering resources of OAI, Google and Anthropic.
This is a smart technique to work around context limitations. Is it anything that an user needs to know?
I've avoided any special configurations (skills, etc.) expecting that the system prompt and harness will incorporate the best techniques in short order.
Why can't I replicate this with "spin up a review subagent"?
Who's saying you can't ?
David Rensin wrote about 7,000 words in the (nested) linked article, apparently to waste 27 minutes of my time; I don't think it contains any reason you couldn't reproduce this Goldfish protocol with "spin up review subagent" in Claude Code
https://drensin.medium.com/elephants-goldfish-and-the-new-go...
Do most harnesses hand off a lot of context to subagents?
Not anymore. Pretty much everyone has converged on treating them like tool calls, maybe with an optional "fork" mode or something that pushes in the context.
why does it work?
Reviewing code in fresh sessions with low to no context avoids the model collapsing over its own slop. I've been doing this manually for a while. This is very promising !