Any benchmarks showing if this actually improves problem solving? Or reduces errors?
The idea is cool, but from own experience in harness engineering, lots of cool sounding ideas can have a negative impact on performance due to emergent and confounding effects.
The TUI and Session manager are straightforward enough. PI is serial by default but can become more DAG-like, where Codex is designed specifically for DAG and this is going more the Codex direction.
I'm much more interested in the memory model and why. As far as I can tell, it's "a vector Db" and not much more is said. Nothing about working memory or procedural memory (there are lots of ways to classify it, https://www.youtube.com/watch?v=BacJ6sEhqMo), but I was disappointed with how "advanced" it seems.
So if I understand correctly, one of the agents is creating the workflow as a DAG dynamically for each new job it's given? That's the most interesting part to me. The rest I was already pretty familiar with.
So maybe that's a big chunk of what you need for an 'AI Company': an agent that manages the goals and hierarchies. Although of course the DAG and agent hierarchy is not quite the same thing. But maybe the workflows and subworkflows are what matter.
I (like presumably many others) have built something similar. My main difference though is that the critics operate on each stage of development before it can move onto the next. The stages are defined by deliverable artifacts: issue, plan, pull request. Critics must approve each artifact before you can move onto the next. So the process is defined by a DAG which defines how to transition successfully from one artifact to the next. It's been fun to work on and I would like to open source it soon, but I assume many others are working on similar systems.
A good tutorial. Generally speaking, the harness is the environment layer between a language model and its task, such as the action set it can call, the state, the context it can see and the memory etc.
However, currently the bigger question comes to my experience during harness is actually not where we use LLM in the system, but where we do NOT use LLM in the system. And the validation of the results becomes more and more important. Any thoughts on this?
A harness [1] was developed by Terrence Tao and some collaborators to prove mathematical results. It has since then been used by others with positive effect. Can someone critique the structure of this harness? I don't know anything about this stuff.
it's not actually proving it though? It's more like stringing it together. A person or LEAN has to actually provde something. I've yet to see anything other than AI-slop produces simulcra of proofs. If it were proving something it'd be <insert mathematician> validates AI proof.
I don't you what you mean. People have used the above harness (or similar) to prove significant results. See this recent paper [1], which claims
> The human authors take full responsibility for the claims and proofs contained
in this paper, and have carefully refined and verified them. The construction and main ideas of the proof were generated entirely by Codex using GPT 5.6 Sol Ultra, using harness ideas generated by the authors based on the UCLA Moonshot Harness [ZHC+26] and [Ope26].
Real break throughs come from integral mathematics and not just a few reorderings. I've no doubt these are talented people recognizing output as useful; however, every time I see these links presented it's never from the "Prominent mathematician verifies AI proof"
Don't put the cart before the horse if you want people to think LLMs are cracking math problems in real terms.
You mis understand, which is why these articles are so empty. I could find some math starved journal and poblish a bunch of logical implications, but that doesn't mean the logic is sound.
Anyone else have related reading that touches on this? I'm building my own custom harness and want to start implementing loop support, etc. But I also want to build some sort of framework so that it's dynamic (e.g. this needs to run x number of iterations, while planning needs to run y number of iterations).
OpenAI and Anthropic both have blog posts about programmatic tool use. Tbh I settled in STARLARK as the intermediary language it's a python dialect so most llms can use it with little help.https://github.com/zarldev/zarlmono/blob/main/zkit%2Fagent%2... it's a wrapper around the calls of the other tool set.
I'll take the "best way to elicit a clarification response on the internet is to state the opposite confidently" bait...
The example listed in the article -- fanning out a few simple get-population, get-timezone, and make-summary calls -- is, in fact, useless overengineering. This is a basic promise chain with extra steps (priced with tokens).
But as with all software pattern learning, we learn the concepts with simple toy examples that generalize into something bigger. It's the generalization that matters here.
This is talking about a few methods and tricks for spawning effective subagents (collectively, that's the "harness"). Those tips and tricks are nice, but to not be considered useless, we need to make sure we understand why spawning subagents is useful in the first place. Yes parallelism is nice for some tasks, but that's not really what this is about.
The real reason is protecting your context. Yeah, we have 1M context windows that can fit all of LotR in it, but these machines work better when they're narrowly focused. Large context windows run into attention issues and forgetfulness ("Yes, you're right, it was stated I should/n't do X but I ignored it, my bad."). So subagents come into play when you don't want all the tokens associated with a subtask to pollute your main/primary context window and degrade task attention. Split that off to a subagent, let that context navigate the details, and just make sure your main one gets just the input/output blackbox results.
The trick is getting a sense for when the complexity of the task warrants that kind of context protection, vs when a single agent is good-enough. Your toy example will never have enough complexity to warrant the setup, but you might one day find a generalization that may.
And its clear that progression is happening on a communith level on all of these and they get integrated later on in commercial offerings like from Anthropic and co.
But also doing a opensource harness and not just giing in to the big companies allows us to have all of this open and transparent and with open models locally.
agreed. All of these are trying to get to something that can't really every be achieved with LLMs with is determinism. Folks are trying to constrain the models to behave in a certain way with all of these tools, but there's far too many edge cases for them to be reliable. Doesn't mean they can't add some value, but it seems very limited. Hoping it's only a matter of time before we go back to engineering and step back from "vibes".
I felt that way initially. It is also a headache when you invest a lot in that and need to continually test ripping that stuff out as new models come out. Or in the case of opus 5, Anthropic says ditch it entirely and trust it.
But there is another aspect which I do enjoy which is closer to the feeling of dialing in key bindings in vim or getting a really good rhythm going with your vscode extensions or zhs plugins. It is that level of "I want my system to do exactly this thing in exactly this way" customization that a lot of technical people crave.
And you can do it with harness and context engineering in many cases. In other cases it introduces friction because it will be like "cool, I will only output 15 words max unless told otherwise" and then in the next turn completely disregards it with an "oops, you did tell me to do that didn't you."
And that frustration compounds when older model versions may have done a better job of that but new models are like "thank you for your suggestion, your opinion, while appreciated, is irrelevant. Now let me get back to overspending on your token budget. "
We won't. AI engineering is here to stay whether people like it or not.
See what some guys like Linus Torvalds, or Eric S. Raymond are saying about. It's not so much about "vibes" but using the tool (yes the AI tool) in a certain way that can propel yourself towards your goal at unprecedented speeds.
So AI is mostly a thin glue between deterministic processes. Doesn't change the fact that that is enough to achieve an extremely large amount of tasks.
But pulling orchestration off is very very tricky.
Even if it is just a small, simple orchestrator.
Ideas like planner, memory, log, subagents, graphs (each on their own) sound great and very promising.
So promising that one would think they must work, how could they not?
I've been there as well!
The challenge is that all these parts of the orchestrator are intertwined with each other
and they are all causing overhead in the main context window in some form or at least overall complexity that is difficult to grasp and predict/engineer for
(even though the idea is to help exactly with the fact that the context window is limited)
To save context window there is also more communication that has 'stille post' ('chinese whispers') like dynamics
Turns out it is very difficult to find out the right context to bubble up and down.
It's very similar to human org communication challenges (think large org stucture vs small teams vs one person that can keep it all in their head)
Yeah, what do you do if one person can't keep it all in their head?
But how great is it when it's possible?
Companies must have figured out how that works right? Maybe we can adopt and implement these ideas?
And yet … easy it is not, especially when you're not dealing with run-of-the-mill well-defined tasks.
But more like with open-ended software development?
I'm not saying it's not possible or that it should not be tried.
On the contrary, I think this is worth pursuing and a bit like the search for the holy grail.
But I also think the other direction of the search space is under-explored.
The holy grail is glamorous.
With 'smol' I'm spelunking on this other extreme (non-orchestration?)
(welcome, join us, we have cookies, and context windows with a lot of room for work items!)
smol is a minimalist agent harness that protects the context window
- no system prompt
- no tool spamming (just 1 tool: sh)
- no agents.md
- no mcp
- no planning, todos, graphs, beads, …
and figuring out how that looks like and performs
it is a worthwhile thread to pull I think
at least from the dozens of benches I'm looking at I see that less stuff in the context window does help a lot
- cheaper per task
- finishing faster
- better tool composition (sh and pipes are great!)
but also for more complicated longer-term tasks the model gets less confused when the context window is not getting spammed
Any benchmarks showing if this actually improves problem solving? Or reduces errors?
The idea is cool, but from own experience in harness engineering, lots of cool sounding ideas can have a negative impact on performance due to emergent and confounding effects.
So I'm a bit skeptical!
The TUI and Session manager are straightforward enough. PI is serial by default but can become more DAG-like, where Codex is designed specifically for DAG and this is going more the Codex direction.
I'm much more interested in the memory model and why. As far as I can tell, it's "a vector Db" and not much more is said. Nothing about working memory or procedural memory (there are lots of ways to classify it, https://www.youtube.com/watch?v=BacJ6sEhqMo), but I was disappointed with how "advanced" it seems.
Question: Any benchmarks showing if this actually improves problem solving? Or reduces errors?
"Answer": a word soup that in no way, shape, or form addresses the question, but does sound jargony and vague enough to be an LLM.
So if I understand correctly, one of the agents is creating the workflow as a DAG dynamically for each new job it's given? That's the most interesting part to me. The rest I was already pretty familiar with.
So maybe that's a big chunk of what you need for an 'AI Company': an agent that manages the goals and hierarchies. Although of course the DAG and agent hierarchy is not quite the same thing. But maybe the workflows and subworkflows are what matter.
Yes, there's a team of agents, each with a different role.
I (like presumably many others) have built something similar. My main difference though is that the critics operate on each stage of development before it can move onto the next. The stages are defined by deliverable artifacts: issue, plan, pull request. Critics must approve each artifact before you can move onto the next. So the process is defined by a DAG which defines how to transition successfully from one artifact to the next. It's been fun to work on and I would like to open source it soon, but I assume many others are working on similar systems.
A good tutorial. Generally speaking, the harness is the environment layer between a language model and its task, such as the action set it can call, the state, the context it can see and the memory etc.
However, currently the bigger question comes to my experience during harness is actually not where we use LLM in the system, but where we do NOT use LLM in the system. And the validation of the results becomes more and more important. Any thoughts on this?
> The plan is a graph
I much prefer giving the LLM a REPL loop, and injecting all the tools as functions inside the REPL loop.
That means that the LLM isn't constrained to writing a DAG, it can write code that loops, exits early, etc.
Do you have an example?
A graph is a fancy way of saying a few async await calls, which a REPL can do
(We added the same to louie.ai, not complicated)
A harness [1] was developed by Terrence Tao and some collaborators to prove mathematical results. It has since then been used by others with positive effect. Can someone critique the structure of this harness? I don't know anything about this stuff.
[1] https://github.com/1stproof/batch-2/tree/main/batch-2-submis...
it's not actually proving it though? It's more like stringing it together. A person or LEAN has to actually provde something. I've yet to see anything other than AI-slop produces simulcra of proofs. If it were proving something it'd be <insert mathematician> validates AI proof.
I don't you what you mean. People have used the above harness (or similar) to prove significant results. See this recent paper [1], which claims
> The human authors take full responsibility for the claims and proofs contained in this paper, and have carefully refined and verified them. The construction and main ideas of the proof were generated entirely by Codex using GPT 5.6 Sol Ultra, using harness ideas generated by the authors based on the UCLA Moonshot Harness [ZHC+26] and [Ope26].
[1] https://arxiv.org/pdf/2607.21551 (Statement on AI usage is at the bottom of page 3).
you keep using the term "used the model" or whatever.
A model is non-deterministic. People prove things, LLM string together a bunch of words and do symbol shunting.
Ensure you understand what symbol shunting is before you make claims. https://ell.stackexchange.com/questions/76400/what-does-one-...
Real break throughs come from integral mathematics and not just a few reorderings. I've no doubt these are talented people recognizing output as useful; however, every time I see these links presented it's never from the "Prominent mathematician verifies AI proof"
Don't put the cart before the horse if you want people to think LLMs are cracking math problems in real terms.
Stringing what together? A sequence of logical implications? The word for that is "proof".
You mis understand, which is why these articles are so empty. I could find some math starved journal and poblish a bunch of logical implications, but that doesn't mean the logic is sound.
A collection of logical implications <> proof.
Anyone else have related reading that touches on this? I'm building my own custom harness and want to start implementing loop support, etc. But I also want to build some sort of framework so that it's dynamic (e.g. this needs to run x number of iterations, while planning needs to run y number of iterations).
OpenAI and Anthropic both have blog posts about programmatic tool use. Tbh I settled in STARLARK as the intermediary language it's a python dialect so most llms can use it with little help.https://github.com/zarldev/zarlmono/blob/main/zkit%2Fagent%2... it's a wrapper around the calls of the other tool set.
Check out https://github.com/strongdm/attractor along with the community implementations at https://factory.strongdm.ai/products/attractor#community.
Check out academic papers about:
1. Hierarchical skills, workflow, skill learning 2. Meta Harness, self-learning harnesses 3. Trace/trajectory representation 4. Common agentic benchmarks
But first more basic things like 5. Blog posts form anthropic 6. How Claude Code/PI/ Hermes!! agent works 7. Agent sessions/ Forking/ Hooks
Sounds like you want something similar to /goal mode in Codex.
And the associated GitHub repo: https://github.com/DataForScience/LLMs
Contrary to the title and intro, this appears to be an agentic _workflow_ builder/runner, not an advanced “agent harness”
A few things:
- they note: “nothing in this post proves it actually works in most cases”
- the DAG sounds good, but LLMs often split tasks into smaller pieces than they need to, which can cause them to lose the forest for the trees
- the forced JSON interplay, in my experience, causes even gpt-5.6-sol to lose a few “IQ points”
For anyone reading this, this tutorial is much more reminiscent of how folks were building “agents” pre-Claude Code.
tl;dr the “orchestrator” here is just a software loop, and the LLM prompts restrict flexibility of the planner/workers
Good stuff, keeping it
why do i hate skillks, harnesses , memory systems whatever. such ideas that everyone thinks they've discovered but are totally useless in practice.
I'll take the "best way to elicit a clarification response on the internet is to state the opposite confidently" bait...
The example listed in the article -- fanning out a few simple get-population, get-timezone, and make-summary calls -- is, in fact, useless overengineering. This is a basic promise chain with extra steps (priced with tokens).
But as with all software pattern learning, we learn the concepts with simple toy examples that generalize into something bigger. It's the generalization that matters here.
This is talking about a few methods and tricks for spawning effective subagents (collectively, that's the "harness"). Those tips and tricks are nice, but to not be considered useless, we need to make sure we understand why spawning subagents is useful in the first place. Yes parallelism is nice for some tasks, but that's not really what this is about.
The real reason is protecting your context. Yeah, we have 1M context windows that can fit all of LotR in it, but these machines work better when they're narrowly focused. Large context windows run into attention issues and forgetfulness ("Yes, you're right, it was stated I should/n't do X but I ignored it, my bad."). So subagents come into play when you don't want all the tokens associated with a subtask to pollute your main/primary context window and degrade task attention. Split that off to a subagent, let that context navigate the details, and just make sure your main one gets just the input/output blackbox results.
The trick is getting a sense for when the complexity of the task warrants that kind of context protection, vs when a single agent is good-enough. Your toy example will never have enough complexity to warrant the setup, but you might one day find a generalization that may.
No clue?
I don't think they are totally usesless.
And its clear that progression is happening on a communith level on all of these and they get integrated later on in commercial offerings like from Anthropic and co.
But also doing a opensource harness and not just giing in to the big companies allows us to have all of this open and transparent and with open models locally.
What each of these is doing, fundamentally, is solving context management in an opinionated way (that and guardrails).
Wrong. They are commonly used by millions.
agreed. All of these are trying to get to something that can't really every be achieved with LLMs with is determinism. Folks are trying to constrain the models to behave in a certain way with all of these tools, but there's far too many edge cases for them to be reliable. Doesn't mean they can't add some value, but it seems very limited. Hoping it's only a matter of time before we go back to engineering and step back from "vibes".
I felt that way initially. It is also a headache when you invest a lot in that and need to continually test ripping that stuff out as new models come out. Or in the case of opus 5, Anthropic says ditch it entirely and trust it.
But there is another aspect which I do enjoy which is closer to the feeling of dialing in key bindings in vim or getting a really good rhythm going with your vscode extensions or zhs plugins. It is that level of "I want my system to do exactly this thing in exactly this way" customization that a lot of technical people crave.
And you can do it with harness and context engineering in many cases. In other cases it introduces friction because it will be like "cool, I will only output 15 words max unless told otherwise" and then in the next turn completely disregards it with an "oops, you did tell me to do that didn't you."
And that frustration compounds when older model versions may have done a better job of that but new models are like "thank you for your suggestion, your opinion, while appreciated, is irrelevant. Now let me get back to overspending on your token budget. "
We won't. AI engineering is here to stay whether people like it or not.
See what some guys like Linus Torvalds, or Eric S. Raymond are saying about. It's not so much about "vibes" but using the tool (yes the AI tool) in a certain way that can propel yourself towards your goal at unprecedented speeds.
So AI is mostly a thin glue between deterministic processes. Doesn't change the fact that that is enough to achieve an extremely large amount of tasks.
so if you don't use skill, harness and memory systems, what do you use?
just out of box models.
Because its all moonshining.
Trying to make gold from pyrite.
I love reading about orchestration concepts.
But pulling orchestration off is very very tricky.
Even if it is just a small, simple orchestrator.
Ideas like planner, memory, log, subagents, graphs (each on their own) sound great and very promising.
So promising that one would think they must work, how could they not?
I've been there as well!
The challenge is that all these parts of the orchestrator are intertwined with each other
and they are all causing overhead in the main context window in some form or at least overall complexity that is difficult to grasp and predict/engineer for
(even though the idea is to help exactly with the fact that the context window is limited)
To save context window there is also more communication that has 'stille post' ('chinese whispers') like dynamics
Turns out it is very difficult to find out the right context to bubble up and down.
It's very similar to human org communication challenges (think large org stucture vs small teams vs one person that can keep it all in their head)
Yeah, what do you do if one person can't keep it all in their head?
But how great is it when it's possible?
Companies must have figured out how that works right? Maybe we can adopt and implement these ideas?
And yet … easy it is not, especially when you're not dealing with run-of-the-mill well-defined tasks.
But more like with open-ended software development?
I'm not saying it's not possible or that it should not be tried.
On the contrary, I think this is worth pursuing and a bit like the search for the holy grail.
But I also think the other direction of the search space is under-explored.
The holy grail is glamorous.
With 'smol' I'm spelunking on this other extreme (non-orchestration?)
(welcome, join us, we have cookies, and context windows with a lot of room for work items!)
smol is a minimalist agent harness that protects the context window
and figuring out how that looks like and performsit is a worthwhile thread to pull I think
at least from the dozens of benches I'm looking at I see that less stuff in the context window does help a lot
but also for more complicated longer-term tasks the model gets less confused when the context window is not getting spammedthe context window is precious
https://x.com/__tosh/status/2084985580144722369
https://github.com/smol-env/smol
I read some of your twits but could find the source for that, do you publish it?
edited with link to repo (I'll add more to it over the next days!)