I can only recommend to regularly measure how many tokens a harness+model combination uses for a certain task
There are huge token efficiency/bloat differences between agents while working on the same tasks, using the same model, in the same environment
Yesterday I ran 10 agentic tasks using GPT 5.6 Sol in an ubuntu 26.04 vm a couple of times with different harnesses and got vastly different token usage.
I'm not surprised that Claude Code is not optimized for an OpenAI model but I was still quite shocked re how much of a difference the harness makes.
Disclaimer: I'm working on 'smol' which is a minimalist harness but it's really nothing special, just a minimal system prompt, no skills files, only tool is shell
Do not underestimate how much popular harnesses are spamming the context window. The context window is very important.
in my book anything that is (I'm sure well intentioned) and injected to help the agent — but doesn't help it — is a waste of tokens
but even injected context that when I read it sounds useful can oversteer the model and make it second guess or take a more complicated route than it normally would
(you can see this when looking at traces with and without that injected context)
often harnesses also mention in their system prompt locations of markdown files that the model can consult if the model thinks they might help
that hint alone as part of the system prompt can be strong enough to make the model read in more tokens than would have been necessary
'spam' is maybe a harsh way to say it
unfortunately I don't see an easy way other than to invest time and tokens into finding out which parts of the added context (in system prompt, injected in turns etc etc) are actually helpful or harmful and when
I'm just doing the easiest thing I could think of: start from nothing or close to nothing
that seems to work better than what most harnesses are doing
Claude Code injects a ton of tools into the system prompt, including their “memory system” that’s like 10k+ tokens. Depending on your task shape, this can easily double your task cost (e.g. a low context-using job that takes many turns, like a monitoring loop).
You should use --disallowed-tools to prune any tools not needed for the task. Note that this is also a perpetual game of whack-a-mole since they’re always adding new tools.
> Claude Code injects a ton of tools into the system prompt
It's so unfortunate they don't let you use the subscription with other harnesses anymore - since even if I used OpenCode they'd still get a bunch of useful data from the API calls, meanwhile I could stretch their tier limits way further.
Calling per token usage of the US closed source labs has always been funny to me, we have a clear model of what it actually costs to host these models from open models.
Your subscription is not subsidised, it is just closer to the actual cost of the model…
According to ccusage I use about $5k of tokens on my $100/mo Claude Max plan and only hit 5-hr windows where I have to switch tools for a couple hours about once a week
It's so unfortunate people don't realize it's cheaper to write their own well working agent instead of insisting with general purpose bloated ones like Claude.
200$/month is a lot of money on Luna/DS4 flash, like really a lot and the results are much better than clowning on bloated CC.
It's absurd how you have more and more organizations encoding their processes on LLMs and "engineers" (charlatan coders) don't even bother optimizing the tool they use most.
> the results are much better than clowning on bloated CC.
I won't argue with the cost effectiveness, but the results are very much not better. Opus and Fable are in a different league than DS4 Flash. Even GPT Terra, which I really like overall, sometimes gets stuck in weird loops and starts to do stupid stuff once its context window fills up. Whereas I can more or less trust the big models to just Do The Thing™ on the first try.
With that said, you get way more value out of a GPT subscription than you do from Claude, partly because of the ability to use more efficient harnesses.
You still pay cache token costs on API calls. Cache cost/token are 90% lower, but you pay it every single turn.
I’m not sure if they let you skip the cache write cost on the first turn. That would imply cross-user caching infrastructure or special casing the default system prompt to give you a discount. Maybe? Away from the computer but you could try a “hello” in a fresh session and see what was billed.
If you're working on smol? How complex is your work? The agent has to do everything using sed? Did you write your own tools? I guess my question is, why aren't you using pi?
The difference in tokens between the two also makes super curious. The system prompt can't be that different (I'd even bet Pi's shorter) and the 4 tools shouldn't make as much of a difference. I'm gonna have to try it.
the system prompt of smol is shorter than the system prompt of Pi
smol has no system prompt
system prompt of Pi 0.83.0
"""
You are an expert coding assistant operating inside pi, a coding agent harness. You help users by reading files, executing commands, editing code, and writing new files.
Available tools:
- read: Read file contents
- bash: Execute bash commands (ls, grep, find, etc.)
- edit: Make precise file edits with exact text replacement, including multiple disjoint edits in one call
- write: Create or overwrite files
In addition to the tools above, you may have access to other custom tools depending on the project.
Guidelines:
- Use bash for file operations like ls, rg, find
- Use read to examine files instead of cat or sed.
- Inspect PI_* environment variables for current model and session details.
- Use edit for precise changes (edits[].oldText must match exactly)
- When changing multiple separate locations in one file, use one edit call with multiple entries in edits[] instead of multiple edit calls
- Each edits[].oldText is matched against the original file, not after earlier edits are applied. Do not emit overlapping or nested edits. Merge nearby changes into one edit.
- Keep edits[].oldText as small as possible while still being unique in the file. Do not pad with large unchanged regions.
- Use write only for new files or complete rewrites.
- Be concise in your responses
- Show file paths clearly when working with files
Pi documentation (read only when the user asks about pi itself, its SDK, extensions, themes, skills, or TUI):
- Main documentation: /usr/local/lib/node_modules/@earendil-works/pi-coding-agent/README.md
- Additional docs: /usr/local/lib/node_modules/@earendil-works/pi-coding-agent/docs
- Examples: /usr/local/lib/node_modules/@earendil-works/pi-coding-agent/examples (extensions, custom tools, SDK)
- When reading pi docs or examples, resolve docs/... under Additional docs and examples/... under Examples, not the current working directory
- When asked about: extensions (docs/extensions.md, examples/extensions/), themes (docs/themes.md), skills (docs/skills.md), prompt templates (docs/prompt-templates.md), TUI components (docs/tui.md), keybindings (docs/keybindings.md), SDK integrations (docs/sdk.md), custom providers (docs/custom-provider.md), adding models (docs/models.md), pi packages (docs/packages.md), environment variables (docs/environment-variables.md)
- When working on pi topics, read the docs and examples, and follow .md cross-references before implementing
- Always read pi .md files completely and follow links to related docs (e.g., tui.md for TUI API details)
Current working directory: /workspace
"""
It is everything. My experience with Claude Code is that you have to decide when to compact to make it efficient. It defaults everything to 1M context and it will never keep it in check. It is strange how little cache reads you hit in smol, that may be a configuration issue.
I will look into it more to see if I have configured it wrong but I think the token efficiency hurts cache use as caching only starts at 1024 tokens so for tasks where smol is under or close to 1024 tokens most of them are uncached
What tools are you using to run the comparison? Or are you just running the prompts and checking them with something like ccusage?
I’m asking because I’ve been looking for agent harness comparison tools too. I’m interested in more than just the inputs and outputs—I also want the system prompts, traces, and tool calls. It’s useful to understand why Codex, for example, uses more tokens while Pi doesn’t.
Fewer tokens aren’t necessarily better if the agent skipped important checks. On the other hand, using more tokens could just mean it’s overthinking the process. Either way, seeing the full execution trace for the same task is really valuable.
I built an ad-hoc custom comparison framework to inspect system prompts, caching behaviour, tool call outputs, exact api requests and responses and so on
I agree fewer tokens is not necessarily better but a bit counter-intuitively often the harness using fewer tokens is not only done faster but has better results
(that said: of course check the results, look at the full traces, agree!)
Running some local models and wondering what insanity was consuming 100k+ tokens to respond to "hello world" was rather eye-opening, yeah. Full of pointless fluff.
While there's a lot of bloat in the harnesses, almost all of it is cached. Your harness appears to be caching almost nothing. That will almost certainly bode extremely poorly for longer-running tasks than the simple ones you mentioned testing. Benchmaxxing for simplistic tasks could leave you not only suffering in completion rate on complex tasks but also possibly even not end up with the cost gains you're hoping for. And maybe that's fine if the only thing you use agents for is the very simple tasks, which is certainly a defensible decision, but probably bears mentioning when you're comparing it against harnesses with a different design goal.
the uncached tokens are also from runs where smol finished a task below 1024 tokens (the minimum amount of tokens needed to activate caching) which is less tokens than other harnesses are using for their system prompt (!)
> GPT-5.6 and later models: Caching is available for prefixes containing at least 1,024 tokens. This is a strict minimum.
so in this specific case the count of uncached tokens for smol makes it look worse than it actually is
that said: it does makes sense to add more tasks that are difficult enough to fill the context window to compare the harnesses for how well they deal with compaction
staying below compaction (or with compaction at fewer compactions) is not only cheaper and faster, it also helps the agent stay on track
> There are huge token efficiency/bloat differences between agents while working on the same tasks, using the same model, in the same environment
Be careful here. Remember these are non-deterministic models at the end of the day, and even with everything being "the same" you can have two runs where the same model, same harness, same tools can arrive at the same conclusion through a wildly different sequence of events.
agree, that makes it a bit tricky to compare (esp if you also want to add different models and reasoning levels into the mix)
I will add more tasks (esp longer ones) and think more about grading, the current tasks were easy to grade because the desired outcomes are well specced but I will also look into more open ended tasks and how to grade those
this benchmark is a very misleading because it doesn't necessarily point to these harnesses saving usage and much of the harness that comes with codex for example is designed at cross functional agentic usage which you will be leaving behind by using a different harness. openai spends a lot of time as does claude. also note that cached is what makes up bulk of the total and codex and claude has huge discounts in that scenario.
the "spamming" is mostly compaction appending files, tool artifacts, images in its summary there is a known github issue for codex. only solution is periodic clean up but its also how a lot of the agentic orchestration is performed and able to work for days.
smol currently is very simple so it definitely does less things, like no subagent orchestration
I will look into how token usage looks like for longer sessions and more complex tasks
re caching: the cache ratio for this bench looks 'bad' for smol because it often finishes a task before caching kicks in (caching starts at 1024 tokens)
It removes money from your pocket into theirs faster than Claude will take money out of your pocket and into theirs. I was a Windsurf then Cursor user and even provided product feedback to the Cursor team on various aspects of their product but the costs for using it killed the value.
1) It's still an IDE. Because of pricing I mostly use Codex, but I always have a VSCode/Cursor IDE open, thus have to juggle between the two. Working directly in the IDE is more comfortable. For full on vibecoding that might be worse, but when you want to do a deep review of the changes, an IDE is way better than reading a diff on github.
2) It supports every model. It's often very helpful to try different models when you don't like the result of the first.
You can use VSCode / Zed, run any CLI coding agent in the integrated terminal and review changes using the Git working tree diff view the IDE offers. Extensions seemingly give more ergonomy, like diffs of agent changes separate from Git, or option to comment on specific file or section, but I found out that agent CLIs don't really need that today -- most of the time they know what I mean without needing to explicitly point at code locations.
They're not as good. I've tried all the options available a couple months ago, and none of them feel nearly as polished as Cursor. For one thing, Cursor Tab is just far better than any other "next edit" prediction I've used. I also happen to like their agent UI the most, but they don't support ACP, so you can't use it with Claude Code, which was a deal-breaker for me.
These days I've switched to Zed, which is good enough (and wicked fast), but I still miss Cursor as an IDE.
The Claude plugin for VSC is amazing and basically "solves programming" for me. It's the primary way that I write code now; I even stopped using Claude Code.
Cursor Tab is nice when you're hand-editing stuff. Not sure it's worth $20 though. I'd totally buy and forget a $5 Cursor subscription just for Cursor Tab.
The $20 price point is just too competitive now and I'd rather use the Claude/Codex plugin over Cursor's agentic coding sidebar.
Recently they also seem to have made big changes that makes Cursor worse for editing code by hand. I'm not sure what they are trying to do, but it doesn't feel like vscode fork anymore.
I'm starting to consider alternatives.
To answer your question though, to me your workflow seems cumbersome. Cursor is more integrated and more frictionless.
I felt the same, but I realized, that if you vibe code more, the changes make sense. Simple example:
Should the AI sidebar be on the left, or on the right? At some point they swapped them automatically, which was jarring (maybe a bug). But now I realize that if I'm not the one primarily writing code and navigating it, then I prefer agents in the "primary" position on the left (esp. if I have the browser on the left half of the screen, the agents are in the middle).
What annoys me the most about Cursor is the random stuff that breaks (due to its nature of VS Code fork).
The tab experience is still unparalleled. The cost might be worse than Codex/Claude sub, but they are all much, much, much cheaper than API pricing, so depends on your POV.
I don't think it offers any. I got tired of switching between Codex and Claude plus I wanted to start using glm and kimi so I switch to pi. Then I got tired of VSCode eating up battery and RAM so I moved to Neovim which has barely any efficiency downside anymore since most code is not written by hand and if you know how to use it well it was only maybe 10% less efficient than VSCode to begin with. I am not sure why a heavy desktop application would be better than nvim + pi in terminal which are both super extensible via vibecoding and very light on resources.
Huh, never thought about that 2nd point - I'm also transitioning to pi, but didn't think to change to neovim as well. I guess your logic holds for claude code users as well though.
I have tried multiple times to switch from Cursor to VSCode + command line tools or VSCode plugin for Claude, but it just doesn't work as well as Cursor itself.
Especially if you are doing "remote" development through SSH. If you are doing stuff where you still have to write some parts of the code manually or you have to fix few things here and there that the AI outputs, you still need a real editor.
Does that mean you’re paying api prices + cursor markup (?), or can you somehow piggyback on your Claude subscription? Fable via the API is really too expensive
You can use Cursor CLI, it's a terminal agent, just like Claude Code and Codex. I use it sometimes, I haven't opened the actual Cursor IDE in ages.
The potential benefit of Cursor CLI (vs CC and Codex) is that you can easily between all major models (by Anthropic, OpenAI, xAI, as well as Kimi K3 and GLM 5.2). I found it useful when reviewing work - e.g. I implement using Opus then review using Sol, etc. Models by different providers tend to have different perspective on things and they can find different issues with the code.
Zed lets you use Claude Code, Codex, OpenCode, etc. all through the same UI. Like you, I use that to cross validate using models from different providers and switch over to GLM 5.2 when I exceed my Claude Code limit.
You can still see what you’re billed on the Spending page. We did accidentally break dollar costs in the Usage CSV export yesterday while cleaning up an old feature flag. That was not intentional and the CSV export is fixed now.
That feature flag also showed a dollar usage graph to some self-serve users. The confusing part was included plan usage shown as $, which is not what you’re billed (on-demand usage is). Some people read it as actual spend, so we decided to remove that graph.
The circular cost indicator near the context usage indicator was also removed. It is so much easier to accidentally leave an expensive model enabled now and not realize it until you have blown through the included credits. And it is hard to imagine that that wasn’t the point of the change.
If you want to say that "admin can" - it doesn't matter, I'm not going to ping admin every day to check how it goes. I'm not going to ask admin about every session to check how cost efficient a model was.
Do you really think that users on a subscription plan aren’t intelligent enough to know that you are showing dollars at API rates if they weren’t on a subscription plan? This is a terrible change.
This is basically what I did. In 2023 I moved from VS Code to Cursor. And in 2025 when Opus 4.7 (or was it 4.6?) did that big leap in December, I switched to Claude Code and VSCode for any edits. I come from Sublime Text so those shortcuts are hardwired in me (which was also the defaults in VSCode). When Cursor hijacked literally every CMD key combo, I got fed up and switched back to VSCode. Maybe I should just go back to Sublime Text now lol, since I really just need a blazing fast code reader.
Cursor has an import assistant from vs code settings. Ahere is no assistant for the inverse. Not even for moving between computers (at least a couple of months ago when I had the need haha).
Elon will be paying his staff in tokens in the future. Grocery stores will have on-demand pricing shown in tokens, and prices will change from the time you pick up the product and you pay at the checkout. It’s Ok, Elon said money won’t exist soon anyway…
Some people don’t use AI and I don’t get it. My wife has never used AI and yesterday I saw her reading a book made out of paper.
I imagine she’s just weighed all the details and options in her mind and plans to french fry tokens from my plate instead of getting her own.
And I hate that. I mean, I love her. She’s definitely value-add. But those are my tokens, right? My brain doesn’t do books. I can’t sit on the beach and read. I must always be swimming somewhere meaningful. So I’ll be vibing the next great Canadian web app and she’ll just casually ask, “hey what do you want to do about dinner?” So now I’m asking Copilot to tell me what I want for dinner. And it’s just… c’mon lady get your own tokens.
Saw this at work yesterday. Absolute insanity to do something as user hostile as removing the cost of the service. There's no way to attempt to brand this as anything but negative for the user, and positive for the company.
Gotta justify a $60B purchase of an IDE and (at the time) a single, decent model.
Agree, I think these companies want to make token usage opaque. Not dissimilar from an AWS bill at most orgs. Not saying you can’t, with great discipline, using good tags, etc, have a good view of where your money is going. But I don’t see it much. And suspect they want to make it super hard to tell the difference between an engineer making wasteful decisions of how to use AI, and someone getting great “value per token.”
Cursor was a great introduction to agentic engineering but I've learned their Claude pricing is largely just batch purchases. Their real moat I think is Composer 2.5 because both their agentic and IDE experiences fall short of Codex and Claude Desktop in my opinion. I think their sales will tell you economically they make the most sense and I would probably agree, but cost isn't everything especially when the spread isn't that wide. At this moment, capability is really king.
These days I'm using Codex and Claude Desktop with Zen when I need to look at code. Codex's real time audio chat feature (not dictation) is also second to none when paired with their agentic flow.
Grok 4.5 is them as well now I guess since they're owned by spacex now. I'd consider grok 4.5 the sonnet and composer the haiku (both capable, fast models).
Cursor has been my corporate vendor for accessing non-anthropic models. Without cost insights and with no ability to proxy API requests its value drops dramatically. They fought hard for a renewal with legacy pricing mode then broke the deal soon thereafter. I’ll be extremely clear to leadership that Cursor usage should be minimized and not renewed. I frankly don’t trust them with IP either, and you shouldn’t either, especially if you’re demonstrably not in a big enterprise.
I think most standard users are intelligent enough to know if they’re on a subscription plan, the dollar amounts shown are if they were paying at API rates. This change is ridiculous.
This month I've noticed that my Cursor model use has gone up faster with Composer 2.5 - I have bothered with Grok yet. Despite the advertised twice-the-usage on Cursor own models I'd say it's burning faster.
I am shocked, shocked, that a company that is part of SpaceXAI would become hostile to its existing paying users and partners. This has never happened before in history.
I’ve been working on a project to solve this problem with Claude code, codex, etc by recording usage and translating to $ in realtime. Looks like I’ll have to add Cursor support now.
cursor is a scam. they pissed me with their UI changes & shady pricing, so i tried claude code recently for the first time and instantly regreted i havent tried earlier
Cursor you have a beautiful comeback story, you have a wonderful model with Composer 2.5, and a terrific behemoth with Grok 4.5 now. A top-tier dev ux with the Cursor agent app, why on earth are you squandering this opportunity and behaving this way?
You came back from the dead pretty much and now you're pissing it away for what exactly?
Do not spite your individual developer customers or you will perish yet again.
Yet another reminder that the companies providing the AI tools everyone is so gaga over are in it to make money off of you. They can and will engage in rugpulling, deception, and other shenanigans to take your money.
I can only recommend to regularly measure how many tokens a harness+model combination uses for a certain task
There are huge token efficiency/bloat differences between agents while working on the same tasks, using the same model, in the same environment
Yesterday I ran 10 agentic tasks using GPT 5.6 Sol in an ubuntu 26.04 vm a couple of times with different harnesses and got vastly different token usage.
https://x.com/__tosh/status/2083593799872237680I'm not surprised that Claude Code is not optimized for an OpenAI model but I was still quite shocked re how much of a difference the harness makes.
Disclaimer: I'm working on 'smol' which is a minimalist harness but it's really nothing special, just a minimal system prompt, no skills files, only tool is shell
Do not underestimate how much popular harnesses are spamming the context window. The context window is very important.
> Do not underestimate how much popular harnesses are spamming the context window. The context window is very important.
Do we have any insight into whether it is actually spam and not useful info such as project or programming language specific context?
in my book anything that is (I'm sure well intentioned) and injected to help the agent — but doesn't help it — is a waste of tokens
but even injected context that when I read it sounds useful can oversteer the model and make it second guess or take a more complicated route than it normally would
(you can see this when looking at traces with and without that injected context)
often harnesses also mention in their system prompt locations of markdown files that the model can consult if the model thinks they might help
that hint alone as part of the system prompt can be strong enough to make the model read in more tokens than would have been necessary
'spam' is maybe a harsh way to say it
unfortunately I don't see an easy way other than to invest time and tokens into finding out which parts of the added context (in system prompt, injected in turns etc etc) are actually helpful or harmful and when
I'm just doing the easiest thing I could think of: start from nothing or close to nothing
that seems to work better than what most harnesses are doing
turns out GPT 5.6 Sol is all you need
Claude Code injects a ton of tools into the system prompt, including their “memory system” that’s like 10k+ tokens. Depending on your task shape, this can easily double your task cost (e.g. a low context-using job that takes many turns, like a monitoring loop).
You should use --disallowed-tools to prune any tools not needed for the task. Note that this is also a perpetual game of whack-a-mole since they’re always adding new tools.
> Claude Code injects a ton of tools into the system prompt
It's so unfortunate they don't let you use the subscription with other harnesses anymore - since even if I used OpenCode they'd still get a bunch of useful data from the API calls, meanwhile I could stretch their tier limits way further.
Subsidized Anthropic subscriptions seem to work fine on the oh-my-pi harness, somehow.
Calling per token usage of the US closed source labs has always been funny to me, we have a clear model of what it actually costs to host these models from open models.
Your subscription is not subsidised, it is just closer to the actual cost of the model…
According to ccusage I use about $5k of tokens on my $100/mo Claude Max plan and only hit 5-hr windows where I have to switch tools for a couple hours about once a week
It's so unfortunate people don't realize it's cheaper to write their own well working agent instead of insisting with general purpose bloated ones like Claude.
200$/month is a lot of money on Luna/DS4 flash, like really a lot and the results are much better than clowning on bloated CC.
It's absurd how you have more and more organizations encoding their processes on LLMs and "engineers" (charlatan coders) don't even bother optimizing the tool they use most.
> the results are much better than clowning on bloated CC.
I won't argue with the cost effectiveness, but the results are very much not better. Opus and Fable are in a different league than DS4 Flash. Even GPT Terra, which I really like overall, sometimes gets stuck in weird loops and starts to do stupid stuff once its context window fills up. Whereas I can more or less trust the big models to just Do The Thing™ on the first try.
With that said, you get way more value out of a GPT subscription than you do from Claude, partly because of the ability to use more efficient harnesses.
The system prompt part is surely cached across all users.
You still pay cache token costs on API calls. Cache cost/token are 90% lower, but you pay it every single turn.
I’m not sure if they let you skip the cache write cost on the first turn. That would imply cross-user caching infrastructure or special casing the default system prompt to give you a discount. Maybe? Away from the computer but you could try a “hello” in a fresh session and see what was billed.
For Anthropic models, yes. But OP was using Claude Code with GPT.
the system prompt still takes up useful space in the context window and steers the model into unnecessary actions and over-thinking patterns
ty re --disallowed-tools
for coding agents 'shell' is often all you need (just make sure the environment has the necessary tools)
appreciate the tip, kind netizen
If you're working on smol? How complex is your work? The agent has to do everything using sed? Did you write your own tools? I guess my question is, why aren't you using pi?
The difference in tokens between the two also makes super curious. The system prompt can't be that different (I'd even bet Pi's shorter) and the 4 tools shouldn't make as much of a difference. I'm gonna have to try it.
smol only has 1 tool: sh
the system prompt of smol is shorter than the system prompt of Pi
smol has no system prompt
system prompt of Pi 0.83.0
""" You are an expert coding assistant operating inside pi, a coding agent harness. You help users by reading files, executing commands, editing code, and writing new files.
Available tools: - read: Read file contents - bash: Execute bash commands (ls, grep, find, etc.) - edit: Make precise file edits with exact text replacement, including multiple disjoint edits in one call - write: Create or overwrite files
In addition to the tools above, you may have access to other custom tools depending on the project.
Guidelines: - Use bash for file operations like ls, rg, find - Use read to examine files instead of cat or sed. - Inspect PI_* environment variables for current model and session details. - Use edit for precise changes (edits[].oldText must match exactly) - When changing multiple separate locations in one file, use one edit call with multiple entries in edits[] instead of multiple edit calls - Each edits[].oldText is matched against the original file, not after earlier edits are applied. Do not emit overlapping or nested edits. Merge nearby changes into one edit. - Keep edits[].oldText as small as possible while still being unique in the file. Do not pad with large unchanged regions. - Use write only for new files or complete rewrites. - Be concise in your responses - Show file paths clearly when working with files
Pi documentation (read only when the user asks about pi itself, its SDK, extensions, themes, skills, or TUI): - Main documentation: /usr/local/lib/node_modules/@earendil-works/pi-coding-agent/README.md - Additional docs: /usr/local/lib/node_modules/@earendil-works/pi-coding-agent/docs - Examples: /usr/local/lib/node_modules/@earendil-works/pi-coding-agent/examples (extensions, custom tools, SDK) - When reading pi docs or examples, resolve docs/... under Additional docs and examples/... under Examples, not the current working directory - When asked about: extensions (docs/extensions.md, examples/extensions/), themes (docs/themes.md), skills (docs/skills.md), prompt templates (docs/prompt-templates.md), TUI components (docs/tui.md), keybindings (docs/keybindings.md), SDK integrations (docs/sdk.md), custom providers (docs/custom-provider.md), adding models (docs/models.md), pi packages (docs/packages.md), environment variables (docs/environment-variables.md) - When working on pi topics, read the docs and examples, and follow .md cross-references before implementing - Always read pi .md files completely and follow links to related docs (e.g., tui.md for TUI API details) Current working directory: /workspace """
> The context window is very important.
It is everything. My experience with Claude Code is that you have to decide when to compact to make it efficient. It defaults everything to 1M context and it will never keep it in check. It is strange how little cache reads you hit in smol, that may be a configuration issue.
I will look into it more to see if I have configured it wrong but I think the token efficiency hurts cache use as caching only starts at 1024 tokens so for tasks where smol is under or close to 1024 tokens most of them are uncached
What tools are you using to run the comparison? Or are you just running the prompts and checking them with something like ccusage?
I’m asking because I’ve been looking for agent harness comparison tools too. I’m interested in more than just the inputs and outputs—I also want the system prompts, traces, and tool calls. It’s useful to understand why Codex, for example, uses more tokens while Pi doesn’t.
Fewer tokens aren’t necessarily better if the agent skipped important checks. On the other hand, using more tokens could just mean it’s overthinking the process. Either way, seeing the full execution trace for the same task is really valuable.
I built an ad-hoc custom comparison framework to inspect system prompts, caching behaviour, tool call outputs, exact api requests and responses and so on
I agree fewer tokens is not necessarily better but a bit counter-intuitively often the harness using fewer tokens is not only done faster but has better results
(that said: of course check the results, look at the full traces, agree!)
Thanks! I ended up built my own too. Just thought there are other better options out there that I might've missed.
Running some local models and wondering what insanity was consuming 100k+ tokens to respond to "hello world" was rather eye-opening, yeah. Full of pointless fluff.
Thanks for sharing. Naming tangent: FWIW I have recent interest in "smolvm" microvms from smolmachines.com, which are unrelated.
Anything else I can read about smol?
also, are you using a tool to collect those metrics? what is it?
smol is basically this 9 line python agent re-implemented in Go
https://news.ycombinator.com/item?id=49006862
I'll have more about it in the next hours/days, you can follow me on twitter in the meantime (https://x.com/__tosh)
So smol is your agent? Do you have a GitHub link?
coming in a few hours
you can follow this org in the meantime https://github.com/smol-env
or on twitter here: https://x.com/__tosh
While there's a lot of bloat in the harnesses, almost all of it is cached. Your harness appears to be caching almost nothing. That will almost certainly bode extremely poorly for longer-running tasks than the simple ones you mentioned testing. Benchmaxxing for simplistic tasks could leave you not only suffering in completion rate on complex tasks but also possibly even not end up with the cost gains you're hoping for. And maybe that's fine if the only thing you use agents for is the very simple tasks, which is certainly a defensible decision, but probably bears mentioning when you're comparing it against harnesses with a different design goal.
smol is also prefix caching
the uncached tokens are also from runs where smol finished a task below 1024 tokens (the minimum amount of tokens needed to activate caching) which is less tokens than other harnesses are using for their system prompt (!)
> GPT-5.6 and later models: Caching is available for prefixes containing at least 1,024 tokens. This is a strict minimum.
https://developers.openai.com/api/docs/guides/prompt-caching
so in this specific case the count of uncached tokens for smol makes it look worse than it actually is
that said: it does makes sense to add more tasks that are difficult enough to fill the context window to compare the harnesses for how well they deal with compaction
staying below compaction (or with compaction at fewer compactions) is not only cheaper and faster, it also helps the agent stay on track
Yes, I do agree that avoiding compactions is extremely desirable, just make sure you're actually testing for how well that happens and the results!
will do!
Same task? How did the results compare?
the tasks were all simple agentic tasks
like creating a checksum of a file, merging csvs and so on, fixing a makefile pipeline
with known 'good' outcomes
all harnesses could reach the outcomes, only cost, time, number of tool uses and so on were different
(Claude Code failed once in 1 task but I think that was just an unfortunate outlier, the tasks aren't that difficult)
> There are huge token efficiency/bloat differences between agents while working on the same tasks, using the same model, in the same environment
Be careful here. Remember these are non-deterministic models at the end of the day, and even with everything being "the same" you can have two runs where the same model, same harness, same tools can arrive at the same conclusion through a wildly different sequence of events.
agree, that makes it a bit tricky to compare (esp if you also want to add different models and reasoning levels into the mix)
I will add more tasks (esp longer ones) and think more about grading, the current tasks were easy to grade because the desired outcomes are well specced but I will also look into more open ended tasks and how to grade those
thank you!
this benchmark is a very misleading because it doesn't necessarily point to these harnesses saving usage and much of the harness that comes with codex for example is designed at cross functional agentic usage which you will be leaving behind by using a different harness. openai spends a lot of time as does claude. also note that cached is what makes up bulk of the total and codex and claude has huge discounts in that scenario.
the "spamming" is mostly compaction appending files, tool artifacts, images in its summary there is a known github issue for codex. only solution is periodic clean up but its also how a lot of the agentic orchestration is performed and able to work for days.
smol currently is very simple so it definitely does less things, like no subagent orchestration
I will look into how token usage looks like for longer sessions and more complex tasks
re caching: the cache ratio for this bench looks 'bad' for smol because it often finishes a task before caching kicks in (caching starts at 1024 tokens)
thank you for flagging this
I was an early and passionate adopter and paying customer of Cursor (since 2023!), but it’s probably been 6 months since I opened it.
These days I “write” code with claude code and codex, and read/review it on GitHub. If I need to read it locally, I use a plain text editor.
Can someone help me understand what value cursor offers in 2026?
It removes money from your pocket into theirs faster than Claude will take money out of your pocket and into theirs. I was a Windsurf then Cursor user and even provided product feedback to the Cursor team on various aspects of their product but the costs for using it killed the value.
> faster than Claude will take money out of your pocket and into theirs
Damn, and I thought Anthropic was fast.
Cursor has two benefits
1) It's still an IDE. Because of pricing I mostly use Codex, but I always have a VSCode/Cursor IDE open, thus have to juggle between the two. Working directly in the IDE is more comfortable. For full on vibecoding that might be worse, but when you want to do a deep review of the changes, an IDE is way better than reading a diff on github.
2) It supports every model. It's often very helpful to try different models when you don't like the result of the first.
You can use VSCode / Zed, run any CLI coding agent in the integrated terminal and review changes using the Git working tree diff view the IDE offers. Extensions seemingly give more ergonomy, like diffs of agent changes separate from Git, or option to comment on specific file or section, but I found out that agent CLIs don't really need that today -- most of the time they know what I mean without needing to explicitly point at code locations.
Can't VS Code extensions solve your use case?
They're not as good. I've tried all the options available a couple months ago, and none of them feel nearly as polished as Cursor. For one thing, Cursor Tab is just far better than any other "next edit" prediction I've used. I also happen to like their agent UI the most, but they don't support ACP, so you can't use it with Claude Code, which was a deal-breaker for me.
These days I've switched to Zed, which is good enough (and wicked fast), but I still miss Cursor as an IDE.
I tried the codex extension many months ago but I didn't like it. I use mostly the Codex App, or sometimes the Codex TUI inside the IDE terminal.
The Claude plugin for VSC is amazing and basically "solves programming" for me. It's the primary way that I write code now; I even stopped using Claude Code.
You can use Intellij Ides, at least free ones.
Cursor Tab is nice when you're hand-editing stuff. Not sure it's worth $20 though. I'd totally buy and forget a $5 Cursor subscription just for Cursor Tab.
The $20 price point is just too competitive now and I'd rather use the Claude/Codex plugin over Cursor's agentic coding sidebar.
Recently they also seem to have made big changes that makes Cursor worse for editing code by hand. I'm not sure what they are trying to do, but it doesn't feel like vscode fork anymore. I'm starting to consider alternatives.
To answer your question though, to me your workflow seems cumbersome. Cursor is more integrated and more frictionless.
Cursor has two “modes” agent and ide.
In agent mode it’s horrible for editing files etc but better if you are juggling multiple chats
I felt the same, but I realized, that if you vibe code more, the changes make sense. Simple example:
Should the AI sidebar be on the left, or on the right? At some point they swapped them automatically, which was jarring (maybe a bug). But now I realize that if I'm not the one primarily writing code and navigating it, then I prefer agents in the "primary" position on the left (esp. if I have the browser on the left half of the screen, the agents are in the middle).
What annoys me the most about Cursor is the random stuff that breaks (due to its nature of VS Code fork).
The tab experience is still unparalleled. The cost might be worse than Codex/Claude sub, but they are all much, much, much cheaper than API pricing, so depends on your POV.
Look at Cursor CLI: https://cursor.com/cli
I don't think it offers any. I got tired of switching between Codex and Claude plus I wanted to start using glm and kimi so I switch to pi. Then I got tired of VSCode eating up battery and RAM so I moved to Neovim which has barely any efficiency downside anymore since most code is not written by hand and if you know how to use it well it was only maybe 10% less efficient than VSCode to begin with. I am not sure why a heavy desktop application would be better than nvim + pi in terminal which are both super extensible via vibecoding and very light on resources.
Huh, never thought about that 2nd point - I'm also transitioning to pi, but didn't think to change to neovim as well. I guess your logic holds for claude code users as well though.
I have tried multiple times to switch from Cursor to VSCode + command line tools or VSCode plugin for Claude, but it just doesn't work as well as Cursor itself.
Especially if you are doing "remote" development through SSH. If you are doing stuff where you still have to write some parts of the code manually or you have to fix few things here and there that the AI outputs, you still need a real editor.
The way they spin up multiple Composer 2.5 before sending it over to the model of your choice is nice.
How do you diff ?
Unless by "text editor" you mean Neovim or Emacs ?
Sounds like on GitHub. That's how I do it too.
I’m surprised. That’s nowhere what you would get out of a tool like GitLens or Sublime Merge.
Diffing through a web interface offered by GitHub or Bitbucket is such a narrow window into changes.
I was running `git diff` before so I'm not sure what I'm missing I guess.
git diff
Code and reasoning generated by Opus/Fable via Cursor seems to be quite better than Claude Code's for the same prompt.
Does that mean you’re paying api prices + cursor markup (?), or can you somehow piggyback on your Claude subscription? Fable via the API is really too expensive
I'm also confused about Cursor. I use Zed. It seems like whatever Cursor has is pure commodity and yet it's worth billions?
You can use Cursor CLI, it's a terminal agent, just like Claude Code and Codex. I use it sometimes, I haven't opened the actual Cursor IDE in ages.
The potential benefit of Cursor CLI (vs CC and Codex) is that you can easily between all major models (by Anthropic, OpenAI, xAI, as well as Kimi K3 and GLM 5.2). I found it useful when reviewing work - e.g. I implement using Opus then review using Sol, etc. Models by different providers tend to have different perspective on things and they can find different issues with the code.
Zed lets you use Claude Code, Codex, OpenCode, etc. all through the same UI. Like you, I use that to cross validate using models from different providers and switch over to GLM 5.2 when I exceed my Claude Code limit.
Design mode: https://cursor.com/blog/design-mode
(I work at Cursor)
You can still see what you’re billed on the Spending page. We did accidentally break dollar costs in the Usage CSV export yesterday while cleaning up an old feature flag. That was not intentional and the CSV export is fixed now.
That feature flag also showed a dollar usage graph to some self-serve users. The confusing part was included plan usage shown as $, which is not what you’re billed (on-demand usage is). Some people read it as actual spend, so we decided to remove that graph.
The circular cost indicator near the context usage indicator was also removed. It is so much easier to accidentally leave an expensive model enabled now and not realize it until you have blown through the included credits. And it is hard to imagine that that wasn’t the point of the change.
> You can still see what you’re billed on the Spending page.
No, you can not: https://www.pasteboard.co/dNXUdT-h8Giy.png
If you want to say that "admin can" - it doesn't matter, I'm not going to ping admin every day to check how it goes. I'm not going to ask admin about every session to check how cost efficient a model was.
Do you really think that users on a subscription plan aren’t intelligent enough to know that you are showing dollars at API rates if they weren’t on a subscription plan? This is a terrible change.
Dead company, purchased completely via SpaceX stock. Hope it doesn't crash more before the cliff period ends.
Cursor achieved fast adoption by making it simple to move from Visual Studio code.
Double-edged sword. It is also simple to move back to VS code and agent extensions.
This is basically what I did. In 2023 I moved from VS Code to Cursor. And in 2025 when Opus 4.7 (or was it 4.6?) did that big leap in December, I switched to Claude Code and VSCode for any edits. I come from Sublime Text so those shortcuts are hardwired in me (which was also the defaults in VSCode). When Cursor hijacked literally every CMD key combo, I got fed up and switched back to VSCode. Maybe I should just go back to Sublime Text now lol, since I really just need a blazing fast code reader.
Cursor has an import assistant from vs code settings. Ahere is no assistant for the inverse. Not even for moving between computers (at least a couple of months ago when I had the need haha).
But doesn’t vscode’s settings.json have most if not all settings stored inside it? Should be a simple copy and paste.
Elon will be paying his staff in tokens in the future. Grocery stores will have on-demand pricing shown in tokens, and prices will change from the time you pick up the product and you pay at the checkout. It’s Ok, Elon said money won’t exist soon anyway…
Some people don’t use AI and I don’t get it. My wife has never used AI and yesterday I saw her reading a book made out of paper.
I imagine she’s just weighed all the details and options in her mind and plans to french fry tokens from my plate instead of getting her own.
And I hate that. I mean, I love her. She’s definitely value-add. But those are my tokens, right? My brain doesn’t do books. I can’t sit on the beach and read. I must always be swimming somewhere meaningful. So I’ll be vibing the next great Canadian web app and she’ll just casually ask, “hey what do you want to do about dinner?” So now I’m asking Copilot to tell me what I want for dinner. And it’s just… c’mon lady get your own tokens.
It sounds like your wife might get left behind by you.
She’s definitely not Mars material.
She'll understand when she's clawing at the door to be let into the singularity.
Saw this at work yesterday. Absolute insanity to do something as user hostile as removing the cost of the service. There's no way to attempt to brand this as anything but negative for the user, and positive for the company.
Gotta justify a $60B purchase of an IDE and (at the time) a single, decent model.
If users start questioning the ROI of your AI product just hide the “I” from them. Problem solved.
Agree, I think these companies want to make token usage opaque. Not dissimilar from an AWS bill at most orgs. Not saying you can’t, with great discipline, using good tags, etc, have a good view of where your money is going. But I don’t see it much. And suspect they want to make it super hard to tell the difference between an engineer making wasteful decisions of how to use AI, and someone getting great “value per token.”
Cursor was a great introduction to agentic engineering but I've learned their Claude pricing is largely just batch purchases. Their real moat I think is Composer 2.5 because both their agentic and IDE experiences fall short of Codex and Claude Desktop in my opinion. I think their sales will tell you economically they make the most sense and I would probably agree, but cost isn't everything especially when the spread isn't that wide. At this moment, capability is really king.
These days I'm using Codex and Claude Desktop with Zen when I need to look at code. Codex's real time audio chat feature (not dictation) is also second to none when paired with their agentic flow.
Grok 4.5 is them as well now I guess since they're owned by spacex now. I'd consider grok 4.5 the sonnet and composer the haiku (both capable, fast models).
I actually use grok as my planner and then composer for coding like opusplan in Claude.
Cursor has been my corporate vendor for accessing non-anthropic models. Without cost insights and with no ability to proxy API requests its value drops dramatically. They fought hard for a renewal with legacy pricing mode then broke the deal soon thereafter. I’ll be extremely clear to leadership that Cursor usage should be minimized and not renewed. I frankly don’t trust them with IP either, and you shouldn’t either, especially if you’re demonstrably not in a big enterprise.
That is how company gets greedy. Just archived that thread, wonder if cursor will close and remove it
Not showing the model cost in the Cursor UI is the biggest failure of Cursor.
I think most standard users are intelligent enough to know if they’re on a subscription plan, the dollar amounts shown are if they were paying at API rates. This change is ridiculous.
This month I've noticed that my Cursor model use has gone up faster with Composer 2.5 - I have bothered with Grok yet. Despite the advertised twice-the-usage on Cursor own models I'd say it's burning faster.
They just don't want you feel obligated to thank them SO MUCH for hidden better prices
I am shocked, shocked, that a company that is part of SpaceXAI would become hostile to its existing paying users and partners. This has never happened before in history.
Small thing but makes a big difference
I’ve been working on a project to solve this problem with Claude code, codex, etc by recording usage and translating to $ in realtime. Looks like I’ll have to add Cursor support now.
The ground-truth is the rudeness is load bearing at human machine interface.
The signal is the ceiling.
cursor is a scam. they pissed me with their UI changes & shady pricing, so i tried claude code recently for the first time and instantly regreted i havent tried earlier
Built an awesome 450k line app with composer 2.5 and grok 4.5. I love it
It's free? Oh it's priced in "tokens".
Kind of (barely) like how Facebook has "friends". Or how a Snickers bar costs $1.99 and not $2.
Abstracting meaning of "cost", reducing value of information. Maximizing profits. Enshittification.
Cursor you have a beautiful comeback story, you have a wonderful model with Composer 2.5, and a terrific behemoth with Grok 4.5 now. A top-tier dev ux with the Cursor agent app, why on earth are you squandering this opportunity and behaving this way?
You came back from the dead pretty much and now you're pissing it away for what exactly?
Do not spite your individual developer customers or you will perish yet again.
If they were smart, they'd rebrand Grok to Composer. Nobody wants to use the Nazi model.
It seems to me that people have a dislike like for Elon then just blanket everything with the same label.
It might be called grok out it was trained by the composer team using a large chunk of their training data.
Given that cursor has been bought by Musk, the nazi branding might be on purpose
please, you're dramatically overestimating who gives a shit about this.
>please, you're dramatically overestimating who gives a shit about this.
And you're underestimating how much of a shit you need to give about this.
“If there's a Nazi at the table and ten other people sitting there talking to him, you got a table with eleven Nazis.”
You’re telling me there are a lot of Nazis we’re going to have to deal with. No problem, we’ve done it before, we’ll do it again.
Well ... don't use Cursor. It's really that simple.
Yet another reminder that the companies providing the AI tools everyone is so gaga over are in it to make money off of you. They can and will engage in rugpulling, deception, and other shenanigans to take your money.
Elon-ification begins...