What I have personally observed with such tools is that they make the AI's dumb, similar to how it makes coders dumb when relying more on AI tools.
These agentic AI's are already smart enough to figure out a highly optimized path to code exploration or search. But, with these tools, they just go very aggressive, partly because the search results from these tools almost in 100% of the cases do not furnish full details, but, just the pointers.
To confirm this behaviour, I did a small test run. This is in no way conclusive, but, the results do align with what I been observing:
---
Task: trace full ingestion and search paths in some okayish complex project. Harness is Pi.
1. With "codebase-memory-mcp": 85k/4.4k (input/output tokens).
2. With my own regular setup: 67k/3.2k.
3. Without any of these: 80k/3.2k.
As we see, such a tool made it worse (not by much, but, still). The outputs were same in quality and informational content.
---
Now, what my "regular setup" mentioned above is?:
Just one line in AGENTS.md and CLAUDE.md: "Start by reading PROJECT.md" .
And PROJECT.md contains just following: 2-3 line description of the project, all relevant files and their one-line description, any nuiances, and finally, ends with this line:
## To LLM
Update this file if the changes you have done are worth updating here. The intent of this file is to give you a rough idea of the project, from where you can explore further, if needed.
Hey, codebase-memory-mcp and semble are not exactly the same, but it's an interesting comparison, I'll put it on the todolist to check that out and add it to our benchmarks if feasible. If you ever get a chance to do this same comparison with semble it would be super useful feedback since these "real" scenarios are hard to benchmark/replicate.
So, I just tested with semble. Your MCP integration did not work, and kept throwing error (Failed to connect to "semble": MCP error -32000: Connection closed) though I installed using documented manner (tried both: pip and ux methods).
Anyways, I made it work by making it generate relevant doc (using semble init), and then copying this into AGENTS.md, and then prompting it with this line:
"""
Start by reading AGENTS.md in current folder. Now, the task::: `Explore the ingestion and search paths. Do not read README.md at all`. Prefer to use `semble` search for code search. Do not do new installation. semble is already available at `/Users/nitinbansal/.local/bin/semble` .
"""
The results are much better. Even better than my own setup, but, vary a lot. I did 4 runs:
Sorry to hear about the MCP integration, that's definitely something we'll look into. If you have any info about your system or how to reproduce it please let me know. Very nice to hear about the results, thanks for checking this! The variance is interesting to see, that's probably non-determinism in the LLM rather than semble since semble is deterministic. But I'm guessing we can make that better with the prompt, I'll look into this.
Interesting. I too have been working in this space, though I took a different approach. Rather than building an index, I worked on making a "smarter grep" by offering search over codebases (and any text content really) with ranking and some structural awareness of the code. Most of my time was spend dealing with performance, and as a result it runs extremely quickly.
I will have to add this as a comparison to https://github.com/boyter/cs and see what my LLMs prefer for the sort of questions I ask. It too ships with MCP, but does NOT build an index for its search. I am very curious to see how it would rank seeing as it does not do basic BM25 but a code semantic variant of it.
This seems to work better for the "how does auth work" style of queries, while cs does "authenticate --only-declarations" and then weighs results based on content of the files, IE where matches are, in code, comments and the overall complexity of the file.
I'd be interested in seeing actual agent benchmarks (eg CC or Copilot CLI with grep removed and this tool instead).
For example, I have explored RTK and various LSP implementations and find that the models are so heavily RL'd with grep that they do not trust results in other forms and will continually retry or reread, and all token savings are lost because the model does not trust the results of the other tools.
Hey, this is something we're actively working on, but this is hard (and expensive) to do well across harnesses/models. The grep pretraining thing is very interesting though, I've noticed the same. E.g. Sonnet 4.6 seems to trust semble but Opus 4.7 less so. I'm hoping we can quantitatively test this and improve it when we have proper benchmarks for this as well. If you do have any feedback though let me know!
Codex CLI is quite happy running RTK. Well with GPT 5.5 xhigh anyway
One thing that irks me is that when it doesn't support eg. a cli flag of find, it gives an error message rather than sending the full output of the command instead. Then the agent wastes tokens retrying, or worse, doesn't even try because the prompting may make them afraid to not run commands without rtk
I can't find the relevant issues in their repo, but I've been somewhat skeptical of their tool over-reporting token savings and there are many issues to that effect in the repo.
I'm not likely to install it again in my latest configuration, instead applying some specific tricks to things like `make test` to spit out zero output exit on unsuccessful error codes, that sort of thing. Anecdotally, I see GPT-5.5 often automatically applying context limiting flags to the bash it writes :shrug:
I've had the same experience with RTK, where my agent got stuck in a loop with a faulty RTK command and could not escape it since RTK hard overwrites anything automatically. I've uninstalled it again for the time being.
Yeah we're also interested in doing this, it's on the roadmap together with optimization of the prompt and descriptions so that models have an easier time using it.
Perhaps anecdotally: we do use this tool ourselves of course, and it's been working pretty well so far. Anthropic models call it and seem to trust the results.
I forced Claude to have a global memory for RTK and my own AI memory system (GuardRails) which it happily uses both, the only times it doesnt use GuardRails is if I dont mention it at all, otherwise it always uses RTK unless RTK falls apart running a tool it does not support.
I did some evals with pi and GPT 5.5. I tested RTK on / headroom on / both on / both off (all with the standard pi system instructions and no AGENTS.md).
I forget the exact tests I used (a couple of the standard agent evals that people use, one python and one typescript because those are what I use).
I don't claim it was an exhaustive test, or even a good one. It's possible I could have spent a day or so tuning my AGENTS.md and the pi system prompt/tool instructions and gotten better results, because if there's one thing running evals taught me it's that subtle differences there can change the results a lot.
However, I got clearly better results with both off, enough to convince me to stop the tests immediately after 3 rounds.
The problem was that while context use did go down (sometimes), the number of turns to complete went up so the overall cost of the conversation was higher.
It's made me very aware of one thing: so many people are sharing these kind of tools, but either with zero evals (or suspiciously hard to reproduce), or in the case of this one, extensive benchmarks testing the wrong thing.
I'm sure this tool does use fewer tokens than grep, and the benchmarks prove it, but that's not what matters here. What matters is, does an agent using it get the same quality of work done more quickly and for lower cost?
We didn't generate this project, we wrote it, a lot of it manually, and trained custom models. We'd been working in the real-time retrieval space for a while, and we thought coding was a good fit for this specific technology.
My comment above wasn't meant to be rude. And you do have extensive benchmarks against grep etc so it's clear you understand the importance of that.
But I still think you're missing the harder but more important proof which is agent evals. Have you done any of that?
I would personally love to find tools in this space which can make agents more efficient and I do believe there's a scope for massive improvements compared to default workflows. But my evals with RTK and Headroom have made me wary that a tool can look like it should work, conceptually make sense, pass non-agentic benchmarks, and still make an actual agentic workflow worse.
yeah I think I'm prone to do the same, it is so easy to create and we get too excited by it instead of first doing the research necessary which is much more boring than actually producing something.
Nice, this sounds great. I want to mention a related issue here, which is that on small codebases, Claude spends a lot of time looking for stuff when it could have just dumped the whole codebase into the context in one go and used very little tokens.
I found a nice workaround which is that you can just dump the whole directory into context, as a startup hook. So then Claude skips the "fumble around blindly in the dark" portion of every task. (I've also seen a great project that worked on bigger repos where it'll give the model an outline with stubs, though I forget what it was called.)
Cool project. I built a custome IDE and coding agent harness and will integrate this into it. If you’re interested in a collaboration, I’d be happy to share revenue to sponsor your open source repo.
The first is hard to test for us unfortunately since we don't use Cursor. But the Claude thing is interesting. I think that providers (especially the ones that directly sell LLM calls like Anthropic) are not incentivised per se to think about token efficiency vs performance, so if you're chasing pure performance, just loading the full codebase into memory might still be the "benchmark topping" way to go. I think the dust hasn't really settled yet and we'll likely see a lot of changes in the coming year about what's the "correct" way to solve it. It might be different based on your harness/budget/model as well.
Seems like a cool idea so I decided to play with it a bit. The test I ran was in the browsercode (https://github.com/browser-use/browsercode) repo with the following prompt:
"Answer this question by only using the `semble` CLI (docs below):
> What tools does Browsercode provide to the agent other than the base OpenCode tools? Provide the exact schema for tool input and tool output and briefly summarize what they do and how they work
"Answer this question by only using the `rg` and `fd` CLIs:
> What tools does Browsercode provide to the agent other than the base OpenCode tools? Provide the exact schema for tool input and tool output and briefly summarize what they do and how they work"
In both cases, I used Pi with gpt-5.4 medium and a very minimal setup otherwise. (And yes, I did verify that either instance only used rg & fd, or only used semble.)
Without Semble, it used 10.9% of the model context and used $0.144 of API credits (or, at least, that's what Pi reported - I used this with a Codex sub so cannot be sure). With Semble, it used 9.8% of the model context and $0.172 of API credits. The resulting responses were also about the same. Very close!
I tried one more test in the OpenCode repo. The question was
> Trace the path from 1) the OPENCODE_EXPERIMENTAL_EXA env var being set to to 1 to 2) the resulting effects in the system prompt or tool provided to the OpenCode agent.
And I included the same instructions/docs as above. The non-Semble version was a bit more detailed -- it went into whether the tool call path invoked Exa based on whether Exa or Parallel was enabled for the web search provider -- but w.r.t. actually answering the question, both versions were accurate. The Semble version used 14.7% context / $0.282 API cost, while the non-Semble version used 19.0% / $0.352. Clearly a win for Semble for context efficiency, but note that the non-Semble version finished about twice as fast as the Semble version.
Try running both on the CK codebase. CK takes like 15 minutes to index itself and gives hundreds of completely irrelevant doc comments as results for “run model on CPU” query. Semble indexes for like 3 seconds and prints out the actual code that runs the model on the CPU.
I also like the index feature form https://maki.sh
Source code has a lot of structure, using a real parser instead of grepping and reading files can potentially save a lot of tokens
Yeah I agree. I have used semble to quickly index a large monorepo and just ask a question about it, it surfaced the right files pretty quickly. Although without an IDE, it's difficult to display them in nice way
Afaik many harnesses ship the "default" which is grep+read (like Claude Code). But I agree, IMO it's a weird gap. To be fair I don't think providers are that incentivised to reduce token burn at the moment, but my guess is that that will change and tools like this will become at least an natively supported option in some harnesses.
This looks great! I built a tool in the same space- and I found that the biggest challenge was often to get the agent to prefer to use the tool over bash tools. What’s your experience with that?
Serena does a lot more than semble (I actually used serena before building this and didn't like how much it does by default). That also made it hard to see if it was actually working well with how many moving parts there are. Semble only does 1 thing: very quick code search, that's it. Context-mode I have not used before though, I will have a look at that, thanks for sharing!
For chunking Semble supports all languages supported by tree-sitter-language-pack. The models we train are trained on 6 languages, but can handle way more.
Hey, this is something we're actively investigating. We recently added a flag, `--include-text-files`, which, when set, also makes Semble index regular documents (i.e., markdown, text, json). This should also work relatively well.
Perhaps Python is their main language (they seem to be ML peeps, which would make that most likely), which means it's easier for them to do manual reviews even if they're using AI for implementing, etc.
What I have personally observed with such tools is that they make the AI's dumb, similar to how it makes coders dumb when relying more on AI tools.
These agentic AI's are already smart enough to figure out a highly optimized path to code exploration or search. But, with these tools, they just go very aggressive, partly because the search results from these tools almost in 100% of the cases do not furnish full details, but, just the pointers.
To confirm this behaviour, I did a small test run. This is in no way conclusive, but, the results do align with what I been observing:
---
Task: trace full ingestion and search paths in some okayish complex project. Harness is Pi.
1. With "codebase-memory-mcp": 85k/4.4k (input/output tokens).
2. With my own regular setup: 67k/3.2k.
3. Without any of these: 80k/3.2k.
As we see, such a tool made it worse (not by much, but, still). The outputs were same in quality and informational content.
---
Now, what my "regular setup" mentioned above is?:
Just one line in AGENTS.md and CLAUDE.md: "Start by reading PROJECT.md" .
And PROJECT.md contains just following: 2-3 line description of the project, all relevant files and their one-line description, any nuiances, and finally, ends with this line:
Hey, codebase-memory-mcp and semble are not exactly the same, but it's an interesting comparison, I'll put it on the todolist to check that out and add it to our benchmarks if feasible. If you ever get a chance to do this same comparison with semble it would be super useful feedback since these "real" scenarios are hard to benchmark/replicate.
So, I just tested with semble. Your MCP integration did not work, and kept throwing error (Failed to connect to "semble": MCP error -32000: Connection closed) though I installed using documented manner (tried both: pip and ux methods).
Anyways, I made it work by making it generate relevant doc (using semble init), and then copying this into AGENTS.md, and then prompting it with this line:
""" Start by reading AGENTS.md in current folder. Now, the task::: `Explore the ingestion and search paths. Do not read README.md at all`. Prefer to use `semble` search for code search. Do not do new installation. semble is already available at `/Users/nitinbansal/.local/bin/semble` . """
The results are much better. Even better than my own setup, but, vary a lot. I did 4 runs:
95k/2.9k
25k/2.7k
71k/2.9k
37k/4.0k
Sorry to hear about the MCP integration, that's definitely something we'll look into. If you have any info about your system or how to reproduce it please let me know. Very nice to hear about the results, thanks for checking this! The variance is interesting to see, that's probably non-determinism in the LLM rather than semble since semble is deterministic. But I'm guessing we can make that better with the prompt, I'll look into this.
Interesting. I too have been working in this space, though I took a different approach. Rather than building an index, I worked on making a "smarter grep" by offering search over codebases (and any text content really) with ranking and some structural awareness of the code. Most of my time was spend dealing with performance, and as a result it runs extremely quickly.
I will have to add this as a comparison to https://github.com/boyter/cs and see what my LLMs prefer for the sort of questions I ask. It too ships with MCP, but does NOT build an index for its search. I am very curious to see how it would rank seeing as it does not do basic BM25 but a code semantic variant of it.
This seems to work better for the "how does auth work" style of queries, while cs does "authenticate --only-declarations" and then weighs results based on content of the files, IE where matches are, in code, comments and the overall complexity of the file.
Have starred and will be watching.
Nice! Let us know if you have any feedback or results to share, would be happy to do the same.
I'd be interested in seeing actual agent benchmarks (eg CC or Copilot CLI with grep removed and this tool instead).
For example, I have explored RTK and various LSP implementations and find that the models are so heavily RL'd with grep that they do not trust results in other forms and will continually retry or reread, and all token savings are lost because the model does not trust the results of the other tools.
Hey, this is something we're actively working on, but this is hard (and expensive) to do well across harnesses/models. The grep pretraining thing is very interesting though, I've noticed the same. E.g. Sonnet 4.6 seems to trust semble but Opus 4.7 less so. I'm hoping we can quantitatively test this and improve it when we have proper benchmarks for this as well. If you do have any feedback though let me know!
I just put something in my global CLAUDE.md (under ~/.Claude) asking it to use the LSP instead of grep and have never had this issue since.
can you share that prompt?
My q would have been this. Lsp solved this no?
Codex CLI is quite happy running RTK. Well with GPT 5.5 xhigh anyway
One thing that irks me is that when it doesn't support eg. a cli flag of find, it gives an error message rather than sending the full output of the command instead. Then the agent wastes tokens retrying, or worse, doesn't even try because the prompting may make them afraid to not run commands without rtk
how effective is RTK for you? worth using?
I can't find the relevant issues in their repo, but I've been somewhat skeptical of their tool over-reporting token savings and there are many issues to that effect in the repo.
I'm not likely to install it again in my latest configuration, instead applying some specific tricks to things like `make test` to spit out zero output exit on unsuccessful error codes, that sort of thing. Anecdotally, I see GPT-5.5 often automatically applying context limiting flags to the bash it writes :shrug:
I've had the same experience with RTK, where my agent got stuck in a loop with a faulty RTK command and could not escape it since RTK hard overwrites anything automatically. I've uninstalled it again for the time being.
Wondering too
Yeah we're also interested in doing this, it's on the roadmap together with optimization of the prompt and descriptions so that models have an easier time using it.
Perhaps anecdotally: we do use this tool ourselves of course, and it's been working pretty well so far. Anthropic models call it and seem to trust the results.
I forced Claude to have a global memory for RTK and my own AI memory system (GuardRails) which it happily uses both, the only times it doesnt use GuardRails is if I dont mention it at all, otherwise it always uses RTK unless RTK falls apart running a tool it does not support.
I did some evals with pi and GPT 5.5. I tested RTK on / headroom on / both on / both off (all with the standard pi system instructions and no AGENTS.md).
I forget the exact tests I used (a couple of the standard agent evals that people use, one python and one typescript because those are what I use).
I don't claim it was an exhaustive test, or even a good one. It's possible I could have spent a day or so tuning my AGENTS.md and the pi system prompt/tool instructions and gotten better results, because if there's one thing running evals taught me it's that subtle differences there can change the results a lot.
However, I got clearly better results with both off, enough to convince me to stop the tests immediately after 3 rounds.
The problem was that while context use did go down (sometimes), the number of turns to complete went up so the overall cost of the conversation was higher.
It's made me very aware of one thing: so many people are sharing these kind of tools, but either with zero evals (or suspiciously hard to reproduce), or in the case of this one, extensive benchmarks testing the wrong thing.
I'm sure this tool does use fewer tokens than grep, and the benchmarks prove it, but that's not what matters here. What matters is, does an agent using it get the same quality of work done more quickly and for lower cost?
with AI the "they could so they never wondered if they should" will be a very frequent thing.
This is a bit rude.
We didn't generate this project, we wrote it, a lot of it manually, and trained custom models. We'd been working in the real-time retrieval space for a while, and we thought coding was a good fit for this specific technology.
My comment above wasn't meant to be rude. And you do have extensive benchmarks against grep etc so it's clear you understand the importance of that.
But I still think you're missing the harder but more important proof which is agent evals. Have you done any of that?
I would personally love to find tools in this space which can make agents more efficient and I do believe there's a scope for massive improvements compared to default workflows. But my evals with RTK and Headroom have made me wary that a tool can look like it should work, conceptually make sense, pass non-agentic benchmarks, and still make an actual agentic workflow worse.
yeah I think I'm prone to do the same, it is so easy to create and we get too excited by it instead of first doing the research necessary which is much more boring than actually producing something.
Nice, this sounds great. I want to mention a related issue here, which is that on small codebases, Claude spends a lot of time looking for stuff when it could have just dumped the whole codebase into the context in one go and used very little tokens.
I found a nice workaround which is that you can just dump the whole directory into context, as a startup hook. So then Claude skips the "fumble around blindly in the dark" portion of every task. (I've also seen a great project that worked on bigger repos where it'll give the model an outline with stubs, though I forget what it was called.)
This is true, agents just don't know a lot about the things they're looking at, e.g., the number of files, file sizes, etc.
Although for small codebases it also holds that whatever you would like to find it easy to find, so search still might help you with cost
Maybe aider? https://aider.chat/2023/10/22/repomap.html
Cool project. I built a custome IDE and coding agent harness and will integrate this into it. If you’re interested in a collaboration, I’d be happy to share revenue to sponsor your open source repo.
https://calendly.com/ryanwmartin/open-office-hours
How does that compare to Cursors’s workspace indexing?
Also curious what the authors think about Claude team explicitly trying out indexing and deciding against it.
The first is hard to test for us unfortunately since we don't use Cursor. But the Claude thing is interesting. I think that providers (especially the ones that directly sell LLM calls like Anthropic) are not incentivised per se to think about token efficiency vs performance, so if you're chasing pure performance, just loading the full codebase into memory might still be the "benchmark topping" way to go. I think the dust hasn't really settled yet and we'll likely see a lot of changes in the coming year about what's the "correct" way to solve it. It might be different based on your harness/budget/model as well.
Seems like a cool idea so I decided to play with it a bit. The test I ran was in the browsercode (https://github.com/browser-use/browsercode) repo with the following prompt:
"Answer this question by only using the `semble` CLI (docs below):
> What tools does Browsercode provide to the agent other than the base OpenCode tools? Provide the exact schema for tool input and tool output and briefly summarize what they do and how they work
---
[the AGENTS.md snippet provided from https://github.com/MinishLab/semble#bash-integration]"
And the equivalent for the non-Semble test:
"Answer this question by only using the `rg` and `fd` CLIs:
> What tools does Browsercode provide to the agent other than the base OpenCode tools? Provide the exact schema for tool input and tool output and briefly summarize what they do and how they work"
In both cases, I used Pi with gpt-5.4 medium and a very minimal setup otherwise. (And yes, I did verify that either instance only used rg & fd, or only used semble.)
Without Semble, it used 10.9% of the model context and used $0.144 of API credits (or, at least, that's what Pi reported - I used this with a Codex sub so cannot be sure). With Semble, it used 9.8% of the model context and $0.172 of API credits. The resulting responses were also about the same. Very close!
I tried one more test in the OpenCode repo. The question was > Trace the path from 1) the OPENCODE_EXPERIMENTAL_EXA env var being set to to 1 to 2) the resulting effects in the system prompt or tool provided to the OpenCode agent.
And I included the same instructions/docs as above. The non-Semble version was a bit more detailed -- it went into whether the tool call path invoked Exa based on whether Exa or Parallel was enabled for the web search provider -- but w.r.t. actually answering the question, both versions were accurate. The Semble version used 14.7% context / $0.282 API cost, while the non-Semble version used 19.0% / $0.352. Clearly a win for Semble for context efficiency, but note that the non-Semble version finished about twice as fast as the Semble version.
Of course this is just me messing around. ymmv.
Wow awesome, thanks for sharing! This is really useful and very much like the experiments we want to be doing in the near future
The instructions on how to install and use it could use some work. I did eventually install it. Will try it later and report back.
Oh sorry that happened. Feel free to open an issue or report it here
Better than grep obviously, but how does this compare to existing LSPs?
Or tools like `ck`: https://beaconbay.github.io/ck/
Try running both on the CK codebase. CK takes like 15 minutes to index itself and gives hundreds of completely irrelevant doc comments as results for “run model on CPU” query. Semble indexes for like 3 seconds and prints out the actual code that runs the model on the CPU.
You didn’t use `ck` directly, you instructed Claude Code to use `ck`, right?
I also like the index feature form https://maki.sh Source code has a lot of structure, using a real parser instead of grepping and reading files can potentially save a lot of tokens
Interesting, will have a look at this, thanks for sharing.
Semantic code search seems like a useful tool for a human too. Not just for agents.
Yeah I agree. I have used semble to quickly index a large monorepo and just ask a question about it, it surfaced the right files pretty quickly. Although without an IDE, it's difficult to display them in nice way
How does this compare with colgrep?
https://github.com/lightonai/next-plaid/tree/main/colgrep
The comparison is in the benchmarks, see the README
fantastic token savings and performance... but unlike grep it's probabilistic search on search terms.
is that an issue? the tiny model might not surface something important
It's not probabilistic, and exact matches will always be preferred over non-exact. So if you search for a function name this will surface it.
Shouldn’t it be a part of the harness at least for local codebase? I wonder how many harnesses are doing that already.
Afaik many harnesses ship the "default" which is grep+read (like Claude Code). But I agree, IMO it's a weird gap. To be fair I don't think providers are that incentivised to reduce token burn at the moment, but my guess is that that will change and tools like this will become at least an natively supported option in some harnesses.
I'm playing with PI as a custom harness ( for Claude code because that what is provided to me )
I will try that ! It make sense and I'm curious to see results, for this or any similar projects mentioned in the thread
allegedly this one is good for PI https://pi.dev/packages/@ff-labs/pi-fff
Would this replace something like codebase-memory-mcp[1] or improve when both is being used?
[1] - https://github.com/DeusData/codebase-memory-mcp
This looks great! I built a tool in the same space- and I found that the biggest challenge was often to get the agent to prefer to use the tool over bash tools. What’s your experience with that?
Setup hooks. Hooks are how your harness forces compliance with your own rules.
How does it compare to context-mode or serina that are both well established now?
Serena does a lot more than semble (I actually used serena before building this and didn't like how much it does by default). That also made it hard to see if it was actually working well with how many moving parts there are. Semble only does 1 thing: very quick code search, that's it. Context-mode I have not used before though, I will have a look at that, thanks for sharing!
Does this support any language or is it limited to a specific set of languages?
For chunking Semble supports all languages supported by tree-sitter-language-pack. The models we train are trained on 6 languages, but can handle way more.
Is the benchmark measuring one-shot retrieval accuracy, or Coding agent response accuracy?
Hey! Co-author here. The benchmark currently only measures retrieval accuracy.
We’re interested in measuring it end to end and also optimizing, e.g. the prompt and tools, for this, but we just haven’t gotten around to it.
Two follow-ups:
1) How do you compare accuracy? by checking if the answer is in any of the returned grep/bm25/semble snippets?
2) How do you measure token use without the agent, prompt, and tools?
1) yes! It’s not accuracy, but ndcg 2) we assume that if the agent gets the correct answer in the returned snippets it does not need to read further
Wouldn't NDCG/token results vary wildly depending on the agent's query and the number of returned items?
e.g. agents often run `grep -m 5 "QUERY"` with different queries, instead of one big grep for all items.
The same holds for semble: the agent can fire off many different semble queries with different k/parameters.
I guess the point we’re trying to make is that you need fewer semble queries to achieve the same outcome, compared to grep+readfile calls.
Does this work well for non-coding documents as well? Say api docs or AI memory files?
Hey, this is something we're actively investigating. We recently added a flag, `--include-text-files`, which, when set, also makes Semble index regular documents (i.e., markdown, text, json). This should also work relatively well.
Congratulations on the release!
Could you add fff to the benchmarks?
We hadn't found that one yet. Will do!
Very useful thanks for sharing!
thanks for sharing!
Nice!
grep doesn't need tokens, so what is 98% fewer than zero?
You need readfile to do something with those tokens. Grep only gives you the matching lines, not the context.
`grep -C $NUM` ? ;)
Even so. Take a look at the NDCG numbers for grep. It's not pretty
ripgrep exists though
The comparison is with ripgrep, see the benchmarks.
very curious to give it a spin but why write a cli in python? would surely be faster and more portable with go or rust?
Perhaps Python is their main language (they seem to be ML peeps, which would make that most likely), which means it's easier for them to do manual reviews even if they're using AI for implementing, etc.
Yes, this is the main reason. We've released some rust stuff in the past, but Python is our main language