It would be cool if you could make this work with Gemini Flash, with keys from AI Studio. I imagine that would expand the set of people who would try it out, because they could use 'free' keys and not worry about unexpected bills.
Cool! Just this week I've been playing around using web chatbots to write userscript. So far, I've just been cutting/pasting code from ViolentMonkey's basic edit window - which is a clunky workflow and a real editor like VSCode would be great. Even though the script is quite short, I've already found I definitely need auto-rollback due to AI regressions. So this sounds very useful!
I use Firefox (on Win) but I guess it might be worth using Chrome just for write/test iteration and then copy over to FF. Looking just now it appears ViolentMonkey on Chrome died in the Manifest V3 apocalypse but is still on Edge.
I'm working on a version of this, https://www.quillmonkey.com/ so you got ahead of me. I imagine there are many versions of this coming. Interesting what set of tools you went with.
Oh that's cool! I've just used wxt to pack extension for firefox and chrome and just used typescript and plain anthropic api. My goal is to make this run fully inside the browser, without any helper binaries, like I've seen with others.
Your project seems pretty close to where mine was a couple weeks ago, where I was focused on a BYOK solution (user-entered Anthropic API key). I saw there was another similar extension already released in the app store (RobotMonkey) which hooks up to their own backend service, and offers subscriptions. For my project, I think that's the right way to go.
It's funny what details about our designs are similar through accident. And what other things are completely different. I can show you my design potentially.
Representing websites in a virtual filesystem is creative and definitely makes it easier for the agent to collect information about the page. But I'm confused between the `Bash` and the `Edit` tools. It seems like one uses the chrome executeScript API, and the other updates the file system. But if it's just doing file writes, are those edits visible in the browser, and persistent across sessions?
Backend service is definitely way to go if you want to serve models for the user.
So Bash and Edit tools are a bit weird, Bash tool is essentially JS execution, and Edit tool automatically generates a script that performs the edits on the page. These tools are needed for the model to explore the page, whatever it does at the end it creates a separate script that will be applied on the page load.
Oh neat. So the edit tool is like a convenient API/wrapper for it to eg add HTML to some element? I guess theoretically that can also be achieved through Bash as well, but the tool fits closer to an interface we know exiting agents are good at.
Awesome! So the agent has access to the DOM/JS running in the browser?
That’s one of my biggest headaches writing user scripts currently: I write the script in an IDE with Claude then copy it to the browser / manually test it in the browser, then copy the results back to Claude or tell it what went wrong.
to my knowledge all the major userscript extensions, at least allow watching for file changes so you don't have to copy it manually, so you can just refresh the page to test
Yup, full access to DOM! Still needs a lot of optimizations, but the trick is that the agent reads the DOM as file, so it can grep parts of it naturally.
I've submitted it to web store, but I'm sure that the review will be very long. The extension requires a lot of permissions, with this kinds of things personally I'll trust more if i can build from source.
It would be cool if you could make this work with Gemini Flash, with keys from AI Studio. I imagine that would expand the set of people who would try it out, because they could use 'free' keys and not worry about unexpected bills.
That's a good point, I'll add support for other models shortly.
Cool! Just this week I've been playing around using web chatbots to write userscript. So far, I've just been cutting/pasting code from ViolentMonkey's basic edit window - which is a clunky workflow and a real editor like VSCode would be great. Even though the script is quite short, I've already found I definitely need auto-rollback due to AI regressions. So this sounds very useful!
I use Firefox (on Win) but I guess it might be worth using Chrome just for write/test iteration and then copy over to FF. Looking just now it appears ViolentMonkey on Chrome died in the Manifest V3 apocalypse but is still on Edge.
I'm working on a version of this, https://www.quillmonkey.com/ so you got ahead of me. I imagine there are many versions of this coming. Interesting what set of tools you went with.
Oh that's cool! I've just used wxt to pack extension for firefox and chrome and just used typescript and plain anthropic api. My goal is to make this run fully inside the browser, without any helper binaries, like I've seen with others.
Your project seems pretty close to where mine was a couple weeks ago, where I was focused on a BYOK solution (user-entered Anthropic API key). I saw there was another similar extension already released in the app store (RobotMonkey) which hooks up to their own backend service, and offers subscriptions. For my project, I think that's the right way to go.
It's funny what details about our designs are similar through accident. And what other things are completely different. I can show you my design potentially.
Representing websites in a virtual filesystem is creative and definitely makes it easier for the agent to collect information about the page. But I'm confused between the `Bash` and the `Edit` tools. It seems like one uses the chrome executeScript API, and the other updates the file system. But if it's just doing file writes, are those edits visible in the browser, and persistent across sessions?
Backend service is definitely way to go if you want to serve models for the user.
So Bash and Edit tools are a bit weird, Bash tool is essentially JS execution, and Edit tool automatically generates a script that performs the edits on the page. These tools are needed for the model to explore the page, whatever it does at the end it creates a separate script that will be applied on the page load.
Oh neat. So the edit tool is like a convenient API/wrapper for it to eg add HTML to some element? I guess theoretically that can also be achieved through Bash as well, but the tool fits closer to an interface we know exiting agents are good at.
Awesome! So the agent has access to the DOM/JS running in the browser?
That’s one of my biggest headaches writing user scripts currently: I write the script in an IDE with Claude then copy it to the browser / manually test it in the browser, then copy the results back to Claude or tell it what went wrong.
Looking forward to trying this.
to my knowledge all the major userscript extensions, at least allow watching for file changes so you don't have to copy it manually, so you can just refresh the page to test
Yup, full access to DOM! Still needs a lot of optimizations, but the trick is that the agent reads the DOM as file, so it can grep parts of it naturally.
<https://web.archive.org/web/20120601101804/https://blog.mozi...>
<https://sites.coecis.cornell.edu/leshed/files/2015/12/coscri...>
<https://ofb.net/~tlau//research/papers/p97-lin.pdf>
<https://github.com/jeffnichols-ibm/coscripter-extension>
Installation should be easier. Why do I need to build the extension or download a release instead of installing it directly from the Chrome Web Store?
I've submitted it to web store, but I'm sure that the review will be very long. The extension requires a lot of permissions, with this kinds of things personally I'll trust more if i can build from source.
Love the decision to edit DOM directly. More LLM tools should carefully consider their training environments instead of treating LLMs like AI Gods.