I’ve been experimenting with embedding an Claude Code/Cursor-style coding agent directly into the browser.
At a high level, the agent generates and maintains userscripts and CSS that are re-applied on page load. Rather than just editing DOM via JS in console the agent is treating the page, and the DOM as a file.
The models are often trained in RL sandboxes with full access to the filesystem and bash, so they are really good at using it. So to make the agent behave well, I've simulated this environment.
The whole state of a page and scripts is implemented as a virtual filesystem hacked on top of browser.local storage. URL is mapped to directories, and the agent starts inside this directory. It has the tools to read/edit files, grep around and a fake bash command that is just used for running scripts and executing JS code.
I've tested only with Opus 4.5 so far, and it works pretty reliably. The state of the file system can be synced to the real filesystem, although because Firefox doesn't support Filesystem API, you need to manually import the fs contents first.
This agent is really useful for extracting things to CSV, but it's also can be used for fun.
I'm working on a version of this, https://www.quillmonkey.com/ so you got ahead of me. I imagine there are many versions of this coming. Interesting what set of tools you went with.
Awesome! So the agent has access to the DOM/JS running in the browser?
That’s one of my biggest headaches writing user scripts currently: I write the script in an IDE with Claude then copy it to the browser / manually test it in the browser, then copy the results back to Claude or tell it what went wrong.
Looking forward to trying this.
It would be cool if you could make this work with Gemini Flash, with keys from AI Studio. I imagine that would expand the set of people who would try it out, because they could use 'free' keys and not worry about unexpected bills.