alt Hacker NewsShow HN: Index – New Open Source browser agent
Hey HN, Robert from Laminar (lmnr.ai) here.
We built Index - new SOTA Open Source browser agent.
It reached 92% on WebVoyager with Claude 3.7 (extended thinking). o1 was used as a judge, also we manually double checked the judge.
At the core is same old idea - run simple JS script in the browser to identify interactable elements -> draw bounding boxes around them on a screenshot of a browser window -> feed it to the LLM.
What made Index so good:
1. We essentially created browser agent observability. We patched Playwright to record the entire browser session while the agent operates, simultaneously tracing all agent steps and LLM calls. Then we synchronized everything in the UI, creating an unparalleled debugging experience. This allowed us to pinpoint exactly where the agent fails by seeing what it "sees" in session replay alongside execution traces.
2. Our detection script is simple but extremely good. It's carefully crafted via trial and error. We also employed CV and OCR.
3. Agent is very simple, literally just a while loop. All power comes from carefully crafted prompt and ton of eval runs.
Index is a simple python package. It also comes with a beautiful CLI.
pip install lmnr-index
playwright install chromium
index run
We've recently added o4-mini, Gemini 2.5 Pro and Flash. Pro is extremely good and fast. Give it a try via CLI.
You can also use index via serverless API. (https://docs.lmnr.ai/index-agent/api/getting-started)
Or via chat UI - https://lmnr.ai/chat.
To learn more about browser agent observability and evals check out open-source repo (https://github.com/lmnr-ai/lmnr) and our docs (https://docs.lmnr.ai/tracing/browser-agent-observability).
Comments
How well does it work with bank websites with non-conventional multi-click logins that sometimes include an "important message" that you have to click through just to get your balance?
Can it actually do something difficult like apply for jobs? So far I know of five or so websites that claim they can apply to jobs for you like sonara.ai and usemassive.com and Skyvern AI but when you try to actually use them all they can do is the one-page job applications and not the much more common Workday 10-page job applications with annoying "create an account" and annoying questions like "Do you have any relatives that work at Sony" and annoying "fill out all your work experience" where you have to click 50 times for one application. That's like half of all job applications. https://jobs.spectrum.com/job/-/-/4673/76746020384?utm_sourc...
Nice.
Can run with `uvx --from lmnr-index --python 3.12 index run`
> Index is the SOTA open-source browser agent for autonomously executing complex tasks on the web.
I've written a handful of pretty hacky Python scripts that just pull down all of the HTML content from a page and toss it over to OpenAI. As you can imagine, these were all extremely simple tasks, e.g., "find out if there's a login button"
What's a good example of a complex task that Index is well-suited for? What's the threshold of minimal complexity where you guys are a really good fit?
My first reaction was to look for MCP server so that I could connect it to Cursor. Just pointing this out in case it helps with new user onboarding. MCP server would work to hook it up to the Claude Desktop Website and most agentic-IDEs (Cursor, Cline, Roo, Windsurf, etc).
What’s the most surprising or complex real-world task you’ve seen it succeed at so far?
Can you open up the options to use other model/versions, especially Gemini-2.5 pro experimental models available through aistudio? Would love to try this but gemini flash fails for even simple tasks. Example: I asked it to extract all the links from comment section of a hackernews comment section and it just scrolled all the way to the end and then nothing. Maybe pro models can do it better.
Impressive and potentially very interesting future work.
One thing I couldn't help but notice was the crazy amount of HTTP requests going on in the demo on the github readme page, and the video looks to be sped up.
I'm all for AI assisting but I wouldn't want to create even 1/10th of these HTTP requests, as a good netizen; unless I'm missing the point.
[dead]
I feel like I see a new company doing this every week. I know there is Skyvern and browser-use in particular. Is there something special about this one?