Warden is interesting. Agents will curl websites, and not humans so returning a markdown structure of that page seems like the best.

Simlar to how everyone started optimising their pages for SEO, pages must be optimised for agents too, and its just simply detecting curl requests and returning structured files with internal linking to other pages.

It should basically be able to navigate the website like a file system. curl to the home page returns basic content with the basic sitemap structure, sitemaps ideally could have a description and token length of that specific page so agents can hit the sitemap route and know all pages/sub pages of the website.

Ideally if we can identify headless requests to a website to then return a markdown, with internal linking kind of layout then that'll be much better for agents to view websites.

Although yes there is firecrawl and cloudflare's new fetch apis, models will default to using curl or fetch on websites, and websites are not going anywhere, agents might need navigating websites more than us humans so it oculd be optimised for it.