I would also suggest requiring adherence to `robots.txt` and adding an option (preferably a default one) to force user acknowledgement of external HTTP requests. Claude Code has a tendency to treat unauthenticated access as an obstacle to work around instead of a sign to stop.[1] An experimental Anubis patch (since removed) took advantage of this behavior to create a honeypot for user data if a Claude user agent is detected.[2] I'm sure more malicious honeypots could be created if a hostile website and/or annoyed sysadmin wanted to do so.

[1] https://blog.xkeeper.net/the-cutting-room-floor/self-hosting...

[2] https://github.com/TecharoHQ/anubis/pull/1895/changes/de2312...