logoalt Hacker News

mjevans05/15/20251 replyview on HN

Static, Near Static (not generated on demand at least; generated only on real content update), and Login seems likely.

AI not caching things is a real issue. Sites being difficult TO cache / failing the 'wget mirror test' is the other side of the issue.


Replies

grishka05/15/2025

What about AI not respecting robots.txt? I myself have never ran into this, but I've seen complaints of many people who did.

show 1 reply