alt Hacker News> There are already “infinite” websites like these on the Internet.
Cool. And how much of the software driving these websites is FOSS and I can download and run it for my own (popular enough to be crawled more than daily by multiple scrapers) website?
Replies
hartator • 01/16/2025
Every not found pages that don’t return a 404 http header is basically an infinite trap.
It’s useless to do this though as all crawlers have a way to handle this. It’s very crawler 101.
Off the top of my head: https://everyuuid.com/
https://github.com/nolenroyalty/every-uuid