Why would an AI crawler do this?

> Attackers found and deliberately targeted URLs that resulted in cache misses, such as non-existent pages with unique paths (404s) as well as temporary redirects (302s)