Simply putting up a basic auth wall that says “Enter any password to proceed” would stop all modern crawlers dead in their tracks, afaik. You could make it more defensible to the trivial overcome by putting a rotating / per-source password in the basicauth message, but honestly, I think they’re all coded not to invite a CFAA hacking lawsuit by trying random passwords on password-protected sites :)