The real problem is most of them just pretend to be normal users now robots.txt only works if they choose to care.