They say they obey robots.txt - isn’t that the easier way?