From the start, robots.txt has always been an indicator of a site's preference with no actual legal significance.