For some reason it seems really important to these AI companies to get the very latest version of your pages as well, so they'll do anything in their power to avoid hitting any caching you may try to set up.