I can't comment on LLMs, but I vaguely remember speculations of it being a mixture of AI training and trying to keep resource utilization at bay. Even generic bots can cause a lot load if you don't have much caching in front of it.

Pre-MS GitHub didn't have code search at all, and once they added it, I think it always required login from the start.