Previously, I had done a fair amount of research into how Google's monopoly on web crawling further entrenches their monopoly in the search engine market. You can read more about this here, https://knuckleheads.club, there is a long report from ~2020 or so that explains how it worked at the time. The club is mothballed, I am doing other things with my life, and I'm happy to say that we played a very small role in the DOJ ordering Google to share their crawl data with qualified competitors (a work in progress, but it's progressing).

Chatbots have super charged this dynamic though, to the point that it is showing up in the robots.txt data. The last few weeks I've been having Claude rerun some old analysis of Common Crawl from back then, when I have spare usage and time. What I've found is that you can see pretty clearly the rise in people outright blocking AI chatbot related crawlers likely because of how aggressive they have become.

  Quarter  Crawl    GPT     Claude  CC      G-Ext   Byte    Bing    Google
  2023 Q1  2023-06  0.00%   0.00%   0.16%   0.00%   0.06%   0.47%   0.39%
  2023 Q2  2023-14  0.00%   0.00%   0.18%   0.00%   0.06%   0.45%   0.38%
  2023 Q3  none       —       —       —       —       —       —       —
  2023 Q4  2023-40  2.21%   0.00%   2.12%   0.04%   0.11%   0.39%   0.27%
  2024 Q1  2024-10  0.53%   0.05%   0.31%   0.09%   0.18%   0.34%   0.31%
  2024 Q2  2024-18  0.55%   0.09%   0.32%   0.11%   0.24%   0.32%   0.31%
  2024 Q3  2024-30  0.68%   0.22%   0.36%   0.20%   0.38%   0.24%   0.33%
  2024 Q4  2024-42  1.10%   0.50%   0.44%   0.32%   0.50%   0.25%   0.40%
  2025 Q1  2025-05  1.14%   0.66%   0.54%   0.42%   0.66%   0.25%   0.44%
  2025 Q2  2025-18  1.37%   0.93%   0.63%   0.70%   0.92%   0.29%   0.19%
  2025 Q3  2025-30  1.42%   1.07%   0.74%   0.62%   1.01%   0.31%   0.27%
  2025 Q4  2025-43  1.92%   1.51%   1.23%   1.15%   1.52%   0.27%   0.19%
  2026 Q1  2026-04  2.13%   1.76%   1.68%   1.58%   1.77%   0.22%   0.15%
  2026 Q2  2026-17  2.80%   2.38%   2.26%   2.13%   2.50%   0.22%   0.14%
  2026 Q3  2026-30  3.45%   3.01%   2.89%   2.71%   3.16%   0.21%   0.14%

GPTBot is OpenAI, ClaudeBot is Anthropic, CCBot is Common Crawl, Google-Ext is a way for website owners to indicate they don't want their content to be used for AI, Bytespider is Bytedance, Bing and Google are the last two. Take these numbers with a truck of salt, haven't had time to verify them.

It's very clear that website owners do not like getting their content scraped and are indicating to GPTBot et al. that they are not welcome. It's a shame that CCBot is caught in the cross fire, but that's life. Bing and Google are doing just fine though, almost like having significant power in the search engine market gives you an advantage in other markets too. Who knew!