740 top websites checked for AI bots: 15% block GPTBot

A scan of 740 sites from Tranco's top 1,500 found 15% block GPTBot in robots.txt, 14% block ClaudeBot and 13% block Google-Extended. Of 109 sites blocking GPTBot, 69 (63%) also block ChatGPT's search agent, including CNN, BBC, Reddit and Amazon.

• 15% of 740 sites block GPTBot, 14% ClaudeBot, 13% Google-Extended
• 69 of 109 GPTBot-blocking sites also block ChatGPT search
• 9% reject AI bot-named requests at the server despite robots.txt
• 14% publish llms.txt, 45% use structured data, 38% lack a sitemap