Study: 22.6% of 261 top sites block GPTBot

An analysis of robots.txt files from 285 popular websites (261 readable) found that 22.6% block GPTBot, 35% block at least one training crawler and 21% block a search crawler. CCBot and Bytespider are the most blocked agents at 28.7% each, while OAI-SearchBot is the least blocked at 11.5%. News sites account for nearly half of all GPTBot blocks.

• 59 of 261 sites (22.6%) disallow OpenAI's GPTBot
• CCBot and Bytespider are the most blocked at 28.7% each
• OAI-SearchBot is the least blocked agent at 11.5%
• 27 of 48 news sites block GPTBot