20.9% of top 1,000 sites block at least one AI crawler

A robots.txt study of the 1,000 most popular sites (Tranco, October 2026) found that 20.9% block at least one of 16 AI bots, while 78.9% allow all of them. Training crawlers are blocked most often: CCBot on 15.5% of sites, GPTBot on 13.8%, while AI search bots are blocked less (OAI-SearchBot on 7.0%).

• 209 of 1,000 sites (20.9%) block at least one AI bot site-wide
• CCBot is blocked by 15.5% of sites, GPTBot by 13.8%, OAI-SearchBot by 7.0%
• 27% of top-100 sites block AI bots versus 17.4% of sites ranked 501–1,000
• 109 sites still list the retired anthropic-ai name and 49 serve a web page instead of robots.txt