23 of 248 Japanese sites allow AI crawlers in robots.txt but return 403 at the CDN

A researcher checked 300 Japanese sites from the Tranco list on October 7, 2026: of 248 sites reachable by a browser, 23 returned 403 to GPTBot, ClaudeBot, PerplexityBot and OAI-SearchBot even though robots.txt allows them. In 14 cases all four agents were blocked.

• 300 sites tested, 248 loaded in a browser; 23 showed the mismatch
• GPTBot blocked on 22 sites, ClaudeBot on 19, PerplexityBot on 16
• On 9 sites only AI agents are blocked while Googlebot and a normal client pass
• Blocks appear on Akamai, CloudFront and Cloudflare, with no single CDN dominating