185-site EU census: who actually blocks AI crawlers
A Pennyforge study dated October 1, 2026 measured AI-crawler blocking across 185 EU sites. Hard opt-outs exist on 35 sites (18.9%), all full Disallow: /; media block most often (66.7%), government portals least (4.2%).
- 121 of 185 sites (65.4%) serve robots.txt, and 77 of those have no rules for any of the 12 AI crawlers
- 35 sites (18.9%) fully disallow AI crawlers; no partial path-level opt-outs were found
- CCBot is blocked by 26 sites, the most of any crawler, with zero explicit allows
- MistralAI-SearchBot has no rule on any of the 185 sites, and no site serves a .well-known/ai-crawler file
Read next
Security