Security1 hOctober 1, 2026, 01:48

Cloudflare may be blocking Googlebot on ad-supported pages since Sept 15

On 2026-09-15 Cloudflare began defaulting new domains and free-tier accounts to disallow AI Training crawlers and block AI Agents on ad-bearing pages. Because Googlebot is classified as mixed-use, it now gets 403 on pages and sitemap fetches even though robots.txt still says Allow.

DEV Community

Open source

Cloudflare may be blocking Googlebot on ad-supported pages since Sept 15

Since Sept 15, 2026 Cloudflare defaults new domains and free-tier accounts to block AI crawlers
Googlebot, Bingbot and Applebot receive 403 on ad-supported pages and sitemap fetches
Googlebot's crawl share on Cloudflare fell from 57.20% to 27.49% year over year
Check: curl as Googlebot returns 403 while a browser request returns 200

#Cloudflare#Googlebot#Google

Elsevier domains briefly redirected to LAPSUS$ "Chapter II" page

Security

Calendar-based phishing jumps 33,000% since May, researchers say

Cloudflare Lets Sites Ask Google to Skip AI Training Without Blocking Search

Policy

Cloudflare may be blocking Googlebot on ad-supported pages since Sept 15

Elsevier domains briefly redirected to LAPSUS$ "Chapter II" page

Calendar-based phishing jumps 33,000% since May, researchers say

Cloudflare Lets Sites Ask Google to Skip AI Training Without Blocking Search

Court upholds Belgian pirate DNS blocking order, OpenDNS exit looms