Cloudflare splits AI training from search indexing with new crawler controls

Cloudflare introduced separate policies for search crawlers, AI training and AI agents, replacing its single Block AI Bots switch. The Disallow AI Training setting publishes a no-training directive in robots.txt while keeping search indexing allowed; Apple, Google and Microsoft committed to honor it.

• Block AI Bots replaced by three categories: Search, Training, Agent
• Disallow AI Training writes a no-training directive into robots.txt
• Bot Preference Sync keeps robots.txt aligned with bot policies
• Apple, Google and Microsoft pledged to honor the directive