chiprook
← AI
AIOctober 17, 2026, 23:18

No AI crawler read llms.txt on a site in 30 days

A developer measured AI crawler traffic to his site via Cloudflare AI Crawl Control over 30 days: robots.txt was fetched 605 times and the post sitemap 192 times, while llms.txt and llms-full.txt got zero requests. He kept the files because they are generated automatically at no cost.

No AI crawler read llms.txt on a site in 30 days
#Cloudflare#ClaudeBot#GPTBot#Googlebot
Read next
AI

23 of 248 Japanese sites allow AI crawlers in robots.txt but return 403 at the CDN

AI

No AI crawler requested llms.txt in 15 days of nginx logs

Software

Cloudflare splits AI crawler controls into Search, Agent and Training

AI

OpenAI crawlers started running JavaScript, traffic spiked