No AI crawler read llms.txt on a site in 30 days
A developer measured AI crawler traffic to his site via Cloudflare AI Crawl Control over 30 days: robots.txt was fetched 605 times and the post sitemap 192 times, while llms.txt and llms-full.txt got zero requests. He kept the files because they are generated automatically at no cost.
- robots.txt fetched 605 times, post sitemap 192, llms.txt zero
- ClaudeBot made 128 of the 192 sitemap requests
- AI crawlers made about 3,000 requests in 30 days
- llms.txt was proposed in September 2024 for on-demand reading, not crawling
Read next
AI