No AI crawler requested llms.txt in 15 days of nginx logs
A developer analyzed 345,808 requests to his site from Aug 28 to Sep 11, 2026. Of 46,155 requests carrying crawler tokens (GPTBot, ClaudeBot, PerplexityBot and others), none requested /llms.txt or /llms-full.txt; all 67 hits on those files came from clients without a crawler token, 53 of them curl.
- 345,808 requests over 15 days, 46,155 with crawler tokens
- 20 distinct crawler tokens, zero requests for llms.txt
- All 67 llms.txt hits lacked a crawler token, 53 were curl
- 698 robots.txt and 282 sitemap.xml requests on Sep 9-11
Read next
AI