chiprook
← AI
AIOctober 5, 2026, 17:28

No AI crawler requested llms.txt in 15 days of nginx logs

A developer analyzed 345,808 requests to his site from Aug 28 to Sep 11, 2026. Of 46,155 requests carrying crawler tokens (GPTBot, ClaudeBot, PerplexityBot and others), none requested /llms.txt or /llms-full.txt; all 67 hits on those files came from clients without a crawler token, 53 of them curl.

No AI crawler requested llms.txt in 15 days of nginx logs
#GPTBot#ClaudeBot#PerplexityBot
Read next
AI

OpenAI crawlers started running JavaScript, traffic spiked

Security

Akamai: verified AI crawlers shift to POST requests on retail sites

AI

185-site EU census: who actually blocks AI crawlers

AI

Crawler: 62 AI models retire in 90 days, 48 lack successors