A Third of AI Crawler Requests to a Site Were Fake

A developer grepped nginx logs and found that about a third of requests claiming to be ClaudeBot, GPTBot, PerplexityBot and Applebot came from outside the operators' published IP ranges. Crawler operators publish their CIDR prefixes in JSON files, and checking logs against them takes about twenty lines of code with no dependencies.

• ClaudeBot made 1136 requests, GPTBot 420 and PerplexityBot 356
• About a third of requests came from unlisted IP addresses
• OpenAI and Apple publish crawler CIDR ranges in JSON files
• Verifying logs against these lists takes about 20 lines of code