Which AI Crawlers Should You Allow, and Which Can You Block?

Split the list in two before you decide anything, because AI crawlers do two different jobs and the cost of blocking them is not the same. ANSWER crawlers feed a surface that links back to you: OAI-SearchBot and ChatGPT-User for ChatGPT, PerplexityBot and Perplexity-User for Perplexity, Claude-User and Claude-Web for Claude. Blocking one of those removes you from that answer surface, so it is a straight loss of visibility and should be treated as a failure, not a preference. TRAINING crawlers collect pages to train a model: GPTBot, ClaudeBot, anthropic-ai, CCBot, Google-Extended, Applebot-Extended, Bytespider and meta-externalagent. Refusing those is a legitimate commercial choice with no direct citation cost, so it is a decision, never a defect. The expensive mistake is blocking by brand. Teams block GPTBot to keep their content out of training and assume ChatGPT is now handled, while OAI-SearchBot, the crawler that actually feeds the citations, was never the one they blocked, or worse, was caught by a broad disallow they forgot was there. Google-Extended is the same trap in reverse: it controls Gemini training and grounding only and has no effect on Google Search, which Googlebot handles.