Which AI Crawlers Should You Allow, and Which Can You Block?
Split the list in two before you decide anything, because AI crawlers do two different jobs and the cost of blocking them is not the same. ANSWER crawlers feed a surface that links back to you: OAI-SearchBot and ChatGPT-User for ChatGPT, PerplexityBot and Perplexity-User for Perplexity, Claude-User and Claude-Web for Claude. Blocking one of those removes you from that answer surface, so it is a straight loss of visibility and should be treated as a failure, not a preference. TRAINING crawlers collect pages to train a model: GPTBot, ClaudeBot, anthropic-ai, CCBot, Google-Extended, Applebot-Extended, Bytespider and meta-externalagent. Refusing those is a legitimate commercial choice with no direct citation cost, so it is a decision, never a defect. The expensive mistake is blocking by brand. Teams block GPTBot to keep their content out of training and assume ChatGPT is now handled, while OAI-SearchBot, the crawler that actually feeds the citations, was never the one they blocked, or worse, was caught by a broad disallow they forgot was there. Google-Extended is the same trap in reverse: it controls Gemini training and grounding only and has no effect on Google Search, which Googlebot handles.
AI crawlers do two different jobs, and the cost of blocking them is not the same, so the list has to be split before any decision is made. Answer crawlers feed a surface that links back to you, and blocking one removes you from that surface. Training crawlers collect pages to train a model, and refusing them has no direct citation cost. Treating the two as one list is how sites quietly remove themselves from AI answers while believing they made a data policy decision. This article gives the full split, the tokens in each half, and the specific brand-level mistakes that cause it.
The Split: Answer Crawlers and Training Crawlers
An answer crawler fetches or indexes pages so an AI product can cite and link them in an answer. A training crawler collects pages to be used as training data for a model. The same company usually operates one of each, under different user-agent tokens, and they must be decided on separately.
Every robots.txt decision about AI comes down to which half of this split a token sits in. Get the half wrong and you have made the opposite of the decision you intended.
Blocking an answer crawler removes you from a surface where people are actively asking your category's questions, and where the citation itself is often the conversion event, because the reader may never click through. That is a loss of visibility with no offsetting benefit.
Refusing a training crawler is a legitimate commercial position. Your content is not used to train a model that competes with you, and your presence in AI answers is unaffected. Reasonable organisations land on both sides of it.
In our own tooling these two are never reported as a single number, because they are two different business decisions. A blocked answer crawler scores as a failure. A refused training crawler scores as a warning, never a failure. One number covering both would tell you nothing you could act on.
Why the Two Are Scored Differently
Most AI readiness scores in the market collapse the crawler question into one figure: how many AI bots can reach you. That figure moves when you refuse a training crawler, which is not a defect, and it moves by the same amount when you block a citation crawler, which is a serious loss. A score that treats a policy choice and a visibility loss as equivalent is not a diagnostic.
So the rule we hold every crawler check to: report the two halves separately, name the surface that is lost in plain language, and never present a deliberate policy choice as a fault. The output a business owner needs is not "you are blocking 6 of 14 bots". It is "you are absent from Perplexity answers, and you have declined to train two models".
| Crawler kind | What blocking it costs | How it should be reported |
|---|---|---|
| Answer | Removal from an AI answer surface that links back to you. Direct visibility loss. | A failure. It needs fixing, and the surface lost should be named. |
| Training | Your pages are not used as training data. No effect on citations. | A warning at most. It is a commercial decision, never a defect. |
The Answer Crawlers, and What Blocking Each One Costs
These are the tokens that feed answer surfaces. Each row states the surface you lose if the token is disallowed.
| User-agent token | Surface you lose | What it does |
|---|---|---|
| OAI-SearchBot | ChatGPT search citations | Builds OpenAI's search index, the source list ChatGPT cites and links when it answers. |
| ChatGPT-User | Live page fetches inside ChatGPT | Fetches a page in real time when someone opens or shares your link inside ChatGPT. |
| PerplexityBot | Perplexity answers and citations | Indexes pages so Perplexity can quote them and link back in its answers. |
| Perplexity-User | Live page fetches inside Perplexity | Fetches a page in real time when a Perplexity user follows or shares your link. |
| Claude-User | Live page fetches inside Claude | Fetches a page in real time when a Claude user asks about your link. |
| Claude-Web | Claude browsing | The browsing agent behind Claude. Keep its rule identical to Claude-User so the two never disagree. |
Note the pairs. OAI-SearchBot and ChatGPT-User are both ChatGPT. PerplexityBot and Perplexity-User are both Perplexity. Claude-User and Claude-Web are both Claude. Allowing one and blocking the other produces a half-present state that is harder to diagnose than a clean block.
The Training Crawlers, and What Refusing Them Costs
These collect pages for model training. Refusing them is a policy decision with no direct citation cost. Make it deliberately, write down why, and keep the file consistent with the decision.
| User-agent token | What it feeds | Worth knowing |
|---|---|---|
| GPTBot | ChatGPT model training | Collects pages to train OpenAI models. It is not the crawler behind ChatGPT citations. |
| ClaudeBot | Claude model training | Anthropic's training crawler. Separate from Claude-User and Claude-Web. |
| anthropic-ai | Legacy Anthropic access | An older token still listed in many files. Keep its rule identical to ClaudeBot. |
| CCBot | The Common Crawl dataset | Common Crawl, the open dataset many labs train on. Blocking it removes you from several models at once. |
| Google-Extended | Gemini training and grounding | Controls Gemini training and grounding only. It has no effect on Google Search, which Googlebot handles. |
| Applebot-Extended | Apple Intelligence training | Controls whether Apple uses your pages for Apple Intelligence. Applebot, for Siri and Spotlight, is a separate token. |
| Bytespider | ByteDance AI products | ByteDance crawler feeding its AI products. Known for heavy request volume on large sites. |
| meta-externalagent | Meta AI and Llama training | Meta's AI crawler, collecting pages for Meta AI and Llama training. |
The Brand-Level Mistake That Does the Damage
The expensive error is not a wrong opinion about AI training. It is deciding at the brand level when the tokens are split by function.
The pattern runs like this. A team decides it does not want its content training OpenAI's models. Someone adds a disallow for GPTBot. The decision is recorded as "we have handled ChatGPT". But GPTBot is the training crawler. OAI-SearchBot is the one that builds the index ChatGPT cites from, and ChatGPT-User is the one that fetches a link a user pastes in. Neither was touched, so the training decision worked and nothing else changed. That case is the harmless version.
The harmful version is the same reasoning applied through a broad rule. A team writes a blanket disallow to keep AI bots out, or inherits a User-agent: * disallow on a section, and the answer crawlers are caught by it alongside the training crawlers. The site now cannot be cited by ChatGPT, Perplexity or Claude, and nothing in analytics reports the loss, because the traffic that never arrived leaves no trace.
Decide per token, never per brand. "We are fine with ChatGPT" and "we are not fine with ChatGPT" are both incoherent instructions, because ChatGPT has three tokens doing two different jobs.
The Google-Extended Trap, in Reverse
Google-Extended produces the same confusion pointing the other way. It is a training and grounding control for Gemini, and it has no effect at all on Google Search. Googlebot handles Search, and it is a separate token with separate rules.
Two mistakes follow. The first is blocking Google-Extended and worrying that organic rankings will fall, which they will not, because Search does not consult it. The second is allowing Google-Extended in the belief that it helps Google Search rankings, which it does not either.
The reverse also holds and is worth stating: a broad disallow written to keep AI crawlers out will catch Googlebot if it is written carelessly, and that does affect Search. Read the file, do not assume it.
Google-Extended controls Gemini training and grounding. Googlebot controls Google Search. Changing one has no effect on the other.
How to Check Your Own File
Reading robots.txt by eye is where this goes wrong, because the rule that applies to a given token is not always the rule with that token's name on it. A broad User-agent: * block can be the effective rule for a bot that is never mentioned in the file, and a specific Allow can be overridden by ordering and specificity in ways that are easy to misread.
The two checks, in order:
- 1.Fetch the file and read it in full: curl -s https://example.com/robots.txt. Look for a User-agent: * block with a Disallow that would catch tokens you never named.
- 2.Resolve the effective rule per token, rather than assuming the named rule wins. For each of the fourteen tokens above, work out which line actually matches it and whether the result is allow or disallow.
- 3.Write the decision down. Which answer crawlers are allowed, which training crawlers are refused, and why. A file nobody can explain gets edited into a worse state within a quarter.
- 4.Re-check after any edge, CDN or security-vendor change. Bot-management rules at the edge can block a crawler that robots.txt allows, and robots.txt will not tell you.
Step two is what our free AI Crawler Checker does: it resolves the effective robots.txt rule for each of these tokens against your live file and shows you the exact rule line that matched, so you are reading the outcome rather than guessing at it.
- Split the list before deciding. Answer crawlers generate citations, training crawlers only feed models, and the cost of blocking them is not comparable
- Blocking an answer crawler is a visibility failure. Refusing a training crawler is a commercial choice and should never be reported as a defect
- GPTBot is training. OAI-SearchBot and ChatGPT-User are the crawlers behind ChatGPT citations
- Google-Extended controls Gemini training and grounding only. Googlebot controls Google Search, and neither affects the other
- Decide per token, never per brand. Each major AI product runs several tokens doing different jobs
- The silent failure is a broad User-agent: * disallow catching answer crawlers that were never named in the file
- Edge and bot-management rules can block a crawler that robots.txt allows, and robots.txt will not tell you
The crawler list, the answer and training classification, and the surface descriptions are taken from the single crawler registry used across Elshorafa Tools, so this article and the product cannot disagree about what a token does. The scoring rule, that a blocked answer crawler is a failure and a refused training crawler is a warning, is the rule that registry encodes. No traffic figure or citation count is claimed here, because the effect of a crawler block is a loss that leaves no trace in analytics and cannot be measured after the fact.
