AI Crawler Access Checker, See Which AI Bots Can Actually Read Your Site
Enter a domain and run the AI crawler test. The tool resolves the effective robots.txt directive for 17 agents, shows the exact rule line behind every verdict, and checks whether the site publishes llms.txt.
Enter a domain above to resolve the effective robots.txt rule for 17 AI crawlers.
Which AI Crawlers Have Access to My Site?
Unless you have written rules that name them, all of them can. robots.txt works on permission by default: if no group in your file matches a crawler and you have no wildcard group, that crawler is allowed. A missing file and a clean 404 resolve the same way, everything is permitted. So the real question is never whether you have a robots.txt, it is which named groups exist in it and which agent each group actually captures.
A failed fetch is a different thing entirely, and this tool refuses to treat it as an answer. If the file comes back as a 403, a 429, a 500, or the connection is reset or times out, we never saw it, so we report Could not verify rather than a verdict. RFC 9309 in fact tells crawlers to assume a complete disallow while a site returns 5xx, which is the opposite of what a permissive default would suggest. Sites behind Cloudflare or Akamai land here often, and that finding matters on its own: most AI crawlers are non-browser clients too.
That is harder to eyeball than it looks. Only one group ever applies to a given crawler, the most specific user-agent match, and it replaces the wildcard group rather than stacking on top of it. A site withUser-agent: * andDisallow: / that later adds an emptyUser-agent: GPTBot group has just granted GPTBot full access to the entire site. Within the winning group, the longest matching path pattern decides the outcome, and on a tie in length Allow beats Disallow. This tool applies all of that and prints the rule line it used, so you can check the working rather than trust the badge.
Should I Block AI Crawlers?
There is no single right answer, and anyone selling you one is selling something. The useful move is to stop treating it as one decision. There are three, and they have different costs.
GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, meta-externalagent, CCBot, Bytespider, cohere-ai
Blocking keeps your content out of future model training. It has no effect on whether you appear in AI answers today.
OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, bingbot, Amazonbot
Blocking removes you from the citations AI assistants show. This is the group that costs you visibility.
ChatGPT-User, Claude-User, Perplexity-User
Blocking means the assistant cannot open your page even when a user pastes the link and asks about it.
Blocking training crawlers costs you nothing in visibility today. It also gains you nothing in the short term. It is a position on whether your work should train a commercial model you do not control and receive no attribution from. Publishers with licensing leverage and original research take that position often. Sites whose growth depends on being found rarely do.
Blocking search and live-fetch agents is the expensive one. AI assistants are becoming a discovery surface, and a blocked page cannot be cited. For most commercial sites the defensible position is to allow the search and live-fetch agents, then decide on training separately and deliberately.
What Is the Difference Between GPTBot and OAI-SearchBot?
They are different agents doing different jobs for the same company, and blocking the wrong one is the most expensive mistake on this page.
Collects pages that may train future OpenAI models. Blocking it removes you from the training corpus and changes nothing about how ChatGPT answers today.
Builds the index behind ChatGPT search. Blocking it removes you from the results ChatGPT shows and cites. This is the visibility cost.
Opens one page in real time because a user asked. Blocking it means a user who pastes your link gets told the page cannot be read.
The same split runs through the other operators. Anthropic separates ClaudeBot, Claude-SearchBot, and Claude-User. Perplexity separates PerplexityBot and Perplexity-User. Google separates Google-Extended, which is a training control only and never affects ranking, from Googlebot, which is your entire presence in Google Search and in AI Overviews. Blanket-blocking a company's name across all its tokens is how sites quietly delete themselves from AI search while believing they only opted out of training.
What Is the Difference Between llms.txt and robots.txt?
One is an access control file that crawlers already obey. The other is a content curation file that most of them do not read yet. They are not alternatives, and only one of them can cost you visibility if you get it wrong.
| Aspect | robots.txt | llms.txt |
|---|---|---|
| What it is | An access control file. Standardised as RFC 9309 in 2022, and in use since 1994. | A content curation file. A community proposal published at llmstxt.org in September 2024, not a ratified standard. |
| What it controls | Whether a named agent may fetch a given path on your domain. | Which pages you point a model at, and what you say each one covers. It grants no access and blocks none. |
| Who obeys it | OpenAI, Anthropic, Google, and Perplexity publish their agent tokens and state that they honour it. Operators with no such commitment can ignore it. | No major AI operator has confirmed it reads the file at crawl time, and Google has stated publicly that it does not use llms.txt. Adoption sits mainly with developer documentation platforms. |
| Format | User-agent groups with Allow and Disallow lines. One group wins per crawler, the most specific match. | Markdown. An H1 site name, a blockquote summary, then H2 sections of curated links with a note on each. |
| Cost of getting it wrong | High. One broad Disallow line can remove you from AI search and from Google. | Low. A missing file costs nothing measurable today, and a stale one only misleads the tools that read it. |
The practical order follows from that table. robots.txt is the file that decides whether an agent may fetch your pages at all, so a single careless Disallow line there removes you from AI search entirely. llms.txt grants nothing and blocks nothing. It is a curated map you hand to a model that has already been let in, and a model that was never let in will never see it.
Fix crawl access first, then publish llms.txt as a cheap bet on an emerging convention. The checker above reports both, so you can see the effective robots.txt verdict per agent and whether llms.txt and llms-full.txt exist in the same pass. When you are ready to write the file, the llms.txt generator builds it in the correct format.
How Do I Test AI Crawler Access on My Site?
- 01Enter your domain above. The tool fetches your live robots.txt and resolves the effective rule for each AI agent.
- 02Read the matched rule line under each verdict. Every result shows the exact line and line number that produced it, so you can audit the finding against your own file.
- 03Decide by purpose. Training, search index, and live fetch are three separate decisions, not one.
- 04Tick the agents you want to allow in the builder. The selection opens at your current position, so it changes nothing until you change it.
- 05Copy the generated file, publish it at the root of your domain, then run the check again to confirm the result.
The generated file repeats your existing disallow rules inside every named agent group. That is deliberate. Once a crawler has a group of its own it stops reading the wildcard group entirely, so an agent group containing onlyAllow: / would hand that agent the admin paths you had closed. If schema, crawl control, and citation readiness need to be built together, that is what our SEO and GEO service covers.
Frequently Asked Questions
What is an AI crawler?
An AI crawler is an automated agent that fetches web pages on behalf of an AI company. Some collect pages to train models, such as GPTBot and ClaudeBot. Some build a search index that grounds AI answers, such as OAI-SearchBot and PerplexityBot. Some fetch a single page live when a user asks about it, such as ChatGPT-User and Claude-User. Each type has a different consequence when you block it.
Does blocking GPTBot remove my content from ChatGPT?
No. GPTBot only controls whether your content can be used to train future OpenAI models. ChatGPT search results are built by a separate agent, OAI-SearchBot, and live page fetches use ChatGPT-User. Blocking GPTBot keeps you out of the training data while leaving you visible in ChatGPT search, provided the other two agents are allowed.
Why does this tool say a crawler is allowed when robots.txt has no rule for it?
Because that is what the standard says. Under RFC 9309, the absence of a rule is permission. If no group in your robots.txt names a crawler and there is no wildcard group, that crawler is allowed. Reporting it as blocked would be wrong, so this tool reports allowed and tells you no group matched.
Is robots.txt legally binding?
No. robots.txt is a voluntary protocol. The major operators including OpenAI, Anthropic, Google, and Perplexity publish their tokens and state that they honour it, and there is reputational cost to being caught ignoring it. Operators with no such commitment can and do ignore it. If you need enforcement rather than a request, you need server-level blocking by user agent or IP, or a web application firewall rule.
Will blocking AI crawlers hurt my Google rankings?
Blocking Google-Extended does not affect Google Search rankings. It only controls whether your content is used to train Gemini and ground Vertex AI. Blocking Googlebot is a different matter entirely and will remove you from Google Search, including AI Overviews. Never block Googlebot unless you intend to leave Google Search.
How long does a robots.txt change take to take effect?
Most crawlers cache robots.txt for up to 24 hours, and Google states it caches for around 24 hours. Expect a change to be respected within a day. Content already collected before the change is not retroactively removed, so blocking a training crawler stops future collection rather than undoing past collection.
What are llms.txt and llms-full.txt?
They are proposed plain-text files at the root of a site that give AI systems a curated map of your content. llms.txt is a short index of your most important pages with brief descriptions. llms-full.txt holds the expanded content itself. Neither is an official standard and no major AI operator has confirmed it consumes them, so treat them as a low-cost experiment rather than a requirement.
Crawl access is the floor, not the strategy.
Letting the agents in is step one. Getting cited takes structured data, entity clarity, and content built to be quoted. We build all three together, from audit to implementation.