AI Crawlers: GPTBot, ClaudeBot, PerplexityBot, and Which to Allow (2026)

AI crawlers are the automated agents AI companies use to fetch pages from your site, and they do three different jobs: collecting text that may train a model, building a search index the assistant queries when it answers, and fetching one page because a user just asked a question about it. That split decides what you should allow. Blocking a training crawler is a licensing decision that costs you nothing in visibility. Blocking a retrieval crawler removes you from that assistant's answers entirely.

Most robots.txt files treat all of them as one category. That is how a site ends up opted out of ChatGPT Search while still feeding the training set, or opted out of Gemini grounding under the impression it protected its Google rankings.

This is the reference, not the playbook. Every bot below is one our free AI SEO audit checks on any URL: who runs it, which of the three jobs it does, what you lose by blocking it, and the exact user-agent token to write. The optimization work for each engine has its own guide, linked at the end.

Key Takeaways

• Training, retrieval, and user-triggered fetching are three separate decisions, and one robots.txt rule cannot express all three.
• Blocking GPTBot does not remove you from ChatGPT Search. OAI-SearchBot is the token that governs that.
• Google-Extended does not affect Google Search ranking or AI Overviews. Google documents both explicitly.
• Two of the user-triggered fetchers state that robots.txt may not apply to them, because a person asked for the page.
• Every major operator publishes verified IP ranges, because impersonation of these user agents is common.

The Three Jobs an AI Crawler Does

Every bot in this guide does one or more of three things, and the difference is the reason a single blanket rule gets sites into trouble.

Training. The crawler collects page text that may be used to build a future model. Nothing you gain or lose here shows up in this quarter's citations. A model trained on your content today ships months from now, and it will not link to you when it does. Allowing training crawlers is a judgment about how your content is used, not a visibility tactic.

Retrieval and indexing. The crawler builds an index the assistant queries at answer time. This is the one that decides whether you can be cited today. Block it and you are absent from that engine's answers no matter how good the page is, which is why it sits at the top of every AI search visibility checklist.

User-triggered fetching. A person asked the assistant about a specific page or topic, and the assistant went and read it in real time. There is no index and no training set involved. Two of the operators below state plainly that robots.txt may not govern these requests, precisely because a human initiated them.

The costly mistake is treating the first as though it were the second. Refusing to feed a training set is a defensible position. Refusing it in a way that also drops you out of the retrieval index is an accident, and it is the single most common one we find.

The AI Crawler Reference

Tokens are case-insensitive in robots.txt, but spelling is not. Google-Extended carries a hyphen, and GoogleExtended matches nothing at all. That typo is common enough in published SEO advice that it is worth checking your own file for it.

The Bots, Operator by Operator

OpenAI: three bots, three separate switches

OpenAI documents OAI-SearchBot, GPTBot, and ChatGPT-User as independent controls, and says so directly: a site can allow OAI-SearchBot to appear in search results while disallowing GPTBot to keep its content out of training. If you allow both, OpenAI may reuse a single crawl for both purposes rather than fetching twice.

OAI-SearchBot is the one that decides visibility. Opt out of it and your site will not be shown in ChatGPT search answers. GPTBot is training only. ChatGPT-User covers pages ChatGPT visits because a user asked, and OpenAI notes that because those actions are user-initiated, robots.txt rules may not apply, and that this agent is not used to decide whether content appears in Search.

OpenAI also runs OAI-AdsBot, which visits landing pages submitted as ads to check them against policy. It is outside the set our audit checks, since it only visits pages you submitted yourself.

Anthropic: training, search, and user requests

Anthropic documents three robots and says all of them honor industry-standard robots.txt directives, including the non-standard Crawl-delay extension. ClaudeBot collects web content that may contribute to model training. Claude-User fetches pages when an individual asks Claude a question. Anthropic's own note on disabling it is worth reading as written: it prevents retrieval of your content in response to a user query, "which may reduce your site's visibility for user-directed web search."

Anthropic also documents Claude-SearchBot, which indexes content to improve search result quality. It is not in the ten-bot set our audit currently checks, so if you want Claude coverage locked down end to end, add it to the same allow group as ClaudeBot and Claude-User. The engine-specific work sits in the Claude SEO guide.

Perplexity: search bot and user agent, and only one obeys you

PerplexityBot surfaces and links sites in Perplexity results, and Perplexity states plainly that it is not used to crawl content for AI foundation models. If you want to appear in Perplexity at all, this one has to be allowed.

Perplexity-User handles pages visited because a user asked a question. Perplexity's documentation is unusually direct about the consequence: since a user requested the fetch, that fetcher generally ignores robots.txt rules. Treat robots.txt as a signal to the indexer, not a lock on the door. The per-engine tactics are in the Perplexity SEO guide.

Google: one crawler, one control token, and a widely repeated error

Googlebot is the crawler. AI Overviews and AI Mode are features of Google Search built on the index Googlebot populates, so the robots.txt directives for Googlebot are what govern your presence there. If you want to limit how your content is used inside those AI answers without leaving Search, the controls are nosnippet, data-nosnippet, max-snippet, and noindex, not a crawler block.

Google-Extended is not a crawler. Google's documentation says it has no separate HTTP request user agent string, that crawling is done with existing Google user agent strings, and that the robots.txt token "is used in a control capacity." What it controls is whether your content can be used for training Gemini models and for grounding in Gemini apps and Vertex AI. Google states that Google-Extended does not impact a site's inclusion in Google Search and is not used as a ranking signal.

So blocking Google-Extended is safe for your rankings and costly for your Gemini presence, which is close to the opposite of what most robots.txt advice implies. The Search side of this is covered in the Google AI Overview SEO guide.

Microsoft: Bingbot carries more than Bing

Bingbot builds the Bing index. That index does double duty, because Microsoft's Copilot products retrieve web results through it, and ChatGPT's live retrieval has historically leaned on Bing as well. A site with strong Google rankings and almost no Bing presence is a pattern we see often, and it is expensive in a way that a Google-only report never shows.

Common Crawl: the one that is not an AI company

CCBot belongs to Common Crawl, a non-profit that maintains an open, downloadable archive of the web. It is not an assistant and it will never cite you. It earns its place on this list because that archive is a standard starting point for people assembling training corpora, so blocking CCBot reaches further than blocking any single vendor's bot, and allowing it reaches further too. Common Crawl publishes reverse-DNS-verifiable IP ranges and notes that other crawlers falsely identify themselves as CCBot.

Three Things Almost Everyone Gets Wrong

Robots.txt groups are not additive. If a crawler finds a group naming its own user agent, that group is the only one it reads, and the User-agent: * group is ignored for it completely. So adding User-agent: GPTBot with a single Crawl-delay line does not layer a delay on top of your wildcard rules. It replaces them, and everything your wildcard group disallowed is now allowed for GPTBot. This is the most common way a robots.txt file does the opposite of what its author intended.

A blanket bot block is a decision you did not make. Plenty of sites carry a Disallow: / under User-agent: *, or a security-tool rule that catches anything with "bot" in the user agent, written years before any of these agents existed. The rule is doing exactly what it says. Nobody chose it.

User-triggered fetches are not reliably covered. Both OpenAI and Perplexity state that robots.txt may not apply to their user-initiated agents. If your goal is visibility, this is good news. If your goal is genuinely keeping content out of an assistant, robots.txt is not the tool, and authentication is.

What to Allow, and What It Costs

The default posture for any brand that wants to be found is straightforward: allow every retrieval and indexing bot, without exception. OAI-SearchBot, PerplexityBot, Googlebot, and Bingbot are the four that decide whether you can be cited at all, and there is no scenario where blocking one of them helps a business that wants customers.

Training is the genuine decision, and it is a business one rather than a technical one. Publishers with licensing leverage often block GPTBot, ClaudeBot, and CCBot deliberately. Most SaaS and ecommerce brands have nothing to license and everything to gain from being described accurately inside a model's weights, so they allow all of it.

If you want to be maximally findable, the file is short:

User-agent: OAI-SearchBot
Disallow:

User-agent: PerplexityBot
Disallow:

User-agent: ClaudeBot
Disallow:

User-agent: Claude-User
Disallow:

User-agent: Googlebot
Disallow:

User-agent: Bingbot
Disallow:

An empty Disallow value means "nothing is disallowed", so each of those groups grants full access. If you would rather be read but not trained on, keep the groups above and add the training opt-outs alongside them:

User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

That combination is coherent: you stay fully visible in ChatGPT Search, Perplexity, Google Search, and AI Overviews, and you decline the training uses. Note the asymmetry in the Anthropic row, where ClaudeBot is the training bot and blocking it is the equivalent move, at the cost of Claude's long-term knowledge of who you are.

Whatever you choose, the file needs re-checking. Operators add agents, split existing ones, and change what a token governs. Google-Extended did not exist before 2023, and Anthropic's three-bot split replaced a single earlier agent. A robots.txt written two years ago is a policy nobody has reviewed since.

How to Check What You Are Blocking Today

Three checks, in order of how often they surface something.

1. Read your own robots.txt. Fetch yourdomain.com/robots.txt and look for a Disallow: / under any group, plus any group naming one of the ten tokens above. Remember the precedence rule: a named group replaces the wildcard rather than adding to it.
2. Check the layer above robots.txt. A WAF, CDN, or bot-management rule can return 403 to these agents no matter what your robots.txt says, and it will not show up in any robots.txt tester. Perplexity, OpenAI, Anthropic, and Common Crawl all publish verified IP ranges specifically so you can allowlist the real agents and drop the impersonators, which is the only reliable way to tell them apart.
3. Check what they see when they get in. Access is not the same as readability. Assume these agents do not execute JavaScript unless the operator says otherwise, which is why our audit counts the words present in your HTML before any script runs. A page that renders its copy client-side is technically unblocked and practically empty.

The free AI SEO audit runs the first and third of these on any URL in about 30 seconds and returns which of the ten bots are blocked and which rule decided it. An llms.txt file is the complementary move once access is clean: it tells the agents that can read you which of your pages are worth reading, and you can build one with the llms.txt generator.

FAQ

Should I block GPTBot? Only if you have a licensing reason to. GPTBot collects content for training OpenAI's foundation models and has no bearing on whether you appear in ChatGPT Search, so blocking it protects your content from training use and costs you no visibility. Most SaaS and ecommerce brands allow it; publishers with content to license often do not.

Does blocking GPTBot remove me from ChatGPT? No. OpenAI documents the two as independent settings and directs site owners to OAI-SearchBot for managing search opt-outs. If you are missing from ChatGPT answers, check OAI-SearchBot and your Bing indexation before you look at GPTBot.

Does blocking Google-Extended hurt my Google rankings? No. Google states that Google-Extended does not impact inclusion in Google Search and is not used as a ranking signal. It controls Gemini training and Search grounding in Gemini apps and Vertex AI. AI Overviews are governed by your Googlebot directives and the standard snippet controls instead.

Is there one robots.txt rule that allows every AI crawler? User-agent: * with an empty Disallow allows all of them, and it is the right default for most sites. The catch is that any group naming a specific agent overrides it for that agent, so a single leftover User-agent: GPTBot block elsewhere in the file silently wins.

How do I tell a real AI crawler from something impersonating one? By IP, never by user-agent string, which anyone can copy. OpenAI, Anthropic, Perplexity, and Common Crawl each publish the address ranges their agents operate from, and Common Crawl supports reverse DNS verification. Impersonation is common enough that all four call it out in their own documentation.

Does llms.txt replace robots.txt? No, they answer different questions. Robots.txt says who may fetch your pages. An llms.txt file says which of your pages matter and what they cover, for the agents already allowed in. Robots.txt is enforced, llms.txt is a courtesy that some engines read. The llms.txt guide covers which ones.

Where to Go Next

Crawler access is the gate, not the strategy. Once every retrieval bot can read you, the work moves to being worth citing, and that is engine-specific:

• ChatGPT SEO for Bing index authority and the structure ChatGPT extracts from.
• Perplexity SEO for freshness and source-led answers.
• Claude SEO for accuracy, documentation depth, and llms.txt.
• Google AI Overview SEO for the Search-side AI answers.
• How to improve AI search visibility for the six levers in priority order, across every engine.
• AI search tracking for measuring whether any of it landed.

If reviewing robots.txt, WAF rules, and render output across a site is not work your team wants to own, it is one of the first things our done-for-you AI SEO service fixes, usually in the first week and usually before anyone writes a word of new content.