Reference · 25 user-agents · checked 2026-09-21

Which AI crawlers read your site, and what does blocking each one cost?

An AI crawler fetches web pages for an AI company, and the three main kinds don’t block each other. Training crawlers collect pages for future models. Search crawlers build the index an assistant cites from. User fetchers open a page because someone asked about it just now. Blocking GPTBot keeps you out of OpenAI’s training data — not out of ChatGPT search, which uses a different bot.

The directory

25 user-agents, grouped by what they do

Every AI crawler user-agent documented by its own operator, plus five tokens with no current documentation behind them. For each one: the operator, what it does, whether it follows robots.txt, and a link to the source. Last checked on 2026-09-21.

Training and data collection 5

Blocking these mainly keeps your pages out of future training data. Two of them, meta-externalagent and Amazonbot, also feed their operator’s products, so a block reaches further.

User-agent	Operator	What it does	robots.txt
GPTBot	OpenAI	Collects pages to train OpenAI’s foundation models. OpenAI documents it as independent of its search crawler: blocking GPTBot does not remove you from ChatGPT search. source	Yes
ClaudeBot	Anthropic	Collects pages that may be used to train Anthropic’s models. Blocking it excludes your future content from training. source	Yes
CCBot	Common Crawl	Builds Common Crawl’s open web archive, which anyone can download. Common Crawl’s FAQ doesn’t say how the archive is used downstream. source	Yes
meta-externalagent	Meta	Collects pages for training foundation AI models or for indexing content directly in Meta products. Meta doesn’t separate the two. source	Yes
Amazonbot	Amazon	Improves Amazon’s products and services, and may be used to train Amazon AI models. source	Yes

Search indexes 7

These build the index an assistant cites from. Blocking one removes your pages from that product’s AI search results.

User-agent	Operator	What it does	robots.txt
OAI-SearchBot	OpenAI	Surfaces websites in ChatGPT’s search features. OpenAI’s documented way to stay citable without joining training: allow this, disallow GPTBot. source	Yes
Claude-SearchBot	Anthropic	Indexes pages to improve Claude’s search results. Anthropic says blocking it may reduce your visibility in them. source	Yes
PerplexityBot	Perplexity	Surfaces and links websites in Perplexity’s search results. Perplexity says it is not used to train AI models. source	Yes
Amzn-SearchBot	Amazon	Improves Amazon’s search experiences. Amazon says it does not crawl for generative AI training. source	Yes
Applebot	Apple	Powers search in Spotlight, Siri and Safari. Applebot-Extended decides whether what it collects may also train Apple’s models. source	Yes
bingbot	Microsoft	Crawls for the Bing search index. source	Yes
Diffbot	Diffbot	Crawls the web to build a general search engine. Diffbot says robots.txt can be overridden where it has an agreement with the site. source	Yes, by default

User-triggered fetchers 6

A person asked about your page just now. Several operators say robots.txt may not apply to these.

User-agent	Operator	What it does	robots.txt
ChatGPT-User	OpenAI	Opens pages for user actions in ChatGPT and Custom GPTs. source	May not apply
Claude-User	Anthropic	Opens a page when a Claude user’s question needs it. Blocking it stops Claude retrieving your content for those questions. source	Yes
Perplexity-User	Perplexity	Visits a page to help answer a user’s question. source	Generally ignored
Amzn-User	Amazon	Supports user actions, such as Alexa questions that need current information. Not used for generative AI training. source	Yes
meta-externalfetcher	Meta	Fetches pages at a user’s request, including AI agents navigating sites to complete tasks. source	May be bypassed
Diffbot-User	Diffbot	Makes requests on behalf of a person browsing a URL. source	Not stated

Control tokens 2

These never visit your site. They only tell an operator what it may do with pages another of its crawlers already fetched.

User-agent	Operator	What it does	robots.txt
Google-Extended	Google	Not a crawler: no request carries it. It controls whether pages Googlebot crawls may train Gemini models and ground Gemini’s answers at prompt time. Google says it does not affect Search inclusion or ranking. source	Token only
Applebot-Extended	Apple	Not a crawler. It controls whether pages Applebot crawls may train Apple’s foundation models. source	Token only

No current documentation 5

No operator documentation stands behind these today. A rule naming one does no harm; it just comes with no stated promise.

User-agent	Operator	What it does	robots.txt
Bytespider	ByteDance	ByteDance publishes no documentation for it: no stated purpose and no robots.txt policy. no source published	No policy
Timpibot	Timpi	Timpi runs an independent search engine, but timpi.io has no documentation for this crawler. source	No policy
cohere-ai	Cohere	Not in Cohere’s documentation. Cohere says it does not currently crawl the web to train models; the token it documents for a future crawler is Coherebot. source	Undocumented
anthropic-ai	Anthropic	Not in Anthropic’s current documentation, which lists only ClaudeBot, Claude-SearchBot and Claude-User. source	Undocumented
Claude-Web	Anthropic	Not in Anthropic’s current documentation, which lists only ClaudeBot, Claude-SearchBot and Claude-User. source	Undocumented

OpenAI also documents OAI-AdsBot, which checks the safety of pages submitted as ads on ChatGPT.

A common setup

Staying citable while opting out of training

# Opt out of training
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /

# Search, user fetchers and everyone else
User-agent: *
Allow: /

A group can name several user-agents before a single rule; that is standard robots.txt under RFC 9309. Every search crawler above falls through to the last group and stays allowed.

Two trade-offs to decide on purpose. Google-Extended also opts you out of Gemini grounding, so if being used in Gemini’s answers matters more than staying out of its training, remove that line. And meta-externalagent and Amazonbot are left out on purpose: blocking them reaches their operators’ products, not just training.

The costly mistake runs the other way: a “User-agent: *” group with “Disallow: /” blocks every compliant crawler in this directory, search included.

How this list is kept: every row comes from the operator’s own documentation, read on 2026-09-21. Where an operator publishes nothing, the row says so instead of repeating what third-party sites claim. Checking which of these your own robots.txt blocks, line by line, is the first of the five checks in a Rankavex audit.

Want to know what an AI sees when it reads your site?

Send the domain. You’ll get the findings back either way.

Ask for a free look at your site

Sources

Cite this

AI crawlers come in three kinds that don’t block each other: training crawlers, search crawlers that build the index an assistant cites from, and fetchers that open a page because a user asked. Blocking GPTBot keeps you out of OpenAI’s training data, not out of ChatGPT search.

Igor Almeida, Rankavex. Last checked 2026-09-21
https://rankavex.com/data/ai-crawlers/

All reference data →