A dozen or so AI bots may visit your shop, and they do three different jobs. Only one of those jobs decides whether an assistant can name you when a shopper asks what to buy. Most advice treats them as one group, which is why so many shops block the wrong ones.

This is the reference table for the first of the three gates in getting your shop recommended by AI assistants. Everything here comes from the crawler documentation each company publishes, linked at the bottom.

Three jobs, not two

A bot that visits your shop is doing one of these:

Building a search index. It fetches your pages now so the assistant can find and cite you later. This is the group that decides whether you can be recommended at all.
Fetching a page because a shopper asked just now. Someone put a question to an assistant, and it went to look. These fetches often ignore your robots.txt file, because a person asked for them.
Collecting text to train a future model. The cost of blocking one of these is real, slow and indirect. It affects a model release rather than today's answer.

The common version of this advice has two groups, training and everything else. Splitting the second group in half is what makes it useful, because the two halves respond to completely different controls.

The crawlers that decide whether you can be cited

These build the indexes the assistants search. They obey robots.txt, so blocking one works, and blocking one is how a shop disappears from that assistant.

Crawler	Feeds	Blocking it means
`OAI-SearchBot`	ChatGPT search	ChatGPT cannot surface you
`Claude-SearchBot`	Claude's search results	Claude cannot surface you
`PerplexityBot`	Perplexity's search results	Perplexity cannot link you
`Bingbot`	Bing, Microsoft Copilot, grounding	You leave Bing and Copilot together
`Googlebot`	Google Search, AI Overviews, AI Mode	You leave Google Search entirely

Four of these are worth a sentence each.

OAI-SearchBot is the one to get right if you only get one right. OpenAI's own documentation tells site owners to allow it in robots.txt if they want to appear in ChatGPT's search features. It is a different bot from GPTBot, and almost nobody outside this field knows that.

PerplexityBot is for search listings only. Perplexity states plainly that it is not used to collect content for training models, so blocking it buys you no training protection and costs you the listing.

Bingbot does two jobs with one name. Microsoft's guidance is that Bing search, Copilot and its grounding results run on the same crawling and indexing foundation, so a robots.txt line aimed at Copilot takes out ordinary Bing results as well.

Googlebot belongs on this list for a reason that surprises people. Google says robots.txt rules for Googlebot are the control for AI Overviews and AI Mode, which are parts of Search. The separate Google-Extended token, which sounds like the AI one, does not govern them.

The fetchers that arrive because a shopper asked

When someone asks an assistant a question and it goes to read a page for them, the fetch comes from a different bot. Whether your robots.txt stops it depends on the vendor, and they do not agree.

Fetcher	Arrives when	Obeys robots.txt
`ChatGPT-User`	a ChatGPT user's question needs it	OpenAI says it may not apply
`Perplexity-User`	a Perplexity user's question needs it	Perplexity says it generally ignores it
`Claude-User`	a Claude user's question needs it	Yes, Anthropic publishes no exception

OpenAI and Perplexity both carve their user-triggered fetchers out explicitly, on the reasoning that a person requested the page, which makes it closer to a browser visit than to a crawl. Anthropic does not. It says its bots honour robots.txt and names no exception for Claude-User.

That difference is the whole practical point, and it is easy to get wrong by grouping these three on how they are triggered rather than on what each vendor actually published. A block on Claude-User works. A block on the other two may be doing nothing at all, which means removing it costs you nothing and adding it buys you nothing.

Either way, none of the three is how you get recommended. The group above is where robots.txt decides that. And if a page must stay out of AI answers altogether, the controls that work are the ones that keep it out of search indexes, such as noindex, rather than a robots.txt line.

The ones that only affect future models

These collect training text. Blocking them is a legitimate choice about how your writing and photography get used. It is not a decision about whether you get recommended this month.

Crawler	Collects for
`GPTBot`	OpenAI's model training
`ClaudeBot`	Anthropic's model training
`CCBot`	Common Crawl, which feeds many models

GPTBot is the name everyone recognises, and it is the reason for the single most common mistake in this whole subject. Shops block GPTBot believing they have made a decision about ChatGPT recommending them. They have not. OpenAI's documentation says disallowing GPTBot indicates the content should not be used to train models, and that is all it does.

CCBot is a non-profit, Common Crawl, which publishes an open archive of the web that many AI companies have drawn on. It obeys robots.txt.

Two names on every list that are not crawlers at all

Google-Extended and Applebot-Extended appear in every article on this subject, usually in a column headed "crawler". Neither of them crawls anything.

Both are permission tokens. You write them in robots.txt to say how data may be used after some other bot has already fetched it.

Apple's documentation is explicit that Applebot-Extended does not crawl webpages, and that pages disallowing it can still appear in search results. It governs what the real crawler, Applebot, is allowed to do with what it already has. Blocking it opts you out of training for Apple's foundation models and changes nothing else.

Google-Extended works the same way for Gemini apps and the Vertex AI API. Google states that using it does not affect a site's inclusion in Search and is not a ranking signal. So it cannot be the thing keeping you out of AI Overviews, whatever a checklist implies.

This matters beyond pedantry. Someone who believes these two are crawlers believes they have a switch for AI visibility that they do not have, and stops looking for the one they do.

The short version

If you read nothing else: allow OAI-SearchBot, Claude-SearchBot, PerplexityBot, Bingbot and Googlebot. Decide about GPTBot, ClaudeBot and CCBot on your own terms, knowing the decision is about training rather than recommendations. Do not expect the -User fetchers or the two -Extended tokens to behave like either group.

Then go and read what your own robots.txt says, because the rules in it were probably written before most of these names existed. The mistakes in it are predictable, and the most common one takes about ten minutes to undo.

Questions people ask

How many of these do I actually need to care about?

Five, and realistically two. OAI-SearchBot and Googlebot cover where most shopping questions are asked today. The other three cost nothing to allow.

Does allowing a search crawler mean my content trains a model?

No, and this is the distinction the two groups exist to draw. OpenAI, Anthropic and Perplexity each use a separate name for indexing and for training, so you can allow one and refuse the other. Perplexity is the clearest case: it says PerplexityBot is not used for training at all.

If I block a training crawler, do I lose recommendations?

Not directly. Your pages still get indexed by the search crawlers and can still be cited. What you give up is your text contributing to a future model's general knowledge, which is a slower and much vaguer effect.

Are these lists stable?

No. Anthropic split its single bot into three separate names, and OpenAI added OAI-AdsBot to check pages submitted as ChatGPT ads. Treat any crawler list, including this one, as needing a look every few months.

Can I tell which ones have visited me?

Yes, from your server's access logs, which record the user agent of every visitor. Your host may expose these directly, or through a plugin. Bing Webmaster Tools also reports which of your pages get cited in Copilot answers.

Where we come into it

Reading this table against one robots.txt file is a ten-minute job, and if that is all you need, the table is the whole answer. It gets harder when you want to know what the rules mean for the specific products you sell, whether the pages those crawlers reach carry enough information to be recommended, and which shops are being named instead of yours.

Shop on Radar reads the rules the way each of these bots would, across the products you pick, and then asks ChatGPT your buyers' questions to see whether the access you thought you had is producing anything. We explain how to fix what we find. We do not touch your shop.

We are not open yet. Join the waitlist and we will tell you when we are.

The AI crawlers that decide whether your shop gets recommended