Shop on Radar
All guides

Your robots.txt is probably blocking the wrong robots

Published 19 September 2026 10 minute read

One small text file decides which AI assistants can read your shop. Here is how to read yours, and the six mistakes that turn up in it most often.

Your shop has a small text file that tells automated visitors which parts of the site they may read. It was written for search engines, usually years ago, and it now decides whether AI assistants can see you. The rules in it are frequently doing something other than what their author intended.

This is the practical half of the first gate in getting your shop recommended by AI assistants. The other half is knowing which bot is which, which you will want open alongside this.

How to read yours in two minutes

Type your shop's address into a browser and add /robots.txt to the end, like yourshop.com/robots.txt. It is plain text. You can read it yourself, and you should, because it is short and it is making decisions on your behalf.

It is made of blocks. Each block names a visitor and then lists what that visitor may and may not fetch:

User-agent: OAI-SearchBot
Allow: /
Disallow: /checkout/

User-agent names the bot. Disallow lists paths it must not fetch. Allow carves exceptions out of a Disallow. A bare Disallow: / means the whole shop. A User-agent: * block means every visitor that has no block of its own, which is the part most people misread, and the third mistake below.

Mistake one: the blanket block that came from somewhere else

The most common serious fault is the shortest:

User-agent: *
Disallow: /

That is the entire shop closed to everything. It gets there in a few ordinary ways. The file was copied from the staging site, where it was correct and deliberate, and nobody changed it at launch. A security plugin added it to keep bots off. A developer put it there during a rebuild and moved on.

It blocks Google as thoroughly as it blocks ChatGPT, so it usually gets caught. When it does not get caught, it is because the shop was already getting most of its traffic from ads and social, and nobody was watching the organic numbers closely enough to notice they were zero.

Mistake two: blocking GPTBot and thinking that settled ChatGPT

This one is almost the opposite. It is careful, deliberate, and aimed at the wrong target:

User-agent: GPTBot
Disallow: /

The shop owner read that AI companies were using websites without asking, found the name everyone quotes, and blocked it. It is a reasonable thing to want.

But GPTBot collects training text. OpenAI's documentation says disallowing it indicates the content should not be used to train models. The bot that decides whether ChatGPT can surface your shop when a shopper asks is OAI-SearchBot, and it is a separate name with separate rules.

So this shop kept its product photography out of a future training run, which may well be what it wanted, and did nothing whatsoever about being recommended. The two are unrelated. Worth knowing before you conclude that blocking was a mistake: often it was not. It just was not the thing the owner thought they were deciding.

Mistake three: a specific block that cancels all your other rules

This is the one nobody expects, and it comes from how the standard actually works rather than from carelessness.

When a bot reads your file, it looks for a block naming itself. If it finds one, it obeys that block and ignores the User-agent: * block completely. The rules are not added together. This is written into RFC 9309, the specification robots.txt follows: a crawler obeys the * group only when no group names it.

Which makes this file do something its author did not intend:

User-agent: *
Disallow: /wp-admin/
Disallow: /checkout/
Disallow: /cart/

User-agent: OAI-SearchBot
Allow: /

The owner meant "keep everyone out of admin and checkout, and let ChatGPT see the shop". What they wrote tells OAI-SearchBot to ignore the first block entirely, so it is now invited into the admin area, the checkout and the cart. Every other bot still respects those three lines. Only the one with its own block escapes them.

It runs in the other direction too, and that version is stranger:

User-agent: *
Disallow: /

User-agent: GPTBot
Crawl-delay: 10

That file closes the shop to everything, then hands GPTBot a block containing no Disallow at all. GPTBot reads its own block, finds nothing forbidden, and crawls freely. The one bot the owner was thinking about is the one bot that gets in.

The fix is to repeat your shared rules inside every block that names a bot:

User-agent: *
Disallow: /wp-admin/
Disallow: /checkout/
Disallow: /cart/

User-agent: OAI-SearchBot
Disallow: /wp-admin/
Disallow: /checkout/
Disallow: /cart/

Repetitive, and correct. There is no syntax for inheriting rules, so repetition is the only way to express it.

Mistake four: blocking Bingbot to keep Copilot out

Bingbot looks like a search engine most shops do not think about much, and blocking it looks cheap. Microsoft's guidance is that Bing search, Copilot and its grounding results run on the same crawling and indexing foundation. One line removes you from all of them, ordinary Bing results included.

Rarely what anyone means to do, and easy to do by accident when trimming crawlers to save bandwidth.

Mistake five: expecting the -Extended tokens to control visibility

Google-Extended and Applebot-Extended read like the AI settings, and they are not.

Neither of them is a crawler. They are permission tokens that say how data may be used after a different bot has fetched it. Apple's documentation states directly that Applebot-Extended does not crawl webpages and that pages disallowing it can still appear in search results. Google states that using Google-Extended does not affect a site's inclusion in Search and is not a ranking signal.

The consequence that matters: whether you appear in Google's AI Overviews and AI Mode is governed by your rules for Googlebot, because those features are part of Search. A shop that carefully set Google-Extended to Disallow and expected to vanish from AI Overviews has changed nothing about them.

Mistake six: treating the file as a lock

robots.txt is a request, and it is public. Anyone can read yours by typing the address, which means a Disallow line is a signed, published list of the paths you would rather nobody looked at. Well-behaved bots honour it. Badly behaved ones read it for ideas.

Two practical consequences. Do not use it to hide anything that matters; use real access control for that. And remember that some of the bots that fetch a page because a person just asked a question may not consult it at all. OpenAI says robots.txt rules may not apply to user-initiated fetches, and Perplexity says its own generally ignores them. Anthropic makes no such exception, so a rule aimed at Claude-User does work. If a page genuinely must stay out of AI answers, keep it out of the search indexes with noindex rather than out of robots.txt.

Two details that bite shops outside English

Paths in robots.txt are not compared as plain text. RFC 9309 requires that anything outside plain ASCII is percent-encoded on both sides before they are matched, which is why a rule written /produkt/buty-męskie is supposed to match the URL your browser shows as /produkt/buty-m%C4%99skie. If your product URLs carry diacritics, and in Polish, German and most languages that are not English they do, every rule you write depends on that step happening correctly.

The rules for spaces and other punctuation are fiddlier than that summary, and this is somewhere real-world checkers genuinely disagree with each other. So if a tool tells you a page is reachable, it is worth confirming the tool handled the encoding rather than comparing the two strings as they look.

And robots.txt only governs the exact host that serves it. If your shop runs on shop.example.com while your blog is on example.com, the file at the blog's address has no authority over the shop. Each needs its own.

What to actually do, in order

  1. Open yourshop.com/robots.txt and read the whole thing. It is usually under thirty lines.
  2. Look for Disallow: / anywhere. Under User-agent: * it closes the shop. Under a named bot it closes the shop to that bot.
  3. Check that OAI-SearchBot, Googlebot, Bingbot, PerplexityBot and Claude-SearchBot are either named and allowed, or not named at all while the * block permits your product pages.
  4. For any bot that does have its own block, confirm the block repeats the rules you thought applied to everyone.
  5. Decide separately, and on your own terms, about GPTBot, ClaudeBot and CCBot. That decision is about training and it does not affect step three.

If you change anything, keep a copy of the old file. It is the cheapest possible way to undo a mistake, and this is a file where a one-character error is expensive.

Questions people ask

Where is the file if I cannot find one?

If yourshop.com/robots.txt returns a 404, you do not have one, and every bot treats that as permission to crawl everything. That is not an emergency. It is a less controlled position than having a deliberate file, but it fails in the safe direction.

Can I edit it myself?

On most platforms, yes. Shopify and WooCommerce both expose it, Shopify through a theme file and WooCommerce through an SEO plugin or the site root. If your shop was built for you and you are not sure, this is a five-minute question for whoever maintains it, and worth asking rather than guessing.

How long until a change takes effect?

Bots re-read robots.txt periodically rather than on every visit, so allow days rather than minutes. Removing a block does not summon a crawler either. It has to come back on its own schedule before anything changes.

Should I just allow everything?

For a shop, close to yes for the search crawlers, because being readable is the entire point of a shop. Keep your admin, cart and checkout paths blocked, since nothing good comes of those being indexed. Training crawlers are a separate and genuine choice.

Does any of this affect my Google ranking?

Blocking Googlebot does, enormously. The AI-specific names do not: Google says Google-Extended is not a ranking signal, and the other companies' crawlers have no bearing on Google at all.

Where we come into it

One file, once, is a job you can do yourself in an afternoon with this page open, and you should. The part that does not scale is everything downstream of it: whether the pages those crawlers can now reach actually say enough to be recommended, which of your products fail that test, and which shops are being named instead of yours when your buyers ask.

We built the robots.txt parser behind Shop on Radar by hand, precedence rules and all, because getting this wrong in either direction is the worst thing an audit can do to a shop. It runs against the products you pick, and then the questions your buyers ask go to ChatGPT to see who gets named. We explain how to fix what we find. We do not touch your shop.

We are not open yet. Join the waitlist and we will tell you when we are.

Sources