I was about to block the AI crawlers on this site.

Then I read OpenAI's own documentation. I had the wrong one.

The internet has spent a year arguing about whether to block AI crawlers.

Almost none of it separates two things that behave nothing alike. The difference decides whether you are protecting your work or quietly removing yourself from the answers.

There are two kinds of crawler, and they are not the same decision

A training crawler builds a model.

It takes your pages, adds them to a dataset, and what comes out is a system that can answer questions about your subject without ever sending you a visitor. The crawl happens once. You may never see any effect at all.

A retrieval crawler fetches a page to answer a question right now.

Someone asks something, the system reads a page, and cites it. The person reading that answer may click through. This one can send traffic, which is why the indexing pipeline that skips the queue matters more to a site than whether a model was trained on it.

OpenAI runs both, and states the difference on its own page:

"GPTBot is used to make our generative AI foundation models more useful and safe. It is used to crawl content that may be used in training our generative AI foundation models."

"OAI-SearchBot is for search. OAI-SearchBot is used to surface websites in search results in ChatGPT's search features. Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers."

Read those two together. Most coverage does not.

Blocking GPTBot stops your content going into training. It does not stop ChatGPT citing you.

Blocking OAI-SearchBot is what stops you being cited.

That one costs you something visible.

Anthropic draws the same line.

Its own help page says ClaudeBot collects content "that could potentially contribute to their training". There is a separate agent for the retrieval case.

Google is a different situation, and more misunderstood

Google does not use a separate crawler for most of this. Its AI features read the same index it already built.

What it gives you instead is a token, and this is where the confusion is worst. From Google's own crawler documentation:

"Google-Extended is a standalone product token that web publishers can use to manage whether their content is used to train Gemini models and for grounding (providing content from the Google Search index to the model at prompt time)"

And then the sentence that ends most of the argument:

"Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal"

And one more, which explains why it behaves oddly in practice:

"Google-Extended doesn't have a separate HTTP request user agent string. Crawling is [done by Googlebot]"

Google-Extended is not a crawler.

There is nothing to block.

You are writing a rule that tells Google what it may do with content it has already indexed, using the Googlebot access it already has. That is why you cannot block Google-Extended and keep Googlebot out of the same pages. They are the same fetch.

So blocking it does one thing. It removes your content from Gemini training and from grounding in AI Overviews.

It does not remove you from Search. It does not change where you rank. Google says that in writing.

What 28 large sites actually do

I wrote a robots.txt parser and checked 30 domains.

Two were unreachable. The counts below are out of 28.

The first attempt at this used a regular expression and produced nonsense.

It reported that Reuters and the New York Times block Googlebot. That would be catastrophic for both, and it is false. Reuters lists hundreds of agents in one shared group. The Times has a Googlebot group that disallows /ads/ and /athletic/search/*. A naive matcher cannot see group structure, so it cannot tell a path rule from a site-wide block.

The parser now walks groups properly, and the numbers here come from that.

                     blocks          blocks          blocks
                     training        retrieval       nothing
                     ------------    ------------    ------------
nytimes.com              5               5
bbc.com                  5               4
cnn.com                  5               5
theguardian.com          4               2
wired.com                4               2
theverge.com             4               4
vox.com                  4               4
axios.com                2               0
medium.com               4               0
quora.com                4               4
healthline.com           5               0
webmd.com                3               0
tripadvisor.com          5               0
allrecipes.com           4               3
reuters.com              1               0
github.com               1               0
forbes.com                                             nothing
businessinsider.com                                    nothing
reddit.com                                             nothing
wikipedia.org                                          nothing
cloudflare.com                                         nothing
wordpress.org                                          nothing
shopify.com                                            nothing
hubspot.com                                            nothing
stripe.com                                             nothing
vercel.com                                             nothing
mozilla.org                                            nothing
mailchimp.com                                          nothing

block BOTH training and retrieval :  9
block TRAINING only               :  7
block NEITHER                     : 12
block the Google-Extended token   :  8

Twelve of twenty-eight block nothing at all.

Not the small ones either. Reddit, Wikipedia, Forbes, Business Insider, Shopify, Stripe, Vercel, Cloudflare, Mozilla, HubSpot, Mailchimp and WordPress.org have no AI crawler restrictions in their robots.txt.

Nine block both, which is the position most coverage describes as the norm. The New York Times, the BBC, CNN, the Guardian, Wired, the Verge, Vox, Quora and Allrecipes.

And seven do the thing almost nobody discusses. Axios, Medium, Healthline, WebMD, Tripadvisor, Reuters and GitHub block the training crawlers and leave the retrieval ones open.

Why those seven are the interesting group

Look at what they gave up and what they kept.

They stopped their content going into training datasets. That is the thing publishers are angriest about, because it is a permanent transfer of value with no traffic attached.

They did not stop being cited. Reuters still appears in AI search answers. GitHub still gets read when someone asks a coding question.

They are not invisible.

That is a coherent position. It is the one the technical facts support.

Training is a one-way transfer. Retrieval is at least capable of sending someone to you.

The other side, stated fairly

I am not going to pretend this is free.

If you are a news organisation, the retrieval crawlers are the argument.

An answer that summarises your reporting takes the reader who would have come to you, and the crawl that produced it took the thing you paid to make.

Nine of the largest sites here block everything, and they are not being naive. They are making a licensing argument. Several of them have signed deals that pay for the access they were otherwise giving away.

And blocking training has a cost that is easy to miss. If the models are trained on everything except you, and people ask those models about your subject, you are the one site they cannot draw on. There is a version of the future where not being in the dataset is the same as not existing in the answer.

That is a real risk. I do not know how it resolves.

Neither does anyone else writing about this today.

What I did on my own site

I did not block anything yet, and I want to be honest about why rather than dress it up as a decision.

Blocking crawlers is a move you make to protect something. This site is new. There is nothing here yet that needs protecting, and the only thing an AI system could take from these pages is the argument, which I would rather it repeated.

When I do block, it will be GPTBot and ClaudeBot, and not OAI-SearchBot. That is the position the vendors' own documentation supports. Take my work out of training, stay in the answers.

I will not touch Google-Extended. Google states that it does not affect Search or ranking, which means the only consequence is disappearing from Gemini training and from AI Overview grounding. Disappearing from AI Overviews in exchange for nothing is a bad trade for a site that needs to be found.

Here is the version I would write, when I write it:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: Applebot-Extended
Disallow: /

Five training crawlers.

OAI-SearchBot, ChatGPT-User, PerplexityBot and Googlebot are not in the list. That is the whole point.

And it goes in robots.txt, which means the same file Google reads. A mistake there can cost you the crawl entirely, which is the lesson in the robots.txt article, and the reason this sits at the bottom rather than the top.

The short version

There are two kinds of AI crawler. They are not the same decision.

Training crawlers build a model and send no traffic. Retrieval crawlers answer a question and can cite you.

OpenAI says this plainly on its own documentation page. GPTBot trains. OAI-SearchBot controls whether you appear in ChatGPT search answers.

Google-Extended is not a crawler, has no user agent, and by Google's own statement does not affect Search or ranking. It controls training and grounding, nothing else.

Of 28 large sites, nine block both, twelve block nothing, and seven block only the training crawlers while staying open to being cited.

That last group is the smallest and the most interesting. It is also the position the technical facts support.

The question is not whether to block AI crawlers. It is which ones, and most of the writing on this has not made the distinction.

It is the same problem as a company telling you to ignore a file it publishes itself, except here the two sides are two different companies rather than two teams inside one.

Sources

  • OpenAI documentation, overview of OpenAI crawlers
  • Anthropic help centre, does Anthropic crawl data from the web and how can site owners block the crawler
  • Google Search documentation, Google common crawlers
  • Direct requests to /robots.txt on 30 domains, parsed with ARTICLES-PLAN/8/check-ai-crawlers.py, 12 October 2026