TL;DR: AI crawlers come in two kinds. Training crawlers (GPTBot, ClaudeBot) collect pages to train models. Search and citation crawlers (OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User) fetch your page to answer a user right now and name you as the source. Blocking a training crawler does not remove you from AI search answers. Blocking a citation crawler does. We published the opposite conclusion in one of our own audits, retracted it in writing and rebuilt our scoring. The rest of this post is how to test your own site per bot and how to decide what to block.
I get one question more than any other from founders who have read a scary article about AI scraping: should I block GPTBot? The honest answer depends on what you want, and the part that trips people up is that blocking it does less than they think in one direction and nothing at all in another. I will start with a mistake we made, because it is the cleanest way to show the problem.
What is the difference between GPTBot and OAI-SearchBot?
GPTBot collects pages that OpenAI may use to train its models. OAI-SearchBot is the crawler behind search results in ChatGPT: it finds and indexes pages so they can be shown and cited when someone asks a question. They are separate user agents with separate purposes, and OpenAI documents them separately so that site owners can treat them differently.
ChatGPT-User is a third one. It fetches a page when a user, or an assistant acting for a user, asks for that specific page in the moment. Perplexity splits the same way: PerplexityBot for its search index and Perplexity-User for fetches on behalf of a person. Anthropic runs ClaudeBot for training data collection.
The two kinds, as we group them
- Training crawlers - GPTBot (OpenAI), ClaudeBot (Anthropic). They collect pages to train models. Nobody is being answered at that moment and nothing is cited.
- Search and citation crawlers - OAI-SearchBot and ChatGPT-User (OpenAI), PerplexityBot and Perplexity-User (Perplexity). They fetch pages to answer a user right now and cite them.
Vendors add and rename agents over time, so treat this list as a snapshot and check each vendor's own crawler documentation before you write rules. The split into two purposes is the part that stays stable.
If I block GPTBot, will I disappear from ChatGPT answers?
No. Blocking a training crawler does not remove you from AI search answers, because the answer is built from pages fetched by the search and citation crawlers. Block OAI-SearchBot and you can drop out of ChatGPT Search results. Block GPTBot and, as far as the documented behaviour goes, you have only said something about training use.
The reverse holds too. If you allow GPTBot and block OAI-SearchBot, you have given away your pages for training and also hidden yourself from the place where a buyer would have been shown your name. That is the worst of both and it is more common than you would expect, usually because someone pasted a rule list from a blog post without reading what each line does.
What did we get wrong on a client audit?
On one client audit we reported that GPTBot received a 403, so the site was invisible to AI search. That conclusion was wrong. We had tested one agent and generalised from it. When we ran a per-bot test, every citation crawler received a 200 and could read the pages. The site was visible to the crawlers that matter for answers. Only the training crawler was refused, which may well have been exactly what the owner wanted.
We retracted the finding in writing to the client. Then we fixed the cause rather than the report: our audit tool now scores only citation crawlers for visibility, and it shows the training crawlers in a separate block with a note explaining that a refusal there is a policy choice, not a visibility problem. I would rather tell you that than have you find an old version of the claim and wonder how many of our other conclusions are built the same way.
How do we weight each crawler in our access score?
In the access part of our audit score, the weights are OAI-SearchBot 30, PerplexityBot 30, ChatGPT-User 20 and Perplexity-User 20. Training bots count for 0. That adds up to 100 across the four crawlers that decide whether you can be retrieved and cited, and it means a site that blocks every training crawler can still score full marks on access.
The weights are our judgement, not a standard. The two search index crawlers carry the most because they build the pool of pages an engine can draw from. The two user-triggered fetchers carry less but still matter, since they are what runs when a person asks about a specific page or when an assistant goes to check a source. If you disagree with the split, the useful part is the principle: score what affects being cited, and report the rest separately.
Does Cloudflare's setting replace my robots.txt?
No. We said it did, and we corrected that too. Cloudflare's managed setting for AI crawlers adds its own block of rules in front of your own file. Your robots.txt is still there and still read, but the managed rules come first, so the file you see in your repository is no longer the whole picture of what a crawler is told.
That matters for two reasons. First, if you are debugging why a crawler is refused, the file you edit may not be the thing refusing it. Open the live robots.txt in a browser or with a plain request and read what is actually served, not what you think you deployed. Second, if you want to allow citation crawlers while keeping training crawlers out, check that the managed rules and your own rules agree with each other, because a managed block placed first can say something different from the lines below it.
Can a firewall block a bot that robots.txt allows?
Yes, and this is the failure that robots.txt reading will never reveal. Robots.txt is a polite request. A firewall, a bot-protection layer or a CDN rule is enforcement. A crawler can be told it is welcome in robots.txt and then receive a 403 or a challenge page at the network edge before it ever reaches your server.
So there are two separate places to check and they can disagree: what robots.txt says, and what the firewall does when the request arrives. If you only read the first, you will tell yourself you are open while the second quietly turns crawlers away. Check both, per bot, every time.
How do I test each AI crawler on my own site?
Request your important pages once per crawler, sending that crawler's user agent string, and record two things for each: the status code, and whether the returned HTML actually contains your content. Do it for each agent in your list, not for one agent standing in for all of them. That is the entire method, and it is the one our first report skipped.
What to record for every page and every bot
- Status code. 200 is what you want from citation crawlers. 403, 429, a redirect to a challenge page or a timeout means that bot is not getting in.
- Body check. Look in the HTML for a heading, a product name or a sentence you know is on the page. A 200 with an empty body is not access.
- Which rule did it. If a bot is refused, find out whether robots.txt, the CDN's managed rules or your own firewall rule is responsible before you change anything.
- Repeat after every infrastructure change. A new CDN setting, a plugin update or a bot-protection toggle can change the result without anyone touching robots.txt.
User agent strings are public in each vendor's documentation, and you can send them with an ordinary command-line request or a short script. One caution: some firewalls verify a crawler by its published network addresses and not by the user agent text alone, so a test from your own machine with a borrowed user agent can be refused or allowed differently from the real crawler. Treat your test as strong evidence, not proof, and confirm against your server or CDN logs where the real bots show up.
Why can a page return 200 and still be empty to a crawler?
Because the status code only says the server answered. If your site renders its content in the browser with JavaScript, the crawler receives the shell and nothing else. We audited a shop whose homepage was 2,591 bytes to crawlers while looking normal to every human visitor. That case has its own write-up in the post on a single-page app that was invisible to AI crawlers, so I will not repeat it here.
The point for this post is that the body check in the test above is not optional. A per-bot status test that stops at 200 would have called that shop healthy. Status first, content second, always both.
Should I block training crawlers, citation crawlers, both or neither?
Decide by what you want, not by what feels safe. If you object to your pages being used for model training, block the training crawlers. If you want to be named in AI answers, keep the citation crawlers open. These two choices are independent and you can make them differently. Blocking everything is a coherent choice only if you are happy to be absent from AI answers.
A decision guide
- You want to be recommended in ChatGPT and Perplexity answers: allow OAI-SearchBot, ChatGPT-User, PerplexityBot and Perplexity-User. Whether you allow training crawlers is a separate call.
- You object to training use of your content but still want citations: disallow GPTBot and ClaudeBot, allow the four citation crawlers, then run the per-bot test to confirm the allow rules really let them through.
- You do not want any AI use of your pages: block both kinds, and accept that you will not be cited. Write that down as a decision so nobody later reports it as a bug.
- You publish paid or members-only content: blocking is reasonable, but put the gate in the firewall, since robots.txt only asks.
I will not tell you which of these is right for you. A publisher worried about its archive and a B2B service company that lives on inbound enquiries have different interests, and both are legitimate. What I will say is that most small companies we see want to be named in answers, and a surprising number of them are blocking the crawlers that would do it because a rule list told them to block anything with AI in the name.
What does blocking training crawlers actually cost me?
As far as the vendors document, it costs you your pages being used to train their models and nothing to do with the live answer. I cannot give you a number for what that is worth, and I would distrust anyone who does. It is a values and licensing question more than a traffic question, which is why I keep it separate from visibility in our scoring.
What does cost you is a blocked citation crawler. In that case the engine cannot fetch your page when a buyer asks, so it answers from other sources and your competitors get named. That is an access problem, it is cheap to fix, and it is the first thing we look at before recommending any other work.
When should you not pay anyone for this?
If your whole problem turns out to be one firewall rule or one line in a managed setting, you do not need an agency. You need an afternoon and the per-bot test described above. You can run it yourself with nothing more than a request tool and the user agent strings from the vendors' documentation.
Where an outside check earns its fee is when the picture is mixed: robots.txt says one thing, a managed CDN block says another, the firewall differs by country or request rate, and the pages are partly rendered in the browser. Untangling that across every crawler is tedious, and tedium is a fair thing to pay for. Our audit starts from $500, and if the finding is just a blocked crawler and one afternoon of work, we will say so and stop there.
What should you do this week?
Four steps, in this order, and none of them need a budget.
- Read your live robots.txt exactly as a crawler receives it, including any block your CDN adds in front of your own rules.
- Run the per-bot test on your homepage and your two or three most important pages, for each of the four citation crawlers, checking status and content.
- Check your firewall and bot-protection rules separately for anything that refuses these agents or challenges them.
- Write down your training policy as a separate decision, then set the training rules to match it without touching the citation rules.
I am telling you about the GPTBot mistake because the category is young and a lot of confident claims in it rest on one test of one bot. If you want a second pair of eyes, the audit at /geo tests each crawler individually and reports training bots separately, so a refusal you chose on purpose is not mistaken for a visibility problem. Ivan / 2pizza.team
We check this per bot in every AI visibility audit. See the GEO audit