Every AI lab publishes the instructions for blocking its own crawler. OpenAI’s crawler documentation tells site owners to add a Disallow rule for GPTBot, and a separate rule for OAI-SearchBot if they want the page out of ChatGPT’s answers. Anthropic’s support documentation tells you to block ClaudeBot and Claude-User while leaving Claude-SearchBot alone. Google’s documentation gives publishers a token called Google-Extended. The mechanics are well documented, and the vendors broadly agree on them.
What almost none of that documentation tells you is how much authority the file itself has. The specification those rules are written against says it in its first section: “These rules are not a form of access authorization” (RFC 9309). A robots.txt file is a request. It is a request that well-behaved crawlers honour, and it costs a crawler nothing to ignore. A site that writes a robots.txt block and stops there has asked nicely, and nothing more.
TL;DR
- robots.txt is not access control. The spec that defines it opens with: “These rules are not a form of access authorization” (RFC 9309).
- The labs split their crawlers by job, so you can refuse training and keep search visibility. OpenAI documents four tokens, Anthropic three, Perplexity two (OpenAI, Anthropic, Perplexity).
- The training token is the one to block. OpenAI states that “Disallowing GPTBot indicates a site’s content should not be used in training generative AI foundation models” (OpenAI).
- The user-triggered tokens are where robots.txt is weakest. OpenAI notes that for
ChatGPT-User, “robots.txt rules may not apply” because the fetch is initiated by a person (OpenAI). Perplexity’s documentation says its user agent generally ignores robots.txt (Perplexity). - A real block has to come from the server. Cloudflare’s AI Crawl Control turns a Block action into “a WAF custom rule on your zone to enforce that block” (Cloudflare).
- Blocking is forward-looking. Anthropic says blocking
ClaudeBotsignals that future site material should be excluded from future training datasets, and its documentation gives no procedure for withdrawing training data already collected (Anthropic). - Check the source address, not the user-agent string. OpenAI and Perplexity both publish the ranges their crawlers use as JSON files (OpenAI, Perplexity).
Know which bot is doing what
The most useful thing to know is that the labs split their crawlers by purpose, which is what lets you keep showing up in search while refusing to become training data.
OpenAI documents four tokens. GPTBot “may be used in training our generative AI foundation models”. OAI-SearchBot “is used to surface websites in search results in ChatGPT’s search features”. ChatGPT-User covers fetches a person triggers by asking ChatGPT a question, and is “not used for crawling the web in an automatic fashion”. OAI-AdsBot “only visits pages submitted as ads, and the data collected by OAI-AdsBot is not used to train generative AI foundation models” (OpenAI).
Anthropic documents three: ClaudeBot, which collects public web content that may contribute to generative-model training; Claude-User, which retrieves websites when someone asks Claude a question; and Claude-SearchBot, which crawls to improve the relevance and accuracy of search responses (Anthropic). Perplexity documents two: PerplexityBot, which is “not used to crawl content for AI foundation models”, and Perplexity-User, which handles user-initiated visits (Perplexity).
Google’s control works differently, because Google is already crawling the web for search. Google’s documentation describes Google-Extended as “a standalone product token that web publishers can use to manage whether content Google crawls…”, and sets out the scope: use of crawled content for training future Gemini models, powering Gemini Apps and the Vertex AI API for Gemini, and for grounding in Gemini Apps and Grounding with Google Search on Vertex AI. Google is explicit about the cost: “Google-Extended does not impact a site’s inclusion in Google Search nor is it used as a ranking signal in Google Search” (Google).
Cloudflare groups the same tokens into four categories in its bot reference: AI Crawler, AI Assistant, AI Search and Search Engine. That list also covers the operators the four labs above do not document directly, including Bytespider, CCBot, meta-externalagent, FacebookBot, Amazonbot, Applebot and MistralAI-User (Cloudflare).
What robots.txt can do, and where it stops
The format is simpler than its reputation. The rules live in a file named “/robots.txt” (all lowercase) in the top-level path of the service, served as UTF-8 text/plain (RFC 9309). Rules are grouped by user-agent token, token matching is case-insensitive, and matching groups get combined. Inside a group, path rules are matched against the URL: “The most specific match found MUST be used. The most specific match is the match that has the most octets.” That is what makes Disallow: /archive/ plus Allow: /archive/public/ give you the public half and nothing else. Two special characters matter for matching a path, * for “0 or more instances of any character” and $ to anchor the end of a path.
Two edge cases cause real damage. A 4xx response means the site is unavailable, and “the crawler MAY access any resources on the server”. A 5xx response means the site is unreachable, and “the crawler MUST assume complete disallow”. A rewrite rule that starts returning 404 quietly reopens the site, while a server that starts returning 500 closes it. Nobody wants either behaviour by accident.
And then the limit that matters most. The spec’s security section adds that the protocol “is not a substitute for valid content security measures”, and notes that listing paths in robots.txt publishes them to anyone who wants to look. Content that has to be genuinely private belongs behind authentication.
The block most people mean to write
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Google-CloudVertexBot
Disallow: /
User-agent: Bytespider
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: meta-externalagent
Disallow: /
User-agent: FacebookBot
Disallow: /Notice what is deliberately missing. OAI-SearchBot, Claude-SearchBot, PerplexityBot and Googlebot stay out of it, because those are the tokens that put you in search results. OpenAI is explicit that the two decisions are separate: “Please use OAI-SearchBot in robots.txt for managing Search opt outs and automatic crawl”, and blocking that bot means blocked sites “will not be shown in ChatGPT search answers, though can still appear as navigational links” (OpenAI).
The user-triggered tokens, ChatGPT-User and Claude-User, are absent for the opposite reason. As the next section explains, robots.txt is least reliable on exactly those fetches, so they need the network layer rather than a line of text.
Common Crawl publishes the same opt-out for its own crawler, CCBot, and adds a warning worth reading twice: “we are aware of crawlers falsely identifying themselves as CCBot”. Its advice is to confirm the source address rather than take the agent string at face value (Common Crawl). Anthropic says its bots respect “do not crawl” signals by honoring industry standard directives in robots.txt, and its documentation adds a non-standard rate control, Crawl-delay. It also advises against handling opt-outs by IP blocking, because that can stop a crawler reaching robots.txt at all (Anthropic).
Where the file is weakest
The user-initiated fetches. These are the requests that arrive when somebody asks a chatbot a question that happens to involve your page, and they are the traffic you most want to be able to refuse, because they are the least like a search engine and the most like a fetch you did not ask for.
OpenAI says plainly that for ChatGPT-User, “Because these actions are initiated by a user, robots.txt rules may not apply” (OpenAI). Anthropic documents Claude-User the same way, and Perplexity’s documentation says Perplexity-User generally ignores robots.txt when handling a user-requested fetch (Perplexity). For those three, the text file is a statement of intent, not a control. Only something in front of the page can act on it.
Make the server say no
This is the part that works, and it is no more complicated than the file it replaces.
If you already run Cloudflare, it ships the control. AI Crawl Control “enables you to take specific action for each AI crawler”, and the dashboard lists each crawler with an action of Allow, Block or Charge. Block is described as the setting that will “completely stop the AI crawler from scraping your webpage”, and Cloudflare explains how it is enforced: “When you block a crawler in AI Crawl Control, the system creates or updates a WAF custom rule on your zone to enforce that block.” The response code can be set to 403 Forbidden or 402 Payment Required on paid plans, and you can still choose to enforce robots.txt separately (Cloudflare). The Charge action is a third option, billing the crawler operator for each successful crawl, though Cloudflare describes it as a closed beta you have to sign up for.
If you run your own server, the same logic applies with the tools you already have. nginx can classify the request with a map block, which its documentation describes as creating “variables whose values depend on values of other variables”, and then refuse it. The map module matches the user-agent header with a regular expression, and the if directive accepts a variable name as its condition, where an empty string or “0” is false. return is allowed inside if, and nginx’s own documentation gives return 403 inside a condition as the worked example (nginx map module, nginx rewrite module).
map $http_user_agent $block_ai_crawler {
default 0;
"~*GPTBot" 1;
"~*ClaudeBot" 1;
"~*Google-Extended" 1;
"~*Google-CloudVertexBot" 1;
"~*Bytespider" 1;
"~*CCBot" 1;
"~*meta-externalagent" 1;
}
server {
# ... your existing configuration ...
if ($block_ai_crawler) {
return 403;
}
}The tokens to put in that list are in the Cloudflare bot reference, and it is worth adding the ones specific to your own traffic as you find them. If you self-host more than a chat UI, the rest of the setup is worth reading alongside this: a private AI home server primer.
One more lever, and it is a search lever, not a training lever. Google’s documentation describes noindex as “Do not show this page, media, or resource in search results”, and X-Robots-Tag as the HTTP header equivalent, which is the way to apply it to PDFs, images and other non-HTML resources. Both require the URL to stay crawlable, or the directive is never discovered (Google). This is worth knowing, but it is a Google Search control. It is not a training opt-out, and treating it as one is a mistake.
Verify by address, not by name
A user-agent string is a claim. It costs nothing to send a request with GPTBot in it and then do something else entirely, so a log line is evidence of intent rather than of identity.
Several operators publish where their crawlers come from. OpenAI serves one JSON file per crawler, with IPv4 and IPv6 prefixes: GPTBot, OAI-SearchBot, ChatGPT-User and OAI-AdsBot. Perplexity publishes PerplexityBot and Perplexity-User. Common Crawl publishes CCBot, and its own guidance is to confirm a request with a reverse DNS lookup on the source address.
The check is simple: take the source IP of the request, look it up in the published range for that crawler, and see whether it falls inside. If a request claims to be GPTBot and arrives from an address that is not in GPTBot’s published list, it is not OpenAI’s crawler, and you have found a spoof. OpenAI’s own guidance points the same way from the other side, recommending you allow requests from the published ranges for OAI-SearchBot.
Two more places to look. Your own access logs, where the token appears verbatim. And Cloudflare’s Directives tab, which shows “how AI crawlers interact with your robots.txt files across your hostnames” and surfaces violations by comparing “current robots.txt directives and past crawler requests” (Cloudflare).
One timing detail before you measure anything. OpenAI notes that “it can take ~24 hours from a site’s robots.txt update for our systems to adjust”, and Perplexity’s documentation gives the same figure for its crawler settings, up to 24 hours (OpenAI, Perplexity). Judge the change the next day, not the next minute.
What this means
The file and the wall are different instruments, and only one of them can say no. A robots.txt block is the polite request; a WAF rule is the wall. The direction of travel is clear, and it is not toward more files. Cloudflare now ships a per-crawler control that compiles down to a WAF rule, and publisher platforms have started using it, as Patreon’s move to block AI training crawlers at the network layer showed in July. A robots.txt block on its own is the older answer to the same question.
The token split is the reason any of this works at all. If one crawler did all three jobs, refusing training would mean refusing search and refusing the user-initiated fetch too, and the sensible choice would be to do nothing. Because the vendors separate training from search from on-demand retrieval, refusing training costs almost nothing in reach.
The ceiling, and it is a real one, is that blocking is forward-looking. Anthropic’s wording is about excluding future site material from future training datasets, and its documentation gives no procedure for withdrawing training data that has already been collected. Where your material is already in weights that exist, the only lever left is to keep it out of the next model.
And for material that has to stay private rather than merely un-harvested, the spec’s own security note points the way: real restriction is an application-layer control, and authentication is the one that holds.
The bottom line
Write the robots.txt block anyway. It costs nothing, it is what all of these vendors document, and it states the intent that a network rule will then enforce. Keep the search tokens out of it, because OAI-SearchBot, Claude-SearchBot and PerplexityBot are how you stay findable.
Then enforce it where the answer can be no. On Cloudflare that is a Block action in AI Crawl Control, which becomes a WAF custom rule returning 403. On your own server it is one map block and one if, as above.
Verify by address. A user-agent string proves nothing, and the published range for each crawler is what lets you check.
Then wait a day before you measure, and do not expect a retraction. Blocking controls what comes next, and nothing else.