The crawler studies on this site check the same 15 AI crawlers. This page says what each one does, in its company’s own words, and how many sites in each study block it. The counts are read from the published data every time the page loads, so they can’t drift from the studies.
Want to act on it? The AI robots.txt generator builds a file from these crawlers, and can start from any site in the three studies.
Three kinds of AI crawler
Training crawlers collect pages that may be used to train AI models. Blocking one asks for your pages to stay out of future training. It doesn’t remove anything already collected.
Search crawlers build an AI company’s own search index, the one its answers draw on. Blocking one can take your pages out of that company’s answers.
User-triggered fetchers read one page because a person asked the assistant to. Some companies say these may ignore robots.txt, because a person started the request.
The studies count search crawlers and fetchers together as retrieval, because both fetch pages to answer questions rather than to train models. The robots.txt generator splits them, because the companies say they treat them differently.
What “blocked” means
A crawler counts as blocked when the rules that apply to it disallow the whole site. That is Disallow: / in a group that names the crawler, or in the * group if nothing names it. Blocking a few folders doesn’t count. Why this definition.
What a cell is
One site checked against one crawler. The news, supermarket and retail studies each have 150 cells: ten sites times 15 crawlers.
Named group or the * group
A group that names a crawler replaces the * group for that crawler. So a file can block every AI crawler without naming any, and naming one can let it in. Example.
Control tokens
Google-Extended and Applebot-Extended aren’t crawlers. They’re names you put in robots.txt to say whether pages that Google’s and Apple’s normal crawlers collect may be used for their AI models.
What robots.txt can’t show
It’s a request, not a lock. It doesn’t stop a crawler that ignores it, and a block made on the server or firewall doesn’t appear in it. These studies count what the file says. More.
| Name in robots.txt | Company | What it does | Kind | robots.txt | UK newsbrands2 Nov 2026 | UK supermarkets10 Nov 2026 | UK retail sites12 Nov 2026 |
|---|---|---|---|---|---|---|---|
GPTBot | OpenAI | Crawls pages that may be used to train OpenAI’s foundation models. Source | Training | Says it follows robots.txt | 7/10 | 0/10 | 3/10 |
OAI-SearchBot | OpenAI | Indexes pages for ChatGPT search. OpenAI says opted-out sites will not be shown in ChatGPT search answers. Source | Retrieval Search index | Says it follows robots.txt | 6/10 | 0/10 | 1/10 |
ChatGPT-User | OpenAI | Fetches a page when a ChatGPT user’s action needs it. OpenAI says robots.txt rules may not apply, because a user started the request. Source | Retrieval User fetch | Says it may not | 4/10 | 0/10 | 1/10 |
ClaudeBot | Anthropic | Collects pages that could be used to train Anthropic’s models. Source | Training | Says it follows robots.txt | 9/10 | 0/10 | 3/10 |
Claude-SearchBot | Anthropic | Crawls to improve Claude’s search results. Anthropic says blocking it stops your pages being indexed for search. Source | Retrieval Search index | Says it follows robots.txt | 5/10 | 0/10 | 1/10 |
Claude-User | Anthropic | Fetches a page when a Claude user’s question needs it. Source | Retrieval User fetch | Says it follows robots.txt | 4/10 | 0/10 | 1/10 |
anthropic-ai | Anthropic | Older Anthropic name. Not listed in Anthropic’s crawler article when checked. Many publishers still name it. Source | Training | No statement found | 9/10 | 0/10 | 1/10 |
Google-ExtendedControl token | A control token, not a crawler. Covers Gemini training and grounding. Google says it has no effect on Google Search inclusion or ranking. Source | Training | Says it follows robots.txt | 5/10 | 0/10 | 2/10 | |
PerplexityBot | Perplexity | Surfaces and links pages in Perplexity search results. Perplexity says it is not used for training. Source | Retrieval Search index | Says it follows robots.txt | 9/10 | 0/10 | 2/10 |
Perplexity-User | Perplexity | Visits a page when a Perplexity user asks a question. Perplexity says it generally ignores robots.txt, because a user requested the fetch. Source | Retrieval User fetch | Says it generally ignores it | 4/10 | 0/10 | 1/10 |
CCBot | Common Crawl | Builds Common Crawl’s open web archive, which anyone can download. Source | Training | Says it follows robots.txt | 9/10 | 0/10 | 3/10 |
Applebot-ExtendedControl token | Apple | A control token, not a crawler. Covers use of Applebot’s data to train Apple’s models. Apple says blocked pages can still appear in its search results. Source | Training | Says it follows robots.txt | 8/10 | 0/10 | 1/10 |
Bytespider | ByteDance | ByteDance’s crawler. No ByteDance documentation found when checked, so any claim about it is unverified. | Training | No statement found | 7/10 | 0/10 | 3/10 |
Meta-ExternalAgent | Meta | Crawls to train AI models or improve products by indexing content. Source | Training | Says it follows robots.txt | 7/10 | 0/10 | 2/10 |
Amazonbot | Amazon | Improves Amazon products. Amazon says it may be used to train Amazon AI models. Source | Training | Says it follows robots.txt | 6/10 | 0/10 | 2/10 |
Meta-ExternalFetcher | Meta | Fetches individual links when a user asks. Meta says it may bypass robots.txt. Source | Not tracked User fetch | Says it may not | Not checked | Not checked | Not checked |
Amzn-SearchBot | Amazon | Improves search in Amazon products such as Alexa. Amazon says it is not used for training. Source | Not tracked Search index | Says it follows robots.txt | Not checked | Not checked | Not checked |
Amzn-User | Amazon | Fetches live pages to answer a user’s request, for example in Alexa. Amazon says it may not follow all robots.txt directives. Source | Not tracked User fetch | Says it may not | Not checked | Not checked | Not checked |
| Blocked cells, all 15 crawlers | 99/150 | 0/150 | 27/150 | ||||