How to read this studyTraining, retrieval, “blocked” and cells, in plain English
Three kinds of AI crawler
Training crawlers collect pages that may be used to train AI models. Blocking one asks for your pages to stay out of future training. It doesn’t remove anything already collected.
Search crawlers build an AI company’s own search index, the one its answers draw on. Blocking one can take your pages out of that company’s answers.
User-triggered fetchers read one page because a person asked the assistant to. Some companies say these may ignore robots.txt, because a person started the request.
The studies count search crawlers and fetchers together as retrieval, because both fetch pages to answer questions rather than to train models. The robots.txt generator splits them, because the companies say they treat them differently.
What “blocked” means
A crawler counts as blocked when the rules that apply to it disallow the whole site. That is Disallow: / in a group that names the crawler, or in the * group if nothing names it. Blocking a few folders doesn’t count. Why this definition.
What a cell is
One site checked against one crawler. The news, supermarket and retail studies each have 150 cells: ten sites times 15 crawlers.
Named group or the * group
A group that names a crawler replaces the * group for that crawler. So a file can block every AI crawler without naming any, and naming one can let it in. Example.
Control tokens
Google-Extended and Applebot-Extended aren’t crawlers. They’re names you put in robots.txt to say whether pages that Google’s and Apple’s normal crawlers collect may be used for their AI models.
What robots.txt can’t show
It’s a request, not a lock. It doesn’t stop a crawler that ignores it, and a block made on the server or firewall doesn’t appear in it. These studies count what the file says. More.
Every crawler these studies track, what it does, and who blocks it →
Checked 2 October 2026.
Latest snapshot: 2 November 2026. The findings below are from 2 October 2026 and stay as published.
Nine of the UK's ten biggest newsbrands block at least one AI crawler. But most of them block the bots that train AI models far more often than the bots that fetch pages when someone asks ChatGPT or Claude a question.
I checked the robots.txt file of each of the UK's ten biggest newsbrands on 2 October 2026 against 15 AI crawlers. Here's what they're actually blocking.
The headline numbers
- 9 of 10 block at least one AI crawler. The Independent is the exception: its robots.txt has no AI-specific rules at all.
- The Daily Mail blocks all 15 crawlers I checked.
- The Telegraph blocks 14 of the 15 AI crawlers I checked but explicitly allows Google-Extended, the token that controls whether Google can use its content for Gemini.
- Only 5 of 10 block Google-Extended. It's the least-blocked training token in the sample.
- ClaudeBot, PerplexityBot and CCBot are each blocked by 9 of 10.
Publishers block training more than answers
Each AI company runs more than one crawler. One collects pages to train models. Another builds a search index. A third fetches a page when a user asks about it. UK publishers treat them very differently.
| Company | Training crawler | Search crawler | User-triggered fetcher |
|---|---|---|---|
| OpenAI | GPTBot: 7 of 10 | OAI-SearchBot: 6 of 10 | ChatGPT-User: 4 of 10 |
| Anthropic | ClaudeBot: 9 of 10 | Claude-SearchBot: 5 of 10 | Claude-User: 4 of 10 |
| Perplexity | Not in study | PerplexityBot: 9 of 10 | Perplexity-User: 4 of 10 |
| Google-Extended: 5 of 10 | Not in study | Not in study |
See how your own file compares with these ten in the AI robots.txt generator.
Added 10 November 2026: the same check on the UK’s ten biggest supermarkets found none of them blocking any of the 15 crawlers. Read the supermarket study, or compare your own file with both groups in the AI crawler benchmark.
Four sites (Sky News, the BBC, the Mirror and the Daily Express) block ClaudeBot but leave Claude-SearchBot open. Only Sky News does the same with OpenAI: it blocks GPTBot but not OAI-SearchBot.
The pattern is the same across all three companies. Publishers are keener to stop their journalism training models than to stop it being fetched and cited in answers. That makes sense if the goal is to keep the referral traffic that AI answers can send while refusing free training data.
Two outliers
The Sun blocks almost everything by default and lets in a named list of crawlers. OpenAI's three crawlers are on that list, with some paths restricted. Every other AI crawler in the sample is blocked.
The Telegraph blocks every other AI crawler I checked, but adds a rule that specifically lets Google-Extended in. No other site in the sample does that.
All ten sites
The whole dataset in one picture
99 of 150 cells blocked. 9 of 10 UK newsbrands block at least one AI crawler.
| Site | GPTBot | ClaudeBot | anthropic-ai | Google-Extended | CCBot | Applebot-Extended | Bytespider | Meta-ExternalAgent | Amazonbot | OAI-SearchBot | ChatGPT-User | Claude-SearchBot | Claude-User | PerplexityBot | Perplexity-User | Blocked |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Daily Mail | Blocked | Blocked | Blocked | Blocked | Blocked | Blocked | Blocked | Blocked | Blocked | Blocked | Blocked | Blocked | Blocked | Blocked | Blocked | 15/15 |
| The Telegraph | Blocked | Blocked | Blocked | Not blocked | Blocked | Blocked | Blocked | Blocked | Blocked | Blocked | Blocked | Blocked | Blocked | Blocked | Blocked | 14/15 |
| BBC | Blocked | Blocked | Blocked | Blocked | Blocked | Blocked | Blocked | Blocked | Blocked | Blocked | Blocked | Not blocked | Not blocked | Blocked | Blocked | 13/15 |
| The Sun | Not blocked | Blocked | Blocked | Blocked | Blocked | Blocked | Blocked | Blocked | Blocked | Not blocked | Not blocked | Blocked | Blocked | Blocked | Blocked | 12/15 |
| Sky News | Blocked | Blocked | Blocked | Blocked | Blocked | Blocked | Blocked | Blocked | Blocked | Not blocked | Not blocked | Not blocked | Not blocked | Blocked | Not blocked | 10/15 |
| The Guardian | Not blocked | Blocked | Blocked | Not blocked | Blocked | Blocked | Blocked | Blocked | Blocked | Not blocked | Not blocked | Blocked | Blocked | Blocked | Not blocked | 10/15 |
| Yahoo! (UK news) | Blocked | Blocked | Blocked | Blocked | Blocked | Not blocked | Blocked | Not blocked | Not blocked | Blocked | Blocked | Blocked | Not blocked | Blocked | Not blocked | 10/15 |
| Mirror | Blocked | Blocked | Blocked | Not blocked | Blocked | Blocked | Not blocked | Blocked | Not blocked | Blocked | Not blocked | Not blocked | Not blocked | Blocked | Not blocked | 8/15 |
| Daily Express | Blocked | Blocked | Blocked | Not blocked | Blocked | Blocked | Not blocked | Not blocked | Not blocked | Blocked | Not blocked | Not blocked | Not blocked | Blocked | Not blocked | 7/15 |
| The Independent | Not blocked | Not blocked | Not blocked | Not blocked | Not blocked | Not blocked | Not blocked | Not blocked | Not blocked | Not blocked | Not blocked | Not blocked | Not blocked | Not blocked | Not blocked | 0/15 |
| Sites blocking | 7 | 9 | 9 | 5 | 9 | 8 | 7 | 7 | 6 | 6 | 4 | 5 | 4 | 9 | 4 | 99/150 |
| Site | AI crawlers blocked (of 15) | GPTBot | ClaudeBot | PerplexityBot | Google-Extended |
|---|---|---|---|---|---|
| Daily Mail | 15 | Yes | Yes | Yes | Yes |
| The Telegraph | 14 | Yes | Yes | Yes | No (explicitly allowed) |
| BBC | 13 | Yes | Yes | Yes | Yes |
| The Sun | 12 | No | Yes | Yes | Yes |
| Sky News | 10 | Yes | Yes | Yes | Yes |
| The Guardian | 10 | No | Yes | Yes | No |
| Yahoo! (UK news) | 10 | Yes | Yes | Yes | Yes |
| Mirror | 8 | Yes | Yes | Yes | No |
| Daily Express | 7 | Yes | Yes | Yes | No |
| The Independent | 0 | No | No | No | No |
The full data for all 15 crawlers is available as a CSV download.
Written about this data
- The Telegraph carve-out, as a decision
- Notes on one file: the Sun
- Training bot, search bot, fetcher
- What “blocked” should mean in a sentence
- robots.txt is a request
- How to block AI training crawlers but still let ChatGPT-User in
Method
- The sample is the ten biggest UK newsbrands by audience in August 2026, as listed in Press Gazette's monthly ranking using Ipsos iris data.
- Each robots.txt file was read on 2 October 2026.
- The files checked were at news.sky.com, theguardian.com, dailymail.com, independent.co.uk, bbc.co.uk, mirror.co.uk, uk.news.yahoo.com, thesun.co.uk, express.co.uk and telegraph.co.uk.
- The 15 crawlers were GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, anthropic-ai, Google-Extended, PerplexityBot, Perplexity-User, CCBot, Applebot-Extended, Bytespider, Meta-ExternalAgent and Amazonbot.
- A crawler counts as blocked when the rules that apply to it disallow the whole site (Disallow: /), either in a group that names it or through the catch-all group when it isn't named.
The standing method note for this series covers the sample, the definitions and the limits in one place; the definitions above follow it.
Limits
- robots.txt is a request, not a wall. It shows what a publisher asks for, not what crawlers do.
- Some sites block crawlers at the network level too. That doesn't show up here.
- Ten sites is a small sample. It covers the biggest audiences, not the whole UK news market.
- These files change. This is a snapshot from one day.
What happens next
I'll re-run this check on the 2nd of every month against the same ten sites and the same 15 crawlers, and keep every month's results. One of my public predictions rests on it.
Follow along on LinkedIn
I share new research, predictions and what I'm seeing in AI search on LinkedIn first.