Written 14 October 2026.
Name the training crawlers in robots.txt with Disallow: / and leave the user-triggered fetchers unnamed. That’s the split most of the UK’s biggest news sites chose.
What I measured
On 2 October 2026 I read the robots.txt files of the UK’s ten biggest newsbrands and checked 15 AI crawlers in each. 9 of the 10 blocked ClaudeBot, Anthropic’s training crawler. 5 of those 9 left Claude-User, the fetcher that reads a page when someone asks, unblocked: the BBC, Sky News, the Mirror, the Daily Express and Yahoo! (UK news).
OpenAI showed the same pattern. GPTBot was blocked by 7 of the 10, ChatGPT-User by 4. Each of the three user-triggered fetchers (ChatGPT-User, Claude-User, Perplexity-User) was blocked by 4.
Full table and CSV: Which UK news sites block GPTBot and other AI crawlers?
The robots.txt
Name the four training tokens in one group:
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Google-Extended
Disallow: /Don’t name ChatGPT-User, Claude-User or Perplexity-User. A crawler that isn’t named follows your User-agent: * group, so if that group allows the site, they can still fetch it.
Two things to check before you copy it. If your * group has Disallow: /, the fetchers are blocked too, because they fall back to it. And if you later give a fetcher a group of its own, only that group applies to it (naming a crawler can unblock it).
The difference between the three kinds of crawler is in Training bot, search bot, fetcher.
Check yours
Open your robots.txt and test each of the 15 crawlers in the method note. A crawler counts as blocked when the group that names it, or the * group if nothing names it, has Disallow: / and no Allow: /.
What I will not claim
- That this keeps your pages out of training. robots.txt is a request, not a wall, and it doesn’t cover text collected before the rule existed.
- That leaving the fetchers open gets you cited or sends traffic. I didn’t test that.
- That one day’s files are the publishers’ policy. They were read on 2 October 2026 and can change without notice.
This is hack 1 on the search hacks page.