Latest posts
-
The Telegraph carve-out, as a decision
The Telegraph blocks 14 of the 15 AI crawlers checked and explicitly allows Google-Extended.
-
Training bot, search bot, fetcher
OpenAI runs three crawlers with different jobs, and UK news sites treat them differently.
-
What “blocked” should mean in a sentence
Why the study counts a crawler as blocked only when the whole site is disallowed for it.
-
Which UK news sites block GPTBot and other AI crawlers?
Nine of the UK’s ten biggest newsbrands block at least one AI crawler, but they block training bots far more often than the bots that fetch pages for AI answers.