Latest posts
-
Study plan: do news sites enforce their robots.txt?
A registered method for checking whether the servers of ten UK news sites act on what their robots.txt files say.
-
Study plan: do news sites that block AI training still appear in AI answers?
A registered method and a fixed list of 30 UK news queries for checking which newsbrands appear in AI answers.
-
Study plan: the Telegraph and the Daily Mail in AI answers
A registered comparison of two papers with almost identical robots.txt files and one difference.
-
Study plan: link, mention or nothing
A registered method for counting how often AI answers to UK news questions link to a news source.
-
Study plan: established newsbrands and lower-credibility sites
A registered method for comparing AI crawler block rates between established UK newsbrands and high-reach sites that fail a set credibility test.
-
Which UK news sites block GPTBot and other AI crawlers?
Nine of the UK’s ten biggest newsbrands block at least one AI crawler, but they block training bots far more often than the bots that fetch pages for AI answers.