Section

Research

Original studies on AI search, using data I collected.

UK news robots.txt, November 2026

Checked 2 November 2026. 0 of the 150 cells changed since 2 October 2026. Each of the UK’s ten biggest newsbrands gave the same answer to each of the 15 AI crawlers as it did a month earlier. I re-read the ten robots.txt files on 2 November 2026 and checked them...