Latest posts
-
BrightonSEO next week will be full of AI claims nobody can mark wrong
BrightonSEO is on 8 and 9 October. Here are the AI sessions I’ll be watching, and what would make each one checkable.
-
How to write a claim you can mark wrong
The four things a claim needs before it can be checked.
-
What I will not claim
Four claims this site won’t make, and why the data can’t support them.
-
Notes on one file: the Sun
The Sun’s robots.txt blocks by default and lets in a short list of named crawlers.
-
What “blocked” should mean in a sentence
Why the study counts a crawler as blocked only when the whole site is disallowed for it.
-
Training bot, search bot, fetcher
OpenAI runs three crawlers with different jobs, and UK news sites treat them differently.
-
The Telegraph carve-out, as a decision
The Telegraph blocks 14 of the 15 AI crawlers checked and explicitly allows Google-Extended.
-
robots.txt is a request
What a robots.txt file can and can’t tell you about AI crawlers.