Archive

Blog

Notes on one file: the Sun

In my October study, nine of the ten robots.txt files had the same basic structure. Crawlers were allowed in by default, and the file named the ones the site wanted to keep out. The Sun’s file, which I read on 2 October 2026, worked the other way round. Blocked...

What I will not claim

This site publishes research about AI crawlers and search. There are some claims I won’t make, because my data can’t support them. I’m listing them here so readers can hold me to it. “The robots.txt block cost them traffic” A robots.txt...

How to write a claim you can mark wrong

A lot of commentary about AI and search can’t turn out to be wrong. “AI search is growing.” “Publishers will need to adapt.” Nothing that happens next year could show either sentence was false, which also means neither tells you much....

robots.txt is a request

My October study counted what the UK’s ten biggest news sites ask AI crawlers to do. Its limits section made one point I think deserves a post of its own: robots.txt is a request. It isn’t a wall. What the file shows A robots.txt file sets out a...

The Telegraph carve-out, as a decision

On 2 October 2026 the Telegraph’s robots.txt blocked 14 of the 15 AI crawlers I checked. The fifteenth was Google-Extended, the token Google uses to let sites opt out of having their content used for its AI models. The Telegraph didn’t simply leave...

Training bot, search bot, fetcher

Most large AI companies run several crawlers, and each one does a different job. A lot of confusion about “blocking AI” comes from people meaning different crawlers without saying so. OpenAI is a useful example because it runs three. GPTBot collects pages...

What “blocked” should mean in a sentence

In my October study, a crawler counts as blocked when the rules that apply to it disallow the whole site. That means Disallow: / in a group that names the crawler, or in the catch-all group if nothing names it. This post explains why I think a definition short enough...