by Jacob Whitmore | 2 October 2026 | Research
Study plan, registered on 2 October 2026. This study has not been run yet. I’m publishing the method before any results exist so it can’t be adjusted to fit them. Results will be added to this page, with the date they were collected. On 2 October 2026, 7... by Jacob Whitmore | 2 October 2026 | Research
Study plan, registered on 2 October 2026. This study has not been run yet. I’m publishing the method before any results exist so it can’t be adjusted to fit them. Results will be added to this page, with the date they were collected. My October crawler... by Jacob Whitmore | 2 October 2026 | Commentary
In my October study, nine of the ten robots.txt files had the same basic structure. Crawlers were allowed in by default, and the file named the ones the site wanted to keep out. The Sun’s file, which I read on 2 October 2026, worked the other way round. Blocked... by Jacob Whitmore | 2 October 2026 | Commentary
This site publishes research about AI crawlers and search. There are some claims I won’t make, because my data can’t support them. I’m listing them here so readers can hold me to it. “The robots.txt block cost them traffic” A robots.txt... by Jacob Whitmore | 2 October 2026 | Predictions
A lot of commentary about AI and search can’t turn out to be wrong. “AI search is growing.” “Publishers will need to adapt.” Nothing that happens next year could show either sentence was false, which also means neither tells you much.... by Jacob Whitmore | 2 October 2026 | Commentary
My October study counted what the UK’s ten biggest news sites ask AI crawlers to do. Its limits section made one point I think deserves a post of its own: robots.txt is a request. It isn’t a wall. What the file shows A robots.txt file sets out a... by Jacob Whitmore | 2 October 2026 | Commentary
On 2 October 2026 the Telegraph’s robots.txt blocked 14 of the 15 AI crawlers I checked. The fifteenth was Google-Extended, the token Google uses to let sites opt out of having their content used for its AI models. The Telegraph didn’t simply leave... by Jacob Whitmore | 2 October 2026 | Commentary
Most large AI companies run several crawlers, and each one does a different job. A lot of confusion about “blocking AI” comes from people meaning different crawlers without saying so. OpenAI is a useful example because it runs three. GPTBot collects pages... by Jacob Whitmore | 2 October 2026 | Commentary
In my October study, a crawler counts as blocked when the rules that apply to it disallow the whole site. That means Disallow: / in a group that names the crawler, or in the catch-all group if nothing names it. This post explains why I think a definition short enough... by Jacob Whitmore | 2 October 2026 | Research
Checked 2 October 2026. Latest snapshot: 2 November 2026. The findings below are from 2 October 2026 and stay as published. Nine of the UK’s ten biggest newsbrands block at least one AI crawler. But most of them block the bots that train AI models far more often...