Checked 2 November 2026.
0 of the 150 cells changed since 2 October 2026. Each of the UK’s ten biggest newsbrands gave the same answer to each of the 15 AI crawlers as it did a month earlier.
I re-read the ten robots.txt files on 2 November 2026 and checked them against the same 15 crawlers, using the definition of blocked in the method note. This is the second snapshot in the series set out in the study plan. The first is the 2 October study.
This month in numbers
- 9 of 10 sites block at least one of the 15 crawlers (9 of 10 on 2 October).
- Training crawlers: 67 of 90 cells blocked (67 on 2 October).
- Retrieval crawlers: 32 of 60 cells blocked (32 on 2 October).
All ten sites
| Site | Blocked on 2 November 2026 (of 15) | Blocked on 2 October 2026 (of 15) |
|---|---|---|
| Daily Mail | 15 | 15 |
| The Telegraph | 14 | 14 |
| BBC | 13 | 13 |
| The Sun | 12 | 12 |
| Sky News | 10 | 10 |
| The Guardian | 10 | 10 |
| Yahoo! (UK news) | 10 | 10 |
| Mirror | 8 | 8 |
| Daily Express | 7 | 7 |
| The Independent | 0 | 0 |
Every changed cell
None. Every one of the 150 cells has the same value it had on 2 October 2026. The change file has a header row and no other rows.
Crawler classes
No crawler changed class. I saved each owner’s crawler documentation page on the day of the snapshot (archive file), and each class on the method note still matches what its owner says the crawler is for. Two gaps are recorded in that file. I found no ByteDance documentation page for Bytespider, and Anthropic’s crawler page doesn’t name the anthropic-ai token. Both crawlers keep the class published on the method note, because there is no documentary evidence of a change.
Other crawler tokens
The ten files name 191 other user-agent tokens that aren’t among the 15 (320 site and token pairs). They’re listed in a separate file and left out of every count on this page, so the months stay comparable.
Predictions affected
None scored this month. P18 calls for fewer than 15 of the 150 cells to differ between 2 October 2026 and 2 April 2027. It is checked on the 2 April 2027 snapshot only, comparing those two dates directly.
What I will not claim
- That one month’s result is a trend. A trend needs at least three snapshots heading the same way.
- Why any file changed or stayed the same.
- Whether any crawler obeys these files.
- Anything about sites outside the ten.
Data
- uk-news-ai-crawlers-2026-11-02.csv: all 150 cells, with the columns scrape_date, brand, domain, bot, bot_class, blocked and source.
- uk-news-ai-crawlers-changes-2026-11-02.csv: every cell that differs from 2 October 2026 (0 rows).
- uk-news-ai-crawlers-other-tokens-2026-11-02.csv: tokens outside the 15.
- Raw robots.txt files as read on 2 November 2026: news.sky.com, theguardian.com, dailymail.com, independent.co.uk, bbc.co.uk, mirror.co.uk, uk.news.yahoo.com, thesun.co.uk, express.co.uk, telegraph.co.uk.
- crawler-docs-2026-11-02.txt: the crawler owners’ documentation pages as read on the same day.
- The 2 October 2026 snapshot stays at its original address.