Study plan, registered on 11 October 2026. This study has not been run yet. I’m publishing the method before any results exist so it can’t be adjusted to fit them. Results will be added to this page, with the date they were collected.
My crawler study read the robots.txt files of the UK’s ten biggest newsbrands on 2 October 2026. From 2 November 2026 I’ll re-read the same files on the 2nd of every month. This page sets out what each monthly snapshot records and how a change is counted, before the second snapshot exists.
What will be measured
The same ten sites, the same 15 crawlers and the same definition of blocked as the method note. Each snapshot gives 150 cells, one per site and crawler. A change is a cell whose blocked value differs from the previous month’s snapshot.
Two records are kept alongside the counts. The first is a copy of every robots.txt file as it was read, so any cell can be checked later against what the file said. The second is a dated copy of each crawler owner’s documentation page for the 15 crawlers, read on the same day. The training or retrieval class of each crawler comes from that documentation. If an owner changes what a crawler is for, the archive shows when, and the change goes in the method change log.
Crawler tokens that appear in a file but aren’t among the 15 are logged separately. They’re never added to the counts, because adding a crawler partway through would make the months impossible to compare.
Schedule
The 2nd of each month. If that day is missed, the snapshot is taken the next day and the real date is given. Nothing is backdated. Each month’s page and CSV stay up unchanged. Corrections go on a dated line on that month’s page.
Dated call
- Claim: the robots.txt policies at these ten sites will mostly hold for six months. Fewer than 15 of the 150 cells will differ between the 2 October 2026 and 2 April 2027 snapshots.
- Called: 11 October 2026.
- Check date: 2 April 2027 (the snapshot for that date).
- Test: compare the two snapshots cell by cell, using the definition of blocked in the method note.
- Hit if: 0 to 14 cells differ.
- Miss if: 15 or more cells differ.
- Void if: a site in the sample stops existing or a crawler token is retired, so the comparison can’t be made as defined.
- Baseline: the 150 cells published on 2 October 2026.
This call is P18 on the predictions page. If it misses, it stays up in red.
The call compares the two dates directly. A cell that changes and changes back in between doesn’t count towards it. The monthly pages will show those.
What I will not claim
- Why any file changed.
- That one month’s change is a trend. A trend needs at least three snapshots heading the same way.
- Whether any crawler obeys the files.
- Anything about sites outside the ten.
Data
One CSV per month, named uk-news-ai-crawlers-YYYY-MM-DD.csv, with the columns scrape_date, brand, domain, bot, bot_class, blocked and source. Each month also gets a change file listing every cell that differs from the previous month, with both values and links to both raw files. The October file stays at its current address.
Snapshots
- 2 October 2026: the baseline, in the crawler study.
- 2 November 2026: UK news robots.txt, November 2026.