Study plan, registered on 14 October 2026. This study has not been run yet. I’m publishing the method before any results exist so it can’t be adjusted to fit them. Results will be added to this page, with the date they were collected.
My crawler study reads what the UK’s ten biggest newsbrands ask AI crawlers to do. This plan applies the same check, with the same 15 crawlers and the same definition of blocked, to a second group that publishes a lot of text AI companies want: the 24 universities of the Russell Group. From 2 December 2026 I’ll read their robots.txt files on the 2nd of every month.
It’s a separate series. The news sample, its crawlers and its definition don’t change.
Sample (frozen)
The 24 members listed on the Russell Group’s Our universities page, read on 14 October 2026. Each domain is the website link on that university’s Russell Group member page, read the same day.
| University | Domain read |
|---|---|
| University of Birmingham | www.birmingham.ac.uk |
| University of Bristol | www.bristol.ac.uk |
| University of Cambridge | www.cam.ac.uk |
| Cardiff University | www.cardiff.ac.uk |
| Durham University | www.durham.ac.uk |
| University of Edinburgh | www.ed.ac.uk |
| University of Exeter | www.exeter.ac.uk |
| University of Glasgow | www.gla.ac.uk |
| Imperial College London | www.imperial.ac.uk |
| King’s College London | www.kcl.ac.uk |
| University of Leeds | www.leeds.ac.uk |
| University of Liverpool | www.liverpool.ac.uk |
| London School of Economics | www.lse.ac.uk |
| University of Manchester | www.manchester.ac.uk |
| Newcastle University | www.ncl.ac.uk |
| University of Nottingham | www.nottingham.ac.uk |
| University of Oxford | www.ox.ac.uk |
| Queen Mary University of London | www.qmul.ac.uk |
| Queen’s University Belfast | www.qub.ac.uk |
| University of Sheffield | www.sheffield.ac.uk |
| University of Southampton | www.southampton.ac.uk |
| University College London | www.ucl.ac.uk |
| University of Warwick | warwick.ac.uk |
| University of York | www.york.ac.uk |
The list stays frozen even if membership changes. If a university leaves or joins, that’s logged on this page with a date. A university that leaves stays in the sample, and a new one isn’t added.
Only the main domain’s robots.txt is read. Subdomains, such as a library catalogue or a repository, have their own files and are out of scope.
Crawlers and definition
These are the same 15 crawlers, with the same training and retrieval classes, as the method note. A crawler counts as blocked when the rules that apply to it disallow the whole site: Disallow: / (or /*) in the group that names it, or in the * group if nothing names it, and no Allow: / in those rules. Path-only rules, an empty Disallow: and a missing file (4xx) count as not blocked. If a file can’t be fetched (5xx or a timeout), it’s retried the same day. If it still fails, the gap is published and nothing is guessed.
Each snapshot has 24 × 15 = 360 cells: 216 training cells (24 × 9) and 144 retrieval cells (24 × 6).
What each snapshot records
- One row per university and crawler, with the same columns as the news CSV: scrape_date, brand, domain, bot, bot_class, blocked, source.
- A raw copy of every robots.txt file as read, so any cell can be checked against what the file said.
- Any redirect, such as from a bare domain to www, with the final URL that was read.
What gets compared
- Within the series: a change is a cell whose blocked value differs from the previous month’s. A trend needs three snapshots heading the same way, as in the news series.
- Against the news sample: same-month block rates side by side, such as the share blocking GPTBot and the training and retrieval rates. These are rates for two different groups. They don’t show why either group does what it does.
What I will not claim
- Why a university blocks or allows a crawler. The file records the rule, not the reason.
- That a block keeps a crawler out, or that it stops a university’s work turning up in AI answers. robots.txt is a request, and research papers are often copied to other sites the university doesn’t control.
- Anything about UK universities outside the Russell Group, or about subdomains.
- That one month’s change is a trend.
Schedule
- Registered: 14 October 2026.
- First snapshot: 2 December 2026, a month after the second news snapshot, so that one ships first.
- Then the 2nd of every month, in the same run as the news snapshot. If the 2nd is missed, the next day is used and the real date given.
Calls resting on this study
P17 and P18 on the predictions page.
Data
uk-university-ai-crawlers-YYYY-MM-DD.csv, one per snapshot, 360 rows, written by a script that uses the same classifier as the news series. Old files are never overwritten.