Checked 30 October 2026
5 of 33 UK news and Russell Group university sites I could read publish an llms.txt file: the Daily Mail and four universities (Durham, Exeter, Imperial and Warwick). The other 28 return a 404, a 410 or a redirect to an unrelated page. UCL’s server returned an error both times, so it isn’t counted either way.
The Daily Mail’s file is the interesting one. It lists a set of its pages as open to AI agents, and its robots.txt has a matching carve-out inside the rules that block its AI crawlers from everything else.
What llms.txt is
llms.txt is a proposal by Jeremy Howard, first published on 3 September 2024 at llmstxt.org. It suggests a Markdown file at the root of a site that tells AI agents what the site is and which pages matter. The only required part is a top-level heading with the site’s name.
It’s a proposal, not a standard. robots.txt asks crawlers what not to fetch; llms.txt offers them a reading list. Whether any AI search product reads it is a separate question this study doesn’t test. For Google, there’s already a call on the board about its own guidance on llms.txt: P27.
The headline numbers
- 5 of 33 readable sites publish llms.txt.
- 1 of 10 news sites: the Daily Mail.
- 4 of 23 readable Russell Group universities: Durham, Exeter, Imperial, Warwick.
- 0 of the four universities’ files has a companion llms-full.txt that I could read (errors on two reads each). Not checked for the Daily Mail.
- 1 gap: UCL returned 502 Bad Gateway on two reads.
The results
| Group | Sites read | llms.txt found | Not found | Gap |
|---|---|---|---|---|
| UK’s ten biggest newsbrands | 10 | 1 | 9 | 0 |
| Russell Group (24) | 23 | 4 | 19 | 1 |
| All | 33 | 5 | 28 | 1 |
| Site | Result | First line of the file |
|---|---|---|
| Daily Mail (dailymail.co.uk, redirects to dailymail.com) | Found | # Daily Mail (after a dated comment line) |
| Durham University | Found | # Durham University |
| University of Exeter | Found | # University of Exeter |
| Imperial College London | Found | # Imperial College London |
| University of Warwick | Found | # University of Warwick - AI Access Guide |
| University of Glasgow | Redirect | Sent to the School of Law’s postgraduate (LLM) degrees page |
| Yahoo News UK | Redirect | Sent to the uk.yahoo.com homepage |
| The Independent | 410 Gone | |
| UCL | 502 twice | Not counted |
| Everyone else | 404 |
Glasgow’s redirect looks like a site search matching “llm” to its Master of Laws degrees. That’s my reading, not something the site says.
The Daily Mail: blocked, with a door
On 2 October 2026 the Daily Mail blocked all 15 AI crawlers in this site’s monthly sample, under the definition that a crawler is blocked when its rules disallow the whole site. That’s still true today. But the group of rules that names its AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended and dozens more) now reads, in order:
Allow: /*_.html
Allow: /llms.txt
Disallow: /Any URL ending in an underscore before .html is allowed. Everything else is disallowed. The llms.txt file, which says it was updated on 26 October 2026, lists pages under 13 section headings, in lifestyle, travel and buying-guide content, and says the URLs with that trailing underscore are unblocked for AI access and may be cited by AI assistants.
When did it change? I didn’t keep a raw copy of the Daily Mail’s file on 2 October, so I checked the Internet Archive:
| Archived copy | Allow: /*_.html | Allow: /llms.txt |
|---|---|---|
| 15 Sep 2026 | Yes | No |
| 2 Oct 2026 | Yes | No |
| 15 Oct 2026 | Yes | No |
| 23 Oct 2026 | Yes | Yes |
| Live, 30 Oct 2026 | Yes | Yes |
So the underscore carve-out was already there when I took the October snapshot. My definition of blocked doesn’t see it, because a path-level allow isn’t a whole-site allow. That’s the definition working as written, and it’s also its blind spot: “blocked” in that study means blocked from the whole site, and the Daily Mail has had a chosen set of exceptions since at least mid-September. The Allow: /llms.txt line arrived between the 15 and 23 October captures.
This is a pattern worth naming. A publisher can block AI crawlers by default and open a curated shelf of pages it wants cited: lifestyle, travel and buying guides, by the file’s own description. robots.txt does the blocking, the URL pattern marks the shelf, and llms.txt is the catalogue. I’ll report it in the November snapshot without changing the frozen definition, since changing it mid-series is on this site’s ban list. The Telegraph carve-out was the first case of a newsbrand writing its AI rules as a decision rather than a blanket. This is the second.
The universities: an open door and a reading list
All four universities with an llms.txt file name no AI crawler in their robots.txt, and none has a whole-site disallow in its catch-all group. So their llms.txt files sit alongside open access. Warwick calls its file an “AI Access Guide” for key user journeys. Imperial’s file was read twice on the day through two different tools and looked different each time (once opening with comment lines, once with the H1), so the version served may vary.
I’ll read the same 24 universities’ robots.txt files from 2 December under the registered university study. That will show whether llms.txt adoption lines up with what each university asks AI crawlers to do.
Should you publish one?
What this data supports:
- Most large UK publishers haven’t. 28 of 33 haven’t, and that includes the BBC, Guardian, Oxford and Cambridge.
- A site that blocks AI crawlers and publishes llms.txt is sending two messages. The Daily Mail resolves that with explicit allow rules. Without them, a crawler that obeys robots.txt can’t fetch the pages your llms.txt points to.
- It costs little to publish. It has no confirmed effect on citations that I’ve found.
What it doesn’t support: any claim that llms.txt gets you cited. This study counts files; it doesn’t test whether anything reads them.
Method
On 30 October 2026 I requested https://<domain>/llms.txt for the same frozen samples this site’s robots.txt series uses: the ten newsbrands in the method note and the 24 universities in the university study plan. Each was opened in a normal Chrome browser and the HTTP status code, final URL after redirects, content type and first line were recorded. A file counts as found when the final response is a 200 with a plain-text body that’s recognisably an llms.txt file. A 4xx, or a redirect to an HTML page, counts as not found. A 5xx is retried; if it fails again, it’s a gap. Query strings added by redirects weren’t recorded. The four universities’ robots.txt files and the Daily Mail’s live file were read the same day. Archive copies are from the Internet Archive’s Wayback Machine.
Limits
- One day. Files appear and change; the Daily Mail edited its own four days before this check.
- Root only. A file elsewhere on a site, or on a subdomain, isn’t counted.
- This counts files, not effects. Nothing here shows any AI system reads them.
- The news sample uses the domains in the method note. The Daily Mail redirects from .co.uk to .com.
- Thirty-three sites in two sectors. Not a measure of UK websites in general.
What happens next
I’ll add llms.txt to the monthly run alongside robots.txt for both samples, as a separate column, from the December snapshot. It won’t change the frozen robots.txt definition.
P36, called 30 October 2026: On 2 April 2027, no more than 3 of the 10 newsbrands in the sample will publish an llms.txt file. Right if 0–3 do. Wrong if 4 or more do.
Download
CSV: 34 sites, status code, final URL, content type, first line. One row per site. Free to cite with a link.
Related: Which UK news sites block GPTBot and other AI crawlers? · The Telegraph carve-out, as a decision · robots.txt is a request · What text do machines actually read on a page?
Sources
Opened on 30 October 2026.
- llmstxt.org: The /llms.txt file
- Daily Mail robots.txt (live) and llms.txt (live)
- Internet Archive copies of the Daily Mail’s robots.txt, linked in the table above
- Every other URL requested is in the CSV