This is the standing method note for the AI crawler series. Each study links here instead of restating the rules, and the rules don't widen between studies.
The sample
The ten biggest UK newsbrands by audience in August 2026, as listed in Press Gazette's monthly ranking using Ipsos iris data: news.sky.com, theguardian.com, dailymail.com, independent.co.uk, bbc.co.uk, mirror.co.uk, uk.news.yahoo.com, thesun.co.uk, express.co.uk and telegraph.co.uk. The sample is frozen: re-runs check these same ten sites even if the ranking changes, so a change in the numbers means a publisher changed its file, not that the sample changed under it.
The snapshot
Each study reads every robots.txt file on a single stated date. Results are a snapshot of that day, because these files change.
What “blocked” means
A crawler counts as blocked when the rules that apply to it disallow the whole site (Disallow: /), either in a group that names it or through the catch-all group when it isn't named. An explicit Allow: / for that crawler overrides. Anything short of a whole-site disallow counts as not blocked, even when some paths are restricted.
The crawlers
This series checks 15 crawlers: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, anthropic-ai, Google-Extended, PerplexityBot, Perplexity-User, CCBot, Applebot-Extended, Bytespider, Meta-ExternalAgent and Amazonbot.
| Crawler | Owner | Class |
|---|---|---|
| GPTBot | OpenAI | training |
| OAI-SearchBot | OpenAI | retrieval (search index) |
| ChatGPT-User | OpenAI | retrieval (user-triggered fetch) |
| ClaudeBot | Anthropic | training |
| Claude-SearchBot | Anthropic | retrieval (search index) |
| Claude-User | Anthropic | retrieval (user-triggered fetch) |
| anthropic-ai | Anthropic | training (older token) |
| Google-Extended | training | |
| PerplexityBot | Perplexity | retrieval (search index) |
| Perplexity-User | Perplexity | retrieval (user-triggered fetch) |
| CCBot | Common Crawl | training (open dataset used to train models) |
| Applebot-Extended | Apple | training |
| Bytespider | ByteDance | training |
| Meta-ExternalAgent | Meta | training |
| Amazonbot | Amazon | training |
Totals: 9 training, 6 retrieval. Each class comes from the owner’s crawler documentation and is rechecked at every snapshot. If an owner changes what a crawler is for, its class changes from that date and the change is logged below.
What robots.txt does not show
- robots.txt is a request, not a wall. It shows what a publisher asks for, not what crawlers do.
- Network-level blocks don't show up in it. A site can allow a crawler in robots.txt and still turn it away at the door.
- It says nothing about whether any crawler actually obeys.
What this site will not claim
- Whether any crawler respects these files.
- Whether blocking or allowing a crawler caused any change in a site's traffic or citations.
- Anything about sites or crawlers outside the stated sample.
- Any figure that doesn't trace back to a published dataset.
- Whether a site has opted out of AI Overviews or AI Mode. Since 31 August 2026 that has been a Search Console setting (Google’s Search generative AI control), not a robots.txt rule, so robots.txt can’t show it.
Sources and tools
The sources and tools behind each study are listed on the What I use page.
AI answer runs
Studies that ask AI engines a question, such as the purple pound and personas posts, follow these rules:
- A fresh session for every run, with memory or personalisation off where the engine allows it. No follow-up questions.
- The prompt pasted word for word, and published with the results.
- Run from the UK, with the date, time window and the model or mode shown on screen recorded.
- Each prompt run at least twice per engine on the same day.
- Any run that is excluded stays in the CSV, with the reason.
- An engine that can’t be run is listed as not run, with the reason. It isn’t filled in from another session.
Follow-up messages. When a study includes a follow-up message, the follow-up is published word for word and pasted into the same chat straight after the first answer, with nothing else added. Both answers are recorded. A study that tests a follow-up includes a control arm with a neutral follow-up, so any effect is measured against it and not against the first answer alone.
Other robots.txt cohorts
The same 15 crawlers, classes and definition of blocked are applied to a second, separate sample: the 24 Russell Group universities, frozen from russellgroup.ac.uk on 14 October 2026 (study plan). Snapshots run on the 2nd of each month from 2 December 2026. Each cohort has its own CSV. Counts from the two cohorts are shown side by side and never pooled.
Checking how my own findings are repeated
Some studies check how AI engines repeat findings first published on this site (study plan). The findings checked are frozen in each study plan, exactly as published, with the date each was measured. An answer counts as crediting the site only if it links to or names jacobwhitmore.com, or names Jacob Whitmore. Exact-phrase searches are only trusted on a day when two control searches pass: a phrase known to be on this site is found, and a made-up string isn't.
This site’s own robots.txt
Set 12 November 2026. The file at /robots.txt lets every crawler read every public page. The only blocked path is the WordPress admin.
It carries one Content-Signal line: Content-Signal: search=yes, ai-input=yes, ai-train=yes. That says yes to three uses: showing a page in search results, an AI tool reading a page to answer someone’s question, and training a model.
Why yes to all three: the research here is published to be found, quoted and checked, and the datasets are already under CC BY 4.0. Blocking training crawlers would work against that.
Two limits. The line is a stated preference and does nothing to stop a crawler that ignores it. I haven’t tested whether any crawler acts on it. How the line works.
The file names no crawler one by one because none is blocked. If the policy changes, the new date and the reason will be written here.
Method change log
| Date | Change |
|---|---|
| 2 Oct 2026 | First crawler snapshot. 10 sites, 15 crawlers, definition of blocked as above. |
| 10 Oct 2026 | Crawler classes, the AI Overviews limit and the AI answer run rules written onto this page. No change to the sample, the crawlers or the definition. |
| 14 Oct 2026 | New robots.txt cohort registered: the 24 Russell Group universities, with the same 15 crawlers and definition. The news series is unchanged. |
| 14 Oct 2026 | AI answer runs: rule added for follow-up messages and control arms. |
| 14 Oct 2026 | Rule added for checking how this site's own findings are repeated: frozen findings list, credit definition and exact-phrase controls. |
Raw data
Every study in the series publishes its full data as a CSV download, so anyone can check the findings or re-run the comparison.
Data licence
Every dataset on this site is published under CC BY 4.0. Reuse it for anything, including commercially, as long as you credit Jacob Whitmore and link to the study it came from. Licence applied 8 November 2026.
Every file is listed on the data page, with an explorer to search and download it. Every headline figure was recomputed from those files on 10 November 2026: results and scripts.