These are experiments I’ve run myself on how AI tools and search engines behave: what they recommend, whether they date their facts, and what websites let them read. Each one starts with the answer in plain English, then shows exactly what I did, what it can’t tell you, and the full data as a download, so anyone can check it.
Every dataset on this site is published under CC BY 4.0. Reuse it for anything, including commercially, as long as you credit Jacob Whitmore and link to the study it came from. Licence applied 8 November 2026.
Every dataset is on the data page, with an explorer to search, filter and download it. Every headline figure was recomputed from those files on 10 November 2026: results and scripts.
New here? Start with four things I’ve found so far.
Studies
Newest first. Every study has a date, a stated sample, its method and a CSV. Filter by topic, or open any study for its limits.
Do the UK’s biggest shopping sites block AI crawlers?
Four of the ten block at least one AI crawler: Amazon 12 of 15, eBay 9, Vinted 5, Temu 1. Six block none. Together 27 of 150 cells, against 99 of 150 for the ten biggest newsbrands.
- Sample
- The ten retail services Ofcom ranks highest by UK audience, checked against the 15 crawlers from the news series. 150 cells.
- Method
- Each site’s robots.txt read in Chrome on 12 November 2026, using the same rules as the news series.
Do the UK’s biggest supermarkets block AI crawlers?
None of the ten blocks any of the 15 AI crawlers. The ten biggest newsbrands block 99 of the same 150 cells. Aldi names nine and allows them all.
- Sample
- The UK’s ten biggest supermarkets, checked against the 15 crawlers from the news series. 150 cells.
- Method
- Each supermarket’s robots.txt read on 10 November 2026, using the same rules as the news series.
How fast did AI answers catch up with a Bank Rate change?
None of the 8 answers was stale. My prediction (P37) said at least one would be. It was wrong, and it stays on the board in red.
- Sample
- 8 answers: 4 AI engines, 2 chats each, 24 hours after the decision.
- Method
- Run 12:02 to 12:07 UK on 6 November 2026, against a test written down on 11 October.
Do AI companies block each other’s crawlers?
3 of 17 block rival AI crawlers, and all three are chat apps: chatgpt.com, claude.ai and grok.com. None of the 11 corporate sites blocks any.
- Sample
- 17 readable AI company sites, checked against the 15 crawlers from the news series.
- Method
- Each site’s robots.txt read on 6 November 2026, using the same rules as the news series (RFC 9309).
Which AI crawlers do the UK’s biggest news sites block?
9 of 10 block at least one AI crawler. They block the bots that train AI models far more often than the bots that fetch pages for AI answers.
- Sample
- 10 biggest UK newsbrands × 15 AI crawlers. 150 checks a month.
- Method
- Each site’s robots.txt is read and saved on the 2nd of every month. What counts as blocked is defined in the method note.
What do AI visibility trackers say about how they measure?
None of the 13 publishes evidence that its numbers are accurate, and none says whether it collects answers signed in or signed out.
- Sample
- 13 AI visibility tracking tools, 9 questions each.
- Method
- Each tool’s own public pages checked against nine questions about how it collects and calculates its numbers.
Do backlinks get you cited in AI answers?
Nobody has published a controlled test. Every study is a correlation, and none of the 16 publishes raw citation data.
- Sample
- 16 published studies.
- Method
- Each study read for what it actually measured, its sample, and whether its data is published.
Do UK news sites and universities publish llms.txt?
5 of 33 publish llms.txt. The Daily Mail’s lists the pages its robots.txt carves out for AI crawlers.
- Sample
- 33 readable sites: 10 UK newsbrands and 23 Russell Group universities.
- Method
- Each domain root checked for an llms.txt file.
Do the AI search statistics everyone repeats hold up?
None of the 32 passes all five checks, and none is about the UK.
- Sample
- 32 statistics that appear on at least 3 of 20 roundup pages.
- Method
- Each statistic traced to the document it came from, then checked for five things: when, how many, where, how, and whether the data is published.
What text do machines actually read on a page?
Body text, headings and tables survived in every tool. The meta description and JSON-LD were dropped by every tool.
- Sample
- 4 HTML-to-text tools, 14 places on one test page.
- Method
- A marker word placed in 14 parts of a page, then searched for in each tool’s output.
Do AI answers say when a “current” figure is from?
7 of 40 answers didn’t say when the figure was from. All 7 came from Google’s two engines.
- Sample
- 40 answers: 10 UK figure questions on 4 AI engines.
- Method
- Every question asked for a figure “now” or “today”, and every figure has an official source that publishes it with a date.
Does ChatGPT recommend different brands to different customers?
Three of the four finance-director answers recommended none of the products the no-persona answers named.
- Sample
- 12 answers: 3 prompts, 2 runs each, on ChatGPT and Gemini.
- Method
- The same question asked with no persona, as a self-employed plumber, and as a finance director of a 250-person manufacturer.
How much is the purple pound worth, according to AI?
Both AI engines that answered gave £274bn, and neither said it was a 2020 estimate.
- Method
- Published estimates traced to their source and year, then compared with what AI engines said.
Registered studies
These studies have a published method and haven’t been run yet. Results will be added to each page with the date they were collected.
Crawlers and robots.txt
Who gets credited
Dates, figures and personas
Several of these carry predictions with check dates. See them on the scoreboard.
How I run studies
- Public data only, collected by me, with the date it was collected.
- Method and limits written up alongside the findings.
- Raw data published so anyone can check it.
- Repeated on a schedule, so changes over time are visible.
The rules for the crawler series, including what counts as blocked, are in the method note. What the data can’t support is in What I will not claim.
Follow along on LinkedIn
I share new research, predictions and what I’m seeing in AI search on LinkedIn first.