Research

These are experiments I’ve run myself on how AI tools and search engines behave: what they recommend, whether they date their facts, and what websites let them read. Each one starts with the answer in plain English, then shows exactly what I did, what it can’t tell you, and the full data as a download, so anyone can check it.

Every dataset on this site is published under CC BY 4.0. Reuse it for anything, including commercially, as long as you credit Jacob Whitmore and link to the study it came from. Licence applied 8 November 2026.

Every dataset is on the data page, with an explorer to search, filter and download it. Every headline figure was recomputed from those files on 10 November 2026: results and scripts.

New here? Start with four things I’ve found so far.

Studies

Newest first. Every study has a date, a stated sample, its method and a CSV. Filter by topic, or open any study for its limits.

Study Data collected 12 Nov 2026

Do the UK’s biggest shopping sites block AI crawlers?

Four of the ten block at least one AI crawler: Amazon 12 of 15, eBay 9, Vinted 5, Temu 1. Six block none. Together 27 of 150 cells, against 99 of 150 for the ten biggest newsbrands.

Sample
The ten retail services Ofcom ranks highest by UK audience, checked against the 15 crawlers from the news series. 150 cells.
Method
Each site’s robots.txt read in Chrome on 12 November 2026, using the same rules as the news series.

Study Data collected 10 Nov 2026

Do the UK’s biggest supermarkets block AI crawlers?

None of the ten blocks any of the 15 AI crawlers. The ten biggest newsbrands block 99 of the same 150 cells. Aldi names nine and allows them all.

Sample
The UK’s ten biggest supermarkets, checked against the 15 crawlers from the news series. 150 cells.
Method
Each supermarket’s robots.txt read on 10 November 2026, using the same rules as the news series.

Study Data collected 6 Nov 2026

Do AI companies block each other’s crawlers?

3 of 17 block rival AI crawlers, and all three are chat apps: chatgpt.com, claude.ai and grok.com. None of the 11 corporate sites blocks any.

Sample
17 readable AI company sites, checked against the 15 crawlers from the news series.
Method
Each site’s robots.txt read on 6 November 2026, using the same rules as the news series (RFC 9309).

Monthly series Since 2 Oct 2026 · latest 2 Nov 2026 · next 2 Dec 2026

Which AI crawlers do the UK’s biggest news sites block?

9 of 10 block at least one AI crawler. They block the bots that train AI models far more often than the bots that fetch pages for AI answers.

Sample
10 biggest UK newsbrands × 15 AI crawlers. 150 checks a month.
Method
Each site’s robots.txt is read and saved on the 2nd of every month. What counts as blocked is defined in the method note.

Registered studies

These studies have a published method and haven’t been run yet. Results will be added to each page with the date they were collected.

Several of these carry predictions with check dates. See them on the scoreboard.

How I run studies

  • Public data only, collected by me, with the date it was collected.
  • Method and limits written up alongside the findings.
  • Raw data published so anyone can check it.
  • Repeated on a schedule, so changes over time are visible.

The rules for the crawler series, including what counts as blocked, are in the method note. What the data can’t support is in What I will not claim.

Follow along on LinkedIn

I share new research, predictions and what I’m seeing in AI search on LinkedIn first.