This is the home of my recurring research into how AI search treats UK websites and businesses. Every study has a date, a fixed sample, a method anyone can repeat and a data file. Results are given as counts with their denominators.
What I observed, what I estimate and what I think are kept apart. A study page reports what was read on a stated date. Forecasts go on the prediction ledger with a check date, and misses stay up.
Latest results
| Sector | Sample | Read | Site and crawler pairs blocked | Sites blocking at least one | Study |
|---|---|---|---|---|---|
| News | 10 UK newsbrands, 15 AI crawlers | 2 Nov 2026 | 99 of 150 | 9 of 10 | November snapshot |
| Supermarkets | 10 UK supermarkets, 15 AI crawlers | 10 Nov 2026 | 0 of 150 | 0 of 10 | Study |
| Retail | 10 UK retail sites, 15 AI crawlers | 12 Nov 2026 | 27 of 150 | 4 of 10 | Study |
Each row is one reading on one day. The news series is read again on the 2nd of every month, so it is the only row that can show change so far.
What the observatory tracks
| Strand | Question | How often | Status |
|---|---|---|---|
| What sites allow | Which AI crawlers do UK sites block in robots.txt? | News monthly. Other sectors as they are read | Running since 2 Oct 2026 |
| What AI says | Do AI answers date their figures, keep up when a fact changes, and credit their sources? | Per study, with reruns | Running |
| Who AI recommends | Which UK businesses do AI answers name, recommend and cite? | Quarterly | AI Visibility 100: plan written, not yet run |
| How it is measured | Can the numbers people quote about AI search be checked? | Per study | Running |
What sites allow
- Which AI crawlers do the UK’s biggest news sites block? The monthly series.
- Do the UK’s biggest supermarkets block AI crawlers?
- Do the UK’s biggest shopping sites block AI crawlers?
- Do AI companies block each other’s crawlers?
- Do UK news sites and universities publish llms.txt?
- AI crawlers explained, for what each crawler is and does.
What AI says
- Do AI answers say when a “current” figure is from?
- How fast did AI answers catch up with a Bank Rate change?
- Does ChatGPT recommend different brands to different customers?
- What text do machines actually read on a page?
How it is measured
- AI search statistics, traced to the source
- What do AI visibility trackers say about how they measure?
- How many times should you run an AI prompt before you trust the number?
- How to use the EntityMap validator, and what it can’t tell you
- BrightonSEO will be full of AI claims nobody can mark wrong
Registered plans
A plan is written and published before the data is collected, so the method can’t be bent to fit the result. The Search Experiment Lab shows which questions have been tested and which haven’t.
- Study plan: which UK universities block AI crawlers, month by month?
- Study plan: does asking “when is that figure from?” fix an undated AI answer?
- Study plan: when AI engines repeat my findings, do they get them right and say where they came from?
- Study plan: do personas change AI brand picks outside accounting software?
- Study plan: do the sources AI engines cite contain the figure?
- Study plan: does an AI crawler stop when my robots.txt asks it to?
- Study plan: UK news robots.txt, month by month
- Study plan: how fast do AI answers catch up with a Bank Rate decision?
- Study plan: undated current figures, every quarter
- Study plan: do news sites enforce their robots.txt?
- Study plan: do news sites that block AI training still appear in AI answers?
- Study plan: the Telegraph and the Daily Mail in AI answers
- Study plan: link, mention or nothing
- Study plan: established newsbrands and lower-credibility sites
Method and data
- Method: the sample rules, the 15 crawlers and what “blocked” means.
- Data: every dataset as a CSV, with its date and row count.
- Reproduce the figures: the scripts that turn the saved files into the published numbers.
- What I will not claim.
Schedule
- 2nd of each month: UK news robots.txt snapshot.
- Prediction checks: on the dates listed on the ledger.
- New results also go out on the RSS feed.
Update history
- 14 November 2026: page created.
Cite this page
Whitmore, J. (2026). UK AI Search Observatory. jacobwhitmore.com. https://jacobwhitmore.com/uk-ai-search-observatory/