Library

Everything published on this site in one searchable list: 79 studies, experiments, datasets, analysis pieces, essays, guides, predictions and tools. Search by word, or filter by type and topic.

79 items, newest first

All items

  1. Experiment · · AI answers

    Ask an AI when its figure is from and it tells you: 80 of 80

    Run 14 November 2026. Asked "When is that figure from?", ChatGPT, Google AI Mode, Gemini and Claude gave a period in 80 of 80 chats. A neutral follow-up got one in 7 of 64. 3 periods…

  2. Research · · Google and search

    Google answered 40 of 40 UK questions before the first link

    Read 18 November 2026. On 40 everyday UK questions, Google showed an AI Overview or its own answer box above the first ordinary link every time. The first link started a median 769…

  3. Research · · Google and search

    The Future of Search Report 2026: what can be checked

    An annual report built only from dated readings on this site, with what each one can and cannot show.

  4. Experiment · · Measurement and evidence

    AI citation tracker: is this site named or linked in AI answers?

    A fixed set of questions asked of AI tools to see whether this site is named or linked. Baseline reading and data.

  5. Research plan · · Measurement and evidence

    Research plans

    Registered plans for studies that have not been run yet, each with its question, sample, measures and limits.

  6. Resource · · Measurement and evidence

    Search Console regex library: 44 tested patterns

    44 regex patterns for Search Console and GA4, each tested, with copy buttons, a CSV and an in-browser tester.

  7. Analysis · · Measurement and evidence

    Does publishing original research get you cited by AI?

    No controlled study shows original research raises AI citations. Why it should help AI use your work, why credit is less certain, and the test this site will score on 2 Jan 2027.

  8. Data publication · · Measurement and evidence

    The Jacob Whitmore Index

    Every headline reading from the research in one table: the figure, its denominator, the date it was read and the study behind it.

  9. Research · · Crawlers and robots.txt

    UK AI Search Observatory

    The home of the recurring research: which AI crawlers UK sites block by sector, what AI answers say, and who they recommend.

  10. Research · · AI answers

    Which UK retailers do AI tools name? 36 answers from four AI tools

    A pilot reading of which UK retailers four AI tools name when asked shopping questions. 36 answers, with the full answer file.

  11. Research plan · · AI answers

    UK businesses in AI answers: the ten-sector plan

    The plan for a quarterly reading of which UK businesses AI tools name, recommend and cite, sector by sector.

  12. Experiment · · Measurement and evidence

    The Search Experiment Lab

    Ten common questions about search and AI discovery, and whether each has been tested here, reviewed, or not tested at all.

  13. Technical guide · · Crawlers and robots.txt

    How to say "search yes, AI training no" in one robots.txt line

    One robots.txt line states whether your pages may be used for search, AI answers or AI training. Vinted is the one top-ten UK retail site using it. Copy the line and read the limits.

  14. Analysis · · Crawlers and robots.txt

    Will AI agents do your shopping? Three of the UK's biggest retail sites have said no in writing

    Checked 12 November 2026. Amazon, eBay and Vinted tell AI shopping agents to stay out of checkout. Seven of the UK's ten biggest retail sites say nothing. What the files and one court…

  15. Research · · Crawlers and robots.txt

    Do the UK's biggest shopping sites block AI crawlers?

    Read 12 November 2026. 4 of the UK's 10 most-visited retail sites block AI crawlers in robots.txt. Amazon blocks 12 of 15, eBay 9, Vinted 5, Temu 1. Six block none.

  16. Resource · · Crawlers and robots.txt

    AI crawlers explained: what each one does and who blocks it

    A plain-English reference to the 15 AI crawlers in the studies: who runs each one and what it is for.

  17. Research · · Crawlers and robots.txt

    Do the UK’s biggest supermarkets block AI crawlers?

    None of the UK’s ten biggest supermarkets blocks any of 15 AI crawlers in robots.txt. The ten biggest newsbrands block 99 of the same 150 cells. Aldi names nine and allows them all.

  18. Data publication · · Measurement and evidence

    The Data Room: every dataset, free to use

    Every dataset behind the research as a plain CSV under CC BY 4.0, with an explorer to search, filter and download rows.

  19. Resource · · Crawlers and robots.txt

    AI crawler benchmark: check your robots.txt

    Paste a robots.txt and see how it compares with the UK sites in the studies, crawler by crawler.

  20. Analysis · · AI answers

    How do ChatGPT, Claude, Gemini, Perplexity and Copilot choose their sources? What the companies actually say

    Checked 8 Nov 2026: what OpenAI, Anthropic, Google, Perplexity and Microsoft say about how their AI answers find and pick sources. Only one explains how it chooses.

  21. Analysis · · Measurement and evidence

    How many times should you run an AI prompt before you trust the number?

    To pin an AI brand-mention rate to within ±10 points you need about 97 runs of one prompt. To show a change worked, about 388 per version. The maths and a free calculator.

  22. Research · · Crawlers and robots.txt

    Do AI companies block each other's crawlers? 17 AI company sites checked

    On 6 November 2026, three of 17 AI company sites blocked rival AI crawlers in robots.txt. All three were chat apps. None of the 11 corporate sites blocked any. Cell-level data.

  23. Prediction · · updated · AI answers

    P37 missed: AI engines had the new Bank Rate within a day

    I called that at least one of four AI engines would still give the old Bank Rate 24 hours after the 5 November 2026 decision. None did. Here is the data and what I got wrong.

  24. Ideas & Perspectives · · Web, publishing and commerce

    What happens after the AI slop?

    AI made publishing almost free. It didn't make trust cheap. Why proof, original data and open methods are the next edge in search and AI answers.

  25. Ideas & Perspectives · · Web, publishing and commerce

    Nobody Wants to Read Your Book

    Writing a book used to be hard, and that difficulty kept the supply of books in line with readers. AI has made writing almost free. Reading still takes ten hours. What happens when books…

  26. Research · · Crawlers and robots.txt

    UK news robots.txt, November 2026

    0 of the 150 cells changed since 2 October 2026. The second monthly snapshot of AI crawler rules in the robots.txt files of the UK’s ten biggest news sites.

  27. Resource · · Crawlers and robots.txt

    AI robots.txt generator

    Build a robots.txt that blocks AI training bots and keeps AI search, with presets, comparisons and exports.

  28. Technical guide · · Measurement and evidence

    How to check a statistic before you use it: five questions

    None of the 32 most-repeated AI search statistics passed five basic checks. Ask any statistic these five questions before you paste it.

  29. Prediction · · Measurement and evidence

    Gartner said search volume would drop 25% by 2026. Can it be scored?

    Gartner's 2024 call that search volume would fall 25% by 2026, checked in October 2026. It can't be scored as written. On the closest public data, it looks like a miss.

  30. Analysis · · Measurement and evidence

    Do backlinks get you cited in AI answers? What 16 studies actually measured

    Checked 30 October 2026. 16 studies on links, authority and rankings vs AI citations. All correlational, none with raw data, one naming a country. Brand mentions beat links.

  31. Analysis · · Measurement and evidence

    Search Console's AI report shows impressions, not clicks. What it can and can't tell you

    Checked 30 October 2026. Search Console's AI report counts impressions only: no clicks, no queries. What it can answer, what it can't, and how to report AI visibility anyway.

  32. Research · · Crawlers and robots.txt

    Do UK news sites and universities publish llms.txt?

    Checked 30 October 2026. 5 of 33 big UK news and university sites publish llms.txt. The Daily Mail's opens AI access to selected URLs while its robots.txt blocks the rest.

  33. Research · · Measurement and evidence

    What do AI visibility trackers say about how they measure? 13 tools checked

    Checked 30 October 2026. 13 AI visibility trackers, nine questions about method. None publishes accuracy evidence. None says if it collects signed in or out.

  34. Research · · Measurement and evidence

    AI search statistics, traced to the source

    Checked 29 October 2026. The 32 AI search statistics repeated most across roundups, each traced to its source. None passed all five checks. None is about the UK.

  35. Analysis · · Google and search

    Google is lying to your rank tracker on purpose

    Google is deliberately serving fake results pages to suspected scrapers. It isn’t blocking them. It answers them, with complete, normal-looking pages whose organic results have been…

  36. Analysis · · Google and search

    Meta’s Muse AI: what it means for SEO, Google and the future of search

    Muse is Meta’s agent, not Facebook search. What Meta announced, how AI Mode differs, the UK position, and a six-layer stack for AI discovery. Dated.

  37. Prediction · · Google and search

    SEO and AI search predictions for 2027

    Nine calls for 2027, each with a confidence figure, a check date and a test.

  38. Analysis · · Web, publishing and commerce

    Consumers will let AI fill the cart. They still won't let it pay.

    Four 2025–26 surveys agree: shoppers let AI agents recommend and fill a basket, not pay. What agents check before they recommend you, and what merchants control.

  39. Research plan · · AI answers

    Study plan: when AI engines repeat my findings, do they get them right and say where they came from?

    A registered quarterly audit of whether AI engines repeat nine findings from this site correctly and credit them.

  40. Research plan · · AI answers

    Study plan: does asking "when is that figure from?" fix an undated AI answer?

    A registered experiment: the same ten UK figure questions on four engines, with a date follow-up against a neutral follow-up, 160 chats.

  41. Research plan · · Crawlers and robots.txt

    Study plan: which UK universities block AI crawlers, month by month?

    A registered monthly check of which AI crawlers the 24 Russell Group universities block in robots.txt, using the same 15 crawlers and definition as the news series.

  42. Technical guide · · AI answers

    Ask an AI answer when its "current" figure is from

    Google AI Mode and Gemini left the date off 7 "current" UK figures. Both errors I found were side figures next to the headline number.

  43. Technical guide · · AI answers

    Keep facts in body text, not in schema or meta tags

    Four common HTML-to-text tools all kept body text and dropped JSON-LD, meta descriptions, aria-labels, data- attributes and comments.

  44. Technical guide · · AI answers

    Put the date in the same sentence as the figure

    7 of 40 AI answers to "current" UK figure questions gave no date. If you publish figures, put the period in the sentence that carries the number.

  45. Technical guide · · Crawlers and robots.txt

    Does naming a crawler in robots.txt override User-agent: *?

    Yes. A crawler follows the group that names it and ignores the * group. The Sun's robots.txt uses this to let OpenAI in.

  46. Technical guide · · updated · Crawlers and robots.txt

    How to block AI training crawlers but still let ChatGPT-User in

    Most of the UK's biggest news sites block AI training crawlers and leave the user-triggered fetchers open. Here's the robots.txt that does it.

  47. Technical guide · · AI answers

    Guides, each one backed by a dated test

    Nine things to do today, each tied to a study on this site with a date, a sample and a CSV.

  48. Experiment · · AI answers

    What text do machines actually read on a page?

    A dated test of which parts of a web page survive when four common tools turn it into text.

  49. Analysis · · Measurement and evidence

    Five labels every AI visibility number needs

    Five labels any AI visibility number needs before anyone can check it: date, prompt, model, runs and one change at a time.

  50. Analysis · · Crawlers and robots.txt

    Crawled is not cited

    An AI crawler in your logs proves a page was fetched. It does not prove the page was used, credited or clicked.

  51. Research plan · · Crawlers and robots.txt

    Study plan: does an AI crawler stop when my robots.txt asks it to?

    A registered method for logging which AI crawlers fetch jacobwhitmore.com, then blocking one and checking whether it stops.

  52. Research plan · · AI answers

    Study plan: do the sources AI engines cite contain the figure?

    A registered method for opening every source AI engines cite for a UK market-size figure and checking whether the figure is in it.

  53. Research plan · · AI answers

    Study plan: do personas change AI brand picks outside accounting software?

    A registered method for testing whether the customer in the prompt changes AI brand recommendations across four UK business categories.

  54. Research plan · · AI answers

    Study plan: undated current figures, every quarter

    A registered method for rerunning the same ten "right now" UK figure questions on four AI engines every quarter.

  55. Research plan · · updated · AI answers

    Study plan: how fast do AI answers catch up with a Bank Rate decision?

    A registered method for asking four AI engines the Bank Rate at fixed times before and after the 5 November 2026 decision.

  56. Research plan · · Crawlers and robots.txt

    Study plan: UK news robots.txt, month by month

    A registered method for re-reading the same ten UK news robots.txt files every month and counting what changed.

  57. Experiment · · AI answers

    Do AI answers say when a "current" figure is from?

    On 10 October 2026, 7 of 40 AI answers to "current" UK figure questions didn't say when the figure was from. All 7 came from Google's two engines.

  58. Experiment · · updated · AI answers

    Does ChatGPT recommend different brands to different customers?

    Checked 9 October 2026 Ask ChatGPT for the best accounting software in the UK and you get Xero, QuickBooks and one more. Tell it you’re the finance director of a 250-person manufacturer…

  59. Experiment · · AI answers

    How much is the purple pound worth? Five figures, and AI doesn't say which year

    Checked 9 October 2026 There’s no single figure. Published estimates of the purple pound, the spending power of disabled people and their households, run from £212bn to £446bn a year. On…

  60. Analysis · · Measurement and evidence

    How to use the EntityMap validator, and what it can't tell you

    I tested the EntityMap validator on 9 October 2026. It checks a file's shape, not whether it's true or whether anything reads it. None of nine UK newsbrands I checked publish one.

  61. Prediction · · Google and search

    I said I'd check my World Cup search predictions in August. I can't.

    In May I said I'd check my World Cup search predictions in August. I've tried. None of them can be marked, and here's why.

  62. Prediction · · Google and search

    The future of search is a smaller click. That part can be checked.

    Most future-of-search posts can't be marked wrong. This one makes three dated calls about links, mentions and the publisher split.

  63. Research · · AI answers

    AI answers quote prices without saying when they were true

    A price without a date is a memory, not a fact. AI engines link sources but rarely say when the price was read.

  64. Analysis · · AI answers

    Ask AI for "the best" local tradesperson and you get a guess

    "Who's the best plumber in Leeds?" has no checkable answer. AI engines answer it anyway, and I think they'll keep doing so.

  65. Analysis · · Crawlers and robots.txt

    Small business sites aren't letting AI in. They're not deciding at all.

    Publishers write AI crawler rules. Most small business sites write nothing, so the platform's default decides for them.

  66. Analysis · · Measurement and evidence

    BrightonSEO next week will be full of AI claims nobody can mark wrong

    BrightonSEO is on 8 and 9 October. Here are the AI sessions I'll be watching, and what would make each one checkable.

  67. Research plan · · AI answers

    Study plan: established newsbrands and lower-credibility sites

    A registered method for comparing AI crawler block rates between established UK newsbrands and high-reach sites that fail a set credibility test.

  68. Research plan · · AI answers

    Study plan: link, mention or nothing

    A registered method for counting how often AI answers to UK news questions link to a news source.

  69. Research plan · · AI answers

    Study plan: the Telegraph and the Daily Mail in AI answers

    A registered comparison of two papers with almost identical robots.txt files and one difference.

  70. Research plan · · AI answers

    Study plan: do news sites that block AI training still appear in AI answers?

    A registered method and a fixed list of 30 UK news queries for checking which newsbrands appear in AI answers.

  71. Research plan · · Crawlers and robots.txt

    Study plan: do news sites enforce their robots.txt?

    A registered method for checking whether the servers of ten UK news sites act on what their robots.txt files say.

  72. Analysis · · Crawlers and robots.txt

    Notes on one file: the Sun

    The Sun's robots.txt blocks by default and lets in a short list of named crawlers.

  73. Prediction · · Measurement and evidence

    How to write a claim you can mark wrong

    The four things a claim needs before it can be checked.

  74. Analysis · · Crawlers and robots.txt

    robots.txt is a request

    What a robots.txt file can and can't tell you about AI crawlers.

  75. Analysis · · Crawlers and robots.txt

    The Telegraph carve-out, as a decision

    The Telegraph blocks 14 of the 15 AI crawlers checked and explicitly allows Google-Extended.

  76. Analysis · · Crawlers and robots.txt

    What “blocked” should mean in a sentence

    Why the study counts a crawler as blocked only when the whole site is disallowed for it.

  77. Research · · Crawlers and robots.txt

    Which UK news sites block GPTBot and other AI crawlers?

    Nine of the UK's ten biggest newsbrands block at least one AI crawler, but they block training bots far more often than the bots that fetch pages for AI answers.

  78. Prediction · · Google and search

    Predictions ledger

    Every dated prediction with its check date and the test written in advance. Right, wrong or could not score, and misses stay up.

  79. Resource · · Measurement and evidence

    Method

    How the studies are run: sample rules, the 15 crawlers, what blocked means and the site robots.txt policy.

How the library is organised

  • Research is a dated reading of something, with a sample, a method and limits. Research plans are published before the data is collected.
  • Experiments are tests I ran myself on AI tools or crawlers. The Search Experiment Lab shows which questions have been tested.
  • Data publications come with a CSV. All of them are also in the Data Room.
  • Analysis explains a development or someone else's study and what it does and doesn't show.
  • Ideas & Perspectives are opinion essays. They are labelled so nobody reads them as research. See Ideas & Perspectives.
  • Technical guides are short how-tos, each tied to a study.
  • Predictions are dated calls and their results. The full record is the ledger.
  • Resources are tools and reference pages. They are grouped on Resources.

Everything here is written by Jacob Whitmore. Dates are first publication dates; an updated date is shown when a page changed after that. New posts also go out on the RSS feed.