by Jacob Whitmore | 12 November 2026 | Research
Read 12 November 2026. Four of the UK’s ten most-visited retail sites block at least one AI crawler in robots.txt. Amazon blocks 12 of the 15 I track, eBay 9, Vinted 5 and Temu 1. The other six block none. Together the ten sites block 27 of 150 cells, against 99... by Jacob Whitmore | 10 November 2026 | Research
Checked 10 November 2026. None of the UK’s ten biggest supermarkets blocks any of the 15 AI crawlers I track in robots.txt. The UK’s ten biggest newsbrands block 99 of the same 150 cells. I read the robots.txt file on each supermarket’s main website on 10 November... by Jacob Whitmore | 6 November 2026 | Research
On 6 November 2026, 3 of the 17 AI company websites I could read blocked at least one of the 15 AI crawlers this site tracks. All three were chat apps: chatgpt.com, claude.ai and grok.com. None of the 11 corporate websites blocked any of them, including openai.com,... by Jacob Whitmore | 2 November 2026 | Research
Checked 2 November 2026. 0 of the 150 cells changed since 2 October 2026. Each of the UK’s ten biggest newsbrands gave the same answer to each of the 15 AI crawlers as it did a month earlier. I re-read the ten robots.txt files on 2 November 2026 and checked them... by Jacob Whitmore | 30 October 2026 | Research
Checked 30 October 2026 5 of 33 UK news and Russell Group university sites I could read publish an llms.txt file: the Daily Mail and four universities (Durham, Exeter, Imperial and Warwick). The other 28 return a 404, a 410 or a redirect to an unrelated page.... by Jacob Whitmore | 30 October 2026 | Research
Checked 30 October 2026 None of the 13 AI visibility trackers I checked publishes any evidence that its numbers are accurate, and none says whether it collects answers signed in or signed out. I read each vendor’s own product, help and documentation pages and...