Checked 10 November 2026.
None of the UK’s ten biggest supermarkets blocks any of the 15 AI crawlers I track in robots.txt. The UK’s ten biggest newsbrands block 99 of the same 150 cells.
I read the robots.txt file on each supermarket’s main website on 10 November 2026 and checked it against the same 15 crawlers, with the same definition of blocked, as the UK news series. Here’s what the files say.
The headline numbers
- 0 of 10 supermarkets block any of the 15 AI crawlers. That is 0 of 150 cells.
- 9 of 10 don’t name a single one of the 15. Their files have rules for all crawlers at once, and those rules cover baskets, logins and search pages, not the whole site.
- Aldi is the only one that names them. It names 9 of the 15, and every one of the nine gets
Allow: /. - On 2 November 2026 the ten biggest newsbrands blocked 99 of 150 cells: 67 of 90 training cells and 32 of 60 retrieval cells.
- GPTBot is blocked by 7 of the 10 newsbrands and 0 of the 10 supermarkets.
All ten supermarkets
| Supermarket | File read | AI crawlers blocked (of 15) | Of the 15, named in the file | How it was read |
|---|---|---|---|---|
| Tesco | www.tesco.com/robots.txt | 0 | 0 | Fetch tool |
| Sainsbury’s | www.sainsburys.co.uk/robots.txt | 0 | 0 | Fetch tool |
| Asda | www.asda.com/robots.txt | 0 | 0 | Fetch tool |
| Aldi | www.aldi.co.uk/robots.txt | 0 | 9 (all allowed) | Fetch tool |
| Morrisons | www.morrisons.com/robots.txt | 0 | 0 | Fetch tool |
| Lidl | www.lidl.co.uk/robots.txt | 0 | 0 | Fetch tool |
| Co-op | www.coop.co.uk/robots.txt | 0 | 0 | Fetch tool |
| Waitrose | www.waitrose.com/robots.txt | 0 | 0 | Fetch tool |
| Iceland | www.iceland.co.uk/robots.txt | 0 | 0 | Fetch tool |
| Ocado | www.ocado.com/robots.txt | 0 | 0 | Chrome |
Aldi says yes out loud
Aldi’s file ends with a list of AI crawlers, each followed by Allow: /. Of the 15 I track it names GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, anthropic-ai, Google-Extended, PerplexityBot, CCBot and Applebot-Extended. It also names four AI tokens outside the 15: Cohere-Ai, OmgiliBot, Omgili and Claude-Web.
An explicit allow changes nothing technically. A crawler that isn’t named falls back to the rules for all crawlers, and Aldi’s rules for all crawlers don’t close the site either. What the list does is put a decision on the record. The other nine files say nothing about AI, so nobody can tell from the file whether anyone decided.
The same file blocks four other crawlers from the whole site by name: CazoodleBot, MJ12bot, dotbot and Gigabot. So Aldi does use robots.txt to refuse crawlers. It chose not to refuse these.
Supermarkets against newsbrands
| Crawler | Job | Newsbrands blocking (of 10), 2 Nov 2026 | Supermarkets blocking (of 10), 10 Nov 2026 |
|---|---|---|---|
GPTBot | Training | 7 | 0 |
OAI-SearchBot | Retrieval | 6 | 0 |
ChatGPT-User | Retrieval | 4 | 0 |
ClaudeBot | Training | 9 | 0 |
Claude-SearchBot | Retrieval | 5 | 0 |
Claude-User | Retrieval | 4 | 0 |
anthropic-ai | Training | 9 | 0 |
Google-Extended | Training | 5 | 0 |
PerplexityBot | Retrieval | 9 | 0 |
Perplexity-User | Retrieval | 4 | 0 |
CCBot | Training | 9 | 0 |
Applebot-Extended | Training | 8 | 0 |
Bytespider | Training | 7 | 0 |
Meta-ExternalAgent | Training | 7 | 0 |
Amazonbot | Training | 6 | 0 |
These are two different kinds of business. A newsbrand sells the words on the page, so a crawler that copies them takes the product. A supermarket sells groceries, and the page is the shop window. The files are consistent with that difference. They don’t prove it is the reason, and I haven’t asked anyone.
To see where your own file sits between the two groups, paste it into the AI crawler benchmark.
What other studies found
Two retail studies I opened on 10 November 2026 point the same way, with bigger samples and different methods.
- Blend Commerce sent an automated browser to 200 UK and US retailers between 8 and 15 September 2026. 29 refused it twice, and Tesco was one of them. Blend could read robots.txt at 24 of the 29, and all 24 allowed the search crawlers of ChatGPT, Perplexity and Claude. So an open file and a closed door can sit on the same site. Tesco’s file is open in my data too.
- Doss read the robots.txt of 500 large retailers on 17 June 2026, 364 of them readable, and reports that about a tenth blocked the AI crawlers behind shopping assistants. Its page doesn’t name a country for the sample.
Neither uses this site’s 15 crawlers or its definition of blocked, so their numbers can’t be added to mine. What this study adds is a like-for-like comparison: two named UK top tens, the same crawlers, the same rule, every cell published.
Method
- The sample is the ten grocers given a market share in Worldpanel’s Great Britain grocery market share release of 11 November 2025 (12 weeks to 2 November 2025), which I opened on 10 November 2026: Tesco, Sainsbury’s, Asda, Aldi, Morrisons, Lidl, Co-op, Waitrose, Iceland and Ocado. Source. The list is frozen.
- For each grocer I read the robots.txt on its main UK website: www.tesco.com, www.sainsburys.co.uk, www.asda.com, www.aldi.co.uk, www.morrisons.com, www.lidl.co.uk, www.coop.co.uk, www.waitrose.com, www.iceland.co.uk, www.ocado.com.
- The 15 crawlers and their training and retrieval classes are the ones in the method note. Nothing was added or removed.
- A crawler counts as blocked when the rules that apply to it disallow the whole site, either in a group that names it or in the
*group when nothing names it. Path-only rules don’t count. - Nine files were read through a web fetch tool, which returned the text of each file. Ocado’s was read in Chrome, because the fetch tool was refused there. Tesco’s was read both ways and the line counts matched: 35 lines, one user-agent group.
- Every cell was classified by script. The same script reproduces all 150 cells of the 2 November news snapshot from the raw files.
Limits
- This study had no registered plan. I ran it the day I thought of it. It reuses the news series’ crawlers and definition unchanged, so there was nothing to tune, but it isn’t pre-registered and I’m saying so.
- robots.txt is a request. It shows what a site asks for, not what crawlers do.
- Big retailers often stop bots at the network, before robots.txt matters. That doesn’t show up here. A supermarket can have an open file and still turn a crawler away.
- One oddity is unexplained. The fetch tool reported that ocado.com’s robots.txt disallowed it, but the file I read in Chrome a minute later has no rule that would. I don’t know why. It’s logged in the data and left out of every count.
- A file read through a fetch tool is a lower grade of evidence than one read in a browser. Five of the ten records are condensed (user-agent lines and whole-site rules only), not full copies. The next run should save full copies of all ten.
- Only the main website was read. A separate shopping subdomain, where a grocer has one, has its own file and is out of scope.
- Ten sites, one day.
What I will not claim
- That supermarkets want to be in AI answers. Nine of the ten files are silent, and silence isn’t a decision I can see.
- That any AI crawler reads these sites, or that any AI answer cites them.
- Why Aldi wrote its list.
- Anything about other retailers.
Data
- uk-supermarkets-ai-crawlers-2026-11-10.csv: all 150 cells, with the columns scrape_date, brand, domain, bot, bot_class, blocked, named_in_file, collection, record and source.
- uk-supermarkets-sample-2026-11-10.csv: the ten grocers, the domain read and the sample source.
- uk-supermarkets-robots-records-2026-11-10.txt: the ten files as recorded. Five are full copies, five are condensed.
- Licence: CC BY 4.0.