How to read this studyTraining, retrieval, “blocked” and cells, in plain English
Three kinds of AI crawler
Training crawlers collect pages that may be used to train AI models. Blocking one asks for your pages to stay out of future training. It doesn’t remove anything already collected.
Search crawlers build an AI company’s own search index, the one its answers draw on. Blocking one can take your pages out of that company’s answers.
User-triggered fetchers read one page because a person asked the assistant to. Some companies say these may ignore robots.txt, because a person started the request.
The studies count search crawlers and fetchers together as retrieval, because both fetch pages to answer questions rather than to train models. The robots.txt generator splits them, because the companies say they treat them differently.
What “blocked” means
A crawler counts as blocked when the rules that apply to it disallow the whole site. That is Disallow: / in a group that names the crawler, or in the * group if nothing names it. Blocking a few folders doesn’t count. Why this definition.
What a cell is
One site checked against one crawler. The news, supermarket and retail studies each have 150 cells: ten sites times 15 crawlers.
Named group or the * group
A group that names a crawler replaces the * group for that crawler. So a file can block every AI crawler without naming any, and naming one can let it in. Example.
Control tokens
Google-Extended and Applebot-Extended aren’t crawlers. They’re names you put in robots.txt to say whether pages that Google’s and Apple’s normal crawlers collect may be used for their AI models.
What robots.txt can’t show
It’s a request, not a lock. It doesn’t stop a crawler that ignores it, and a block made on the server or firewall doesn’t appear in it. These studies count what the file says. More.
Every crawler these studies track, what it does, and who blocks it →
Read 12 November 2026.
Four of the UK’s ten most-visited retail sites block at least one AI crawler in robots.txt. Amazon blocks 12 of the 15 I track, eBay 9, Vinted 5 and Temu 1. The other six block none. Together the ten sites block 27 of 150 cells, against 99 of 150 for the UK’s ten biggest newsbrands.
The headline numbers
- 4 of 10 sites block at least one of the 15 AI crawlers. 6 of 10 block none.
- 27 of 150 cells are blocked (18%). For the ten newsbrands on 2 November it was 99 of 150 (66%). For the ten supermarkets on 10 November it was 0 of 150.
- Training crawlers: 20 of 90 cells blocked. Retrieval crawlers, the ones that fetch pages for AI search and live answers: 7 of 60.
- Six of those seven retrieval blocks are Amazon’s. Take Amazon out and the other nine sites block one retrieval cell of 54.
- The four sites that block are the four where other people do the selling: Amazon, eBay, Temu and Vinted. That grouping is mine, not Ofcom’s.
All ten sites
| Site | File read | Blocked (of 15) | Named (of 15) | How it was read |
|---|---|---|---|---|
| Amazon | amazon.co.uk | 12 | 12 | Chrome |
| eBay | ebay.co.uk | 9 | 13 | Chrome |
| Temu | temu.com | 1 | 9 | Chrome |
| Apple | apple.com | 0 | 0 | Chrome |
| Tesco | tesco.com | 0 | 1 | Chrome |
| Argos | argos.co.uk | 0 | 0 | Chrome |
| Asda | asda.com | 0 | 0 | Chrome |
| Lidl | lidl.co.uk | 0 | 0 | Chrome |
| Vinted | vinted.co.uk | 5 | 5 | Chrome |
| Shein | shein.co.uk | 0 | 0 | Chrome |
Amazon blocks nearly everything, including the bots that answer shoppers
Amazon’s UK file has 100 groups that each name one crawler and disallow the whole site. One name, ClaudeBot, appears twice, so that is 99 different crawlers.
Twelve of my 15 are among them: every OpenAI, Anthropic and Perplexity crawler I track, plus Google-Extended, CCBot, Bytespider and Meta-ExternalAgent. That includes ChatGPT-User, Claude-User and Perplexity-User, which only fetch a page when a person asks an AI tool about it.
The three it leaves open are Applebot-Extended, the retired anthropic-ai token, and Amazonbot, which is Amazon’s own.
The list goes well past my 15. It names GoogleAgent-Mariner and GoogleAgent-Shopping, along with dozens of other agents and scrapers.
eBay splits training from search
eBay has two AI groups. The first stacks ten crawlers and disallows the whole site, with three paths allowed back in (/help, /ebaylive/seller, /sellercentre). Nine of my 15 are in it: GPTBot, ClaudeBot, anthropic-ai, CCBot, Applebot-Extended, Bytespider, Meta-ExternalAgent, Amazonbot and PerplexityBot.
The second group names OAI-SearchBot, ChatGPT-User, Claude-SearchBot and Claude-User. It gives them close to the same path rules as everyone else. So eBay refuses OpenAI’s and Anthropic’s training crawlers and lets their search and live-fetch crawlers in.
Perplexity gets different treatment. PerplexityBot is in the blocked group. Google-Extended and Perplexity-User aren’t named at all.
By my definition a crawler is blocked when the rules disallow the root of the site. eBay’s nine count as blocked even with the three paths allowed back.
Temu puts GPTBot in the same group as Googlebot
Temu’s file names nine of my 15 and blocks one. Eight of them (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot and Google-Extended) share a single group with Googlebot and Bingbot. Whatever Google’s search crawler may read, they may read.
The one block is Bytespider, ByteDance’s crawler, which sits in a group with a string of Yahoo crawlers under DisAllow: /*.
Vinted blocks five and adds a signal
Vinted stacks GPTBot, ClaudeBot, Google-Extended, Amazonbot and CCBot in one group and disallows the site. All five are training crawlers. It names none of the retrieval crawlers, so they fall under the general rules, which allow the site.
Vinted is also the only one of the ten with a Content-Signal line. Its general group says ai-train=no, search=yes, ai-input=yes. The comments at the top of the file say automated agents may not create accounts, add to basket or buy.
Six sites say nothing
Apple, Argos, Asda, Lidl and Shein name none of the 15 and block none. Tesco names one, meta-externalagent, and only keeps it out of /account/.
Three of these were in the supermarket study two days ago. Lidl’s rules are the same as on 10 November.
Tesco’s file has changed. On 10 November it had one User-agent line and 26 Disallow rules, counted in Chrome. On 12 November it had two User-agent lines and 24 Disallow rules, also counted in Chrome. The new line is the Meta group. It still blocks none of the 15.
Asda served two different files. My browser got one with a header dated 21 August 2025 and nine rules. A web fetch tool got one dated 28 October 2024 with eleven rules, on both 10 and 12 November. I don’t know why. Neither version names a crawler or shuts anything out of the whole site.
Retail against news and supermarkets
| Crawler | Type | Retail sites blocking (of 10) | Newsbrands blocking (of 10) | Supermarkets blocking (of 10) |
|---|---|---|---|---|
| GPTBot | Training | 3 | 7 | 0 |
| ClaudeBot | Training | 3 | 9 | 0 |
| CCBot | Training | 3 | 9 | 0 |
| Bytespider | Training | 3 | 7 | 0 |
| Google-Extended | Training | 2 | 5 | 0 |
| Meta-ExternalAgent | Training | 2 | 7 | 0 |
| Amazonbot | Training | 2 | 6 | 0 |
| anthropic-ai | Training | 1 | 9 | 0 |
| Applebot-Extended | Training | 1 | 8 | 0 |
| PerplexityBot | Retrieval | 2 | 9 | 0 |
| OAI-SearchBot | Retrieval | 1 | 6 | 0 |
| ChatGPT-User | Retrieval | 1 | 4 | 0 |
| Claude-SearchBot | Retrieval | 1 | 5 | 0 |
| Claude-User | Retrieval | 1 | 4 | 0 |
| Perplexity-User | Retrieval | 1 | 4 | 0 |
News sites sell words, and most of them block. Supermarkets sell groceries, and none of them block. The big shopping sites sit between the two, and the ones that block are the ones whose pages are full of other people’s listings.
What other reporting found
Amazon’s blocks are well covered. Search Engine Journal counted 101 user-agent groups in amazon.com’s file on 22 September 2026, 100 of them disallowing the whole site. My count for amazon.co.uk matches that 100.
Modern Retail reported on 8 December 2025 that eBay had added an agent policy to its robots.txt, and then explicit blocks on bots from Perplexity, Anthropic, Amazon and others.
What this study adds is a fixed UK list, the same 15 crawlers and the same definition as the news and supermarket studies, so the three sectors can be put side by side.
Method
- Sample. The ten services in Figure 41 of Ofcom’s Online Nation 2025 report, “Top ten retail services”, ranked by UK adult audience in May 2025 (Ipsos iris). The list is frozen.
- Files. One robots.txt per service, from its main UK shopping site. Apple and Temu serve the UK from apple.com and temu.com.
- Crawlers. The same 15 as the news series: nine training and six retrieval. See Method.
- Definition. A crawler is blocked when the rules that apply to it disallow the whole site, through a group that names it or through the wildcard group when nothing names it. Path-only rules don’t count. More in What “blocked” should mean.
- Reading. All ten files were opened in Chrome on 12 November 2026. Nine were first read through a web fetch tool. The Chrome reading matched the fetch tool on every one of the 150 cells.
- Classifying. The records were run through the same script published on Reproduce the figures.
Limits
- One day. This says nothing about direction.
- It wasn’t pre-registered.
- Only three of the ten records are full copies. The rest keep the rules that decide the result. The data file marks which.
- I read the sample table in a copy of Ofcom’s report PDF, not in the file on Ofcom’s own site. Ofcom’s audience data is from May 2025.
- Asda returned different files to a browser and to a fetch tool. Other sites may do the same without my seeing it.
- robots.txt is a request. It doesn’t show what a server does, and Amazon has also used its terms of use against an AI agent. See robots.txt is a request.
- Ofcom ranks organisations, not domains. I chose the domain for each.
What I will not claim
- That marketplaces block AI crawlers as a rule. Four of four in a list of ten is a pattern worth a second look and no more.
- That the six silent sites have decided to allow AI. Saying nothing is not a decision I can see.
- That any of these blocks work.
- That I know why Tesco’s file changed.
Data
- uk-retail-sites-ai-crawlers-2026-11-12.csv: all 150 cells, with the columns scrape_date, brand, domain, bot, bot_class, blocked, named_in_file, collection, record and source.
- uk-retail-sites-sample-2026-11-12.csv: the ten services, the domain read, and Ofcom’s audience and reach figures.
- uk-retail-sites-robots-records-2026-11-12.txt: the ten files as recorded. Three are full copies.
- Licence: CC BY 4.0.