Checked 12 November 2026.
Three of the UK’s ten most-visited retail sites now use robots.txt to tell AI shopping agents to stay out of the checkout: Amazon, eBay and Vinted. The other seven say nothing about agents at all. And the one court that has ruled on this so far sided with the agent.
So the answer to “will AI agents replace search for shopping?” depends less on the technology than on who is allowed through the door. This is what the door says today.
Why this matters
An AI answer that recommends a product is one thing. An AI agent that opens the shop, fills the basket and pays is another. The second one skips the search results, the adverts and the “customers also bought” row.
I wrote in October that shoppers will let AI fill the cart but not pay. That was about what consumers accept. This is about what shops accept.
What the three files say
I read the robots.txt files of the ten retail services Ofcom ranks highest by UK audience. The crawler counts are in the study. Here I’m looking at something the counts don’t show: rules and notes aimed at agents that act, as opposed to crawlers that read.
Amazon names the agents. Its UK file has 100 groups that each shut one named bot out of the whole site. Among them are GoogleAgent-Mariner and GoogleAgent-Shopping, which by their names are Google’s browsing and shopping agents, and the live-fetch bots of OpenAI, Anthropic and Perplexity.
eBay writes a policy. Its file opens with a comment block headed “Robots & Agent Policy”. One line reads: “Checkouts are strictly for human users.” The block goes on to rule out buy-for-me agents and bots that place orders with no human review, and points approved partners to eBay’s API. Modern Retail spotted the policy in December 2025 and noted the wording is close to a policy Shopify introduced earlier that year.
Vinted does both jobs in comments. The top of its file says AI systems may index public pages for search and may not use them for training. A second block says automated agents may not create accounts, add items to a basket or buy.
| Site | Blocks named AI crawlers | Names agent bots | Written agent policy in the file |
|---|---|---|---|
| Amazon | Yes (12 of 15) | Yes | No |
| eBay | Yes (9 of 15) | No | Yes |
| Vinted | Yes (5 of 15) | No | Yes |
| Temu | One (Bytespider) | No | No |
| Apple, Tesco, Argos, Asda, Lidl, Shein | No | No | No |
robots.txt is the weakest tool for this job
A robots.txt rule works by naming a bot and asking it to stay away. That has two problems when the visitor is an agent.
First, the agent has to have a name. On 20 September 2026 Amazon blocked Meta’s shopping agent, Muse. Search Engine Journal’s account points out that Muse sends no user-agent string of its own, so there was nothing for Amazon’s file to name. Amazon used its Conditions of Use on the shopper’s account instead.
Second, the agent has to listen. robots.txt is a request. An agent driving a normal browser on a signed-in account isn’t crawling in the sense the file was written for.
That’s why eBay didn’t stop at the file. Modern Retail’s report says the policy warns of action under eBay’s User Agreement, and that is where a rule can bind a customer.
The court sided with the agent
Amazon sued Perplexity in November 2025 over Comet, a browser whose assistant can shop on Amazon for its user. A district court blocked Comet in March 2026.
On 4 August 2026 the Ninth Circuit appeals court threw that order out. As eMarketer summarised it, the panel thought Amazon unlikely to win on its computer-fraud claim, because when a user tells the assistant to do something on Amazon, the user is the one accessing Amazon’s computers.
Two things stay open. The case itself goes on. And the ruling left alone the question of whether a platform can keep agents out through its terms of service. Just under seven weeks later Amazon used that route against Muse.
This is a US case. I’ve found no UK ruling on the point.
What the other seven are doing
Nothing I can see. Apple, Argos, Asda, Lidl and Shein name no AI crawler and no agent. Tesco names one Meta crawler and keeps it out of account pages.
Temu is the odd one. It names eight AI crawlers and puts them in the same group as Googlebot, so OpenAI’s and Anthropic’s bots get the rules Google’s search crawler gets.
I called this the default vote when I wrote about small business sites: a site that says nothing has voted for whatever happens. Here it applies to some of the biggest shops in the country.
If you sell online
- If you sell on Amazon or eBay, the platform has made the agent decision for you. An outside agent sent to buy your listing goes against the platform’s stated rules, whatever the shopper wants.
- If you run your own shop, you’re where Argos and Asda are: no stated position. That’s fine until an agent turns up at your checkout. Decide before then whether an order placed by software is one you’ll accept, and put the answer in your terms, because the file alone won’t hold.
- Keep reading and buying separate. eBay lets OpenAI’s and Anthropic’s search bots read its listings and bars agents from the checkout. Being found in AI answers and letting AI pay are two decisions. The robots.txt generator handles the first. Only your terms handle the second.
- Check what you already say. Paste your file into the AI crawler benchmark to see which of the 15 crawlers it blocks.
What I will not claim
- That agents are being kept out. I’ve read what the files ask for. I haven’t tested what the servers do.
- That the seven silent sites welcome agents.
- That the US ruling tells you anything about UK law.
- That three of ten is a trend. It’s one reading on one day.
What would change this picture
Three things, any of which can be checked: a fourth site in this list adding agent wording to its file; the Amazon and Perplexity case reaching a final ruling; or a retailer in the list announcing that outside agents may check out. I’ve seen none of the three as of today.