How to say “search yes, AI training no” in one robots.txt line

Checked 12 November 2026.

Add a Content-Signal line to your robots.txt. It states what a crawler may do with your pages after it has read them, without you having to name every bot. One of the UK’s ten most-visited retail sites does this: Vinted.

The line

User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=no
Allow: /

• search covers a search index and ordinary results.
• ai-input covers an AI tool reading your page to answer someone’s question.
• ai-train covers training or fine-tuning a model.

Each takes yes or no. Leave one out and you’ve said nothing about it. The definitions are Cloudflare’s, published 24 September 2025.

What the study found

On 12 November 2026 I read the robots.txt files of the ten retail services Ofcom ranks highest by UK audience. One of the ten had a Content-Signal line. Vinted’s general group says ai-train=no, search=yes, ai-input=yes. Its group for five training crawlers says ai-input=no and blocks the site as well.

The other nine have no such line, including Amazon, which names 99 crawlers one at a time instead.

Which to pick

• You want to be found and quoted, and not trained on: the block above.
• You want ordinary search only: set ai-input=no as well.
• You don’t mind any of it: you don’t need the line.

Limits

• It’s a stated preference. Cloudflare’s own post says some companies may ignore it and that it isn’t a technical barrier.
• I haven’t tested whether any crawler changes what it does when it sees the line.
• It doesn’t replace a block. If you want a named crawler kept out, block it by name too, as Vinted does.
• One site in ten is one site. This shows the line is in use by a large UK retailer and nothing more.