Pick which AI crawlers to block and get a robots.txt file for WordPress, Next.js or any other site. Each crawler shows what its own company says it does, and how many of the UK’s ten biggest news sites blocked it on 2 October 2026.
Nothing you type leaves your browser. Vendor statements were checked on 1 November 2026.
1 Your site
2 Pick the crawlers to block
Quick starts set the switches below. Change any of them after.
Share of the UK’s ten biggest newsbrands blocking that token on . Grey dash: not in that study.
Check a file you already have
Paste a robots.txt. Nothing leaves your browser. The check reads the file the way RFC 9309 says a crawler should: a group that names the bot replaces the * group for that bot.
| Token | Result | Rule used |
|---|
Vendor statements checked against each vendor’s own page on . robots.txt is a request. It does not stop a crawler that ignores it, and it does not remove anything already collected.
What this tool can’t do
robots.txt is a request. A crawler that ignores it isn’t stopped, and blocking a crawler doesn’t remove anything it already collected (why robots.txt is a request). Blocking an AI company’s own crawler also doesn’t keep you out of every AI answer, because some engines search other companies’ indexes (how AI answer engines choose sources).
Where the numbers come from
The newsbrand figures are from my study of the UK’s ten biggest news sites, checked on 2 October 2026, using the same definition of “blocked” as the method page. The split between training, search and user fetchers is explained in training bot, search bot, fetcher.