Paste your robots.txt and see which of 15 AI crawlers it blocks, next to what the UK’s ten biggest newsbrands and ten biggest supermarkets do. Benchmarks checked 2 and 10 November 2026.
Open yoursite.com/robots.txt, copy everything, paste it here. The check runs in your browser. Nothing you paste is sent anywhere.
What the benchmarks are
- Newsbrands: the UK’s ten biggest by audience, robots.txt read on 2 November 2026. Study and data.
- Supermarkets: the UK’s ten biggest by grocery market share, robots.txt read on 10 November 2026. Study and data.
- The 15 crawlers and the definition of blocked are in the method note.
What this tool can’t tell you
- Whether any crawler obeys your file. robots.txt is a request.
- Whether your server or CDN blocks crawlers before they read the file.
- What you should block. To write a file, use the AI robots.txt generator.