site,robots_url,checked,GPTBot,OAI-SearchBot,ChatGPT-User,ClaudeBot,Claude-SearchBot,Claude-User,anthropic-ai,Google-Extended,PerplexityBot,Perplexity-User,CCBot,Applebot-Extended,Bytespider,Meta-ExternalAgent,Amazonbot
Sky News,https://news.sky.com/robots.txt,2026-10-02,blocked (named),not blocked (named),not blocked (named),blocked (named),not blocked (named),not blocked (named),blocked (named),blocked (named),blocked (named),not blocked (named),blocked (named),blocked (named),blocked (named),blocked (named),blocked (named)
The Guardian,https://www.theguardian.com/robots.txt,2026-10-02,not blocked (via * rule),not blocked (via * rule),not blocked (via * rule),blocked (named),blocked (named),blocked (named),blocked (named),not blocked (via * rule),blocked (named),not blocked (via * rule),blocked (named),blocked (named),blocked (named),blocked (named),blocked (named)
Daily Mail,https://www.dailymail.com/robots.txt,2026-10-02,blocked (named),blocked (named),blocked (named),blocked (named),blocked (named),blocked (named),blocked (named),blocked (named),blocked (named),blocked (named),blocked (named),blocked (named),blocked (named),blocked (named),blocked (named)
The Independent,https://www.independent.co.uk/robots.txt,2026-10-02,not blocked (via * rule),not blocked (via * rule),not blocked (via * rule),not blocked (via * rule),not blocked (via * rule),not blocked (via * rule),not blocked (via * rule),not blocked (via * rule),not blocked (via * rule),not blocked (via * rule),not blocked (via * rule),not blocked (via * rule),not blocked (via * rule),not blocked (via * rule),not blocked (via * rule)
BBC,https://www.bbc.co.uk/robots.txt,2026-10-02,blocked (named),blocked (named),blocked (named),blocked (named),not blocked (via * rule),not blocked (via * rule),blocked (named),blocked (named),blocked (named),blocked (named),blocked (named),blocked (named),blocked (named),blocked (named),blocked (named)
Mirror,https://www.mirror.co.uk/robots.txt,2026-10-02,blocked (named),blocked (named),not blocked (via * rule),blocked (named),not blocked (via * rule),not blocked (via * rule),blocked (named),not blocked (via * rule),blocked (named),not blocked (via * rule),blocked (named),blocked (named),not blocked (via * rule),blocked (named),not blocked (via * rule)
Yahoo! (UK news),https://uk.news.yahoo.com/robots.txt,2026-10-02,blocked (named),blocked (named),blocked (named),blocked (named),blocked (named),not blocked (via * rule),blocked (named),blocked (named),blocked (named),not blocked (via * rule),blocked (named),not blocked (via * rule),blocked (named),not blocked (via * rule),not blocked (via * rule)
The Sun,https://www.thesun.co.uk/robots.txt,2026-10-02,not blocked (named),not blocked (named),not blocked (named),blocked (via * rule),blocked (via * rule),blocked (via * rule),blocked (via * rule),blocked (via * rule),blocked (via * rule),blocked (via * rule),blocked (via * rule),blocked (via * rule),blocked (via * rule),blocked (via * rule),blocked (via * rule)
Daily Express,https://www.express.co.uk/robots.txt,2026-10-02,blocked (named),blocked (named),not blocked (via * rule),blocked (named),not blocked (via * rule),not blocked (via * rule),blocked (named),not blocked (via * rule),blocked (named),not blocked (via * rule),blocked (named),blocked (named),not blocked (via * rule),not blocked (via * rule),not blocked (via * rule)
The Telegraph,https://www.telegraph.co.uk/robots.txt,2026-10-02,blocked (named),blocked (named),blocked (named),blocked (named),blocked (named),blocked (named),blocked (named),not blocked (named),blocked (named),blocked (named),blocked (named),blocked (named),blocked (named),blocked (named),blocked (named)