scrape_date,brand,domain,bot,bot_class,blocked,named_in_file,collection,record,source
2026-11-10,Tesco,www.tesco.com,GPTBot,training,no,no,fetch tool; line counts cross-checked in Chrome,condensed record,https://www.tesco.com/robots.txt
2026-11-10,Tesco,www.tesco.com,OAI-SearchBot,retrieval,no,no,fetch tool; line counts cross-checked in Chrome,condensed record,https://www.tesco.com/robots.txt
2026-11-10,Tesco,www.tesco.com,ChatGPT-User,retrieval,no,no,fetch tool; line counts cross-checked in Chrome,condensed record,https://www.tesco.com/robots.txt
2026-11-10,Tesco,www.tesco.com,ClaudeBot,training,no,no,fetch tool; line counts cross-checked in Chrome,condensed record,https://www.tesco.com/robots.txt
2026-11-10,Tesco,www.tesco.com,Claude-SearchBot,retrieval,no,no,fetch tool; line counts cross-checked in Chrome,condensed record,https://www.tesco.com/robots.txt
2026-11-10,Tesco,www.tesco.com,Claude-User,retrieval,no,no,fetch tool; line counts cross-checked in Chrome,condensed record,https://www.tesco.com/robots.txt
2026-11-10,Tesco,www.tesco.com,anthropic-ai,training,no,no,fetch tool; line counts cross-checked in Chrome,condensed record,https://www.tesco.com/robots.txt
2026-11-10,Tesco,www.tesco.com,Google-Extended,training,no,no,fetch tool; line counts cross-checked in Chrome,condensed record,https://www.tesco.com/robots.txt
2026-11-10,Tesco,www.tesco.com,PerplexityBot,retrieval,no,no,fetch tool; line counts cross-checked in Chrome,condensed record,https://www.tesco.com/robots.txt
2026-11-10,Tesco,www.tesco.com,Perplexity-User,retrieval,no,no,fetch tool; line counts cross-checked in Chrome,condensed record,https://www.tesco.com/robots.txt
2026-11-10,Tesco,www.tesco.com,CCBot,training,no,no,fetch tool; line counts cross-checked in Chrome,condensed record,https://www.tesco.com/robots.txt
2026-11-10,Tesco,www.tesco.com,Applebot-Extended,training,no,no,fetch tool; line counts cross-checked in Chrome,condensed record,https://www.tesco.com/robots.txt
2026-11-10,Tesco,www.tesco.com,Bytespider,training,no,no,fetch tool; line counts cross-checked in Chrome,condensed record,https://www.tesco.com/robots.txt
2026-11-10,Tesco,www.tesco.com,Meta-ExternalAgent,training,no,no,fetch tool; line counts cross-checked in Chrome,condensed record,https://www.tesco.com/robots.txt
2026-11-10,Tesco,www.tesco.com,Amazonbot,training,no,no,fetch tool; line counts cross-checked in Chrome,condensed record,https://www.tesco.com/robots.txt
2026-11-10,Sainsbury's,www.sainsburys.co.uk,GPTBot,training,no,no,fetch tool,condensed record,https://www.sainsburys.co.uk/robots.txt
2026-11-10,Sainsbury's,www.sainsburys.co.uk,OAI-SearchBot,retrieval,no,no,fetch tool,condensed record,https://www.sainsburys.co.uk/robots.txt
2026-11-10,Sainsbury's,www.sainsburys.co.uk,ChatGPT-User,retrieval,no,no,fetch tool,condensed record,https://www.sainsburys.co.uk/robots.txt
2026-11-10,Sainsbury's,www.sainsburys.co.uk,ClaudeBot,training,no,no,fetch tool,condensed record,https://www.sainsburys.co.uk/robots.txt
2026-11-10,Sainsbury's,www.sainsburys.co.uk,Claude-SearchBot,retrieval,no,no,fetch tool,condensed record,https://www.sainsburys.co.uk/robots.txt
2026-11-10,Sainsbury's,www.sainsburys.co.uk,Claude-User,retrieval,no,no,fetch tool,condensed record,https://www.sainsburys.co.uk/robots.txt
2026-11-10,Sainsbury's,www.sainsburys.co.uk,anthropic-ai,training,no,no,fetch tool,condensed record,https://www.sainsburys.co.uk/robots.txt
2026-11-10,Sainsbury's,www.sainsburys.co.uk,Google-Extended,training,no,no,fetch tool,condensed record,https://www.sainsburys.co.uk/robots.txt
2026-11-10,Sainsbury's,www.sainsburys.co.uk,PerplexityBot,retrieval,no,no,fetch tool,condensed record,https://www.sainsburys.co.uk/robots.txt
2026-11-10,Sainsbury's,www.sainsburys.co.uk,Perplexity-User,retrieval,no,no,fetch tool,condensed record,https://www.sainsburys.co.uk/robots.txt
2026-11-10,Sainsbury's,www.sainsburys.co.uk,CCBot,training,no,no,fetch tool,condensed record,https://www.sainsburys.co.uk/robots.txt
2026-11-10,Sainsbury's,www.sainsburys.co.uk,Applebot-Extended,training,no,no,fetch tool,condensed record,https://www.sainsburys.co.uk/robots.txt
2026-11-10,Sainsbury's,www.sainsburys.co.uk,Bytespider,training,no,no,fetch tool,condensed record,https://www.sainsburys.co.uk/robots.txt
2026-11-10,Sainsbury's,www.sainsburys.co.uk,Meta-ExternalAgent,training,no,no,fetch tool,condensed record,https://www.sainsburys.co.uk/robots.txt
2026-11-10,Sainsbury's,www.sainsburys.co.uk,Amazonbot,training,no,no,fetch tool,condensed record,https://www.sainsburys.co.uk/robots.txt
2026-11-10,Asda,www.asda.com,GPTBot,training,no,no,fetch tool,full copy,https://www.asda.com/robots.txt
2026-11-10,Asda,www.asda.com,OAI-SearchBot,retrieval,no,no,fetch tool,full copy,https://www.asda.com/robots.txt
2026-11-10,Asda,www.asda.com,ChatGPT-User,retrieval,no,no,fetch tool,full copy,https://www.asda.com/robots.txt
2026-11-10,Asda,www.asda.com,ClaudeBot,training,no,no,fetch tool,full copy,https://www.asda.com/robots.txt
2026-11-10,Asda,www.asda.com,Claude-SearchBot,retrieval,no,no,fetch tool,full copy,https://www.asda.com/robots.txt
2026-11-10,Asda,www.asda.com,Claude-User,retrieval,no,no,fetch tool,full copy,https://www.asda.com/robots.txt
2026-11-10,Asda,www.asda.com,anthropic-ai,training,no,no,fetch tool,full copy,https://www.asda.com/robots.txt
2026-11-10,Asda,www.asda.com,Google-Extended,training,no,no,fetch tool,full copy,https://www.asda.com/robots.txt
2026-11-10,Asda,www.asda.com,PerplexityBot,retrieval,no,no,fetch tool,full copy,https://www.asda.com/robots.txt
2026-11-10,Asda,www.asda.com,Perplexity-User,retrieval,no,no,fetch tool,full copy,https://www.asda.com/robots.txt
2026-11-10,Asda,www.asda.com,CCBot,training,no,no,fetch tool,full copy,https://www.asda.com/robots.txt
2026-11-10,Asda,www.asda.com,Applebot-Extended,training,no,no,fetch tool,full copy,https://www.asda.com/robots.txt
2026-11-10,Asda,www.asda.com,Bytespider,training,no,no,fetch tool,full copy,https://www.asda.com/robots.txt
2026-11-10,Asda,www.asda.com,Meta-ExternalAgent,training,no,no,fetch tool,full copy,https://www.asda.com/robots.txt
2026-11-10,Asda,www.asda.com,Amazonbot,training,no,no,fetch tool,full copy,https://www.asda.com/robots.txt
2026-11-10,Aldi,www.aldi.co.uk,GPTBot,training,no,yes,fetch tool,full copy,https://www.aldi.co.uk/robots.txt
2026-11-10,Aldi,www.aldi.co.uk,OAI-SearchBot,retrieval,no,yes,fetch tool,full copy,https://www.aldi.co.uk/robots.txt
2026-11-10,Aldi,www.aldi.co.uk,ChatGPT-User,retrieval,no,yes,fetch tool,full copy,https://www.aldi.co.uk/robots.txt
2026-11-10,Aldi,www.aldi.co.uk,ClaudeBot,training,no,yes,fetch tool,full copy,https://www.aldi.co.uk/robots.txt
2026-11-10,Aldi,www.aldi.co.uk,Claude-SearchBot,retrieval,no,no,fetch tool,full copy,https://www.aldi.co.uk/robots.txt
2026-11-10,Aldi,www.aldi.co.uk,Claude-User,retrieval,no,no,fetch tool,full copy,https://www.aldi.co.uk/robots.txt
2026-11-10,Aldi,www.aldi.co.uk,anthropic-ai,training,no,yes,fetch tool,full copy,https://www.aldi.co.uk/robots.txt
2026-11-10,Aldi,www.aldi.co.uk,Google-Extended,training,no,yes,fetch tool,full copy,https://www.aldi.co.uk/robots.txt
2026-11-10,Aldi,www.aldi.co.uk,PerplexityBot,retrieval,no,yes,fetch tool,full copy,https://www.aldi.co.uk/robots.txt
2026-11-10,Aldi,www.aldi.co.uk,Perplexity-User,retrieval,no,no,fetch tool,full copy,https://www.aldi.co.uk/robots.txt
2026-11-10,Aldi,www.aldi.co.uk,CCBot,training,no,yes,fetch tool,full copy,https://www.aldi.co.uk/robots.txt
2026-11-10,Aldi,www.aldi.co.uk,Applebot-Extended,training,no,yes,fetch tool,full copy,https://www.aldi.co.uk/robots.txt
2026-11-10,Aldi,www.aldi.co.uk,Bytespider,training,no,no,fetch tool,full copy,https://www.aldi.co.uk/robots.txt
2026-11-10,Aldi,www.aldi.co.uk,Meta-ExternalAgent,training,no,no,fetch tool,full copy,https://www.aldi.co.uk/robots.txt
2026-11-10,Aldi,www.aldi.co.uk,Amazonbot,training,no,no,fetch tool,full copy,https://www.aldi.co.uk/robots.txt
2026-11-10,Morrisons,www.morrisons.com,GPTBot,training,no,no,fetch tool,full copy,https://www.morrisons.com/robots.txt
2026-11-10,Morrisons,www.morrisons.com,OAI-SearchBot,retrieval,no,no,fetch tool,full copy,https://www.morrisons.com/robots.txt
2026-11-10,Morrisons,www.morrisons.com,ChatGPT-User,retrieval,no,no,fetch tool,full copy,https://www.morrisons.com/robots.txt
2026-11-10,Morrisons,www.morrisons.com,ClaudeBot,training,no,no,fetch tool,full copy,https://www.morrisons.com/robots.txt
2026-11-10,Morrisons,www.morrisons.com,Claude-SearchBot,retrieval,no,no,fetch tool,full copy,https://www.morrisons.com/robots.txt
2026-11-10,Morrisons,www.morrisons.com,Claude-User,retrieval,no,no,fetch tool,full copy,https://www.morrisons.com/robots.txt
2026-11-10,Morrisons,www.morrisons.com,anthropic-ai,training,no,no,fetch tool,full copy,https://www.morrisons.com/robots.txt
2026-11-10,Morrisons,www.morrisons.com,Google-Extended,training,no,no,fetch tool,full copy,https://www.morrisons.com/robots.txt
2026-11-10,Morrisons,www.morrisons.com,PerplexityBot,retrieval,no,no,fetch tool,full copy,https://www.morrisons.com/robots.txt
2026-11-10,Morrisons,www.morrisons.com,Perplexity-User,retrieval,no,no,fetch tool,full copy,https://www.morrisons.com/robots.txt
2026-11-10,Morrisons,www.morrisons.com,CCBot,training,no,no,fetch tool,full copy,https://www.morrisons.com/robots.txt
2026-11-10,Morrisons,www.morrisons.com,Applebot-Extended,training,no,no,fetch tool,full copy,https://www.morrisons.com/robots.txt
2026-11-10,Morrisons,www.morrisons.com,Bytespider,training,no,no,fetch tool,full copy,https://www.morrisons.com/robots.txt
2026-11-10,Morrisons,www.morrisons.com,Meta-ExternalAgent,training,no,no,fetch tool,full copy,https://www.morrisons.com/robots.txt
2026-11-10,Morrisons,www.morrisons.com,Amazonbot,training,no,no,fetch tool,full copy,https://www.morrisons.com/robots.txt
2026-11-10,Lidl,www.lidl.co.uk,GPTBot,training,no,no,fetch tool,full copy,https://www.lidl.co.uk/robots.txt
2026-11-10,Lidl,www.lidl.co.uk,OAI-SearchBot,retrieval,no,no,fetch tool,full copy,https://www.lidl.co.uk/robots.txt
2026-11-10,Lidl,www.lidl.co.uk,ChatGPT-User,retrieval,no,no,fetch tool,full copy,https://www.lidl.co.uk/robots.txt
2026-11-10,Lidl,www.lidl.co.uk,ClaudeBot,training,no,no,fetch tool,full copy,https://www.lidl.co.uk/robots.txt
2026-11-10,Lidl,www.lidl.co.uk,Claude-SearchBot,retrieval,no,no,fetch tool,full copy,https://www.lidl.co.uk/robots.txt
2026-11-10,Lidl,www.lidl.co.uk,Claude-User,retrieval,no,no,fetch tool,full copy,https://www.lidl.co.uk/robots.txt
2026-11-10,Lidl,www.lidl.co.uk,anthropic-ai,training,no,no,fetch tool,full copy,https://www.lidl.co.uk/robots.txt
2026-11-10,Lidl,www.lidl.co.uk,Google-Extended,training,no,no,fetch tool,full copy,https://www.lidl.co.uk/robots.txt
2026-11-10,Lidl,www.lidl.co.uk,PerplexityBot,retrieval,no,no,fetch tool,full copy,https://www.lidl.co.uk/robots.txt
2026-11-10,Lidl,www.lidl.co.uk,Perplexity-User,retrieval,no,no,fetch tool,full copy,https://www.lidl.co.uk/robots.txt
2026-11-10,Lidl,www.lidl.co.uk,CCBot,training,no,no,fetch tool,full copy,https://www.lidl.co.uk/robots.txt
2026-11-10,Lidl,www.lidl.co.uk,Applebot-Extended,training,no,no,fetch tool,full copy,https://www.lidl.co.uk/robots.txt
2026-11-10,Lidl,www.lidl.co.uk,Bytespider,training,no,no,fetch tool,full copy,https://www.lidl.co.uk/robots.txt
2026-11-10,Lidl,www.lidl.co.uk,Meta-ExternalAgent,training,no,no,fetch tool,full copy,https://www.lidl.co.uk/robots.txt
2026-11-10,Lidl,www.lidl.co.uk,Amazonbot,training,no,no,fetch tool,full copy,https://www.lidl.co.uk/robots.txt
2026-11-10,Co-op,www.coop.co.uk,GPTBot,training,no,no,fetch tool,full copy,https://www.coop.co.uk/robots.txt
2026-11-10,Co-op,www.coop.co.uk,OAI-SearchBot,retrieval,no,no,fetch tool,full copy,https://www.coop.co.uk/robots.txt
2026-11-10,Co-op,www.coop.co.uk,ChatGPT-User,retrieval,no,no,fetch tool,full copy,https://www.coop.co.uk/robots.txt
2026-11-10,Co-op,www.coop.co.uk,ClaudeBot,training,no,no,fetch tool,full copy,https://www.coop.co.uk/robots.txt
2026-11-10,Co-op,www.coop.co.uk,Claude-SearchBot,retrieval,no,no,fetch tool,full copy,https://www.coop.co.uk/robots.txt
2026-11-10,Co-op,www.coop.co.uk,Claude-User,retrieval,no,no,fetch tool,full copy,https://www.coop.co.uk/robots.txt
2026-11-10,Co-op,www.coop.co.uk,anthropic-ai,training,no,no,fetch tool,full copy,https://www.coop.co.uk/robots.txt
2026-11-10,Co-op,www.coop.co.uk,Google-Extended,training,no,no,fetch tool,full copy,https://www.coop.co.uk/robots.txt
2026-11-10,Co-op,www.coop.co.uk,PerplexityBot,retrieval,no,no,fetch tool,full copy,https://www.coop.co.uk/robots.txt
2026-11-10,Co-op,www.coop.co.uk,Perplexity-User,retrieval,no,no,fetch tool,full copy,https://www.coop.co.uk/robots.txt
2026-11-10,Co-op,www.coop.co.uk,CCBot,training,no,no,fetch tool,full copy,https://www.coop.co.uk/robots.txt
2026-11-10,Co-op,www.coop.co.uk,Applebot-Extended,training,no,no,fetch tool,full copy,https://www.coop.co.uk/robots.txt
2026-11-10,Co-op,www.coop.co.uk,Bytespider,training,no,no,fetch tool,full copy,https://www.coop.co.uk/robots.txt
2026-11-10,Co-op,www.coop.co.uk,Meta-ExternalAgent,training,no,no,fetch tool,full copy,https://www.coop.co.uk/robots.txt
2026-11-10,Co-op,www.coop.co.uk,Amazonbot,training,no,no,fetch tool,full copy,https://www.coop.co.uk/robots.txt
2026-11-10,Waitrose,www.waitrose.com,GPTBot,training,no,no,fetch tool,condensed record,https://www.waitrose.com/robots.txt
2026-11-10,Waitrose,www.waitrose.com,OAI-SearchBot,retrieval,no,no,fetch tool,condensed record,https://www.waitrose.com/robots.txt
2026-11-10,Waitrose,www.waitrose.com,ChatGPT-User,retrieval,no,no,fetch tool,condensed record,https://www.waitrose.com/robots.txt
2026-11-10,Waitrose,www.waitrose.com,ClaudeBot,training,no,no,fetch tool,condensed record,https://www.waitrose.com/robots.txt
2026-11-10,Waitrose,www.waitrose.com,Claude-SearchBot,retrieval,no,no,fetch tool,condensed record,https://www.waitrose.com/robots.txt
2026-11-10,Waitrose,www.waitrose.com,Claude-User,retrieval,no,no,fetch tool,condensed record,https://www.waitrose.com/robots.txt
2026-11-10,Waitrose,www.waitrose.com,anthropic-ai,training,no,no,fetch tool,condensed record,https://www.waitrose.com/robots.txt
2026-11-10,Waitrose,www.waitrose.com,Google-Extended,training,no,no,fetch tool,condensed record,https://www.waitrose.com/robots.txt
2026-11-10,Waitrose,www.waitrose.com,PerplexityBot,retrieval,no,no,fetch tool,condensed record,https://www.waitrose.com/robots.txt
2026-11-10,Waitrose,www.waitrose.com,Perplexity-User,retrieval,no,no,fetch tool,condensed record,https://www.waitrose.com/robots.txt
2026-11-10,Waitrose,www.waitrose.com,CCBot,training,no,no,fetch tool,condensed record,https://www.waitrose.com/robots.txt
2026-11-10,Waitrose,www.waitrose.com,Applebot-Extended,training,no,no,fetch tool,condensed record,https://www.waitrose.com/robots.txt
2026-11-10,Waitrose,www.waitrose.com,Bytespider,training,no,no,fetch tool,condensed record,https://www.waitrose.com/robots.txt
2026-11-10,Waitrose,www.waitrose.com,Meta-ExternalAgent,training,no,no,fetch tool,condensed record,https://www.waitrose.com/robots.txt
2026-11-10,Waitrose,www.waitrose.com,Amazonbot,training,no,no,fetch tool,condensed record,https://www.waitrose.com/robots.txt
2026-11-10,Iceland,www.iceland.co.uk,GPTBot,training,no,no,fetch tool,condensed record,https://www.iceland.co.uk/robots.txt
2026-11-10,Iceland,www.iceland.co.uk,OAI-SearchBot,retrieval,no,no,fetch tool,condensed record,https://www.iceland.co.uk/robots.txt
2026-11-10,Iceland,www.iceland.co.uk,ChatGPT-User,retrieval,no,no,fetch tool,condensed record,https://www.iceland.co.uk/robots.txt
2026-11-10,Iceland,www.iceland.co.uk,ClaudeBot,training,no,no,fetch tool,condensed record,https://www.iceland.co.uk/robots.txt
2026-11-10,Iceland,www.iceland.co.uk,Claude-SearchBot,retrieval,no,no,fetch tool,condensed record,https://www.iceland.co.uk/robots.txt
2026-11-10,Iceland,www.iceland.co.uk,Claude-User,retrieval,no,no,fetch tool,condensed record,https://www.iceland.co.uk/robots.txt
2026-11-10,Iceland,www.iceland.co.uk,anthropic-ai,training,no,no,fetch tool,condensed record,https://www.iceland.co.uk/robots.txt
2026-11-10,Iceland,www.iceland.co.uk,Google-Extended,training,no,no,fetch tool,condensed record,https://www.iceland.co.uk/robots.txt
2026-11-10,Iceland,www.iceland.co.uk,PerplexityBot,retrieval,no,no,fetch tool,condensed record,https://www.iceland.co.uk/robots.txt
2026-11-10,Iceland,www.iceland.co.uk,Perplexity-User,retrieval,no,no,fetch tool,condensed record,https://www.iceland.co.uk/robots.txt
2026-11-10,Iceland,www.iceland.co.uk,CCBot,training,no,no,fetch tool,condensed record,https://www.iceland.co.uk/robots.txt
2026-11-10,Iceland,www.iceland.co.uk,Applebot-Extended,training,no,no,fetch tool,condensed record,https://www.iceland.co.uk/robots.txt
2026-11-10,Iceland,www.iceland.co.uk,Bytespider,training,no,no,fetch tool,condensed record,https://www.iceland.co.uk/robots.txt
2026-11-10,Iceland,www.iceland.co.uk,Meta-ExternalAgent,training,no,no,fetch tool,condensed record,https://www.iceland.co.uk/robots.txt
2026-11-10,Iceland,www.iceland.co.uk,Amazonbot,training,no,no,fetch tool,condensed record,https://www.iceland.co.uk/robots.txt
2026-11-10,Ocado,www.ocado.com,GPTBot,training,no,no,chrome,condensed record,https://www.ocado.com/robots.txt
2026-11-10,Ocado,www.ocado.com,OAI-SearchBot,retrieval,no,no,chrome,condensed record,https://www.ocado.com/robots.txt
2026-11-10,Ocado,www.ocado.com,ChatGPT-User,retrieval,no,no,chrome,condensed record,https://www.ocado.com/robots.txt
2026-11-10,Ocado,www.ocado.com,ClaudeBot,training,no,no,chrome,condensed record,https://www.ocado.com/robots.txt
2026-11-10,Ocado,www.ocado.com,Claude-SearchBot,retrieval,no,no,chrome,condensed record,https://www.ocado.com/robots.txt
2026-11-10,Ocado,www.ocado.com,Claude-User,retrieval,no,no,chrome,condensed record,https://www.ocado.com/robots.txt
2026-11-10,Ocado,www.ocado.com,anthropic-ai,training,no,no,chrome,condensed record,https://www.ocado.com/robots.txt
2026-11-10,Ocado,www.ocado.com,Google-Extended,training,no,no,chrome,condensed record,https://www.ocado.com/robots.txt
2026-11-10,Ocado,www.ocado.com,PerplexityBot,retrieval,no,no,chrome,condensed record,https://www.ocado.com/robots.txt
2026-11-10,Ocado,www.ocado.com,Perplexity-User,retrieval,no,no,chrome,condensed record,https://www.ocado.com/robots.txt
2026-11-10,Ocado,www.ocado.com,CCBot,training,no,no,chrome,condensed record,https://www.ocado.com/robots.txt
2026-11-10,Ocado,www.ocado.com,Applebot-Extended,training,no,no,chrome,condensed record,https://www.ocado.com/robots.txt
2026-11-10,Ocado,www.ocado.com,Bytespider,training,no,no,chrome,condensed record,https://www.ocado.com/robots.txt
2026-11-10,Ocado,www.ocado.com,Meta-ExternalAgent,training,no,no,chrome,condensed record,https://www.ocado.com/robots.txt
2026-11-10,Ocado,www.ocado.com,Amazonbot,training,no,no,chrome,condensed record,https://www.ocado.com/robots.txt