scrape_date,brand,domain,token,whole_site_disallow,source
2026-11-02,Sky News,news.sky.com,AI2Bot,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-news-sky-com-2026-11-02.txt
2026-11-02,Sky News,news.sky.com,AwarioRssBot,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-news-sky-com-2026-11-02.txt
2026-11-02,Sky News,news.sky.com,AwarioSmartBot,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-news-sky-com-2026-11-02.txt
2026-11-02,Sky News,news.sky.com,cohere-ai,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-news-sky-com-2026-11-02.txt
2026-11-02,Sky News,news.sky.com,Diffbot,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-news-sky-com-2026-11-02.txt
2026-11-02,Sky News,news.sky.com,FacebookBot,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-news-sky-com-2026-11-02.txt
2026-11-02,Sky News,news.sky.com,magpie-crawler,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-news-sky-com-2026-11-02.txt
2026-11-02,Sky News,news.sky.com,omgili,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-news-sky-com-2026-11-02.txt
2026-11-02,Sky News,news.sky.com,omgilibot,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-news-sky-com-2026-11-02.txt
2026-11-02,Sky News,news.sky.com,PanguBot,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-news-sky-com-2026-11-02.txt
2026-11-02,Sky News,news.sky.com,Scrapy,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-news-sky-com-2026-11-02.txt
2026-11-02,Sky News,news.sky.com,TurnitinBot,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-news-sky-com-2026-11-02.txt
2026-11-02,Sky News,news.sky.com,Webzio-Extended,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-news-sky-com-2026-11-02.txt
2026-11-02,Sky News,news.sky.com,Claude-Web,no,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-news-sky-com-2026-11-02.txt
2026-11-02,Sky News,news.sky.com,MistralAI-User,no,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-news-sky-com-2026-11-02.txt
2026-11-02,Sky News,news.sky.com,NewsNow,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-news-sky-com-2026-11-02.txt
2026-11-02,Sky News,news.sky.com,news-please,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-news-sky-com-2026-11-02.txt
2026-11-02,Sky News,news.sky.com,DataForSeoBot,no,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-news-sky-com-2026-11-02.txt
2026-11-02,The Guardian,theguardian.com,Mediapartners-Google,no,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-theguardian-com-2026-11-02.txt
2026-11-02,The Guardian,theguardian.com,NewsNow,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-theguardian-com-2026-11-02.txt
2026-11-02,The Guardian,theguardian.com,TurnitinBot,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-theguardian-com-2026-11-02.txt
2026-11-02,The Guardian,theguardian.com,PetalBot,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-theguardian-com-2026-11-02.txt
2026-11-02,The Guardian,theguardian.com,MoodleBot,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-theguardian-com-2026-11-02.txt
2026-11-02,The Guardian,theguardian.com,FacebookBot,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-theguardian-com-2026-11-02.txt
2026-11-02,The Guardian,theguardian.com,Mojeek,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-theguardian-com-2026-11-02.txt
2026-11-02,The Guardian,theguardian.com,JenkersBot,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-theguardian-com-2026-11-02.txt
2026-11-02,The Guardian,theguardian.com,Seekr,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-theguardian-com-2026-11-02.txt
2026-11-02,The Guardian,theguardian.com,YouBot,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-theguardian-com-2026-11-02.txt
2026-11-02,The Guardian,theguardian.com,Arquivo-web-crawler,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-theguardian-com-2026-11-02.txt
2026-11-02,The Guardian,theguardian.com,coccocbot-web,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-theguardian-com-2026-11-02.txt
2026-11-02,The Guardian,theguardian.com,SeznamBot,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-theguardian-com-2026-11-02.txt
2026-11-02,The Guardian,theguardian.com,yacy,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-theguardian-com-2026-11-02.txt
2026-11-02,The Guardian,theguardian.com,AwarioRssBot,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-theguardian-com-2026-11-02.txt
2026-11-02,The Guardian,theguardian.com,AwarioSmartBot,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-theguardian-com-2026-11-02.txt
2026-11-02,The Guardian,theguardian.com,SentiOne,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-theguardian-com-2026-11-02.txt
2026-11-02,The Guardian,theguardian.com,ImageSift,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-theguardian-com-2026-11-02.txt
2026-11-02,The Guardian,theguardian.com,YandexAdditional,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-theguardian-com-2026-11-02.txt
2026-11-02,The Guardian,theguardian.com,YandexAdditionalBot,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-theguardian-com-2026-11-02.txt
2026-11-02,The Guardian,theguardian.com,scalepostAI,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-theguardian-com-2026-11-02.txt
2026-11-02,The Guardian,theguardian.com,Buck,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-theguardian-com-2026-11-02.txt
2026-11-02,The Guardian,theguardian.com,amazon-QBusiness,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-theguardian-com-2026-11-02.txt
2026-11-02,The Guardian,theguardian.com,DuckAssistBot,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-theguardian-com-2026-11-02.txt
2026-11-02,The Guardian,theguardian.com,Google-CloudVertexBot,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-theguardian-com-2026-11-02.txt
2026-11-02,The Guardian,theguardian.com,Amzn-SearchBot,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-theguardian-com-2026-11-02.txt
2026-11-02,The Guardian,theguardian.com,AhrefsBot,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-theguardian-com-2026-11-02.txt
2026-11-02,The Guardian,theguardian.com,AhrefsSiteAudit,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-theguardian-com-2026-11-02.txt
2026-11-02,Daily Mail,dailymail.com,Googlebot-News,no,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-dailymail-com-2026-11-02.txt
2026-11-02,Daily Mail,dailymail.com,Mediapartners-Google,no,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-dailymail-com-2026-11-02.txt
2026-11-02,Daily Mail,dailymail.com,CrystalSemanticsBot,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-dailymail-com-2026-11-02.txt
2026-11-02,Daily Mail,dailymail.com,archive-it.org,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-dailymail-com-2026-11-02.txt
2026-11-02,Daily Mail,dailymail.com,archive.org_bot,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-dailymail-com-2026-11-02.txt
2026-11-02,Daily Mail,dailymail.com,ArchiveBot,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-dailymail-com-2026-11-02.txt
2026-11-02,Daily Mail,dailymail.com,Arquivo-web-crawler,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-dailymail-com-2026-11-02.txt
2026-11-02,Daily Mail,dailymail.com,Authory,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-dailymail-com-2026-11-02.txt
2026-11-02,Daily Mail,dailymail.com,babel,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-dailymail-com-2026-11-02.txt
2026-11-02,Daily Mail,dailymail.com,bl.uk_ldfc_bot,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-dailymail-com-2026-11-02.txt
2026-11-02,Daily Mail,dailymail.com,bl.uk_ldfc_renderbot,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-dailymail-com-2026-11-02.txt
2026-11-02,Daily Mail,dailymail.com,bne.es_bot,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-dailymail-com-2026-11-02.txt
2026-11-02,Daily Mail,dailymail.com,bnf.fr_bot,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-dailymail-com-2026-11-02.txt
2026-11-02,Daily Mail,dailymail.com,ecoresearch,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-dailymail-com-2026-11-02.txt
2026-11-02,Daily Mail,dailymail.com,ecoresearchCrawler,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-dailymail-com-2026-11-02.txt
2026-11-02,Daily Mail,dailymail.com,Europarchive.org,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-dailymail-com-2026-11-02.txt
2026-11-02,Daily Mail,dailymail.com,kb.dk_bot,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-dailymail-com-2026-11-02.txt
2026-11-02,Daily Mail,dailymail.com,Kulturarw3,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-dailymail-com-2026-11-02.txt
2026-11-02,Daily Mail,dailymail.com,LivelapBot,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-dailymail-com-2026-11-02.txt
2026-11-02,Daily Mail,dailymail.com,mediacloud,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-dailymail-com-2026-11-02.txt
2026-11-02,Daily Mail,dailymail.com,mirrorweb,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-dailymail-com-2026-11-02.txt
2026-11-02,Daily Mail,dailymail.com,MSIECrawler,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-dailymail-com-2026-11-02.txt
2026-11-02,Daily Mail,dailymail.com,nationalarchives.gov.uk,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-dailymail-com-2026-11-02.txt
2026-11-02,Daily Mail,dailymail.com,netarkivet.dk,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-dailymail-com-2026-11-02.txt
2026-11-02,Daily Mail,dailymail.com,netEstate NE Crawler,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-dailymail-com-2026-11-02.txt
2026-11-02,Daily Mail,dailymail.com,Nicecrawler,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-dailymail-com-2026-11-02.txt
2026-11-02,Daily Mail,dailymail.com,nl-israel_iaharvester2026,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-dailymail-com-2026-11-02.txt
2026-11-02,Daily Mail,dailymail.com,nlnbot,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-dailymail-com-2026-11-02.txt
2026-11-02,Daily Mail,dailymail.com,nlncrawler,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-dailymail-com-2026-11-02.txt
2026-11-02,Daily Mail,dailymail.com,PageFreezer,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-dailymail-com-2026-11-02.txt
2026-11-02,Daily Mail,dailymail.com,SafeDNS,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-dailymail-com-2026-11-02.txt
2026-11-02,Daily Mail,dailymail.com,SmarshBot,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-dailymail-com-2026-11-02.txt
2026-11-02,Daily Mail,dailymail.com,special_archiver,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-dailymail-com-2026-11-02.txt
2026-11-02,Daily Mail,dailymail.com,tkl.iis.u-tokyo.ac.jp,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-dailymail-com-2026-11-02.txt
2026-11-02,Daily Mail,dailymail.com,TurnitinBot,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-dailymail-com-2026-11-02.txt
2026-11-02,Daily Mail,dailymail.com,webarchiv.cz,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-dailymail-com-2026-11-02.txt
2026-11-02,Daily Mail,dailymail.com,WordPress.com mShots,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-dailymail-com-2026-11-02.txt
2026-11-02,Daily Mail,dailymail.com,XY-Archive-Compliance-Archiver,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-dailymail-com-2026-11-02.txt
2026-11-02,Daily Mail,dailymail.com,XY-Archive-Compliance-Crawler,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-dailymail-com-2026-11-02.txt
2026-11-02,Daily Mail,dailymail.com,AhrefsBot,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-dailymail-com-2026-11-02.txt
2026-11-02,Daily Mail,dailymail.com,AI2Bot,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-dailymail-com-2026-11-02.txt
2026-11-02,Daily Mail,dailymail.com,Ai2Bot-Dolma,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-dailymail-com-2026-11-02.txt
2026-11-02,Daily Mail,dailymail.com,amazon-kendra-,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-dailymail-com-2026-11-02.txt
2026-11-02,Daily Mail,dailymail.com,bedrockbot,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-dailymail-com-2026-11-02.txt
2026-11-02,Daily Mail,dailymail.com,CloudflareBrowserRenderingCrawler,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-dailymail-com-2026-11-02.txt
2026-11-02,Daily Mail,dailymail.com,ChatGLM-Spider,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-dailymail-com-2026-11-02.txt
2026-11-02,Daily Mail,dailymail.com,Claude-Web,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-dailymail-com-2026-11-02.txt
2026-11-02,Daily Mail,dailymail.com,cohere-ai,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-dailymail-com-2026-11-02.txt
2026-11-02,Daily Mail,dailymail.com,cohere-training-data-crawler,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-dailymail-com-2026-11-02.txt
2026-11-02,Daily Mail,dailymail.com,Cotoyogi,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-dailymail-com-2026-11-02.txt
2026-11-02,Daily Mail,dailymail.com,DeepSeekBot,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-dailymail-com-2026-11-02.txt
2026-11-02,Daily Mail,dailymail.com,Diffbot,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-dailymail-com-2026-11-02.txt
2026-11-02,Daily Mail,dailymail.com,DuckAssistBot,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-dailymail-com-2026-11-02.txt
2026-11-02,Daily Mail,dailymail.com,EchoboxBot,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-dailymail-com-2026-11-02.txt
2026-11-02,Daily Mail,dailymail.com,FacebookBot,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-dailymail-com-2026-11-02.txt
2026-11-02,Daily Mail,dailymail.com,FriendlyCrawler,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-dailymail-com-2026-11-02.txt
2026-11-02,Daily Mail,dailymail.com,Gemini-Deep-Research,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-dailymail-com-2026-11-02.txt
2026-11-02,Daily Mail,dailymail.com,Google-CloudVertexBot,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-dailymail-com-2026-11-02.txt
2026-11-02,Daily Mail,dailymail.com,GoogleOther-Image,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-dailymail-com-2026-11-02.txt
2026-11-02,Daily Mail,dailymail.com,GoogleOther-Video,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-dailymail-com-2026-11-02.txt
2026-11-02,Daily Mail,dailymail.com,GPTBot-Search,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-dailymail-com-2026-11-02.txt
2026-11-02,Daily Mail,dailymail.com,Grok,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-dailymail-com-2026-11-02.txt
2026-11-02,Daily Mail,dailymail.com,iaskspider/2.0,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-dailymail-com-2026-11-02.txt
2026-11-02,Daily Mail,dailymail.com,ICC-Crawler,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-dailymail-com-2026-11-02.txt
2026-11-02,Daily Mail,dailymail.com,ImagesiftBot,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-dailymail-com-2026-11-02.txt
2026-11-02,Daily Mail,dailymail.com,img2dataset,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-dailymail-com-2026-11-02.txt
2026-11-02,Daily Mail,dailymail.com,ISSCyberRiskCrawler,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-dailymail-com-2026-11-02.txt
2026-11-02,Daily Mail,dailymail.com,Kangaroo Bot,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-dailymail-com-2026-11-02.txt
2026-11-02,Daily Mail,dailymail.com,KunatoCrawler,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-dailymail-com-2026-11-02.txt
2026-11-02,Daily Mail,dailymail.com,LinerBot,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-dailymail-com-2026-11-02.txt
2026-11-02,Daily Mail,dailymail.com,Meltwater,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-dailymail-com-2026-11-02.txt
2026-11-02,Daily Mail,dailymail.com,Meta-ExternalFetcher,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-dailymail-com-2026-11-02.txt
2026-11-02,Daily Mail,dailymail.com,Meta-WebIndexer,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-dailymail-com-2026-11-02.txt
2026-11-02,Daily Mail,dailymail.com,MistralAI-User,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-dailymail-com-2026-11-02.txt
2026-11-02,Daily Mail,dailymail.com,OAI-Operator,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-dailymail-com-2026-11-02.txt
2026-11-02,Daily Mail,dailymail.com,omgili,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-dailymail-com-2026-11-02.txt
2026-11-02,Daily Mail,dailymail.com,omgilibot,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-dailymail-com-2026-11-02.txt
2026-11-02,Daily Mail,dailymail.com,PanguBot,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-dailymail-com-2026-11-02.txt
2026-11-02,Daily Mail,dailymail.com,PetalBot,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-dailymail-com-2026-11-02.txt
2026-11-02,Daily Mail,dailymail.com,QualifiedBot,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-dailymail-com-2026-11-02.txt
2026-11-02,Daily Mail,dailymail.com,Scrapy,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-dailymail-com-2026-11-02.txt
2026-11-02,Daily Mail,dailymail.com,Seekr,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-dailymail-com-2026-11-02.txt
2026-11-02,Daily Mail,dailymail.com,Sidetrade indexer bot,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-dailymail-com-2026-11-02.txt
2026-11-02,Daily Mail,dailymail.com,TaraGroup Intelligent Bot,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-dailymail-com-2026-11-02.txt
2026-11-02,Daily Mail,dailymail.com,TikTokSpider,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-dailymail-com-2026-11-02.txt
2026-11-02,Daily Mail,dailymail.com,Timpibot,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-dailymail-com-2026-11-02.txt
2026-11-02,Daily Mail,dailymail.com,VelenPublicWebCrawler,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-dailymail-com-2026-11-02.txt
2026-11-02,Daily Mail,dailymail.com,WARDBot,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-dailymail-com-2026-11-02.txt
2026-11-02,Daily Mail,dailymail.com,Webzio-Extended,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-dailymail-com-2026-11-02.txt
2026-11-02,Daily Mail,dailymail.com,wpbot,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-dailymail-com-2026-11-02.txt
2026-11-02,Daily Mail,dailymail.com,WRTNBot,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-dailymail-com-2026-11-02.txt
2026-11-02,Daily Mail,dailymail.com,xAI-SearchBot,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-dailymail-com-2026-11-02.txt
2026-11-02,Daily Mail,dailymail.com,YouBot,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-dailymail-com-2026-11-02.txt
2026-11-02,Daily Mail,dailymail.com,GoogleOther,no,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-dailymail-com-2026-11-02.txt
2026-11-02,The Independent,independent.co.uk,Nutch,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-independent-co-uk-2026-11-02.txt
2026-11-02,BBC,bbc.co.uk,magpie-crawler,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-bbc-co-uk-2026-11-02.txt
2026-11-02,BBC,bbc.co.uk,omgili,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-bbc-co-uk-2026-11-02.txt
2026-11-02,BBC,bbc.co.uk,omgilibot,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-bbc-co-uk-2026-11-02.txt
2026-11-02,BBC,bbc.co.uk,Claude-Web,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-bbc-co-uk-2026-11-02.txt
2026-11-02,BBC,bbc.co.uk,cohere-ai,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-bbc-co-uk-2026-11-02.txt
2026-11-02,BBC,bbc.co.uk,PetalBot,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-bbc-co-uk-2026-11-02.txt
2026-11-02,BBC,bbc.co.uk,Scrapy,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-bbc-co-uk-2026-11-02.txt
2026-11-02,BBC,bbc.co.uk,Google-CloudVertexBot,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-bbc-co-uk-2026-11-02.txt
2026-11-02,BBC,bbc.co.uk,YandexAdditional,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-bbc-co-uk-2026-11-02.txt
2026-11-02,BBC,bbc.co.uk,YandexAdditionalBot,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-bbc-co-uk-2026-11-02.txt
2026-11-02,BBC,bbc.co.uk,TurnitinBot,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-bbc-co-uk-2026-11-02.txt
2026-11-02,BBC,bbc.co.uk,Brightbot,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-bbc-co-uk-2026-11-02.txt
2026-11-02,BBC,bbc.co.uk,ApifyBot,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-bbc-co-uk-2026-11-02.txt
2026-11-02,BBC,bbc.co.uk,ApifyWebsiteContentCrawler,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-bbc-co-uk-2026-11-02.txt
2026-11-02,BBC,bbc.co.uk,Diffbot,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-bbc-co-uk-2026-11-02.txt
2026-11-02,BBC,bbc.co.uk,Diffbot-User,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-bbc-co-uk-2026-11-02.txt
2026-11-02,BBC,bbc.co.uk,ExaBot,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-bbc-co-uk-2026-11-02.txt
2026-11-02,BBC,bbc.co.uk,TavilyBot,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-bbc-co-uk-2026-11-02.txt
2026-11-02,BBC,bbc.co.uk,ShapBot,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-bbc-co-uk-2026-11-02.txt
2026-11-02,BBC,bbc.co.uk,YouBot,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-bbc-co-uk-2026-11-02.txt
2026-11-02,BBC,bbc.co.uk,FirecrawlAgent,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-bbc-co-uk-2026-11-02.txt
2026-11-02,BBC,bbc.co.uk,Amzn-SearchBot,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-bbc-co-uk-2026-11-02.txt
2026-11-02,BBC,bbc.co.uk,Amzn-User,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-bbc-co-uk-2026-11-02.txt
2026-11-02,BBC,bbc.co.uk,ProRataInc,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-bbc-co-uk-2026-11-02.txt
2026-11-02,BBC,bbc.co.uk,CloudflareBrowserRenderingCrawler,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-bbc-co-uk-2026-11-02.txt
2026-11-02,BBC,bbc.co.uk,AhrefsBot,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-bbc-co-uk-2026-11-02.txt
2026-11-02,Mirror,mirror.co.uk,daumoa,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-mirror-co-uk-2026-11-02.txt
2026-11-02,Mirror,mirror.co.uk,Sosospider,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-mirror-co-uk-2026-11-02.txt
2026-11-02,Mirror,mirror.co.uk,rogerbot,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-mirror-co-uk-2026-11-02.txt
2026-11-02,Mirror,mirror.co.uk,semetrical,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-mirror-co-uk-2026-11-02.txt
2026-11-02,Mirror,mirror.co.uk,Googlebot-News,no,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-mirror-co-uk-2026-11-02.txt
2026-11-02,Mirror,mirror.co.uk,AhrefsBot,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-mirror-co-uk-2026-11-02.txt
2026-11-02,Mirror,mirror.co.uk,Perplexity-ai,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-mirror-co-uk-2026-11-02.txt
2026-11-02,Mirror,mirror.co.uk,YouBot,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-mirror-co-uk-2026-11-02.txt
2026-11-02,Mirror,mirror.co.uk,Claude-Web,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-mirror-co-uk-2026-11-02.txt
2026-11-02,Mirror,mirror.co.uk,grapeshot,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-mirror-co-uk-2026-11-02.txt
2026-11-02,Mirror,mirror.co.uk,bingbot,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-mirror-co-uk-2026-11-02.txt
2026-11-02,Mirror,mirror.co.uk,AmazonAdBot,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-mirror-co-uk-2026-11-02.txt
2026-11-02,Mirror,mirror.co.uk,ozone,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-mirror-co-uk-2026-11-02.txt
2026-11-02,Yahoo! (UK news),uk.news.yahoo.com,ADmantX,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-uk-news-yahoo-com-2026-11-02.txt
2026-11-02,Yahoo! (UK news),uk.news.yahoo.com,AlphaBot,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-uk-news-yahoo-com-2026-11-02.txt
2026-11-02,Yahoo! (UK news),uk.news.yahoo.com,AwarioRssBot,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-uk-news-yahoo-com-2026-11-02.txt
2026-11-02,Yahoo! (UK news),uk.news.yahoo.com,AwarioSmartBot,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-uk-news-yahoo-com-2026-11-02.txt
2026-11-02,Yahoo! (UK news),uk.news.yahoo.com,BLEXBot,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-uk-news-yahoo-com-2026-11-02.txt
2026-11-02,Yahoo! (UK news),uk.news.yahoo.com,Buzzbot,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-uk-news-yahoo-com-2026-11-02.txt
2026-11-02,Yahoo! (UK news),uk.news.yahoo.com,claritybot,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-uk-news-yahoo-com-2026-11-02.txt
2026-11-02,Yahoo! (UK news),uk.news.yahoo.com,Claude-Web,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-uk-news-yahoo-com-2026-11-02.txt
2026-11-02,Yahoo! (UK news),uk.news.yahoo.com,cohere-ai,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-uk-news-yahoo-com-2026-11-02.txt
2026-11-02,Yahoo! (UK news),uk.news.yahoo.com,Diffbot,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-uk-news-yahoo-com-2026-11-02.txt
2026-11-02,Yahoo! (UK news),uk.news.yahoo.com,FacebookBot,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-uk-news-yahoo-com-2026-11-02.txt
2026-11-02,Yahoo! (UK news),uk.news.yahoo.com,FriendlyCrawler,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-uk-news-yahoo-com-2026-11-02.txt
2026-11-02,Yahoo! (UK news),uk.news.yahoo.com,huggingface,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-uk-news-yahoo-com-2026-11-02.txt
2026-11-02,Yahoo! (UK news),uk.news.yahoo.com,ImagesiftBot,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-uk-news-yahoo-com-2026-11-02.txt
2026-11-02,Yahoo! (UK news),uk.news.yahoo.com,img2dataset,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-uk-news-yahoo-com-2026-11-02.txt
2026-11-02,Yahoo! (UK news),uk.news.yahoo.com,magpie-crawler,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-uk-news-yahoo-com-2026-11-02.txt
2026-11-02,Yahoo! (UK news),uk.news.yahoo.com,Meltwater,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-uk-news-yahoo-com-2026-11-02.txt
2026-11-02,Yahoo! (UK news),uk.news.yahoo.com,Neevabot,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-uk-news-yahoo-com-2026-11-02.txt
2026-11-02,Yahoo! (UK news),uk.news.yahoo.com,news-please,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-uk-news-yahoo-com-2026-11-02.txt
2026-11-02,Yahoo! (UK news),uk.news.yahoo.com,NewsNow,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-uk-news-yahoo-com-2026-11-02.txt
2026-11-02,Yahoo! (UK news),uk.news.yahoo.com,Nutch,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-uk-news-yahoo-com-2026-11-02.txt
2026-11-02,Yahoo! (UK news),uk.news.yahoo.com,omgili,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-uk-news-yahoo-com-2026-11-02.txt
2026-11-02,Yahoo! (UK news),uk.news.yahoo.com,omgilibot,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-uk-news-yahoo-com-2026-11-02.txt
2026-11-02,Yahoo! (UK news),uk.news.yahoo.com,panscient.com,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-uk-news-yahoo-com-2026-11-02.txt
2026-11-02,Yahoo! (UK news),uk.news.yahoo.com,Perplexity-ai,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-uk-news-yahoo-com-2026-11-02.txt
2026-11-02,Yahoo! (UK news),uk.news.yahoo.com,PetalBot,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-uk-news-yahoo-com-2026-11-02.txt
2026-11-02,Yahoo! (UK news),uk.news.yahoo.com,PiplBot,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-uk-news-yahoo-com-2026-11-02.txt
2026-11-02,Yahoo! (UK news),uk.news.yahoo.com,scoop.it,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-uk-news-yahoo-com-2026-11-02.txt
2026-11-02,Yahoo! (UK news),uk.news.yahoo.com,Scrapy,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-uk-news-yahoo-com-2026-11-02.txt
2026-11-02,Yahoo! (UK news),uk.news.yahoo.com,Seekr,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-uk-news-yahoo-com-2026-11-02.txt
2026-11-02,Yahoo! (UK news),uk.news.yahoo.com,SentiBot,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-uk-news-yahoo-com-2026-11-02.txt
2026-11-02,Yahoo! (UK news),uk.news.yahoo.com,SeznamBot,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-uk-news-yahoo-com-2026-11-02.txt
2026-11-02,Yahoo! (UK news),uk.news.yahoo.com,TurnitinBot,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-uk-news-yahoo-com-2026-11-02.txt
2026-11-02,Yahoo! (UK news),uk.news.yahoo.com,YouBot,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-uk-news-yahoo-com-2026-11-02.txt
2026-11-02,Yahoo! (UK news),uk.news.yahoo.com,ZumBot,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-uk-news-yahoo-com-2026-11-02.txt
2026-11-02,The Sun,thesun.co.uk,Googlebot,no,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-thesun-co-uk-2026-11-02.txt
2026-11-02,The Sun,thesun.co.uk,Googlebot-Image,no,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-thesun-co-uk-2026-11-02.txt
2026-11-02,The Sun,thesun.co.uk,Googlebot-Video,no,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-thesun-co-uk-2026-11-02.txt
2026-11-02,The Sun,thesun.co.uk,Googlebot-News,no,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-thesun-co-uk-2026-11-02.txt
2026-11-02,The Sun,thesun.co.uk,Google-InspectionTool,no,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-thesun-co-uk-2026-11-02.txt
2026-11-02,The Sun,thesun.co.uk,GoogleOther,no,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-thesun-co-uk-2026-11-02.txt
2026-11-02,The Sun,thesun.co.uk,APIs-Google,no,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-thesun-co-uk-2026-11-02.txt
2026-11-02,The Sun,thesun.co.uk,Mediapartners-Google,no,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-thesun-co-uk-2026-11-02.txt
2026-11-02,The Sun,thesun.co.uk,DuckDuckBot,no,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-thesun-co-uk-2026-11-02.txt
2026-11-02,The Sun,thesun.co.uk,Slurp,no,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-thesun-co-uk-2026-11-02.txt
2026-11-02,The Sun,thesun.co.uk,Twitterbot,no,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-thesun-co-uk-2026-11-02.txt
2026-11-02,The Sun,thesun.co.uk,Bingbot,no,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-thesun-co-uk-2026-11-02.txt
2026-11-02,The Sun,thesun.co.uk,adidxbot,no,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-thesun-co-uk-2026-11-02.txt
2026-11-02,The Sun,thesun.co.uk,MicrosoftPreview,no,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-thesun-co-uk-2026-11-02.txt
2026-11-02,The Sun,thesun.co.uk,Applebot,no,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-thesun-co-uk-2026-11-02.txt
2026-11-02,The Sun,thesun.co.uk,AdsBot-Google,no,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-thesun-co-uk-2026-11-02.txt
2026-11-02,The Sun,thesun.co.uk,AmazonAdBot,no,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-thesun-co-uk-2026-11-02.txt
2026-11-02,The Sun,thesun.co.uk,facebookexternalhit,no,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-thesun-co-uk-2026-11-02.txt
2026-11-02,The Sun,thesun.co.uk,Yahoo Ad Monitoring,no,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-thesun-co-uk-2026-11-02.txt
2026-11-02,The Sun,thesun.co.uk,parse.ly scraper,no,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-thesun-co-uk-2026-11-02.txt
2026-11-02,The Sun,thesun.co.uk,Screaming Frog SEO Spider,no,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-thesun-co-uk-2026-11-02.txt
2026-11-02,The Sun,thesun.co.uk,SemrushBot,no,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-thesun-co-uk-2026-11-02.txt
2026-11-02,The Sun,thesun.co.uk,OnCrawl,no,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-thesun-co-uk-2026-11-02.txt
2026-11-02,The Sun,thesun.co.uk,Chrome-Lighthouse,no,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-thesun-co-uk-2026-11-02.txt
2026-11-02,The Sun,thesun.co.uk,Slackbot-LinkExpanding,no,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-thesun-co-uk-2026-11-02.txt
2026-11-02,The Sun,thesun.co.uk,Slack-ImgProxy,no,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-thesun-co-uk-2026-11-02.txt
2026-11-02,The Sun,thesun.co.uk,Slackbot,no,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-thesun-co-uk-2026-11-02.txt
2026-11-02,The Sun,thesun.co.uk,Snap URL Preview Service,no,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-thesun-co-uk-2026-11-02.txt
2026-11-02,The Sun,thesun.co.uk,proximic,no,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-thesun-co-uk-2026-11-02.txt
2026-11-02,The Sun,thesun.co.uk,GumGum-Bot,no,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-thesun-co-uk-2026-11-02.txt
2026-11-02,The Sun,thesun.co.uk,TTD-Content,no,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-thesun-co-uk-2026-11-02.txt
2026-11-02,The Sun,thesun.co.uk,Pubmatic,no,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-thesun-co-uk-2026-11-02.txt
2026-11-02,The Sun,thesun.co.uk,ias_crawler,no,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-thesun-co-uk-2026-11-02.txt
2026-11-02,The Sun,thesun.co.uk,outbrain,no,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-thesun-co-uk-2026-11-02.txt
2026-11-02,The Sun,thesun.co.uk,Leikibot,no,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-thesun-co-uk-2026-11-02.txt
2026-11-02,The Sun,thesun.co.uk,weborama-fetcher,no,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-thesun-co-uk-2026-11-02.txt
2026-11-02,The Sun,thesun.co.uk,AHC/2.1,no,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-thesun-co-uk-2026-11-02.txt
2026-11-02,The Sun,thesun.co.uk,SirdataBot,no,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-thesun-co-uk-2026-11-02.txt
2026-11-02,The Sun,thesun.co.uk,peer39_crawler/1.0,no,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-thesun-co-uk-2026-11-02.txt
2026-11-02,The Sun,thesun.co.uk,EchoboxBot/1.0,no,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-thesun-co-uk-2026-11-02.txt
2026-11-02,The Sun,thesun.co.uk,CriteoBot/0.1,no,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-thesun-co-uk-2026-11-02.txt
2026-11-02,The Sun,thesun.co.uk,AdDefend Site Context Crawler/1.0,no,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-thesun-co-uk-2026-11-02.txt
2026-11-02,The Sun,thesun.co.uk,SmartologyBot,no,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-thesun-co-uk-2026-11-02.txt
2026-11-02,The Sun,thesun.co.uk,Dianomi,no,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-thesun-co-uk-2026-11-02.txt
2026-11-02,The Sun,thesun.co.uk,snews,no,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-thesun-co-uk-2026-11-02.txt
2026-11-02,Daily Express,express.co.uk,grapeshot,no,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-express-co-uk-2026-11-02.txt
2026-11-02,Daily Express,express.co.uk,Googlebot-News,no,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-express-co-uk-2026-11-02.txt
2026-11-02,Daily Express,express.co.uk,ia_archiver,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-express-co-uk-2026-11-02.txt
2026-11-02,Daily Express,express.co.uk,Nutch,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-express-co-uk-2026-11-02.txt
2026-11-02,Daily Express,express.co.uk,ias_crawler,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-express-co-uk-2026-11-02.txt
2026-11-02,Daily Express,express.co.uk,ias_wombles,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-express-co-uk-2026-11-02.txt
2026-11-02,Daily Express,express.co.uk,ias-ie,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-express-co-uk-2026-11-02.txt
2026-11-02,Daily Express,express.co.uk,daumoa,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-express-co-uk-2026-11-02.txt
2026-11-02,Daily Express,express.co.uk,Sosospider,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-express-co-uk-2026-11-02.txt
2026-11-02,Daily Express,express.co.uk,rogerbot,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-express-co-uk-2026-11-02.txt
2026-11-02,Daily Express,express.co.uk,semetrical,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-express-co-uk-2026-11-02.txt
2026-11-02,Daily Express,express.co.uk,AhrefsBot,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-express-co-uk-2026-11-02.txt
2026-11-02,Daily Express,express.co.uk,Perplexity-ai,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-express-co-uk-2026-11-02.txt
2026-11-02,Daily Express,express.co.uk,YouBot,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-express-co-uk-2026-11-02.txt
2026-11-02,Daily Express,express.co.uk,Claude-Web,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-express-co-uk-2026-11-02.txt
2026-11-02,Daily Express,express.co.uk,AmazonAdBot,no,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-express-co-uk-2026-11-02.txt
2026-11-02,Daily Express,express.co.uk,ozone,no,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-express-co-uk-2026-11-02.txt
2026-11-02,The Telegraph,telegraph.co.uk,endeca,no,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-telegraph-co-uk-2026-11-02.txt
2026-11-02,The Telegraph,telegraph.co.uk,AI2Bot,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-telegraph-co-uk-2026-11-02.txt
2026-11-02,The Telegraph,telegraph.co.uk,Ai2Bot-Dolma,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-telegraph-co-uk-2026-11-02.txt
2026-11-02,The Telegraph,telegraph.co.uk,bedrockbot,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-telegraph-co-uk-2026-11-02.txt
2026-11-02,The Telegraph,telegraph.co.uk,ChatGLM-Spider,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-telegraph-co-uk-2026-11-02.txt
2026-11-02,The Telegraph,telegraph.co.uk,Claude-Web,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-telegraph-co-uk-2026-11-02.txt
2026-11-02,The Telegraph,telegraph.co.uk,cohere-ai,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-telegraph-co-uk-2026-11-02.txt
2026-11-02,The Telegraph,telegraph.co.uk,Cotoyogi,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-telegraph-co-uk-2026-11-02.txt
2026-11-02,The Telegraph,telegraph.co.uk,DeepSeekBot,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-telegraph-co-uk-2026-11-02.txt
2026-11-02,The Telegraph,telegraph.co.uk,Diffbot,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-telegraph-co-uk-2026-11-02.txt
2026-11-02,The Telegraph,telegraph.co.uk,FacebookBot,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-telegraph-co-uk-2026-11-02.txt
2026-11-02,The Telegraph,telegraph.co.uk,FriendlyCrawler,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-telegraph-co-uk-2026-11-02.txt
2026-11-02,The Telegraph,telegraph.co.uk,Google-CloudVertexBot,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-telegraph-co-uk-2026-11-02.txt
2026-11-02,The Telegraph,telegraph.co.uk,GoogleOther,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-telegraph-co-uk-2026-11-02.txt
2026-11-02,The Telegraph,telegraph.co.uk,GoogleOther-Image,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-telegraph-co-uk-2026-11-02.txt
2026-11-02,The Telegraph,telegraph.co.uk,GoogleOther-Video,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-telegraph-co-uk-2026-11-02.txt
2026-11-02,The Telegraph,telegraph.co.uk,Grok,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-telegraph-co-uk-2026-11-02.txt
2026-11-02,The Telegraph,telegraph.co.uk,iaskspider/2.0,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-telegraph-co-uk-2026-11-02.txt
2026-11-02,The Telegraph,telegraph.co.uk,ICC-Crawler,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-telegraph-co-uk-2026-11-02.txt
2026-11-02,The Telegraph,telegraph.co.uk,ImagesiftBot,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-telegraph-co-uk-2026-11-02.txt
2026-11-02,The Telegraph,telegraph.co.uk,img2dataset,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-telegraph-co-uk-2026-11-02.txt
2026-11-02,The Telegraph,telegraph.co.uk,ISSCyberRiskCrawler,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-telegraph-co-uk-2026-11-02.txt
2026-11-02,The Telegraph,telegraph.co.uk,Kangaroo Bot,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-telegraph-co-uk-2026-11-02.txt
2026-11-02,The Telegraph,telegraph.co.uk,KunatoCrawler,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-telegraph-co-uk-2026-11-02.txt
2026-11-02,The Telegraph,telegraph.co.uk,Meltwater,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-telegraph-co-uk-2026-11-02.txt
2026-11-02,The Telegraph,telegraph.co.uk,omgili,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-telegraph-co-uk-2026-11-02.txt
2026-11-02,The Telegraph,telegraph.co.uk,omgilibot,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-telegraph-co-uk-2026-11-02.txt
2026-11-02,The Telegraph,telegraph.co.uk,PanguBot,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-telegraph-co-uk-2026-11-02.txt
2026-11-02,The Telegraph,telegraph.co.uk,PetalBot,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-telegraph-co-uk-2026-11-02.txt
2026-11-02,The Telegraph,telegraph.co.uk,QualifiedBot,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-telegraph-co-uk-2026-11-02.txt
2026-11-02,The Telegraph,telegraph.co.uk,Scrapy,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-telegraph-co-uk-2026-11-02.txt
2026-11-02,The Telegraph,telegraph.co.uk,Seekr,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-telegraph-co-uk-2026-11-02.txt
2026-11-02,The Telegraph,telegraph.co.uk,Sidetrade indexer bot,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-telegraph-co-uk-2026-11-02.txt
2026-11-02,The Telegraph,telegraph.co.uk,TikTokSpider,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-telegraph-co-uk-2026-11-02.txt
2026-11-02,The Telegraph,telegraph.co.uk,Timpibot,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-telegraph-co-uk-2026-11-02.txt
2026-11-02,The Telegraph,telegraph.co.uk,VelenPublicWebCrawler,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-telegraph-co-uk-2026-11-02.txt
2026-11-02,The Telegraph,telegraph.co.uk,WARDBot,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-telegraph-co-uk-2026-11-02.txt
2026-11-02,The Telegraph,telegraph.co.uk,Webzio-Extended,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-telegraph-co-uk-2026-11-02.txt
2026-11-02,The Telegraph,telegraph.co.uk,WRTNBot,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-telegraph-co-uk-2026-11-02.txt
2026-11-02,The Telegraph,telegraph.co.uk,DuckAssistBot,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-telegraph-co-uk-2026-11-02.txt
2026-11-02,The Telegraph,telegraph.co.uk,Meta-ExternalFetcher,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-telegraph-co-uk-2026-11-02.txt
2026-11-02,The Telegraph,telegraph.co.uk,MistralAI-User,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-telegraph-co-uk-2026-11-02.txt
2026-11-02,The Telegraph,telegraph.co.uk,OAI-Operator,yes,https://jacobwhitmore.com/wp-content/uploads/2026/11/robots-telegraph-co-uk-2026-11-02.txt