This site publishes research about AI crawlers and search. There are some claims I won’t make, because my data can’t support them. I’m listing them here so readers can hold me to it.
“The robots.txt block cost them traffic”
A robots.txt rule is a request about crawling. Traffic depends on plenty of other things, including the news cycle, redesigns, algorithm updates and paywalls. My studies look at the request. Saying it caused a change in traffic would need evidence that robots.txt files can’t provide.
“This brand ranks in ChatGPT”
AI answers change with the date, the wording of the question, the location and the account asking. There’s no fixed ranking to check. If I ever study how often brands appear in AI answers, it will use a fixed list of queries on a stated date, and the findings will only cover that date.
“Blocking a crawler keeps it out”
On 2 October 2026 the Daily Mail’s robots.txt blocked all 15 crawlers I checked. That was the strictest file in my sample. But robots.txt is a request, and I didn’t test what the Mail’s server does when a crawler ignores it. So I can say the Mail asks every crawler I checked to stay out. I can’t say it keeps them out.
“One day’s file is the publisher’s policy”
robots.txt files change, often without any announcement. Everything I’ve published from this snapshot is dated 2 October 2026. If a later snapshot shows something different, that’s new information, and catching changes like that is why I re-run the check every month.
The approach is the same in each case. I report what the file shows and the date I read it. Bigger claims might get more attention, but I couldn’t show they were true. The method note sets out the same limits.
The data behind this post is in the study: Which AI crawlers do the UK’s biggest news sites block?