Study plan, registered on 2 October 2026. This study has not been run yet. I’m publishing the method before any results exist so it can’t be adjusted to fit them. Results will be added to this page, with the date they were collected.
On 2 October 2026 the Daily Mail’s robots.txt blocked all 15 AI crawlers I checked. The Telegraph blocked 14 and explicitly allowed Google-Extended. The two files are almost the same apart from that one rule, which makes these two papers a useful pair to compare.
What will be measured
For each query and engine, I’ll record whether each paper is linked, named without a link, or absent. The results will be counted by engine. It uses the same 30 queries, engines and date as the AI answers study plan, where the full query list is published.
The conclusion will only cover what the table shows. If the two papers appear at similar rates, that will be the finding.
Limits
Google-Extended controls whether Google can use a site for training its AI models. It isn’t a switch for search, so this study looks at how often each paper appears, not at why. The two papers also differ in other ways, such as their paywalls and the topics they cover, and this study doesn’t control for those.
What would show the result was wrong
Running the same queries on the same day and getting clearly different counts for either paper.
Data
Results will be published here as a CSV when the study runs. The robots.txt data for both papers is in the October crawler study.