Some links on this page are affiliate links. If you buy through one, I may get a commission at no extra cost to you. Each one is labelled. I only list things I actually use.
Running this site
Day-to-day SEO
- Crawling: Screaming Frog SEO Spider
- Search data: Google Search Console
- Analytics: Google Analytics 4
- Demand and trends: Google Trends
- Links and keywords: Ahrefs
- Page speed: PageSpeed Insights
- Debugging: Chrome DevTools
- AI: Claude and ChatGPT
Research sources
Every number in my research comes from one of these. If a source isn’t listed, I didn’t use it.
- The sample: Press Gazette’s monthly ranking of the top 50 UK news websites, which uses Ipsos iris audience data. The October 2026 study used the August 2026 top ten.
- The data: each site’s own robots.txt file, read directly from the site on the snapshot date.
- The crawler names: the user agent tokens the AI companies publish for their own crawlers.
Research tools
- A web browser, to read each robots.txt file as the site serves it.
- A script that applies the definition of blocked to each file, the same way every time.
- A CSV for each study, published alongside it.
The research uses no paid SEO tools, no third-party crawler databases and no estimated data. If that changes for a study, the study will say so. Affiliate relationships never decide what gets measured or how.