Reproduction scripts for jacobwhitmore.com research. Written 10 November 2026.
Python 3.9+, standard library only.

1. Download the CSV and robots-*.txt files linked from https://jacobwhitmore.com/data/ and the study pages into one folder, e.g. ./data
2. python3 verify_uk_news.py ./data     -> re-classifies all 150 cells of the 2 Nov 2026 UK news snapshot from the raw robots.txt files and compares with the published CSV and the October file
3. python3 verify_all.py ./data         -> recomputes the headline figures of every other study from its CSV
4. python3 build_supermarkets.py ./supermarkets ./data/uk-news-ai-crawlers-2026-11-02.csv  -> rebuilds the supermarket dataset from the ten records

robots_engine.py is the classifier: RFC 9309 group selection (named group, else *), longest match on the path "/", Allow wins ties.
Licence: CC BY 4.0. Corrections: contact@jacobwhitmore.com
