Study plan, registered on 11 October 2026. This study has not been run yet. I’m publishing the method before any results exist so it can’t be adjusted to fit them. Results will be added to this page, with the date they were collected.
My studies of news sites can say what a robots.txt file asks for, but not whether crawlers obey it. That needs server logs, and the only server logs I have are my own. This study uses them.
It’s the one study on this site that doesn’t use public data. Only the rows for the 15 crawlers are published, with IP addresses shortened. Requests from people are never published.
What will be measured
Which of the 15 crawlers in the method note request pages from jacobwhitmore.com, and whether one of them stops after this site’s robots.txt disallows it.
- Phase A, 30 days: robots.txt stays as it is on 11 October 2026. It disallows /wp-admin/ for all crawlers and has no AI-specific rules.
- The change: robots.txt adds a group for CCBot with
Disallow: /. Nothing else changes. - Grace period, 48 hours: crawlers cache robots.txt, so requests in the first 48 hours after the change aren’t scored.
- Phase B, 30 days: every request that identifies as CCBot is logged.
CCBot is Common Crawl’s crawler. It collects an open dataset used to train models, and Common Crawl doesn’t run an answer engine. Blocking it doesn’t change whether answer engines can read this site.
Definitions
A request counts as a crawler’s when its user-agent contains that crawler’s token. Where the owner publishes its crawler IP addresses, each request is also checked against them and marked verified or unverified. A user-agent can be faked, so verified and unverified requests are reported separately.
When it starts
Phase A starts once I’ve confirmed that my host gives me the raw access logs for the full 30 days. The start date, the change date and the check date will be added to this page as a dated line before any log is read, and the call below goes on the predictions page the same day. Until then it has no ID.
Call, to be dated when phase A starts
- Claim: CCBot will make no requests to this site after the grace period in phase B.
- Hit if: zero requests identifying as CCBot after the grace period.
- Miss if: one or more.
- Void if: CCBot made no requests in phase A, so there’s nothing for the change to stop.
What I will not claim
- Anything about other sites. This is one small site and one crawler.
- Anything about traffic.
- That a crawler which never appears was blocked. My host may block some bots before they reach the logs.
Data
own-site-crawler-log-YYYY-MM-DD.csv, with the columns timestamp_utc, phase, user_agent, token_matched, ip_prefix, ip_verified, owner_range_source, path, status_code, robots_rule_applied and within_grace.