Plan published 19 November 2026, before any file was audited.

Question

In the robots.txt files this site already tracks, how many User-agent lines aimed at an AI company name a crawler that company does not list in its own documentation?

It matters because a site may believe it has opted out when the line it wrote matches nothing the company says it sends.

Sample

The saved robots.txt copies from the most recent reading of each tracked sector: the ten newsbrands in the monthly news series, the ten supermarkets and the ten retail sites. Thirty files. No site is added or removed.

The reference list

For each company behind the 15 crawlers in the method, I read that company’s own crawler documentation on the check date and save the list of user-agent tokens it gives, with the URL and the date. Only the company’s own page counts. Third-party lists don’t.

What gets recorded

One row per User-agent line that names one of those companies or its products, compared without regard to capital letters: the site, the sector, the token exactly as written, the company it points at, whether the token is on that company’s documented list, the documentation URL and the date.

A token that is not on the list is recorded as “undocumented”. The result is a count of undocumented lines out of all lines aimed at these companies, and a count of sites with at least one.

When

With the next news reading on 2 December 2026, using the documentation as it stands that day.

What this can’t show

  • Undocumented does not mean ignored. A company may still honour an old name it has stopped listing. This counts what is documented, not what crawlers do.
  • Documentation changes. The reference list is one day’s reading and is saved with its date.
  • Some saved copies are condensed records, with long wildcard groups summarised. User-agent lines are kept in full.
  • It covers the companies behind the 15 tracked crawlers only.

Change log

  • 19 November 2026: plan published.