Written 14 October 2026.
Yes. A crawler follows the group that names it and ignores the User-agent: * group. So a site-wide Disallow: / stops applying to any crawler you give a group of its own.
That’s the robots.txt standard, RFC 9309: a crawler uses the group that matches its token, and falls back to * only when nothing matches.
What I measured
On 2 October 2026 the Sun’s robots.txt disallowed the whole site in its * group. It then gave OpenAI’s three crawlers groups of their own, with path-level rules only. Result: GPTBot, OAI-SearchBot and ChatGPT-User were not blocked. The other 12 crawlers I checked were blocked by the catch-all.
More on that file: Notes on one file: the Sun. The other nine files are in the crawler study.
The trap
This file does not block GPTBot:
User-agent: *
Disallow: /
User-agent: GPTBot
Disallow: /private/It keeps GPTBot out of /private/ and nothing else. The Disallow: / in the * group never reaches it.
The Sun does this on purpose. If you didn’t mean to, repeat the full-site rule inside the named group:
User-agent: *
Disallow: /
User-agent: GPTBot
Disallow: /Check yours
For every crawler your file names, read only its own group. If Disallow: / isn’t in that group, the crawler isn’t blocked, whatever the * group says. That’s also how my definition of blocked works.
What I will not claim
- Why the Sun’s file is built this way. The file shows the rule, not the reason.
- That every crawler follows the standard. robots.txt is a request.
- Anything beyond one file read on 2 October 2026.
This is hack 5 on the search hacks page.