Do AI companies block each other’s crawlers? 17 AI company sites checked

On 6 November 2026, 3 of the 17 AI company websites I could read blocked at least one of the 15 AI crawlers this site tracks. All three were chat apps: chatgpt.com, claude.ai and grok.com. None of the 11 corporate websites blocked any of them, including openai.com, anthropic.com and x.ai.

So the companies are open where they publish and closed where their product lives.

What I checked

The same 15 crawler tokens and the same definition of “blocked” as the UK news series: a crawler is blocked when the rules that apply to it contain Disallow: / (or /*) and no Allow: /. If the file names the crawler, its own group applies. If it doesn’t, the * group does. That follows RFC 9309, which says crawlers use the * group when no group matches them.

I also recorded a second column the news series doesn’t need: whether the file names the crawler at all. A file can block a crawler without naming it, by closing the whole site to everyone.

Sites: the corporate site and, where there is one, the chat product’s site for each company that runs one of the 15 crawlers, plus six other AI companies. 19 hosts. Google’s main domain, google.com, is left out because its robots.txt governs Google Search; gemini.google.com stands for Google. Two couldn’t be read (meta.ai and amazon.com: my browser extension isn’t permitted on them and Anthropic’s fetch tool was refused), so they are in the CSV as not observed and left out of every count below. That leaves 17 hosts and 255 cells.

What the files say

Product sites: 3 of 6 block something. Corporate sites: 0 of 11.

chatgpt.com closes the door on everyone, then lists rivals by name

chatgpt.com allows its homepage and a long list of public pages, then ends the * group with Disallow: /. Under this site’s definition that blocks every one of the 15 crawlers, OpenAI’s own GPTBot, OAI-SearchBot and ChatGPT-User included, because none of them is named and they fall back to *. They still get the homepage and the listed landing pages.

Above that, it names rivals one by one with a whole-site block: CCBot, Google-Extended, anthropic-ai, Claude-Web, Bytespider, PerplexityBot, and others outside the 15 (img2dataset, Omgili, FacebookBot, magpie-crawler).

Two details stand out.

The Anthropic tokens it names are anthropic-ai and Claude-Web. Anthropic’s crawler page lists ClaudeBot, Claude-User and Claude-SearchBot and doesn’t mention either of the older two. ClaudeBot is blocked on chatgpt.com anyway, by the closed * group, not by name. And in 2024 an Anthropic spokesperson told 404 Media that ClaudeBot had been set to respect rules written for the two old tokens, so the anthropic-ai line may still do what it was meant to. Under RFC 9309 alone it wouldn’t: a crawler matches its own token.

The Perplexity line reads User-agent: Perplexity‑User. The hyphen in it is not a hyphen. It is U+2011, a non-breaking hyphen, which looks the same on screen. I checked the character codes in the browser. A crawler matching its name as Perplexity-User (U+002D) won’t match that line. In this case it changes nothing: the closed * group still applies, and Perplexity says Perplexity-User “generally ignores robots.txt rules” because a user requested the fetch.

claude.ai names OpenAI and Google, and nobody else

claude.ai has one group that blocks four crawlers by name: GPTBot, OAI-SearchBot, ChatGPT-User and Google-Extended. Everything else falls to a * group that is open apart from /api/ and /lti/. PerplexityBot, CCBot, Bytespider, Meta’s and Amazon’s crawlers are not blocked.

One of the four may not listen. OpenAI says of ChatGPT-User: “Because these actions are initiated by a user, robots.txt rules may not apply.”

grok.com blocks six rivals; x.ai invites training

grok.com names six of the 15 in one group (GPTBot, ChatGPT-User, PerplexityBot, ClaudeBot, Google-Extended, Applebot-Extended), lets them see the homepage and a few product pages, and disallows the rest.

xAI’s corporate site, x.ai, blocks nothing and carries a Cloudflare Content Signals line: Content-Signal: ai-train=yes, search=yes, ai-input=yes. Of the 17 files, it is the only one with a content signal, and it says yes to all three uses.

Everyone else

gemini.google.com, perplexity.ai and copilot.microsoft.com close their app paths (/app/, /chat/, /chats/*, search URLs) to every crawler alike. None names an AI crawler. The 11 corporate sites name none of the 15 either. Apple’s file names only Chinese search crawlers; Meta’s names a handful of non-AI bots; Common Crawl’s blocks two Indeed bots.

What this does and doesn’t show

Fact: three chat-product sites block rival AI crawlers in robots.txt; no corporate site in the sample does.

Interpretation (mine): the product domains have little public content to lose. A logged-in chat app is mostly an app shell, so a block there costs the company almost nothing. The corporate sites hold the docs, research and announcements a company wants repeated in other people’s answers. Blocking a rival’s crawler there would mean being absent from the rival’s answers about yourself. That reading fits the pattern but I haven’t tested it. Another explanation fits too: the product blocks may be about shared chats and generated pages, which are user content the company may not want copied.

Hypothesis: if the first reading is right, corporate sites will stay open even as product sites tighten. A quarterly rerun will show it.

What it doesn’t show:

• robots.txt is a request. These files say what each company asks of crawlers, not what any server does or what any crawler obeys.
• Nothing about traffic, training or citations.
• 17 hosts chosen by me. The list is in the CSV and anyone can add to it.

Compared with earlier work

StackFox noted in February 2026 that “ChatGPT.com still leads with 10 AI bots blocked”, with openai.com, anthropic.com and google.com blocking none. Their count used their own list of 95+ tokens and no raw data is linked from the page. I count 12 tokens with a whole-site block on chatgpt.com today, on a different list. The two numbers aren’t a measured change, but they point the same way, and the openai.com, anthropic.com result matches.

Method notes

• Collected 6 November 2026, 15:12 to 15:25 UK.
• Read in Chrome (with a SHA-256 of the file computed in the page): chatgpt.com, openai.com, perplexity.ai, apple.com. Fetched raw with curl: claude.ai. Read through Anthropic’s web fetch tool: the other 12. The fetch tool returns text after processing, so those copies carry a lower confidence and are marked in the collection column.
• Classifier: the same parser as the news series, run on the saved copies. The CSV gives one row per host per crawler, with blocked, named_in_file, own_company_bot and the hash of the copy.
• Not observed: www.meta.ai, www.amazon.com.

Data: CSV, 285 rows · copies of all 17 files, with hashes · definition of blocked: method

Next check: 6 February 2027, same hosts, same 15 crawlers.