Written 12 October 2026.
An AI crawler visiting your page tells you one thing: the page was fetched. It doesn’t tell you the page was used, shown or credited.
It’s easy to see GPTBot or PerplexityBot in your server logs and read it as good news. It might be. But a log line is the first step of a long chain, and most pages drop out somewhere along it.
The chain
For a page to send you a visitor from an AI answer, all of these have to happen:
- Fetched. A crawler downloads the page. This is the only step your logs show.
- Kept. The page is stored, as training data or in a search index. Fetching doesn’t guarantee either.
- Retrieved. Someone asks a question and the engine picks your page as a possible source.
- Used. The answer actually draws on your page.
- Credited. The answer names you or links to you.
- Clicked. Someone follows the link.
Each step is a filter. A log full of crawler hits proves step 1 happened a lot. It says nothing about steps 2 to 6.
Different crawlers, different jobs
Not every AI crawler is there for the same reason. Some collect training data, some build search indexes and some fetch a page live because a user asked about it. I set out the difference in Training bots, search bots and fetchers.
That matters here. A training crawl feeds what a model learns, not the links it shows. The links in AI answers come from search indexes and live fetches. A live fetch is the closest to an answer, but it still isn’t proof the answer used you.
What to measure instead
If you want to know whether AI engines use your pages, look at the answers, not the logs:
- Write down the questions your customers would actually type.
- Run them on a stated date, on named engines, and record whether you’re named or linked.
- Repeat them on a schedule, with the same wording, so a change means something.
Logs are still useful. They show which crawlers come and how often. My crawler study measures something else again: what sites ask crawlers to do in robots.txt. Neither tells you whether anyone gets cited.
What I will not claim
- That crawler visits never lead to citations. They’re a necessary first step for some engines. They aren’t evidence of the later steps.
- How often each step happens. I haven’t measured it, and I haven’t seen anyone publish it with a method I could check.
Related: robots.txt is a request covers the other side: what a site asks crawlers to do isn’t the same as what happens.