AI crawl-to-refer ratios: why a single published figure misleads

Cloudflare Radar measured Anthropic's crawler at 47,779 pages crawled per referral sent back in January 2026. In September it measured 549. Perplexity went the other way over the same months, from 112 to 2,373. Mistral started the year at 23 and has had no measurable referrals since August.

These figures get quoted as properties of a crawler. They are closer to properties of a sample: a source, a window and a traffic mix. If you are about to block an operator on the strength of one number, check the date on it first.

Why does the same crawler get different ratios?

Because the window changes the answer, even inside one dataset. Here is Anthropic on Cloudflare Radar alone, depending on how the window is cut:

January on its own and the first eleven weeks of the year differ by 2x. May and June, one month apart, differ by almost 4x. Anyone who quoted "Anthropic is about 24,000 to 1" in the spring was quoting a real number. By the autumn it was off by a factor of forty.

Most of that fall came from referrals growing from a very small base. By September, claude.ai sent 0.37% of all referral traffic on Cloudflare's network, close to ChatGPT's 0.41%. The crawling eased too: ClaudeBot's share of all crawler traffic dropped from 19.5% in July to 11.6% in September.

Which crawl-to-refer ratios are stable?

The ones backed by large referral volumes. Here are July and September side by side on Radar:

Google barely moves, because it sends enough traffic that the ratio settles wherever you measure it. Operators sending almost nothing have ratios that swing on small changes in either crawling or referrals, and in both directions at once: over the same two months, Anthropic improved threefold while Perplexity deteriorated eightfold.

Should you block AI crawlers based on a published ratio?

Not on one figure. Published recommendations that sort crawlers into block, evaluate and allow are built on one window of one dataset. Mistral was the most reciprocal operator in Radar's January data at 23:1. By July it was at 29,944:1, and in August and September Radar counted no referrals from it at all. Perplexity spent the year getting steadily more extractive while its reputation as the answer engine that links out stayed put.

Blocking behaviour lags further still. In Cloudflare's 5 October 2026 scan of top-domain robots.txt files, PerplexityBot was disallowed by 348 files and explicitly allowed by 344. MistralAI-User did not appear among the 40 user agents most often disallowed.

Blocking also has a cost that never appears in a referral report. An operator can shape how an assistant describes your brand without sending you a single click. We have no clean way to measure that, so be careful with blanket blocks.

How do you measure your own crawl-to-refer ratio?

From your own server logs and analytics, because network aggregates were never built to describe your site. On Cloudflare's network, Claude now sends almost as large a share of referrals as ChatGPT. On one SEO-tool site that published its September analytics, ChatGPT sent more than 120 times as many sessions as Claude. Both are accurate.

1. Pull 30 days of server logs segmented by user agent: GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, MistralAI-User, Meta-ExternalAgent.
2. Divide each operator's requests by referral sessions from its matching surface. If you already run Microsoft Clarity, its Scrape-to-Referral card, added in August 2026, does this calculation for your site.
3. Write the date range and the source next to every figure you produce.
4. Re-run it quarterly. Anything older than a quarter is a hypothesis.

Your measured ratio will read worse than reality, because referral attribution undercounts by design. Clicks from native apps arrive with no referrer and get filed as direct traffic. A model that names you without linking sends the visit through branded search days later, if the visit arrives at all.