Technical SEO
The crawl, render and markup details that decide whether search engines and AI crawlers can read your pages at all.
Divide Search Console's generative AI impressions by total Web impressions and you get the share of your Google visibility happening inside AI Overviews and AI Mode. Here's how to calculate it, segment it and keep it honest.
The free tools that answer "is my site blocking AI crawlers" read robots.txt. A block can also sit at the edge, in an X-Robots-Tag header, or in a meta tag you never wrote. We failed all four on our own site.
Cloudflare Radar put Anthropic at 47,779 pages crawled per referral in January 2026 and 549 in September. Perplexity moved the other way, from 112 to 2,373. A single ratio is a dated sample, not a property of a crawler.
Since 15 September 2026, Cloudflare sorts crawlers into Search, Training and Agent. Disallow AI Training keeps Googlebot in search; the hard Block setting now stops it, search included. Here is how to tell which one your zone uses.
GPTBot, ClaudeBot and PerplexityBot read the raw HTML response and don't execute JavaScript. If your content only appears after rendering, answer engines can't retrieve it or cite it.
What does AI say about you?
We ask ChatGPT, Gemini, Google AI Mode and AI Overviews the questions your customers ask, and write down every answer.