Does Cloudflare's AI crawler block stop Googlebot? It depends on one setting

Only if Training is set to Block. Since 15 September 2026, Cloudflare sorts AI crawlers into Search, Training and Agent, and the Training control has two very different ways to say no. Disallow AI Training writes a no-training preference into robots.txt and keeps Googlebot, Applebot and Bingbot crawling for search. Block stops those same crawlers outright, search included, because Googlebot crawls for search and for Google's AI products under one user agent.

Existing training blocks, including the legacy "Block AI bots" toggle, were migrated to Disallow AI Training in the week after 15 September. If anyone on your team has since set Training to Block, Googlebot is out.

What changed on 15 September?

Mixed-use crawlers lost their exemption from Block. Before, a training block left Googlebot alone. Now the setting you pick decides:

New ad-supported domains start with Training on Disallow AI Training and Agent on Block on pages with ads. That differs from the July announcement, which would have blocked training on ad pages outright, and Cloudflare's developer documentation still carries the July wording.

Which crawler operators count as Accountable?

Apple, Google and Microsoft, plus Amazon, Anthropic, Meta and OpenAI. Disallow AI Training only keeps a mixed-use crawler in search if Cloudflare labels its operator Accountable. The label requires a training opt-out, an AI-summaries opt-out, URL-level visibility into training use and search metrics, and an assurance that opting out of training won't affect search. Apple, Google and Microsoft qualify partly on time-bound commitments. Amazon, Anthropic, Meta and OpenAI are listed because they already run separate search and training crawlers, so their training bots stay blocked while their search bots don't.

Bing is the gap. Microsoft doesn't yet read a no-training preference from robots.txt, and support is targeted for early 2027. Until then the opt-out for Bing is the NOARCHIVE meta tag, which also removes your links from Chat and Copilot answers.

One vendor decides who counts as accountable, and that vendor sits in front of more than 20% of the web, so its defaults set the opening terms of machine access for a large share of new sites. Reasonable enough as a practical matter, and worth watching.

Were publishers already blocking training and allowing search?

Yes. A robots.txt snapshot taken across Cloudflare's network on 31 August 2026 shows training crawlers blocked and answering crawlers allowed:

The split runs by purpose rather than by company. OpenAI's training bot is blocked at 2.33:1 while its search bot collects more allows than blocks. Crawling moved the same way: on Cloudflare Radar, search and answer retrieval grew from 2.6% of AI-bot requests in January 2026 to 10.6% in September, while training held at about 44%.

In Cloudflare's figures, mixed-use crawlers account for 36.6% of verified crawler traffic, the largest single category. Under 1% of sites block search crawlers, and 17% restrict AI training.

Are sites ready to serve AI agents?

Mostly not. Cloudflare's 5 October 2026 scan of 111,711 top domains found explicit AI-crawler rules in 80% of robots.txt files, markdown served to agents on 3.3%, an MCP server card on 0.5%, and Web Bot Auth on fewer than 0.1%. Two years of argument about who to let in, and very little work on what to hand them once they arrive.

There is a cheaper fix to try before blocking anything. A crawler that revalidates an unchanged page downloads it in full unless you send correct Last-Modified and ETag headers and answer with a real 304. Fixing that cuts bot bandwidth without costing you any citations.

What should you check in Cloudflare this week?

1. Open each zone's AI crawler settings and note what Training is set to. Disallow AI Training keeps Googlebot in search; Block takes it out.
2. Check whether the zone is new since 15 September and carries ads, because Agent is then blocked on ad pages by default.
3. Compare Googlebot request volume in your logs for the two weeks either side of 15 September.
4. Decide what you want from Bing separately, since its no-training opt-out still costs Copilot visibility.
5. Separate training crawlers from user-triggered ones in your logs before making another blocking decision.