Cloudflare has introduced a new site setting called Disallow AI Training that solves a long-standing dilemma for website owners: how to stay indexed in search results while refusing to let that same crawler train AI models on your content.
The problem stems from mixed-use crawlers, bots run by large organizations that serve both search indexing and AI training with a single crawler. Until now, site owners had no way to allow one function without allowing the other. Blocking the crawler entirely meant disappearing from search results too.
A robots.txt directive with teeth
The new setting publishes a specific Disallow directive in a site’s robots.txt file. Cloudflare says it goes further than a standard robots.txt entry by identifying who is actually crawling, classifying why, and blocking crawlers that ignore the stated preference. Cloudflare also reports on Radar what each crawler operator actually does with that preference.
According to Cloudflare’s own data, fewer than 1% of sites on its network choose to block search crawlers, while 17% enable some mechanism to block AI training. That gap illustrates why the company built granular controls rather than a single blanket “Block AI” switch.
The Accountable designation
Cloudflare created a designation called Accountable for bot operators that meet, or have committed to a timeline for meeting, four requirements: an opt-out mechanism for AI training, an opt-out mechanism for AI summaries, URL-level visibility into which pages were used for training versus search, and assurance that opting out of training will not hurt search rankings. Apple, Google, and Microsoft currently qualify.
What changes for site owners
With Disallow AI Training in place, Cloudflare’s existing Block and “Block on pages with ads” settings now apply uniformly to all training crawlers, including mixed-use ones. Training-only crawlers run by Amazon, Anthropic, Meta, and OpenAI can be blocked outright without any search impact, since they don’t perform indexing.
Disallow AI Training is only offered as a Training control, not for Search or Agent behaviors, and there is no ads-only version of it because Cloudflare says the list of ad-serving pages changes too fast to enumerate reliably in a static robots.txt file.
What’s coming next
Cloudflare says AI summary controls are next on its roadmap. Rather than an all-or-nothing toggle, the company plans to let site owners set how much of their content can appear in AI-generated summaries, configured once on Cloudflare instead of separately with each AI operator. That capability is targeted for early next year.