When Cloudflare announced its Content Independence Day initiative twelve months ago, the company pointed to a fundamental shift in how the internet’s economics worked. A year on, its follow-up data report shows that shift has accelerated faster than the company itself anticipated.
AI Crawler Traffic Has More Than Doubled in Share
According to Cloudflare’s June 2026 data, AI training crawlers now account for 52% of all crawler requests, up from 22% in Spring 2025. An additional 36% comes from what Cloudflare calls mixed-use crawlers, which blend search indexing, agent activity, and training functions. Traditional, pure search crawlers represent a small and shrinking fraction of total activity, even though they remain important for publisher visibility.
The blurring of crawler purposes creates a real problem for site owners. Mixed-use bots force a difficult choice: stay discoverable in an agentic world, or risk having your most valuable content consumed for AI training without compensation.
More Than Half of Internet Traffic Is Now Non-Human
Cloudflare says agent traffic crossed a significant threshold this year: over 50% of traffic on the internet is now non-human. That milestone reflects a broader behavioral shift. For every hour people spend online looking for information, only 15 minutes is now spent on the open web itself. AI-driven discovery is displacing the traditional search-and-click model at a rapid pace.
The company also notes that generative AI has been adopted by roughly 2.5 billion active users in approximately 3.5 years, a pace it describes as more than twice as fast as smartphone adoption.
Publishers Losing Traffic Across Every Sector
The original business model of the open web was simple: publishers provided content, search engines provided visibility, and referral traffic created economic value. That exchange is breaking down. AI systems increasingly answer questions, complete research tasks, and summarize product comparisons without sending users back to the original source.
Cloudflare reports that some heavily crawled content categories have seen human traffic decline by as much as 40% in under a year. The impact, initially concentrated in news and media, has spread to retail, software, IT, and finance. Some publishers are actively preparing for what they call a scenario where search referrals effectively disappear.
Cloudflare’s Response: Scarcity, Transparency, and a Content Marketplace
When the initiative launched, Cloudflare committed to three things: giving site owners transparency and control over how their content is accessed, creating tools that restore scarcity to content, and building a marketplace where publishers and AI companies can negotiate content licensing directly.
The company says all three pillars are now taking shape. New domains on Cloudflare have AI training crawlers blocked by default unless the owner opts in. Cloudflare describes the broader goal as ensuring the internet remains a sustainable and healthy ecosystem, not just for publishers, but for the global economy that depends on freely circulating, credible information.
What This Means for Site Owners
- Review your crawler permissions in Cloudflare’s dashboard. The defaults now block AI training bots, but mixed-use crawlers still require careful consideration.
- Monitor referral traffic trends. A sustained decline in search-driven visits may be an early indicator that AI systems are consuming your content without returning visitors.
- Watch the emerging content licensing market. Cloudflare and others are building infrastructure for publishers to negotiate direct compensation from AI companies.
The data makes one thing clear: the period when publishing to the open web automatically translated into audience and revenue is over. Site owners who treat crawler access as a negotiable asset rather than a default will be better positioned in the agentic era ahead.