Cloudflare has announced a new analytics dashboard called Attribution Business Insights, aimed at helping website owners understand the commercial impact of AI crawler traffic on their properties. The tool is now available to all Cloudflare Bot Management customers.
The Broken Crawl-to-Referral Bargain
For decades, the web operated on an informal agreement: search engines crawled content and, in return, sent readers back to the source. Cloudflare describes this as the SEO era, characterized by a relatively balanced crawl-to-referral ratio. That balance has shifted sharply with the rise of AI crawlers.
According to Cloudflare’s data, major AI crawlers now operate at dramatically skewed ratios. The company observed crawl-to-referral ratios ranging from 118:1 to nearly 50,000:1. In practical terms, an AI crawler may consume a site’s content tens of thousands of times for every single visitor it actually sends back. Publishers absorb the infrastructure costs of serving that traffic while receiving little or no referral benefit in return.
What the Dashboard Shows
The Attribution Business Insights dashboard is built around the data points publishers have said matter most. Key features include:
- Bot vs. human traffic breakdown: An overview of overall bot traffic volume alongside organic human visits to content pages.
- Crawl-to-referral ratios: Site-wide ratios viewable over 24-hour, seven-day, and 30-day windows, with per-bot-operator breakdowns available.
- Top bots by volume: A ranked list showing each bot’s country of origin, bandwidth consumption, and current allow or block status.
- Updated crawler classification: Bots are categorized beyond a generic “AI Crawler” label into three types: Training (feeding LLM development), Search (refreshing Retrieval-Augmented Generation databases), and Agent (serving agentic, real-time query responses).
Why the Classification Matters
The distinction between crawler types has real strategic value. A bot classified as Training is consuming content to build a competing model’s knowledge base, with no referral benefit whatsoever. A Search-type crawler may at least contribute to surfaces where the original source could be cited. An Agent-type crawler is fulfilling live queries on behalf of end users, a scenario that may or may not result in traffic depending on how the AI product presents its answers.
Cloudflare frames this as a shift from the SEO era into what it calls the AEO (Answer Engine Optimization) and GEO (Generative Engine Optimization) era, where the old assumptions about discoverability no longer hold.
Availability
Attribution Business Insights is available today to Cloudflare Bot Management customers. The company positions it as a decision-making tool for publishers and business owners, not just security teams, providing the data needed to make informed choices about which crawlers to allow, restrict, or block outright.