Cloudflare’s network now sits in front of more than 20% of the web, and the company’s Data team is responsible for moving the resulting telemetry, logs, and analytics data from Cloudflare’s global edge to its central data centers. By May of this year, that pipeline was delivering roughly two billion events per second, a mix of customer traffic and internal telemetry.
The challenge, as Cloudflare describes it, is a familiar one for any large-scale operator: traffic grows exponentially while hardware budgets grow linearly. Over the past five years, Cloudflare says a series of pipeline changes have effectively deferred about two and a half years’ worth of hardware expansion, letting the same physical footprint absorb far more data.
Breaking up the monolith
In 2022, Cloudflare retired a single monolithic data schema that packed every product’s fields into one firehose stream, forcing internal teams to ingest everything and filter out what they didn’t need. Replacing it with product-specific schemas, even though some fields had to be duplicated across messages, cut backbone bandwidth usage by 35 to 40 percent, equivalent to about a year of compressed traffic growth.
Switching compression algorithms
By 2024, daily network congestion pushed the team to swap its compression algorithm from LZ4 to zstd. Earlier tests had shown zstd using more CPU, so Cloudflare rolled the change out cautiously, first to a handful of servers and then across the fleet, tracking CPU metrics closely. The payoff was a nearly 40 percent reduction in compressed traffic throughput with little added CPU cost, letting the same size pipe support more than 1.5 times the previous traffic volume, roughly equivalent to ten months of deferred growth.
Three more fixes in 2026
Cloudflare says it is now pursuing three additional strategies aimed at cutting uncompressed data volume. One involves rethinking how disaster-resilient data replication works. Historically, every non-regional analytics event was double-written in full to two separate centralized regions for resilience, which meant Logfwdr traffic doubled across the board as more analytics streams adopted this redundancy. Cloudflare had already solved a similar problem for its Logpush product by treating centralized data centers as a shared pool of receivers, sending each message to only one region and failing over automatically if a destination becomes unreachable. Applying an equivalent model to analytics is more complex, since ClickHouse clusters in each region still need a complete copy of non-regional data rather than a single delivered copy.
The broader pattern across all these changes, schema redesign, compression upgrades, and now replication and sampling rework, reflects an approach of continuously re-engineering the data path itself rather than simply adding more servers to keep pace with growth.