The most expensive chips in modern data centers are spending more than half their time waiting. According to analysis of Model Floating Point Operations Per Second (FLOPs) Utilization, AI labs are achieving only 35 to 40 percent Model Flop Utilization (MFU) on Nvidia H100 GPUs during trillion-parameter training runs. The culprit is not the compute hardware itself but the network fabric connecting it.
Bandwidth Is Growing Faster Than Switching Can Handle
AI training workloads are already pushing from 400 Gb/s toward 800 Gb/s per link, with 1.6 Tb/s line rates expected to be necessary by 2027. Raw link speed, however, is only part of the challenge. As GPU clusters scale to thousands of nodes, the harder problem becomes how efficiently the switching fabric coordinates data movement across all of them simultaneously. That is a substantially more difficult engineering task than simply increasing per-link bandwidth.
The financial stakes are significant. Networking’s share of data center capital expenditure is rising from roughly 5 to 10 percent today toward an estimated 15 to 20 percent by 2030, shifting from an infrastructure overhead line item into a primary cost driver.
Faster Interconnects Alone Do Not Solve the Problem
The instinctive response to a bandwidth gap is faster transceivers and denser cables. That approach runs into a hard limit: as per-link bandwidth increases, the demands placed on every switching node in the network grow proportionally. A switch that was marginally adequate at 400 Gb/s becomes a hard ceiling at 800 Gb/s. Bypassing the switching layer by building clusters purely through point-to-point interconnects multiplies laser sources, drives power consumption up nonlinearly, and compounds complexity with every node added. The switch cannot be routed around.
A Disaggregated Industry Trying to Solve a System-Level Problem
The AI infrastructure stack developed as a collection of independently optimized components: accelerators, transceivers, interconnects, and switches each designed to their own performance envelope and then handed to architects expected to integrate them. The result is systematic overengineering. Network fabrics are specified for generic workloads that match no actual deployment, and designers must assume worst-case conditions at every interface.
Closing the gap requires a different design philosophy. Rather than assembling a fabric from the best available parts, network architecture must begin with the workload and reason backward. In practice, that means three things:
- Co-optimization across the stack: The interposer, interconnect, and switching layer are not independent variables. Gains in one layer are routinely offset by constraints in another.
- Architecture-specific design: AI training, inference, and HPC workloads carry fundamentally different traffic patterns, latency tolerances, and bandwidth profiles. Generic reference architectures serve none of them well.
- Photonic packet-level reconfigurable switching: Electronic packet switches face hard scaling limits in power dissipation and latency. Photonic switching offers a path beyond those limits, but circuit-switched optical architectures reconfigure too slowly for AI traffic, which shifts dynamically between training and inference. Packet-level, reconfigurable photonic architectures preserve low latency and energy efficiency while adapting to asymmetric, dynamic workloads.
What This Means for Infrastructure Buyers
For hosting operators and enterprise buyers planning GPU infrastructure, the practical takeaway is straightforward: switching architecture deserves the same scrutiny as compute selection. Nvidia has expanded its networking division substantially, recognizing that returns on compute investment depend directly on whether the network can deliver data without congestion or idle time. Organizations specifying AI clusters that treat the network as a commodity layer are likely underestimating both their costs and their performance gaps.
The switching layer must be treated as a first-class design input, not an afterthought. The clusters being designed today will carry these architectural decisions for years.