For years, organizations have leveraged AI/ML for specific tasks—from computer vision in video analytics to Google’s BERT powering advanced search and ML models driving predictive analytics. Industry leaders have long pioneered research in natural language processing (NLP) models for speech recognition, text summarization, and sentiment analysis, but the launch of ChatGPT marked the beginning of the AI renaissance and the generative AI era. Overnight, AI shifted from a futuristic concept to an essential daily collaborative tool. The rise of large language models (LLMs) and generative AI has further accelerated global innovation, disrupted established workflows, and forced every major industry to reimagine products and solutions.

Today, every organization—from small enterprises to the largest cloud providers—must decide how to integrate this technology and stay ahead amidst the rapidly evolving AI landscape.

The foundations of AI fabrics

The building blocks of a high-performance AI cluster environment include accelerators, storage servers, and the network fabrics that connect these servers (Figure 1).

The frontend fabric: This acts as the gateway to your GPU cluster, handling standard data center traffic, including north-south user access, API calls, logging, and data ingestion from storage or data lakes into the GPUs.

• Design: It carries both RDMA and non-RDMA traffic. Because traffic is often bursty, oversubscription is acceptable and commonly implemented in this fabric.
• Cisco Advantage: Silicon One–powered Cisco N9000 and 8000 series disaggregated solutions deliver the robust functionality required to support this fabric.

The backend fabric: Also known as the “scale-out” fabric, this provides a high-speed, low-latency interconnect for GPU-to-GPU collective communication (east-west traffic).

• Design: A lossless, non-blocking (1:1 subscription) fabric, critical for synchronized communication. Its size directly depends on the number of GPUs and associated NIC interface speed.
• Cisco Advantage: Silicon One–powered Cisco N9000 and 8000 series disaggregated solutions provide the advanced load balancing and congestion management needed to maintain this high throughput.

When AI ambition outgrows the data center

As AI infrastructure requirements grow, hyperscalers, neoclouds, sovereign clouds, and large enterprises must rethink how they build AI clusters. Faced with limited power and space at individual locations, operators must build distributed data centers to meet the surging training and inference demands of trillion-parameter LLMs.

Because a single cluster cannot handle workloads of this scale, a distributed ecosystem supported by a scale-across fabric is required.

Unifying the distributed ecosystem with scale-across fabric

As data centers supporting AI/ML applications become more distributed, the scale-across fabric serves as a vital interconnect. This architecture allows AI clusters to transcend single-site limits, spanning multiple facilities to overcome space and power constraints. By extending geographically, operators can optimize cost and energy while unlocking capacity pooling, extended resilience, and seamless, continuous growth. By interconnecting data centers spanning hundreds of kilometers, operators aren’t just adding links—they are forming a mega-scale AI infrastructure that spans multiple locations.

Whatever fabric architecture is in place, Cisco champions an Ethernet-based approach because it provides the scale, interoperability, and ecosystem maturity required for modern AI infrastructure (Figure 2). As interface speeds evolve to 1.6 Tbps and beyond, Ethernet remains the only viable path that offers the massive east-west bandwidth and performance needed to future-proof AI investment.

Designing a scale-across fabric for optimal performance

Scale-across is not a one-size-fits-all architectural solution. Every environment is unique, and design constraints from distance to traffic patterns can significantly impact AI cluster performance. To architect a scale-across architecture effectively, several critical questions must first be addressed, including:

• Is geographic distance a factor influencing frontend and backend fabrics?
• Can both frontend and backend fabrics be extended? If so, are there performance trade-offs?
• Is deep buffering required? At what distance does it become a necessity?
• How should RDMA NICs, switches, and the AI/ML stack be tuned to handle distance and oversubscription?
• How should training and inference designs evolve to ensure optimal performance?

The core purpose of a scale-across fabric is to connect multiple AI clusters, so they behave as a single logical entity (Figure 3). However, because the frontend and backend fabrics serve completely different purposes, extending them introduces distinct architectural challenges.

• Extending the backend: This acts as the high-speed engine of the cluster and is ultra-sensitive to packet loss and delay. As distance grows, bottlenecks emerge, stalling GPU synchronization requiring network fabric tuning and workload redesign to mitigate impacts.
• Extending the frontend: This serves as the gateway to the infrastructure. Although a certain level of packet loss and latency may be tolerable depending on the nature of the workload, proactive network fabric tuning is required to optimize performance.

Understanding the variables of distributed AI success

When network reach is extended beyond one data center, the underlying design principles change. Organizations are no longer dealing with one simple network topology. Four critical variables define the success of this architecture, including:

• Distance: While intra–data center distances are typically 50 to 100 meters, scale-across fabrics can span hundreds of kilometers. Distance is the single largest factor in RDMA performance degradation and determines exactly how far the network fabric can be pushed before encountering severe performance degradations.
• Oversubscription: Building a non-blocking fabric remains the gold standard, but it is often cost-prohibitive for long-haul bandwidth. Network architects must balance the oversubscription ratio, keeping it lower for the backend to ensure robust GPU-to-GPU communication, while allowing a higher ratio when extending the frontend fabric.
• Deep buffering: This variable depends entirely on environmental constraints, and there is no generic solution. Whether an architecture requires deep buffers or not depends completely on the specific distance profile and oversubscription ratios deployed.
• Long-range optics: Signal degradation over distances is an unavoidable physical constraint. Chromatic dispersion, fiber attenuation, and nonlinear effects degrade signal integrity. Choosing the right optics—from short-range (SR) to long-range extended coherent solutions (ZR+)—is vital, and rigorous testing remains the only way to confirm actual performance.

Evidence-based architecture: Insights from the Cisco Zeus AI Lab

Interconnecting distributed data center locations introduces profound architectural complexities (Figure 4). Enterprises need not face interconnect hurdles through trial and error.

Cisco’s dedicated AI Infrastructure Benchmarking Engineering Team conducts rigorous, empirical research to establish proven, validated reference designs. This systematic testing methodology encompasses critical networking features, including:

Performance evaluation tools and key metrics

• PerfTest measures raw RDMA performance, establishing the foundational benchmark for all network communications across both frontend and backend fabrics. The primary metric tracked is session throughput.
• NVIDIA Collective Communications Library (NCCL) benchmarking evaluates the efficiency of GPU-to-GPU collective communication patterns, which directly influence higher-level application performance.
• MLCommons MLPerf benchmarking quantifies end-to-end application metrics, specifically job completion time (JCT) and token throughput, serving as the ultimate indicators of operational agility and infrastructure ROI.

Test topology and architectural use cases

• Frontend extension validates a scale-across fabric architecture that interconnects the frontend networks of two distinct data center locations.
• Backend extension evaluates a scale-across fabric architecture that links the high-speed backend networks of two geographically separated data center sites.

Physical and operational environments

• Fiber infrastructure tests span a diverse range of physical distances, ranging from 100 meters to 100’s kilometers.
• Optical transceivers are deployed from a tiered selection of optical modules—including Far Reach (FR), Data center Reach (DR), Long Reach (LR), and coherent ZR/ZR+ technologies—aligned strictly with the specific distance ratings and attenuation profiles of the fiber paths.

Performance tuning and infrastructure optimization

• Congestion management establishes baseline performance metrics before implementing active tuning strategies to mitigate degradation. This process focuses on counteracting the dual impacts of physical distance and oversubscription ratios.
• Congestion avoidance mechanisms deploy flow-level load balancing (FLLB) within the frontend fabric to manage the mixed profile of RDMA and non-RDMA traffic, while utilizing highly efficient adaptive routing within the backend fabric.
• NIC and NCCL parameter optimization adjust critical variables at both the RDMA and NCCL layers. This optimization is vital as distances grow, ensuring the network pipeline remains fully saturated even as the round-trip time (RTT) increases.
• Model parallelism alignment analyzes how geographic distance dictates the structural design of AI workloads, guiding the optimal configuration of tensor, pipeline, and data parallelism schemes to achieve peak performance and high ROI.

The architectural blueprint for next-generation AI infrastructure

Although formal industry standards for distributed AI fabrics are not yet established, research defines three distinct tiers to guide AI infrastructure architecture (Figure 5):

• Campus scale-across that can span 100 meters to several kilometers
• Metro scale-across that can span several kilometers up to tens of kilometers
• Regional scale-across that can span hundreds of kilometers and beyond

The future of AI infrastructure is no longer defined solely by isolated computational capacity; it is defined by fabric connectivity that transcends physical geographical boundaries. As organizations refine data center strategies for the next wave of innovation, the defining architectural question remains: Is your scale-across strategy prepared for what comes next?

The evolution of mega-scale AI infrastructure introduces unprecedented complexities. In an upcoming technical blog series, we will deconstruct each of these scale-across architectural tiers. This series will deliver empirical insights and data-driven recommendations derived from the Cisco Zeus AI Lab, enabling organizations to deploy and scale distributed clusters with predictability and confidence.

Power your network for AI at scale with Cisco AI Networking

Additional resources:

• Unlock the power of scale-across with Cisco
• MLPerf benchmarking results