MLPerf™ Inference v6.1 results from a distributed inference service spanning NVIDIA H200 and AMD Instinct™ MI350X systems over a Cisco Silicon One™-based Ethernet fabric.
The AI data center market is entering a new phase. As AI expands into agentic and automated workflows, it consumes far more tokens than traditional LLM applications, driving a larger, more continuous build-out of accelerator capacity. Fleets also refresh incrementally, so new systems are arriving alongside installed hardware that is still productive. To scale economically, organizations need the flexibility to combine accelerators from multiple technology providers and hardware generations rather than commit to a single ecosystem, shifting capacity across the fleet instead of stranding it.
This diversity can also improve supply-chain resilience by reducing exposure to component shortages, rising costs, and ecosystem lock-in. This also accelerates a shift towards Ethernet as the common fabric across accelerator choices. Hyperscalers, neoclouds, and enterprises all need peak performance while preserving this architectural flexibility.
At Cisco, we built a distributed inference service for our MLPerf™ Inference submission to demonstrate how the network can connect different types of accelerator systems. We are proud that it is the industry’s first multi-vendor accelerator submission. It shows that organizations can orchestrate workloads across mixed-accelerator pools with minimal cross-architecture overhead, improving unit economics. Notably, in our MLPerf™ runs, the combined service reached up to 104.7% of what the two systems delivered on their own, with no net penalty for crossing architectures.
Scaling the inference service
LLM inference looks like one operation to the user, but the accelerator handles two distinct phases. Prefill processes the prompt and creates the key-value (KV) cache, shaping time to first token (TTFT). Decode generates output one token at a time, repeatedly reading model weights and the growing cache, shaping time per output token (TPOT).
As models and request volumes grow, distributed serving offers several ways to scale. Model parallelism splits execution across systems when a model cannot fit or run efficiently on one. Data-parallel serving adds replicas to handle more requests. Neither approach alone scales prefill and decode independently.
These phases can be served in two ways. One keeps them together: with whole-request in-flight batching, a router sends each request to a single model-serving replica that runs both prefill and decode and keeps the request’s KV state local, so nothing crosses the backend network.
The other separates them. Prefill-decode (PD) disaggregation splits the phases into different serving groups so each can scale on its own, which matters when long prompts raise prefill demand while long responses extend decode work. The trade-off is that the request-specific KV state must move across the backend network between the two phases.
Heterogeneous accelerators expand the options
Heterogeneity can span accelerators from multiple vendors, architectures, and generations, with each platform taking on different roles as workload needs change. Once the model and serving stack run correctly on each system, a request can stay on one replica or cross systems at the prefill/decode boundary. The best placement shifts with the model, request mix, latency target, and available capacity, letting heterogeneous systems join as one distributed inference service rather than operating as isolated pools.
PD-disaggregation stresses the accelerators in each phase differently, so teams can assign accelerators to the phases where their strengths fit best. Accelerators that excel at compute-heavy prefill can focus on prompt processing, while one strong at data movement handles decode. This can improve performance and better optimize available resources.
Measuring the complete system
To see whether the design held up in practice, Cisco connected eight NVIDIA H200 and eight AMD Instinct™ MI350X GPUs through a Cisco Silicon One™ G200-based Ethernet fabric and ran a vLLM-based distributed inference service across the two systems. The MLPerf™ Inference v6.1 submission evaluated multiple models and serving configurations, including PD disaggregation for the mixture-of-experts GPT-OSS 120B model.
A fast network transfer alone would not prove that the design worked. The complete service also processed the prescribed inputs, generated outputs that met the accuracy target, and stayed within the required latency limits.
MLPerf™ Inference provides those controls. It fixes the model, dataset, accuracy target, and load-generation rules, then evaluates the system through scenarios that represent different operating conditions:
- Offline: All samples are issued in one query to measure bulk throughput, without online p99 TTFT or TPOT limits.
- Server: Queries follow a Poisson arrival pattern and must meet benchmark-specific p99 TTFT and TPOT limits.
- Interactive: The arrival model is similar to Server, with tighter p99 TTFT and TPOT limits.
In the GPT-OSS 120B Server run, the NVIDIA H200 group handled prefill while the AMD Instinct™ MI350X group handled decode. This matched the compute demands of prefill with the H200 system and the data-movement demands of decode with the AMD Instinct™ MI350X system. Both platforms used a compatible KV-cache format, allowing the decoder to read the prefiller’s FP8 KV cache directly without conversion between architectures. NIXL, with UCCL as its cross-platform RDMA backend, moved the request-specific state across the fabric.
Using the same vLLM-based PD-disaggregation stack, we measured each system separately and added their throughput rates to establish a 70,085.48-token-per-second baseline. Running the systems together produced 73,357.62 tokens per second. That was 104.7% of the baseline, an increase of 3,272.14 tokens per second.
Relative to the arithmetic sum of the standalone measurements, this was a superlinear end-to-end result. Moving KV-cache state across the Ethernet fabric still consumed network resources but assigning prefill and decode to separate systems more than offset that cost. The connected configuration introduced no net throughput penalty and delivered higher throughput than the standalone results would suggest, while remaining within the applicable p99 TTFT and TPOT limits.
The network carries the critical KV path
In a PD-disaggregated service, the network is part of the response path rather than just the connection between servers. Decode cannot begin until the KV state created during prefill arrives, so network latency, bandwidth, and reliability directly affect optimal performance.
The Cisco Silicon One™ G200-based fabric provides the high-bandwidth, low-latency Ethernet path required to move that request-specific state between serving groups without turning the handoff into a bottleneck. In the submitted configuration, the Cisco Silicon One™ G200-based fabric carried KV-cache transfers over RDMA on a leaf-local path, helping the combined service exceed the arithmetic sum of its standalone throughput measurements while remaining within the required p99 latency limits.
“AI infrastructure is no longer a single architecture or a single generation. Operators are adding new accelerators while installed systems still have useful life, and they cannot afford to strand either one. The first multi-vendor MLPerf Inference result matters because it tests that operating model. Prefill and decode stress different parts of the system. When those phases can run on different accelerators, and the fabric can move request-specific KV state without a throughput penalty, the network stops being plumbing. It becomes part of the inference solution itself. Ethernet is the common layer that makes that mix possible. Organizations that invest in the fabric can keep productive hardware in the pool, add capacity in stages, and improve unit economics without locking the inference service to one interconnect or one supplier.” Alan Weckel, Founder and Technology Analyst, 650 Group
Conclusion
The future of AI infrastructure will be defined less by any single accelerator architecture than by how efficiently organizations combine the infrastructure they already own with the infrastructure they add next.
This is the practical value of an open, high-performance Ethernet network: more usable capacity, greater freedom of choice, and a smoother path to AI scale. On a common fabric, organizations can keep productive hardware in service, adopt new accelerators on their own terms, and improve unit economics without proprietary lock-in.
Heterogeneous AI infrastructure is the foundation for a more flexible, resilient, and efficient data center.
Additional resources:
Cisco is breaking new ground with industry-first multi-vendor accelerator MLPerf™ inference benchmarking
Discover how
