The Perils of Equipment Testing, Revisited
Based on a deluge of questions in my inbox, I guess its time to re-visit the perilous topic of equipment testing again. As I recently noted, you can have rigorously designed tests executed in a conscientious manner that may still only provide an incomplete picture at best. And sometimes you get tests that reveal more about the tester than the actual product that was purportedly being tested.
In case you missed it, a competitor recently released a report they commissioned that shows that their blade system offered better throughput than the Cisco UCS because of what they proposed was an inherent design flaw—over-subscription of the blade uplinks from each server. Unfortunately for them, it was their test plan that was flawed, not our UCS architecture or blade server innovation.
The fundamental fault with the test was it was conducted with a mis-configured Cisco UCS. Essentially, the testing firm matched a normally running system from our competitor against a Cisco UCS in its failover state.
Let me explain: each Cisco UCS 8-blade chassis supports 2 FEX modules, each with 4 x 10GbE uplinks, resulting in 80Gb of total uplink bandwidth. Typical design best practices recommend you run these FEX in an active-active mode with half the blades using one FEX as their primary and the rest of the blades using the other FEX (see the figure above). Now, you can do the math on your fingers for this: 8 blades with 80Gb of bandwidth = 10Gb/blade. Now, in a failover scenario, all blades will fail over to the remaining FEX without loss of connectivity—which is how our system was tested. The report did include a number of more equitable testing scenarios that did not involve using the Cisco UCS in a failover mode—in those tests, the Cisco UCS consistently outperformed the competitor’s system. For example, with two pairs of blades exchanging 10Gb data streams, UCS aggregate throughput was 36.59 Gbps or 760Mbps faster than the other system. In fact, one of the things the testing firm inadvertently revealed is how good our QoS is—the crippled UCS shared the remaining bandwidth equitably per default policy, which is actually an important, real-world consideration—if you take the time to create policy to ensure certain traffic is prioritized—say storage traffic or key application traffic—you also want to have some degree of comfort that the policy will actually be followed.
From my perspective, the report illustrates, at best, the testing firm and sponsor’s lack of understanding of the Cisco UCS architecture and the underlying tenets of Unified Computing and instead, exposes a viewpoint shaped by a dated view of compute infrastructure and networking technology. The ultimate irony is that the testing that was intended to show Cisco UCS weakness with regards to I/O throughput, in fact, shows, in apples-to-apples comparisons, the incorrectly configured Cisco UCS still outperformed our competitor.
At the end of the day, superior I/O capabilities and other meaningful innovations such as the Cisco UCS Extended Memory Management and the stateless computing model enabled by UCS Manager clearly show that Cisco UCS was designed around features that matter to customers in real-world environments. For some other perspectives on the Cisco UCS, here are a few other folks offering their thoughts:
- Cisco Unified Computing System Is a Robust Platform for Physical, Virtual Data Center Ops (eWeek)
- Cisco UCS Wows (InfoWorld)
- How UCS Reinvents the Datacenter (InfoWorld)