Cisco Blogs


Cisco Blog > Mobility

Responding to CRN’s 802.11n Testing

CRN recently published an article profiling 802.11n performance testing of the Aironet 1250 Series access point. The article titled”Where’s Cisco with Wireless N?” (http://www.crn.com/networking/206905063#community) profiles the results of their testing of Cisco’s 802.11n solution. The article was part of a larger story testing 802.11n products including Meru and Ruckus. Cisco declined to participate in the CRN product review due to basic differences in product testing methodology. Cisco asserts that the review of its 802.11n access point as presented by CRN misses some of the basic principles of product testing and as such is inconclusive on many levels. Firstly, CRN tested a product that was operating with an early version of software. As with any new hardware, users should seek the latest software to ensure optimal performance. The code for Cisco’s 802.11n access point has evolved quickly and the version used by CRN had already been through 4 subsequent releases. The most recent product code is always available for download from the Cisco website -a point that was overlooked by the CRN staff.Secondly, CRN tested the access point with several client devices that were using old device drivers and had not been Wi-Fi Certified for the 802.11n draft 2.0 standard. In fact, the Toshiba Portg M700-S7002 -the only device to have Wi-Fi 802.11n certification -was tested using drivers from early 2007, predating any 802.11n Wi-Fi certification. Wi-Fi certification is imperative when benchmarking performance results. The refusal to update device drivers clearly added to incompatibilities and limited performance results. Furthermore, CRN tested only the 2.4GHz radio with little detail given as to the use of 20-MHz or 40-MHz channels. While 802.11n operates effectively in the 2.4GHz frequency, optimal performance results are realized in the 5GHz frequency. The methodology behind the CRN performance tests is questionable. The Cisco access point was tested without a WLAN controller -differing from its intended configuration. No attention was paid to using the latest software drivers or product software. CRN provides no transparency to product sourcing and configuration methodology and refused to offer Cisco the opportunity to troubleshoot for fear of revealing the source of the product and its configurations.For an unbiased and transparent review of Cisco’s product performance, readers should shift their attention to the Network Computing performance review. The performance review is summarized in the article,”First Look: Cisco’s 1250 802.11n AP” and can be read here: http://www.networkcomputing.com/channels/wireless/showArticle.jhtml?articleID=206901320. This review shows the performance of the Aironet 1250 as reaching 154.9 Mbps. The 300 Mbps as referenced by CRN is the link speed. It is well understood that the TCP/UDP throughput reported in performance tests is approximately half that of the link speed (due to overhead associated with the 802.11 standard). Cisco has achieved phenomenal success with the Aironet 1250 with over 30,000 units shipped -- the fastest selling access point in Cisco history. Leading customers like Duke University have shown real-world performance results just shy of 130 Mbps. CRN fell short on delivering a complete and transparent test due to flawed testing methodology. Only by testing products with the latest drivers and software code can CRN expect to receive results commensurate with the true performance of the product.

In an effort to keep conversations fresh, Cisco Blogs closes comments after 60 days. Please visit the Cisco Blogs hub page for the latest content.

2 Comments.


  1. In the real world of course everyone always has the latest device drivers loaded on their PC so it would be totally unreasonable to consider testing in anything other than a sterile lab environment… because tha’ts the real word…isn’t it………..NO !!! Having a test done with old device drivers might actually be more representative of the real world than making sure everything is up to date. The reality is that in the real world many/most people do have older device drivers on their PCs – we are not all equipped with the latest Wi-Fi Alliance certified Apple or Dell 11n Notebooks Oh, and one other thing – is a review only ”unbiased and transparent” when it produces the results you want the world to see ?

       0 likes

  2. Chris Kozup

    Thanks Bob for your blog post.Agreed that it’s important to distinguish between the real world and lab environments when conducting product tests. However, when publications conduct product tests with the purpose of advising customers on performance numbers, they should follow proper testing methodology – a methodology that involves testing the products in the configuration in which they are intended to be used. Not only did CRN not use the latest device drivers, but the devices they used weren’t even Wi-Fi Certified for 802.11n draft 2.0. It is thus misleading to represent the data as 802.11n performance data. Cisco is pleased to work with different testing facilities provided they follow a clear testing methodology and clearly represent the purpose of the tests without obscuring important nuances in how the tests were conducted.

       0 likes