Cisco Blogs
Share

Usability Testing with Customers Everywhere


January 24, 2009 - 5 Comments

One of the things we do relentlessly at Cisco is to test how easy it is for you, our customers, to use our web sites and products. It’s called “usability testing” and in the old days it would all be done in person, which a research expert and customer using a web page or software interface in one room, and observers (from the web team or product team) in an adjacent room behind a two-way mirror. For instance, we might ask you to pretend you were interested in replacing your phone and communication system in your company and then have you go to the web to observe how you’d go about researching that topic. We would be watching attentively in the back room to learn about what works and what doesn’t, and how to make your task easier. In the past, I have even on occasion traveled to other cities to do tests with local folks outside of the Bay Area. It was a good way to meet a diverse range of customers, but expensive! In the last two years, we’ve done a record-breaking number of usability tests. But increasingly, we are saving time and money and extending our reach by doing the tests using online collaboration tools: Thanks to tools like Cisco WebEx Meeting Center, we can now test our web sites with customers from across the US and indeed around the globe. The customers, and many of the observers are all remote from the scene. And we can test complicated scenarios we never would have dreamed of attempting in a single lab setting previous. For instance, below is a picture of a test we ran recently where we were testing interaction between (remote) customers, (remote) call center reps, and (remote) Cisco partners who were working with the customers. Pretty much the only “local” San Jose, California people involved were the ones you see in the picture, who were all observers of the test. image WebEx is a favorite of usability experts, because it lets them reach out to customers across the globe and also allows the user to view and control web site protoypes or software applications that are in your test enviornment (that is, you don’t have to push your test site live or make the user install your app). Some companies I know are now doing 95% of their usability testing via remote tools. In preparation for a talk I am giving next week on this subject, I asked some trusted colleges for pluses, minuses, and tips for remote usability testing. Here’s some of their wisdom.

Pluses:

  • Better for customers: Local customers don’t have to drive to your site or a third-party lab to do a test; they just click a mouse and pick up their phone
  • Much cheaper on your travel budget: Rather than flying experts to cities to run tests with customers, you connect effortlessly online.
  • Easier to recruit customers to the test: Because customers don’t have to travel, they’re more willing to participate.
  • More realistic: Test happen in the customers’ own environment (with all the typical distractions) so they’re more realistic than a sterile lab. And, sometimes customers will be willing to share non-confidential aspects of their desktop tools and environment, so you can understand how your application and web site works in their environment.
  • More reliable: A hobgoblin of traditional usability tests is “no show” users who got lost or got stuck in traffic; remote tests using online collaboration have much better attendance records because there is less to go wrong.

Minuses:

  • Of course, nothing is quite the same as meeting people in person (except Cisco Telepresence, perhaps)

Tips:

  • Be very explicit about timezone. Also send an invite via Outlook so customers have it. Send them reminders. Get cell phone and email address so you can contact them if they aren’t at their desks at the appointed time.
  • Have a task list online if users are remote. For instance, create a google doc and open in it a second tab of their browser.
  • Use electronic electronic gift certificates (e.g. for Amazon) as a thank you gift. (But sometimes they get caught in the customer’s spam filter…. tell people to be sure to check in a followup email. .)
  • Record all of the sessions. For instance, WebEx Meeting has a way to record the session. Then, rather than try to edit highlights, point to a “most typical” session and recommend that people watch that one.
  • Think about the technical profile you are recruiting for the test. My friend Meghan Ede just finished doing an extensive test of consumer users of retail sites, and the whole thing was done remotely. She used WebEx Meeting Center and reports that the test went flawlessly even with an audience of non-techies. Though, Scott Kincaid of Usability Sciences (who does a lot of international testing) says: “The rule of thumb I use to my clients, ‘if you are trying to test my mom, who is not very technical, don’t do the test remote. If you are doing a test for electrical engineers, yes, do the test remotely.’ As with many things, your mileage may vary.

Thanks to Meghan Ede, Catherine Courage, Kit Lofgren, Scott Kincaid, and Goli Jendreski for the above wisdom. By the way, on January 27 at 6:45-9 PM, Cisco is hosting a session of BayCHI (Bay Area Chapter of ACM SIGCHI Computer-Human Interaction group) on this very topic. We will discuss:

  • How remote collaboration will change the way we work, live, play, and learn (Lin Brown)
  • User Experience design approaches for remote collaboration (Cordell Ratzlaff)
  • How to leverage collaboration technologies for remote usability testing (Goli Jendreski)
  • A live user test conducted with remote users in another country. (Martin Hardee)
  • Telepresence in usability testing (Alison Ruge)

Anyone with an interest in usability is welcome! Here’s the BayCHI website for details. http://www.baychi.org/calendar/20090127/

In an effort to keep conversations fresh, Cisco Blogs closes comments after 60 days. Please visit the Cisco Blogs hub page for the latest content.

5 Comments

  1. I am new to the usability test option but it seems like a great innovative way to test products and let the customer survey it. Thanks for sharing this valuable information with us.

  2. I will recommend not to wait until you get enough money to order all you need! You can just take the loans or just term loan and feel yourself fine"

  3. Great article, Martin. A number of colleagues have recommended WebEx, and I can see how it would be easy to implement and run these sessions. This will be a great complement to our existing in-person learning labs.

  4. I am new to the usability test option but it seems like a great innovative way to test products and let the customer survey it. It is nothing like interacting with the customer in person but that is becoming obsolete I think.

  5. Hi Martin,Not sure if you went to the IA Summit this year, but I'm sad we didn't have a chance to meet up. I've had the chance to use your design comics for a few internal pitches of including stories"" in the design process.Also, I'm happy to report that I use WebEx extensively for remote tests. Do you happen to have any artifacts from the BayCHI meeting that you could share? Something on Slideshare.net, for example?"