Toolbar: Cautionary Tale for Fellow Webmasters
Usually web teams congratulate themselves for adding things to web sites. But this past weekend, we were happy to remove something instead: The toolbar on the bottom of Cisco.com pages.
The idea of the toolbar was to provide kind of a “swiss army knife” function that could contain features that would work across the site. We wanted to start small with the features, but expand over time. The toolbar was my idea, and one we modeled after similar functions on other sites across the web. Here it is, in all it’s glory:
This past fall, we rolled out a new design across the two million pages of Cisco.com. It included a number of navigational and other improvements (some of them pretty subtle but useful) and it included this toolbar (which we thought was subtle, but was not, as it turns out). This was an epic redesign (in the traditional sense of that word), and the feedback, reviews, and site metrics bear out that it is working well.
But, it turns out there was a huge problem with that toolbar: It really got in the way of people’s work, especially in one important way: In the Firefox web browser, if you used the Control-F function to find text on a long page, the toolbar would often obscure the results! So, this new ‘feature” actually ended up unexpectedly breaking the standard behavior of a key browser. Yikes.
How could we let something like that happen, you ask? Well, that’s a story.
I’ve worked on web sites for a very long time (basically since the web started), and thought I knew every twist and turn that could go right or wrong. When we got the idea for the toolbar – which was eventually supposed to have a lot of useful features – we did all of the classically right stuff to vet the idea. But we didn’t vet hard enough, and we fell victim to some early schedule dependencies that forced us to decide about implementing this “feature” before we should have. First, here’s the review and vetting we did, just to give you an idea of our thoroughness:
Concept reviews: The thing we liked about the toolbar idea was that it gave us a place to put new features that could work universally across the web site, which would make them guaranteed to be available everywhere and work in a consistent way. And, it seemed to place things “out of the way” so they wouldn’t clutter our pages. We walked through the concept and even included it in all of our online prototypes. It seemed to solve some problems.
Best practices check: But, we wondered… do these tool bars work well in real life? So, I personally talked to the web folks at a number of other companies and web sites that had implemented similar bars. The advice we got boiled down to: “They do get used, but make sure it’s collapsible for people using notebooks and other short screens.” Noted as a feature: Make sure the darn thing is collapsible!
Usability testing: We tested a dozen different prototypes of the new Cisco.com, with customers on all continents except Antarctica, and almost all of those prototypes included the toolbar. (Below is a picture of one of the tests we ran at CiscoLive! last summer.) We weren’t focused on the toolbar per se, but ran into very few problems. From tests in Germany and Japan came the feedback that the country switching and My Cisco functions would be clearer at the top of the page. But by then it was baked into a template system that couldn’t be changed without a rollout risk.
Testing of other sites: Just for good measure, we did online usability tests of several sites that use toolbars, to understand whether users see them and interact with them. (They do, when the bars are obvious, although not at first.)
We also set up an internal preview of the site so that hundreds of people could see how the new site would work at rollout.
But, Some Fatal Mistakes
By the time we were done with all of this testing, we were feeling pretty confident that the toolbar would be a useful tool and would be a fine addition to the site as long as we were sure to make it collapsible.
But there was one thing we couldn’t do at the time, for technical reasons: We couldn’t roll out the toolbar or other site features out live to have tens of thousands of customers play with them in a real-time beta. Obviously, this would have surfaced any problems that customers encounter in real-world use.
Oh, and also for technical reasons, we couldn’t roll out that “collapse” feature with the site update; it got “descoped.” “Ah, well,” I told myself, “it hasn’t been that big an issue for the other sites we talked to, so we’ll get it in a later release.”
Boy, were we wrong. The first week of rollout. We got comments like these:
And then, there’s was Greg Ferro’s epic blog post about the toolbar on Ethereal Mind. (Artwork tweaks by Bob Plankers from lonesysadmin.net, BTW.). Sigh.
OK, not everyone in the world felt that way, but when the best compliment people could muster was “well, it’s not really that bad,” we knew we had a problem.
Fixing the Situation
So, we got the collapse feature implemented as soon as we could – but even that took weeks to deploy, because it’s a complex site. And then we set about on the path to remove the bar completely. Unfortunately, again for technical and site architecture reasons, it wasn’t easy to remove the toolbar and relocate the functions easily. It required months to get the work scheduled, done, QA’d, and finally rolled out on April 15th.
I’ll post separately about the process of designing the removal, which was its own journey (remember we have two million pages and 90 country web sites).
So, What Should You Do?
So, what could we have done better? What should you do if you’re ever in a similar situation? There are a few big lessons I will share with you:
Have a “live beta” capability that can be applied to a subset of visitors, and make sure you use it. In our case, this was difficult in some areas of the site because of legacy architecture, but you can be sure we’re working on it now. (Not just because of the toolbar, but because we need it for other reasons.)
Be able to undo changes quickly. In our case, it would have been very nice to be able to undo just the toolbar function, but it was more baked into the site framework than we would’ve liked. Having the ability to change *just* how the bar worked would have been ideal.
Evangelize internal alphas: We had a preview of the new site for months before the new site went live, but we didn’t proprerly evangelize the changes to all of the technical experts within the company. While they probably saw the announcements of the preview site, we didn’t shout from the rooftops “HEY, THIS IS SOMETHING YOU HAVE TO USE IN YOUR DAILY WORK RIGHT NOW to make sure it’s an improvement.”
Have common tasks as part of your core use cases. We knew people use Control-F all the time to navigate our pages. But we didn’t have that use case baked into our testing rigorously enough, especially for multiple browsers. Frankly, it’s hard to predict every problem and use case – so the internal alphas and external betas are the best thing to have in place – but having the Control-F use case would have helped as a backstop.
Maintain flexibility even in long development schedules: Finally, we got boxed into making a go/no go decision on the toolbar well before we had enough data. The ability to late-bind on decisions like this is really important on any project, no matter how hard it may seem to build in that flexibility. So, when you are architecting a system, make sure it is flexible enough to roll with the changes at every level.
The toolbar episode was a small negative in a long positive list of great things we have been doing to improve Cisco.com over the last year, such as search improvement, navigational improvements, better content and work on the responsiveness and performance of pages. But it’s an example of how a little annoyance can sometimes overshadow great work and improvements.
So, I am going to leave this picture on my door for a while longer.
P.S. I am hoping that @etherealmind has now actually exploded with joy:
@etheralmind April 6:
It’s confirmed. Cisco Website Toolbar dies on April 15. http://bit.ly/g9qVEg | I’m vibrating with unexploded joy.