Cisco Blogs

Toolbar: Cautionary Tale for Fellow Webmasters

- April 22, 2011 - 13 Comments

Usually web teams congratulate themselves for adding things to web sites.  But this past weekend, we were happy to remove something instead: The toolbar on the bottom of pages.

The idea of the toolbar was to provide kind of a “swiss army knife” function that could contain features that would work across the site. We wanted to start small with the features, but expand over time.  The toolbar was my idea, and one we modeled after similar functions on other sites across the web. Here it is, in all it’s glory:

Expanded Toolbar

This past fall, we rolled out a new design across the two million pages of It included a number of navigational and other improvements (some of them pretty subtle but useful)  and it included this toolbar (which we thought was subtle, but was not, as it turns out).  This was an epic redesign (in the traditional sense of that word), and the feedback, reviews, and site metrics bear out that it is working well.

But, it turns out there was a huge problem with that toolbar: It really got in the way of people’s work, especially in one important way: In the Firefox web browser, if you used the Control-F function to find text on a long page, the toolbar would often obscure the results!  So, this new ‘feature” actually ended up unexpectedly breaking the standard behavior of a key browser. Yikes.

How could we let something like that happen, you ask?  Well, that’s a story.

I’ve worked on web sites for a very long time (basically since the web started), and thought I knew every twist and turn that could go right or wrong.  When we got the idea for the toolbar – which was eventually supposed to have a lot  of useful features – we did all of the classically right stuff to vet the idea. But we didn’t vet hard enough, and we fell victim to some early schedule dependencies that forced us to decide about implementing this “feature” before we should have. First, here’s the review and vetting we did, just to give you an idea of our thoroughness:

Concept reviews: The thing we liked about the toolbar idea was that it gave us a place to put new features that could work universally across the web site, which would make them guaranteed to be available everywhere and work in a consistent way.  And, it seemed to place things “out of the way” so they wouldn’t clutter our pages. We walked through the concept and even included it in all of our online prototypes. It seemed to solve some problems.

Best practices check: But, we wondered… do these tool bars work well in real life?  So, I personally talked to the web folks at a number of other companies and web sites that had implemented similar bars. The advice we got boiled down to: “They do get used, but make sure it’s collapsible for people using notebooks and other short screens.”  Noted as a feature: Make sure the darn thing is collapsible!

Usability testing: We tested a dozen different prototypes of the new, with customers on all continents except Antarctica, and almost all of those prototypes included the toolbar.  (Below is a picture of one of the tests we ran at CiscoLive! last summer.) We weren’t focused on the toolbar per se, but ran into very few problems.  From tests in Germany and Japan came the feedback that the country switching and My Cisco functions would be clearer at the top of the page. But by then it was baked into a template system that couldn’t be changed without a rollout risk.

Testing of other sites: Just for good measure, we did online usability tests of several sites that use toolbars, to understand whether users see them and interact with them. (They do, when the bars are obvious, although not at first.)

We also set up an internal preview of the site so that hundreds of people could see how the new site would work at rollout.

But, Some Fatal Mistakes

By the time we were done with all of this testing, we were feeling pretty confident that the toolbar would be a useful tool and would be a fine addition to the site as long as we were sure to make it collapsible.

But there was one thing we couldn’t do at the time, for technical reasons: We couldn’t roll out the toolbar or other site features out live to have tens of thousands of customers play with them in a real-time beta.  Obviously, this would have surfaced any problems that customers encounter in real-world use.

Oh, and also for technical reasons, we couldn’t roll out that “collapse” feature with the site update; it got “descoped.”  “Ah, well,” I told myself, “it hasn’t been that big an issue for the other sites we talked to, so we’ll get it in a later release.”

Boy, were we wrong.  The first week of rollout. We got comments like these:

And then, there’s was Greg Ferro’s epic blog post about the toolbar on Ethereal Mind. (Artwork tweaks by Bob Plankers from, BTW.). Sigh.

OK, not everyone in the world felt that way, but when the best compliment people could muster was “well, it’s not really that bad,” we knew we had a problem.

Fixing the Situation

So, we got the collapse feature implemented as soon as we could – but even that took weeks to deploy, because it’s a complex site. And then we set about on the path to remove the bar completely.  Unfortunately, again for technical and site architecture reasons, it wasn’t easy to remove the toolbar and relocate the functions easily. It required months to get the work scheduled, done, QA’d, and finally rolled out on April 15th.

I’ll post separately about the process of designing the removal, which was its own journey (remember we have two million pages and 90 country web sites).

So, What Should You Do?

So, what could we have done better? What should you do if you’re ever in a similar situation? There are a few big lessons I will share with you:

Have a “live beta” capability that can be applied to a subset of visitors, and make sure you use it.  In our case, this was difficult in some areas of the site because of legacy architecture, but you can be sure we’re working on it now. (Not just because of the toolbar, but because we need it for other reasons.)

Be able to undo changes quickly. In our case, it would have been very nice to be able to undo just the toolbar function, but it was more baked into the site framework than we would’ve liked.  Having the ability to change *just* how the bar worked would have been ideal.

Evangelize internal alphas: We had a preview of the new site for months before the new site went live, but we didn’t proprerly evangelize the changes to all of the technical experts within the company. While they probably saw the announcements of the preview site, we didn’t shout from the rooftops “HEY, THIS IS SOMETHING YOU HAVE TO USE IN YOUR DAILY WORK RIGHT NOW to make sure it’s an improvement.”

Have common tasks as part of your core use cases. We knew people use Control-F all the time to navigate our pages. But we didn’t have that use case baked into our testing rigorously enough, especially for multiple browsers. Frankly, it’s hard to predict every problem and use case – so the internal alphas and external betas are the best thing to have in place – but having the Control-F use case would have helped as a backstop.

Maintain flexibility even in long development schedules: Finally, we got boxed into making a go/no go decision on the toolbar well before we had enough data.  The ability to late-bind on decisions like this is really important on any project, no matter how hard it may seem to build in that flexibility. So, when you are architecting a system, make sure it is flexible enough to roll with the changes at every level.

The toolbar episode was a small negative in a long positive list of great things we have been doing to improve over the last year, such as search improvement, navigational improvements, better content and work on the responsiveness and performance of pages. But it’s an example of how a little annoyance can sometimes overshadow great work and improvements.

So, I am going to leave this picture on my door for a while longer.

P.S. I am hoping that @etherealmind has now actually exploded with joy:

@etheralmind April 6:
It’s confirmed. Cisco Website Toolbar dies on April 15. | I’m vibrating with unexploded joy.

In an effort to keep conversations fresh, Cisco Blogs closes comments after 60 days. Please visit the Cisco Blogs hub page for the latest content.


  1. Just goes to show you that what's broken can be fixed.

  2. I would suggest you force your internal customers/employees to use your latest website dogfood. That will get you lots of beta users, and definitely get you lots of feedback as well. Making users have to opt-in to changes will not get the scope of users you need to sufficiently test. The company I work for has a strong dogfood culture, and it has definitely paid off when products have been less than stellar. Its also hard if not impossible to opt-out no matter who you are, everyone's work is business critical afterall.

    • Hi Bill, Great point, and we actually do this. But perhaps not aggressively enough. One issue with relying on "dogfood tests" alone is that they don't always find issues that customers will find. That's because customers often have different use cases and edge cases that are hard to identify. But, even an "edge case" on a web site with millions and millions of visitors can translate into tens of thousands of users being irked if their edge case doesn't work out optimally.

  3. I'm Martin's manager, and I want to publicly and loudly applaud him and the team for the openness of this blog post and the team's behavior around the toolbar adventure. We think we have a great website in, entirely due to the hard work and commitment of all the terrific folks who work so hard to make it great. But we are not perfect, we do make mistakes. But we pay attention to visitors, we listen to feedback, and we work our tails off to improve the site based on that feedback. I was one of those flogging Martin regularly about the toolbar, but let me be among those standing up and cheering the honesty and the transparency in the way they dealt with the situation. Kudo's to you and the team Martin! Mike

    • Like I said recently, I do have the smartest boss in the world. :-) I should really thank Greg, Chad, James, Katie, Marilyn, Chris P, Kit, Laura and the My Cisco team, and all the others who were involved wholly or tangentially on this one. It was more complicated than it seemed, as you can probably tell from the blog post.

      • P.S. Just so readers don't think we're relaxing, we do have quite a large "to do" list of items to improve or fix. I have a meeting on these scheduled for early next week, using some blog posts, tweets and web site feedback comments as out punch list.

    • As a vocal critic of the website redesign and the toolbar in particular, let me also congratulate Martin and web team for responding so positively. It's nice to see that some parts of Cisco still understand customers and listen to us. I believe that the toolbar was personal issue for Martin and it might have been painful to see an idea get crushed. So credit for taking the challenge and turning it around. I would especially like to say thanks for the outreach from Martin's team in terms in discussing the problems and sharing their progress. And being willing to discuss what was happening. I would like to see a lot more open thinking, and community integration from Cisco in the future and engaging with customers as well as resellers. It's time to recognise that the Internet is changing that communication process and Cisco is almost the last IT vendor that continues to ignore social media. greg

    • as a whole is indeed a great place to visit and this blog has been one on my preferred places to keep coming back. For any website not matter how big or small it maybe, it is important to take user suggestions seriously and try to make changes based on those ideas and feedback's. And talking about the mistakes, then IMO. we cannot term a move or an action to a 'Mistake' unless it is repeated despite knowing about the consequences. Moreover, when working online it need be the case where something that has worked for a competing site would work for ours too. An endeavor to try better things is something that is important. So Martin did his part, now few users might have felt that really helpful, while few might have got irritated with the toolbar that is been referred in the article. IMO. that's absolutely fair enough. The best part here as it seems is that it was taken down after the user suggestions. But, over and above this site has been a perfect place for me to learn newer information. Thanks.

  4. Martin -- Great stuff, as always. I have to respect the transparency of you and your team's work - the DoD could probably learn a thing or two from you guys. Have you considered making sure that the live betas include some of your more "vocal" users/critics? I know that, personally, I enjoy the opportunity to provide useful feedback when I know someone at the other end is paying attention and considering it. I'm sure I'm not the only one ... -- Mike C

    • That's a good point, Mike. I've been talking with Bill Skeet who leads user experience specifically for the Support areas about this very thing -- either betas-for-everyone or at least working previews for the most active members of the community. We often build prototypes of things as we're working on them -- and do usability tests with some of you on those -- but in retrospect I think should be just making the more widely available for people to really bang on as "release candidates."

  5. Great post Martin - we've collectively lived through lots of web enhancements which we truly believed would make things better for people coming to, but in fact, either didn't make a difference at all, or made things worse. I hope that folks understand that it's never our intention to make their lives more difficult! Most often, it's simply fallout from our "perpetual beta" efforts to provide the best web experience possible -- led/championed (thankfully!) by you! Thanks for all you do - and for your honest and transparency throughout!

    • Thanks! And, I'll add that we do make a lot of seemingly small improvements that end up making a big difference, like the quicklinks to downloads and documentation in the search results, the quick "megamenu" navigation from the top bar to support the top tasks that people do on the site. A lots more that we've mentioned here. Of course, we have a pet list of label and functionality fixes for the next few months.

  6. Martin, this is a great post! I actually didn't know the full scope of testing and diligence that went into the site redesign. I kind of knew it was going on but not the details... thanks for that! I love the idea of being able to roll out new features at the beta stage either opt in like google or just to a (seemingly) random subset of users as I've heard amazon, ebay and facebook can do. Lastly agility in projects is a kind of holy grail with a surprisingly complex path to get there... I agree with you it's doable though and well worth it!!!