Surviving 800,000 E-Mails an Hour Without Breaking a Sweat
I have to admit, I have always been fascinated by e-mail systems. Some of this is rooted in the fact that one of my first jobs was as an All-in-1 administrator—think Office365 running on a DEC VAX. Beyond that, e-mail typifies many of the challenges of the data center: supporting increasing scale, maintaining a consistent user experience, handling ever increasing storage requirements, supporting mobile users and delivering bulletproof availability.
Curious as to what we do at Cisco, I had a chat the other day with Ken Pauley from Cisco IT. Ken has been with Cisco for a little over 4 years, running the Design & Engineering Team for Messaging & Calendaring. He has a 25+ year IT career that has been primarily focused around Messaging & Calendaring technologies for medium to large scale enterprises so he has some useful perspective on things.
By way of background about our Microsoft Exchange environment–last quarter we collectively sent about 900 M messages and received about 870 M messages. Our current environment is deployed in six different locations. From a storage perspective we have 123TB of storage in Richardson, 123TB in two SJ locations, 82TB in Amsterdam, 82TB in Hong Kong and 41TB in Bangalore. Richardson and San Jose both have 3 PODS of servers each, Amsterdam has 2 PODs, the rest have 1 POD each. A POD contains between eight and 20 servers and supports up to 11,200 users. We have about 130 servers supporting e-mail across Cisco.
Omar Sultan: What is the most challenging thing about Cisco’s e-mail environment?
Ken Pauley: The diversity. In many enterprise deployments IT will limit choices to minimize support costs. Cisco has a great model of choice for folks; from hardware to software, which is really nice for individual needs. However, it really creates quite a bit of challenges in the email world as not all clients are the same. There has been a lot of pressure from the employee community for us to upgrade our email infrastructure to support Mac clients better, which we will be doing, but it will not solve all of these problems. Every client has different features and there will always be gaps between them which will introduce some unique issues as well as support problems. I would add that scale is another challenge, more so on the calendaring end of the service. Scheduling resources in a large enterprise is really demanding and there really are no products out there that provide this service well.
OS: How do you expect the move to Cisco UCS to impact your job?
KP: Our new deployment is going to have large quotas as well as some data replication capabilities. We have not completed our architecture but have some estimates. We’ll be deploying in two physical locations with a total of around 68 servers in each location (roughly equal split of physicals vs virtuals) and about 2.3PB of storage across those systems. We are expecting UCS to have a big impact on our deployment in FY12. It is part of the solution that will help us to dramatically reduce the amount of infrastructure that we would deploy otherwise. We are also looking forward for a quicker deployment than what was seen in the initial Exchange 2003 deployment.
OS: With the explosion of collaboration technologies at Cisco, do you see e-mail usage going up, going down, or saying the same?
KP: We see the usage changing. We’re partnering with many of the different technologies like IWE powered by Cisco Quad and some content virtualization tools coming from other IT teams to provide options for employees on how they can send attachments, which really are the big storage hog in the email system. Depending on who you are communicating with, there may be other methods for delivering those attachments in the future. In fact, with IWE today, we’re seeing many people using the new Post tool to remove some attachments from email. That said, this adoption will likely take some time and we have a large community that we communicate with externally where we may not have as much of an influence over. We will be watching this closely, it may have an impact and change some volume and/or quantity of messages, but in an organization of this size, how material of a change that will be remains to be seen.
OS: Where do you see enterprise messaging going in the next 5-10 years? Some folks predict e-mail will eventually be marginalized in favor of other modes of communication–what do you see in your crystal ball?
KP: I see better integration with tools that will allow people to communicate effectively to all of the communities that are out there. These tools need to be flexible to the demands of the individual at the time that they are communicating for them to be successful, which means some great integration. Email and calendaring are very mature, completely adopted and can be used just about anywhere. New tools will continue to come along and will need to interact well with a service like email or they have the potential of alienating some of their audience. So will email go away, I don’t think so. Will it look different than it does today, yes, it sure will. When you get that email in the future, you’ll likely be able to very easily link that email to other threads, conversations, communities, etc to bring more context to that message.
OS: What can our customers learn from our Exchange environment–best practices, or things you might do differently next time?
KP: Flexibility is key I believe. We learned that from our initial deployment. It was a very successful and reliable deployment, but it lacked the ability to change with the times as IT progressed. Infrastructure like this tends to be around for 5-7 years which is a pretty long lifecycle, a lot can change during that time. Our new deployment will hopefully solve some of those issues so we can adapt to change much quicker than we could before.
For more details on Cisco+Microsoft solutions, you can go here.