Better, Together – Cisco IT at the Tokyo Olympics

“Faster, Higher, Stronger – Together.” The Tokyo 2020 Olympic and Paralympic Games motto highlighted an immense feeling of solidarity across the world. Isolated for so long during the pandemic, these Games signified coming together and forging forward, not just as individuals, but as teams. And Cisco was there for it.

Cisco is renowned for supporting large scale sporting events and this year was the official Network Equipment Partner for Tokyo 2020. While Cisco IT has been heavily involved in previous events, including Rio 2016 and London 2012, this time was different.

As a team, we had to figure out how to run a 24×7 operation from our home offices, delivering the best experience possible. We were responsible for the stability of our internal infrastructure, applications, and support processes for our business stakeholders such as our Technical Assistance Center (TAC) engineers and Customer Experience engineers. Our engineers were supporting the wired and wireless network in more than 80 competition and non-competition venues across Japan.

On reflection, we learned a lot of lessons about our people, processes, and technology.

Our people are the heartbeat of the operation

For a program of this scale to be successful, we needed strong partnerships built on trust right from the planning phase. We found this to be even more important as we were running this 24×7 war room operations virtually over Webex.  When you’re under pressure and solving for issues during a critical time, having that established rapport and trust really helps break down those barriers.

An example of this is when we experienced an issue during the very early hours of Japan’s morning with one of the applications used heavily by our customer facing Technical Assistance Centre (TAC). The IT application team raised a change request to add new code, however the Enterprise Operations Centre (EOC) and the EMEAR IT Duty Manager made a combined decision on behalf of the APJC program leads not to deploy this code as they felt it could be quite risky. Instead, a short-term solution was deployed that would have minimum impact on the Games. This was an example of a trusted relationship where we acted on the best interest of Cisco.

We also relied very heavily on our local Regional IT and Workplace Resources (WPR) teams in Japan who did a lot of the groundwork preparing the Game Time Operations (GTO) war room for our CX Duty Managers, Marketing and other business teams. Configuring the GTO war room with Webex boards, IP Phones, printers and activating all the network ports became quite challenging because of pandemic restrictions— the IT and WPR teams could only come into the office on their specified days. Planning and coordination between these teams was important to ensure everything was ready before the big day.

Our processes kept us on track

A lot goes on behind the scenes for such a large-scale event like the Olympics. With two years in the making, we had huge effort and commitment from many teams in Cisco IT.

Our Enterprise Operation Centre (EOC) team and Technical Support team already have clearly defined operational processes to ensure that the network and our applications are running like clockwork. We built another layer of protection and rigor into these processes

With the support of our Enterprise Release Management Organization (ERMO), we set up the Tokyo 2020 Change Control Board. This board made final decisions around change requests to the network or to our applications during the Games (e.g., downtime for maintenance, upgrades, hot/cold backups).

We needed full visibility and transparency of any incidents to our IT services and tagged close to 6,000 configuration items (e.g., databases, hosts) with ‘Tokyo 2020.’ Any issues on our network or application that had a dependent CI (Configuration Item) tagged were treated as high priority incidents no matter their designated criticality.

Once the Games were underway, IT duty managers were the central point of contact between the business teams and the IT service teams.  To ensure we had 24 x 7 operations, we used a ‘follow the sun’ model by engaging the regional IT team across APJC, Americas/Latam and EMEAR.

Having the IT duty manager coordinating across teams was incredibly helpful because of their holistic view and understanding of an issue’s impact on all stakeholders.  For instance, when some employees had difficulty connecting through VPN, initially only users experiencing the issue were provided a workaround solution and the number of incidents raised with technical support continued to increase.  The IT duty manager ensured that email communication was sent to all employees in Japan, informing them of the workaround, and the number of incidents dropped significantly.

Our technology made it possible

While technology was a key enabler in the way we operated as a cohesive team, it also gave us foresight into potential incidents.  We leveraged the following to operate our virtual environment:

• Webex – We simulated the physical war room experience in Japan through Webex. Since all team members from all time zones were in the one space, everyone had visibility of any issues. Handovers between duty managers were seamless.
• ThousandEyes – We always strive to identify issues and resolve them before they impact customers and partners. ThousandEyes gave us the ability to detect issues early in the network and prevent them from becoming major issues.
• AppDynamics – Wherever we had AppD application monitoring enabled, we looked at what was going on in the system, for example, was there something stuck in the database or was garbage collection high in the database?  This detection helped us identify issues more quickly than the traditional way of going through logs in the systems and reduced the MTTR (mean time to resolution).
• Dashboards – We used homegrown tools to gain a business view of critical incidents and how they might impact our Cisco engineers supporting the Olympic and Paralympic Games.

Planning is well underway for the Olympic and Paralympic Games in Paris and our IT teams are looking forward to supporting these Games with some of the best practices we have learnt from Tokyo 2020!