Cisco Ultra Cloud Core Repels Pro Hackers

Security is critical when developing new technology and that’s never going to change. Those with ill intent work 24 hours per day, seven days per week to discover vulnerabilities in servers, routers, and security devices that can be exploited. Whether it’s to release viruses or collect millions of dollars in ransomware, tech criminals will never stop committing crimes. The best we can do is protect our hardware, software, and firmware with zero trust and the utmost cyber security built in as a foundational concept rather than bolted on later as an afterthought.

With security at the forefront of everything we do, Cisco recently collaborated in a 5G hackathon with Finnish Transport and Communications Agency Traficom and Junction, an international non-profit ethical hacking and tech community which hosts the annual 48-hour event in Helsinki. This hackathon, held June 18-20, 2021, brought together entrepreneurs, developers, designers, and students from around the globe with a goal of helping local and multinational companies build viable, secure solutions to real-world challenges in 5G networks and services.

Cisco participated in this event, alongside Ericsson, Nokia and others, with an offer of 10,000 Euros in prize money to try to hack into the Cisco Ultra Cloud Core. Some of world’s leading cyber security professionals were asked to search for hidden flags and discover/document any vulnerabilities found in our 5G core. We knew going in that collaboration between vendors, researchers, regulatory authorities, solution developers, and the ethical hacker community would benefit everyone involved when building secure networks and services.

To set the stage for this event, appropriately named “I Bet You Can’t Get In”, the Mobility Technical Marketing Engineer (TME) group constructed this challenge in a way that created a bona fide environment in which to work while still protecting our core lab assets from nefarious activity. The mobility TME began by building a VMware cluster fully isolated from the lab with internet connectivity via a Cisco Adaptive Security Virtual Appliance (ASAv) firewall. Once we had a secure, remote-access sandbox environment prepared, a second team worked to bring up the 5G core on the VMware cluster.

More than 80 global participants organized into teams joined the Cisco challenge and were given access to the test lab through Anyconnect VPN. The challenge was then broken up into three primary facets: collecting user traffic to find the “needle” (a motivational quote from Steve Jobs), obtaining Command-Line Interface (CLI) access to the User Plane Function (UPF) to discover administrator account details, and exploiting a REST API to retrieve any information possible.

On a Friday evening (now that’s dedication!), we led the kickoff meeting to lay out the challenge via a live Webex session. The event commenced at 8 pm local time and the teams of hackers worked through the night and into the morning. On Saturday we held a check-in meeting to examine the progress they’d made. At that time, many of the hackers had given up. Aside from port scan results, no one had managed to gain access or even come close to meeting the challenge. Our team then dropped a few hints such as revealing the password of the UPF’s operator account. Following this hint, two teams utilized the monitor interface tool to locate the Steve Jobs quote, thus solving the first challenge.

The second part of the challenge, SSH CLI interfaces on 5G NFs, held its own, as did the third portion involving REST APIs. The only “hardening” involved was to disable SSH access to the Kubernetes (K8s) nodes, and in the end, only the first challenge was solved (buried in http and gtpu).

Finally, Team cKobclz79 was awarded the first prize of 5,000 Euros based on their ability to solve the first challenge and find the quote. Team Steamy Jofa was awarded the second prize of 3,000 Euros based on continued effort and overall tenacity. However, no other teams achieved enough progress to earn the third prize, so we decided to donate that money to a local Finnish charity.

At the conclusion of the event, we had a chance to speak with the winning three-member Team cKobclz79. As alumni of Aalto University in Finland, they frequently receive notifications of various events involving their alma mater. With an interest in misusing APIs and searching for design flaws in network architecture, they felt a 5G cybersecurity hack offered the perfect chance to keep up with the trends and technologies and acquire penetration testing skills in the industry.

The team began by scanning the network to locate an entry point, and once they found some hosts with open ports, they started searching for vulnerabilities. While digging deeper and sniffing internal traffic, they expanded their knowledge of 5G architecture, learning new terms and better understanding connections between nodes. The team found different parts of the challenge difficult due to having divergent backgrounds. One member struggled to understand the architecture and documentation but had no trouble with the tech stack, another wasn’t sure if they were on the right track because of so many hosts, while the third found several attack points that were not required for the challenge but nevertheless interesting.

With an attitude that no technology is truly safe, the team knew most security vulnerabilities reside in misconfigurations and leaked internal tech stack information, but during initial reconnaissance, they gained access through a system design flaw. They found that a websocket in a staging server would allow them to send commands. A check of other ports then revealed a Docker image whose source code provided a window into how commands were sent. By performing remote code execution, they established the reverse shell as root. This allowed them to install tools to scan the network and examine internal traffic. Enumerating the internal network gave them insight into other nodes based on the 5G architecture documentation provided to start the challenge.

The team then found a Docker registry from which they could pull the image and get the code for CLI access. This websocket GUI allows system administrators to connect to cluster manager and maintain the operation of Ultra Cloud Core.

Following an internal network scan, they found the K8s master node that could be reached from cluster manager. With no firewall to block communication from cluster manager to K8s master, they discovered that a service account credential to connect to the K8s cluster would have allowed them to run commands using kubectl they installed in cluster manager. They could then disrupt existing services or deploy malicious images to trick customers into connecting to them since all services are deployed, controlled, and managed by K8s master.

This event was a huge success and every participant derived value from it. Although we can’t definitively say that Cisco Ultra Cloud Core can’t possibly be hacked, this experience proved that it’s far more difficult than one might imagine. The event allowed technology vendors to put their 5G core to the test, and although these dedicated hackers did their best, they couldn’t hack our core.

As security becomes an increasing concern with the proliferation of 5G and IoT technologies, this event served as a great learning experience for the Cisco mobility team. As new threats exploit the wider footprint of these emerging technologies, challenges like this serve as one of many methods for gaining insights into future attacks so that we can design and build more resilient networks and services.

We encourage you to learn more about the industry-leading cloud native 5G core and take a closer look at some of our security protocols keeping this innovative technology safe.