It was so amazing to return to London for the Black Hat Europe 2021 Network Operations Center (NOC). Produced by Informa Tech, and built by the security partners, the mission of the NOC is to quickly build a conference network that is secure, stable and accessible for the briefings, sponsors and attendees.
It is a team effort, where collaboration combines a robust backbone (Gigamon), firewall protection (Palo Alto Networks), segmented wireless network (Commscope Ruckus) and network full packet capture & forensics, with identity (RSA NetWitness). Cisco Secure supports the NOC operations with DNS visibility and architecture intelligence (Cisco Umbrella and Cisco Umbrella Investigate) and automated malware analysis and threat intelligence (Cisco Secure Malware Analytics (Threat Grid), backed by Cisco Talos Intelligence and Cisco SecureX).
Cisco Secure also protected 14 iPads used for the Black Hat conference registration and 48 iPhones for sponsor lead retrieval, with the Cisco Meraki Systems Manager (SM) mobile device management (MDM) platform, with security through the Cisco Secure Endpoint for iOS/Security Connector.
Mobile Device Setup, by Paul Fidler
The first challenge we faced was configuring the iPads and iPhones with the contractor in Germany before they shipped to London. We wanted the relevant settings, restrictions and applications on the devices before they arrived, so that they could be used for guest registration, with no or little end user intervention, whilst keeping the devices secure, allowing for further changes in the future to be made remotely. We also needed to be able to see the inventory of the devices, including OS version, location, SSID and applications.
Apple devices, since 2010, have had Mobile Device Management (MDM) capability, allowing them to be enrolled remotely into 3rd party MDM solutions. There’s several ways that this can be achieved:
- User Enrolled
- Device Enrollment
- Supervision
Supervision is a process that devices go through, where the owner of the device has met enough criteria that Apple have deemed them actual owners of the device. Because of this, it allows the owner elevated privileges: Granting them the ability install DNS, Global Proxies and many other capabilities.
Using a combination of Apple Configurator and Meraki, the devices were enrolled into the Systems Manager. First, we asked the contractor to put the devices under Supervision, and then enroll them using a Meraki SM QR code that we emailed to them. They just had to point the camera at the code and click on the pop-up link.
Once the devices were managed, applications were authorized in Apple Business Manager (a portal that allows for applications to be ‘bought’ on behalf of users), removing the need for an iTunes account to be provisioned on the device, and then installed using Meraki SM. WiFi settings were also installed remotely, as well as restrictions and other settings, then the devices were shipped to England.
Apple’s MDM Protocol is extensive and gives a holistic view of devices. However, for privacy reasons, and other issues, there’s the need to supplement this with information from the Meraki SM client on the device. This includes Location, connected SSID and whether the device is jailbroken; Once the app was opened manually at the conference, we started to see full inventory for the devices:
Another challenge for the team was that the contractor who supplied the iPads had put PIN codes on the devices…. For 70 devices, this could have been painful, and as there’s no systematic way of provision a PIN on Apple devices (it is possible, however, to provision a PIN Policy!), Meraki SM was used to remotely remove both the PIN policy and PIN. This was only possible because the device was supervised.
As the devices were now managed and supervised, it also allowed for Cisco Secure Endpoint for iOS with Umbrella DNS to be installed and configured remotely from the Umbrella dashboard.
After some usage of the devices, data started to be visible in the Umbrella dashboard. There’s more about this configuration shortly.
We also deployed a WiFi profile to connect to the SSID reserved for the conference administration, by MAC address, with a unique sixteen-character password for each iPad, for connecting to the Commscope Ruckus access points. With the satellite map view, we were able to see the location of the iOS devices, and one were to ‘walk away’ from the conference, we had the ability to remotely wipe all the data and ‘brick’ the device.
During the conference, we needed to repurpose a few iPhones as mobile registration devices, incase there was a backlog at registration. This was easily accomplished remotely and registration was a breeze.
At the conclusion of the conference, we employed one of the strongest use cases for leveraging Meraki SM: en masse remote wiping before being returned to the contractor. This ensured that any data and applications that resided on the device were removed. Again, wiping 70 devices (Black Hat USA 2021 had 300 devices!) would take a considerable amount of time: from the Meraki SM dashboard, this took three mouse clicks!
Leveraging SecureX device insights beta for iOS inventory and security, by Aditya Sankar
With the Meraki SM as the MDM provider, Umbrella for roaming DNS security and Secure Endpoint for iOS, we had a trifecta of integrated solutions, for an extremely high level of protection for our mobile device deployment.
How do these technologies work together to protect each device? The answer is via the Cisco Security Connector (CSC) application for iOS. The CSC app was developed by Cisco, in partnership with Apple, and it has two components to ensure full stack protection on mobile devices. First is the Umbrella roaming security to provide DNS-layer enforcement and encryption, and customizable URL based protection with intelligent proxy even when a device is off network. The other half is Clarity for iOS. This offers application auditing and correlation, logging of encrypted URL requests without SSL decryption, and full visibility of network traffic from the device, by taking advantage of the supervised mode iOS API’s. We had been using the DNS roaming feature of the CSC app at Black Hat USA 2021, but adding in Clarity for iOS in London brought a whole other layer of protection to the iPads and iPhones.
Meraki, Umbrella, and Secure Endpoint offer a tight integration that’s also very easy to configure. Let’s go over the simple configuration steps to add Clarity to the CSC app. First grab your API credentials from the Meraki SM dashboard and paste them into the Secure Endpoint console under Accounts -> Organization -> MDM Integration:
Next, we need to deploy Clarity for iOS from the Endpoint console under Management -> Deploy clarity for iOS. Here you can select the Group in which those connectors will reside, choose your Meraki organization and the network. Once you click Update, the Clarity content filter setting will automatically be pushed to the Meraki profile. Now just ensure the latest Meraki profile is pushed to all your device and that’s it!
With the iOS devices fully protected and configured, it was a great opportunity to try out a new feature in SecureX that is still in beta called SecureX device insights. Device insights provides a seamless, agentless, unified view of the assets in your environment for attack surface reduction. SecureX device insights can take data from multiple sources and merge them to create a single unified record to have all the information about that particular endpoint in one place. For us at Black Hat, we used device insights to show us a single record for our iPads and iPhones with data from both Secure Endpoint and Meraki SM.
I did notice a difference in the number of devices in the Meraki SM portal and in the AMP console. Using device insights, I was able to easily filter for devices that were in Meraki SM, but not in Secure Endpoint to find six iPhones that the contractor held back in Germany. I also found one device running iOS 12.2 which is end of life!
In the NOC, we kept the Umbrella organization (org) for the iPads and iPhones is separate from the Umbrella org for the conference network. This allows us to keep a distinct separation between which traffic is from the iPads/iPhones and which is from the conference network. The conference Umbrella org does not block any requests since we do not want to interrupt demos or any network activity. However, the iPad/iPhone org has full DNS security protection. We added both Umbrella orgs as separate tiles in the SecureX dashboard, so we can have a unified method of monitoring high level statistics.
Right on the SecureX dashboard, we noticed a few blocks in the “Umbrella-iDevices” tile. In this case, we were able to see some folks trying to use the iPhones to access Facebook and LinkedIn, which were blocked.
With SecureX device insights, you can drill into the inventory record and view the device details, where it is seen in the network and the secure policies.
Insights also integrates with Cisco Duo and Orbital, along with partners Ivanti MobileIron, VMWare WorkspaceONE/AirWatch, Microsoft InTune and JAMF.
We see you! – Black Hat, Umbrella and Device Attribution, by Alejo Calaoagan
For several years, Cisco Umbrella provided DNS security for every Black Hat Conference around the globe. From Las Vegas to London to Singapore, we’ve monitored and analyzed DNS traffic with Umbrella, hunted and validated threats with Umbrella Investigate, and (when needed) mitigated domain or cloud application-based threats via DNS.
We’ve done a phenomenal job of protecting both the attendees and the conference itself, however something was missing: device and identity attribution.
Activity visibility is a foundational element of security. Sure, you can survive with limited visibility, and Umbrella is intelligent enough to set it and forget it in many cases. However, at an event like Black Hat, an event with a huge cyber target on its back, deep attribution is paramount to ensuring threats are contained.
In years past, while Umbrella has monitored and protected Black Hat by identifying requests made to malicious destinations, we were not capturing where the requests were coming from, as they were all masked by the Ruckus access point that the request came through. This year, we decided to up our game by obtaining permission from Bart and Grifter to roll out an Umbrella Virtual Appliance (VA) in front of the attendee network (a great idea by Aditya Sankar and thank you to our partners at PAN for allowing us to connect).
Traditionally, Umbrella VA are used for active directory policy integration and user attribution. By leveraging our Umbrella VAs here in London, we’re able to see the source IP for all requests that were made, giving us the ability dig deeper into devices that are making suspicious requests. This allowed us to zero in on the full breadth of activity on a potentially compromised machine. Chances are that if a machine is making a request to malicious destination, it will likely call out to other malicious destinations as well.
Rather than just reacting to threats we see, this attribution enabled us to proactively hunt for threats, effectively helping us stay ahead of attackers versus simply responding to attacks. This proactive approach is what separates good security teams from great ones.
Threat Hunting – RAT Attack, by Jessica Bair
The trainers, briefers and sponsors need to be able to access and demonstrate malicious code and network activity; without infecting attendees or other networks, or experiencing an outage. It is a balancing act that the NOC team enjoys creating at each conference. The NOC was closed to attendees again, but was streamed live and available to be viewed from outside of the NOC and at home via their Twitch channel, with presentations from NOC leaders Neil Wyler (@grifter801) and Bart Stump (@thestump3r).
Threat hunting is a core mission of the Cisco Secure team, while monitoring the DNS activity for potentially malicious activity. Also, to review the automated malware analysis of samples submitted by RSA NetWitness for maliciousness.
The Cisco threat hunting team investigates potential threats in SecureX threat response, which was integrated with 20+ Cisco and partner threat intelligence platforms.
Ian Redden, who manages the SecureX threat response 3rd party integrations ecosystem, built a custom integration with RSA NetWitness for the Black Hat NOC.
While threat hunting in Cisco Umbrella, I observed multiple connections to a domain identified by Secure Malware Analytics / Threat Grid feed integration as Remote Access Trojan (RAT) DarkComet Network Communications.
Investigation in SecureX threat response confirmed the malicious nature by several intelligence sources, including Recorded Future, IBM X-Force Exchange and Cisco Talos.
The Ruckus team tracked the movements of the attendee across the conference, different levels and rooms, indicating a likely infected host vs. a demo machine. The IronNet observed the attendee was then searching for Pokemon cards to purchase.
With this information, the NOC management authorized the use of a captured webpage notification, which we use for users who connected to the Black Hat network and were found to be infected with malware, shared credentials in the clear or were running cryptomining. The notifications were done by moving affected users into a group within the PAN Firewall. This way, those who are delivering presentations and demos can still reach their attended target, but unaware attendees can be protected.
Malware Analysis -> PII breach or fraud? by Ian Redden
RSA NetWitness Orchestrator carved the files off the network stream and sent them to Cisco Secure Malware Analytics (Threat Grid). During the conference week, over 500 samples were sent for analysis. I created a physical light on monitor, that would flash when sample analysis was taking place. Unfortunately, it never flashed ‘RED’ indicating a malware conviction.
At most Black Hat conferences, we observe a breach of personal information. RSA Netwitness Orchestrator submitted a PDF document to Secure Malware Analytics the last hours of the conference.
Malware Analytics analyzed the file using the Random Cursor Movement with Image Recognition playbook. While analyzing the file, I noticed the PDF file appeared to be a negative COVID-19 test.
On page 2 of the PDF document, a QR code appears. Using a QR code decoder, the encoded text is a URL to a website that contains the same PDF (websitehome.co.uk). The personal identifiable information contained within the PDF is for an EU citizen.
Further investigation by the NOC team, including RSA NetWitness, IronNet and Cisco found the website where the PDF originated still hosted the PDF, that the PDF appeared to be manufactured by a non-existent ab, and that most likely the certificate was fraudulent.
The final file submitted of the conference was pxxxx-sa-cxxx-sxxxxx-s-covid-19_bulgarisch.pdf. The document is notable as the filename appears to contain the name, country and the word “COVID-19”. The file contains the URL to the Reinickendorf district of Berlin’s website (berlin.de/ba-reinickendorf/corona) on Coronavirus.
In anticipation of the NOC, I developed multiple versions of a tower light to showcase the power of Cisco SecureX and the ability to use its API to integrate with anything…. even a light. The idea was to turn lights on or off, pulse, animate or flash depending on the alert or severity.
During Blackhat Europe, the tower light was integrated with Malware Analytics. The use case was as follows:
– RED PULSING – Submitted sample score greater than 90 (i.e. critical/very malicious)
– YELLOW MARQUEE – Submitted sample that is currently being processed
– OFF – No samples being submitted
Over the course of the week, no samples were submitted with a malware score higher than 90 to trigger the red pulsing light.
This proof-of-concept is currently using a Raspberry Pi 4 with 3 x Adafruit Neopixels connected to GPIO 18 (PWM) and 5 volts and ground for power. Future enhancements include using a ESP32 module using Micropython, a 3D printed enclosure, and clear plastic light covers for more color. The software is currently written using Python 3 and runs as a docker container. The code for the light will be hosted on github.com/ciscosecurity over the next few weeks.
Real or mirror, mirror? By Christian Clasen
New websites are created on the Internet every second. It’s an unbelievably difficult task to keep tabs on the destinations that may appear and make determinations in real-time about whether they pose a threat. One of the most valuable features of Cisco Umbrella is the category “Newly Seen Domains.”
Umbrella is uniquely positioned to spot new domains as they are registered. By placing a skeptical eye on those short-lived Internet properties can prevent attacks before they are fully launched and protect devices at the earliest possible stage.
In the NOC, we got just such an alert about a domain.
It was registered the day before the Black Hat Europe conference started.
In Umbrella investigate, we could also quickly see the global queries for the domain.
Although the nascency of the domain triggered Umbrella’s security alert warning, another data point that Umbrella Investigate provides is the geographical disparity between where the domain is registered, and where the requests for it originate. The domain was registered in Iceland, but majority of connections to it were coming from the United States.
Using an off-site, sandboxed machine, we had a look at what the webpage actually looked like. You can also do this in Secure Malware Analytics / Threat Grid for a set period of time, by submitting a URL. We found a single folder on a not-so-modern looking page:
Inside the directory was a copy of the “Mirror” news site.
Copies (or “mirrors”) of well-known sites are often used to trick users into clicking on links that look familiar, but are in fact harmful and can install malware or steal sensitive information from users. Whatever the use of this site, it is something that Umbrella was uniquely positioned to alert against, and in this case, pointed us to something that was well deserving of the NOC’s attention.
DNS Stats
We saw a marked decrease in DNS activity, with the hybrid event and reduced in-person attendance, compared previous years.
The Umbrella Activity Volume report allowed us to quickly aggregate events, and also use it to drill in for threat hunting.
In 2021, over 2,162 apps connected to the conference network and made DNS requests, far out pacing any other Black Hat Europe conference.
It reflects the move to mobile, that will continue to grow.
Acknowledgements: Special thanks to the Cisco Secure Black Hat NOC team: Jonny Noble, Alejo Calaogan, Christian Clasen, Aditya Sankar, Ian Redden and Paul Fidler. Also, to our NOC partners RSA (especially the RSA NetWitness team led by Percy Tucker), Palo Alto Networks (especially James Holland), Commscope Ruckus (especially Jim Palmer), Gigamon, IronNet (especially Bill Swearington), and the entire Black Hat / Informa Tech staff (especially Marissa Parker – Queen of the NOC, Steve Fink – Chief Architect, Neil Wyler, Bart Stump and James Pope).
We are all so very hopeful to reunite for Black Hat Asia in May 2022.
Note: The staff of the NOC were all vaccinated against COVID-19 and underwent COVID-19 testing before and after the conference.
About Black Hat
For more than 20 years, Black Hat has provided attendees with the very latest in information security research, development, and trends. These high-profile global events and trainings are driven by the needs of the security community, striving to bring together the best minds in the industry. Black Hat inspires professionals at all career levels, encouraging growth and collaboration among academia, world-class researchers, and leaders in the public and private sectors. Black Hat Briefings and Trainings are held annually in the United States, Europe and Asia. More information is available at: blackhat.com. Black Hat is brought to you by Informa Tech.
We’d love to hear what you think. Ask a Question, Comment Below, and Stay Connected with Cisco Secure on social!
Cisco Secure Social Channels
CONNECT WITH US