Avatar

This year marks the first time Cisco Talos Incident Response (Talos IR) has supported the Security Operations Centre (SOC) at the GovWare Conference and Exhibition in Singapore, providing dedicated threat hunting expertise.

It was an exceptional opportunity to leverage the live conference environment as a real-world threat hunting exercise, applying Cisco’s technologies and Talos IR’s threat hunting methodologies against an active attack surface. Unlike enterprise environments, conference networks represent quite a unique environment for threat hunting. We don’t have months of baseline data available, devices come and go all the time, thousands of devices flood the network for just a few days, and each device has its own security posture, protection mechanisms, and patch levels, making them either resistant or very vulnerable to exploitation.

The objective of Talos IR’s engagement at GovWare was to detect and investigate any malicious attempts targeting the infrastructure and visitors, as well as to identify potential compromises affecting attendees’ systems at the conference. To achieve this, we developed and pursued four hypothesis-driven questions, aligning them with data from multiple Cisco and partner technology stacks deployed within the GovWare SOC. The primary hypothesis questions were:

  • Are there any attendees attempting to breach each other’s systems?
  • Are there any attendees attempting to subvert any SOC or conference systems?
  • Are there any attendees compromised, and could we warn them?

Each of these questions required a slightly different investigation plan to ensure we identified what data to examine and validate our visibility into potential attacks. We extend our sincere appreciation to our partners at Endace, alphaMountain, Pulsedive, Stealth Mole and several others for granting access to their technologies. Their collaboration ensured that our threat hunting efforts extended beyond Cisco infrastructure, enabling us to integrate contextual intelligence from multiple security platforms into one XDR view.

In addition to technology contributions, the collaboration and expertise of partner teams supporting GovWare were instrumental in enhancing the depth and effectiveness of our joint threat hunting mission.

During conferences, Talos IR sees all kinds of traffic coming through the infrastructure. Some encrypted, some clear text, but we also see all kinds of different protocols. The following table summarize the ‘clear text’ traffic observed during GovWare:

Observed clear text protocol stats from Cisco Secure Firewall
Table 1: Observed clear text protocol stats from Cisco Secure Firewall

Since we had deep packet inspection from Endace feeding directly into the Cisco Splunk index, we pulled some statistics, as well. The following statistics Endace show different service classifications applied to traffic passing thought the conference networks:

Endace classified traffic
Table 2: Endace classified traffic

Finally, as we were monitoring the DNS through Cisco Umbrella, we created a summary of observed categories for the DNS traffic. Over a few days of the conference, we saw what was expected — a lot of standard Software/Business/SaaS/PaaS calls.

Top 20 Cisco Umbrella classifications
Table 3: Top 20 Cisco Umbrella classifications

As we did not block or filter any traffic, everything was allowed throughout the conference.

As vendors and individuals from all across the world showed up to the conferences, the SOC observed all kind of interesting malware that traveled across the world, too. It is not uncommon for individuals and organizations to travel with compromised or unpatched systems, and while the physical location might change for individuals, the malware still communicates as soon as devices are booted up. We observed all kind of interesting beaconing, phishing visits, and attempts to download malware. Here is the breakdown of statistics related to malware-related categories we observed in Cisco Umbrella:

Malware-related observations in Umbrella
Table 4: Malware-related observations in Umbrella

Each visit to the categories in table above was an opportunity to investigate using cross-correlation between different tools. Here is an example of malware that was identified by correlating firewall traffic observations with communication to a specific IP address known to be malicious.

XDR-level investigation into a suspicious host
Fig. 1: XDR-level investigation into a suspicious host

In one case, Talos IR found a tunnel related to a known threat actor that our colleagues at Google Mandiant also identified. The connections to the malicious IP appeared every second, and the IP used a domain generated algorithm (DGA) to rapidly change DNS assignment to avoid detection and blocking. In XDR, we observed rapid correlation between destination IP, its classification, network traffic, and DNS records to confirm that an attempt was being made to establish a malicious tunnel.

XDR correlation for suspicious tunneling event
Fig. 2: XDR correlation for suspicious tunneling event

Around the time of this investigation, we saw the DNS records in Table 5 being rapidly assigned to the IP to avoid detection.

DNS records related to malicious tunnel identified on the network
Table 5: DNS records related to malicious tunnel identified on the network

There were several other malware, crytomining, tunnels, phishing, and port scanning instances over three days of the conference, which we observed in different products such as Secure Networks Analytics.

SNA alert related to port scanning
Fig. 3: SNA alert related to port scanning

Our investigations typically began with this type of correlation (e.g., Figure 4) before diving into more detailed data in Splunk or Endace.

Malicious connections observed in Cisco XDR
Fig. 4: Malicious connections observed in Cisco XDR

About 20% of conference traffic was unencrypted. Surprisingly, this is quite a small number compared to threat hunting in the SOC during other conferences like Black Hat or during major events.

Overall, I evaluated that the conference was relatively clean in terms of clear text traffic. One of the most interesting findings in this area was the presence of clear text credentials used to authenticated against email systems. Talos IR detected instances of POP3, IMAP, and HTTP usage during several conference days. There were repeated attempts to log in to email servers using insecure protocols, transfer files via email, and interact with services that should clearly be transitioning to encrypted communication.

POP3 passwords present in the network
Fig. 5: POP3 passwords present in the network

Likewise, some systems were authenticating using basic authentication over HTTP, switching to clear text websocket protocols for interaction.

Basic authentication over HTTP
Fig. 6: Basic authentication over HTTP

Thanks to our friends at Endace for enabling us to hunt across their system to identify these issues. In each case the SOC observed, it is our job to notify impacted individuals to let them know that they are transferring credentials in clear. The GovWare team was also present to assist and facilitate this communication, helping keep the internet and its users safe.

Every conference SOC or NOC functions like a hackathon. We always find something that needs to be coded, fixed, addressed, or correlated to get the data ready for analysis. That is mainly because there are so many different systems at play to help monitoring environment. Every conference has its own technology providers, and various vendors bring their equipment and systems. Having platforms like XDR help to provide visibility, but sometimes the SOC needs to access the API to control execution or analysis.

This year, I got familiar with the latest Cisco platform for sandboxing, Splunk Attack Analyzer, so I had the opportunity to code some edge-case tools:

  1. One tool submitted multiple URLs extracted from cleartext protocols and full packet capture from Endace. This allowed me to analyze new and previously unseen URLs extracted from network traffic at GovWare. We don’t break encryption at conferences and we want to ensure network security, so we do not actively proxy connections; the use case here was limited, but it was still helpful to use full packet capture to extract URLs and perform searches across them. The source code for this tool can be found on Github.
  2. Another tool submitted multiple files extracted from the wire. As we were able to acquire packets in cleartext at the conference (about 20%) and extract files, we could use that data to see if any malware was being transmitted over the network. We actually found a couple of interesting file instances on the wire that we could submit to the sandbox and automate submission via API. The source code for this tool can be found on Github.

With Cisco Splunk aggregating telemetry from all monitoring systems in the environment, we could efficiently explore and correlate network activity. The following examples highlight several Splunk queries we used to surface noteworthy insights, such as cleartext credential exchanges that required further analysis.

Clear text passwords search query for Endace index
Fig. 7: Clear text passwords search query for Endace index

When investigating file activity across the network, we used syslog data from Cisco Firewall to identify which files have traversed the network using the following query:

File search query for Cisco Firewall
Fig. 8: File search query for Cisco Firewall

Finally, to identify systems that were under remote control within the environment, we used the following query:

Remote control search query for Cisco Umbrella
Fig. 9: Remote control search query for Cisco Umbrella

During GovWare, Cisco’s Splunk platform proved to be an excellent tool for rapidly navigating multiple indexes; identifying systems of interest; correlating their activities across DNS, network, and tunneling data; and validating observations coming from other products. Splunk was especially helpful for searching across diverse data sources to confirm findings and link them to broader investigations into phishing, malware, tunneling, and other threats seen on the wire.

Threat hunting is all about finding previously unnoticed compromises — Talos IR never waits for alerts to pop up. During the GovWare event, our team built a collaborative environment powered by the Cisco technology stack together with our partners. Multiple solutions were unified through two main platforms: Cisco XDR and Splunk. Endace provided complete packet visibility and delivered metadata into Splunk, while Cisco XDR correlated alerts from tools such as Secure Network Analytics, Cisco Firewall, and others, contributing this intelligence to a unified platform.

Threat hunting technology stack at GovWare
Table 6: Threat hunting technology stack at GovWear

This level of integration gave the SOC a single, consolidated view of network activity. In a conference setting like this, successful threat hunting depends on having a clear, repeatable process and complete visibility into the internal environment to detect any signs of malicious activity. This process defines how to respond when XDR raises an alert and how to proactively search across systems that do not generate alerts. For Talos IR, that methodology is based on the PEAK framework (Prepare, Execute, and Act with Knowledge) which guides our threat hunting operations. This approach is consistent across Cisco Talos IR engagements and enables us to scale our work regardless of the size or complexity of the environment. Forming a hypothesis, testing it, and confirming with evidence are the core steps of threat hunting.

After four days of live operations at GovWare, including a day of setup and three days of active monitoring, we evaluated our initial hypotheses and confirmed the following outcomes:

Are attendees attempting to breach each other’s systems?

A few participants conducted basic port scans against other systems on the network, but no major intrusions were observed between attendees.

Are attendees trying to subvert SOC systems?

Some scanning and reconnaissance activity was directed toward the SOC environment itself. Cisco’s architecture prevented any successful compromise, though a few individuals attempted to probe or bypass security controls. It was difficult to determine whether these actions were deliberate adversarial behavior or simple curiosity, as endpoint-level visibility is not available during live conference events.

Are any attendees compromised, and could we warn them?

A small number of attendees were notified after signs of compromise were detected, including beaconing activity, malware infections, and exposure of cleartext credentials.

Check out the other blogs by my colleagues in the GovWare SOC.

About GovWare

GovWare Conference and Exhibition is the region’s premier cyber information and connectivity platform, offering multi-channel touchpoints to drive community intel sharing, training, and strategic collaborations.

A trusted nexus for over three decades, GovWare unites policymakers, tech innovators, and end-users across Asia and beyond, driving pertinent dialogues on the latest trends and critical information flow. It empowers growth and innovation through collective insights and partnerships.

Its success lies in the trust and support from the cybersecurity and broader cyber community that it has had the privilege to serve over the years, as well as organisational partners who share the same values and mission to enrich the cyber ecosystem.


We’d love to hear what you think! Ask a question and stay connected with Cisco Security on social media.

Cisco Security Social Media

LinkedIn
Facebook
Instagram
X

Authors

Yuri Kramarz

Principal Engineer

Cisco Talos