Read Me First
As a network and workload security strategy leader, I spend a lot of time thinking about the future of the good old network firewall. Everyone has been using and abusing the “next-generation” qualifier to describe any modern firewall product for far too long, so it is appropriate to drop this extraneous prefix and talk about what truly comes next for this technology. There will be no sales pitches or product announcements here today, but rather my vision for where this industry is going – preferably followed by a healthy, yet passionate debate in the comments.
Spoiler alert: I’m not going to join the cool club of pronouncing the firewall dead. Whether you run cloud native applications, or host them in a public cloud, or go full-on software-as-a-service (SaaS), or even delegate your threat protection to a Secure Access Service Edge (SASE) solution, they all rely on some form of a network for connectivity. A future firewall may look and feel very different, but it will be there – even if hidden behind some policy abstraction layers or managed entirely by someone else. After all, the cloud is just your stuff running in someone’s else data center. The two main problems for the firewall to overcome in all those new deployment scenarios are insertion and visibility.
Insert Your Firewall Here
For starters, the network firewall (or network security in general) term is somewhat misleading. Very few of us deploy a firewall to protect the network infrastructure itself. It is all about securing our applications and data, whether on the client or service side. The “network” qualifier refers to how we insert those controls into the traffic path and what kind of messages they inspect. For better or worse, IP networks continue to be the most universally supported interconnect method for devices and applications alike, keeping network security products very relevant. How this insertion happens is changing quite rapidly, though. Back in the day, inserting a firewall in front of or between applications was as easy as plugging in a few cables and maybe configuring some VLANs on a switch. As applications moved into the virtual machine form factors, lateral threat protection with a firewall became cumbersome. With the proliferation of cloud native microservices and public cloud deployments, effective firewall insertion became next to impossible. At the same time, application security vendor pitches went from micro- to nano- to femto-segmentation as fast as the respective marketing powerhouses could look those up in a dictionary. It’s obvious that the firewall needs to sit in the network stack as close to each application as possible, but we need new tools to not only assist this insertion but also sharpen the protection profile and preserve precious processing resources.
Battling Foggy Lenses
It’s no secret that firewalls see all transit communication as network flows with IP addresses and TCP/UDP ports. Most additional context around these flows is discovered through deep packet inspection (DPI) which goes all the way up to the application layer when necessary. This approach has been used for years in application-level gateways (ALG) which were primarily implemented to modify embedded IP and transport port information in packet payloads for network address translation (NAT) purposes. The big breakthrough of next-generation firewalls (NGFW) was in their ability to use that upper protocol layer information to identify specific applications and use those names to abstract security policies into something more relevant to humans and business processes. Some of those early NGFW implementations took shortcuts to improve performance by simply treating all UDP/53 traffic as DNS and TCP/23 as Telnet, but eventually everyone invested in good enough DPI to get beyond those obvious pitfalls. Over time, NGFW added more and more DPI functions, such as intrusion prevention (IPS), malware detection, and data loss prevention (DLP). Finally, all NGFW products enforced security policies and block threats by looking at each packet of each network flow through a fine lens of application message parsers and known attack patterns. Then entered pervasive flow encryption with Transport Layer Security (TLS) to wreak havoc on this well-oiled operation.
Since firewalls rely on DPI to associate user and application contextual data with TCP/IP network flows, TLS makes the whole process a lot more difficult. Some higher-level flow attributes, such as URL categories, can still be extracted from the cleartext protocol headers without applying decryption. However, features like file blocking and IPS require TLS to be stripped off before inspection and then applied back after. Depending on whether the firewall protects a client or a server, full TLS decryption may or may not be possible. In those cases when decryption happens, the firewall performance drops significantly even when state-of-the-art TLS hardware acceleration is used. It soon becomes clear that relying on DPI alone for threat protection does not scale, and a firewall needs to learn new tricks to enrich flow context and win back some threat visibility.
Keeping Employees Out of Trouble
Edge firewalls commonly inspect outbound traffic to prevent company assets from being used for naughty stuff (formally called an acceptable use policy) and to stop confidential data from leaking out (DLP). Since TLS decryption typically requires a private Certificate Authority (CA) certificate to be installed on each client, it only works on managed endpoints without significantly impairing the end user experience. To make matters worse, most SaaS offerings with thick software clients or mobile device apps deploy techniques like mutual certificate authentication (mTLS), which make transit decryption completely impossible. Considering the sheer volume of Internet bound traffic and the performance impact from decryption, performing DPI on outbound user traffic is largely impractical. While the insertion of a network firewall at the edge of an enterprise network or a branch is straight forward – even if one consumes it as a cloud-delivered SASE service, the flow visibility is severely degraded.
One approach to identifying popular SaaS application flows is by keeping a database of well-known destination IP addresses. However, some cloud software providers offer dozens of individual applications in a single suite with a single set of IP addresses used to host all of them. How does one permit employees to use a business-critical chat application while preventing them from uploading files onto a cloud drive? Cloud Access Security Broker (CASB) solutions address a similar problem by integrating with the SaaS via an API and tracking user activity within an enterprise account. This approach can be extended to an edge firewall which can integrate with a CASB or directly with a particular SaaS API to associate network flows with specific user activities. This exact functionality has been employed for some time by Cisco Umbrella. It associates specific microapplications with outbound network connections via an integration with Cisco Cloudlock which in turn tracks user OAuth sessions into popular SaaS productivity suites as a CASB. And while it borders on violating my promise of not pitching a product, this is an example of how a network firewall – even when delivered as a cloud service – gets application visibility without DPI.
What about those applications which don’t have predefined destination IP addresses, and especially those which do not want to be detected? In a mix of undecryptable HTTP-over-TLS flows, the firewall has a very hard time distinguishing a legitimate web browser from sneaky anonymizer software. However, there is enough variation in the outer header fields and specific behavioral patterns, which could allow the firewall to identify a specific application with a high confidence level and no need for DPI. This sounds like magic, but it is really the power of machine learning (ML). One of those clever solutions is Cisco Mercury which is an open-source package for application fingerprinting across a variety of network protocols, such as TLS or HTTP. It is extremely accurate, and it could be used by a network firewall for far more than mere application detection – like identifying malware communication or data exfiltration. It could also allow the firewall to apply stricter DPI policies to those anomalous connections. I call this selective firewalling which directs precious processing resources where they are needed the most rather than inspecting all flows uniformly. You are free to call foul on this, but I’m absolutely not counting this as a product pitch, just a very cool technology which I get easily excited about.
Inference based flow context enrichment is great, but what about asking a managed client endpoint directly? After all, it knows everything about each outgoing network flow. This is indeed what Cisco AnyConnect with Network Visibility Module (NVM) can export toward an external collector for each TCP or UDP connection, even when the user is not connected to VPN. If that external collector is an edge firewall, NVM can provide it with all kinds of contextual flow data in near-real time. This includes a globally unique endpoint identifier across all Cisco cloud services, a logged-in user identity, a true geolocation, an operating system version, even names and unique hashes for both child and parent processes which created the connection. The edge firewall can immediately consume this information to make a policy decision without spending any cycles on TLS decryption or DPI. What would otherwise seem like a benign web browser session to any regular DPI device becomes a red flag when the web browser is forked by a known malware process running as the administrator. Armed with this evidence, the firewall can further interrogate or even quarantine the endpoint by tapping a cloud security service and referencing the global device identifier from the flow telemetry. However exciting this is, I’m definitively calling a second strike on the product pitch here.
Guarding Crown Jewels
The good news for a data center edge firewall which protects applications from inbound threats is that at least the TLS decryption is very possible. Since both the protected applications and firewall are typically under the same administrative domain, the applications’ private keys which TLS uses for encryption and decryption can be made available to the firewall as well. This creates a completely painless experience for the incoming users who can no longer differentiate between a TLS session terminating on the firewall and one going straight to the application. This allows the firewall to go all-in on DPI from user identify and posture validation (pro tip: never write a security blog without a relevant Zero Trust reference) to IPS to web application security. Even if your modern applications talk API over RPC, HTTP/2, and TLS, those languages are either already understood by a network firewall or easily implemented as new inspection engines. The performance impact is still there, but it is certainly not the biggest problem. You can’t have it all, and just when you got some flow visibility back, you’ve suddenly lost the point of insertion.
As we had established earlier, inserting network controls in between virtual and especially container-based workloads is a major pain. When your application stack is fully distributed into tens of thousands of microservices which constantly shift across private and multiple public clouds, hairpinning those inter-workload connections through an external network firewall is the very definition of a bad idea. The best outcome is a few nervous laughs from the DevOps team in appreciation of an assumed joke, but further persistence in executing upon that plan could easily result in relatively major body injuries. If these application connections would not come close to the firewall, maybe the firewall can bring itself closer to those connections.
One approach is to instantiate a firewall right next to the workload, but it’s difficult to do without some friendly local service which can help with landing and expanding. It also would not hurt to get some visibility into the application environment and its communication patterns for better policy abstraction and more focused protection. This is where a software agent on the host operating system can provide a lot of benefit. It tracks microservice instantiation and decommission events, package names and versions, machine- and user-defined attributes, Common Vulnerability and Exposures (CVE) manifests, and many other application properties which are otherwise foreign to a network firewall. The firewall integrates with this agent to enrich network flow information with this newly available data and incorporate it into security policy constructs. Furthermore, knowing specific application vulnerabilities helps the firewall to tune its IPS and web application security functions toward specific attacks and thus preserve valuable processing resources. But the real benefit of the agent comes with a more intelligent firewall insertion.
Some cloud native network security solutions are installed onto all application nodes to indiscriminately inspect all incoming and outbound traffic. This creates a certain level of initial deployment complexity, but also impacts the overall application performance by wasting precious CPU cycles on full DPI. An application host agent can assist in spinning up a local network firewall service and then direct only certain flows through it, leveraging the knowledge of specific workload communication patterns and exposure telemetry. It can also insert that firewall service above the encryption layer within the application network stack in service meshes which deliver TLS processing as another underlying function. This eliminates the extra performance impact from performing TLS decryption and provides DPI visibility into inter-application communication even in mTLS-enabled microservice deployments. Now you can start layering in API gateways, Zero Trust reverse proxies, and other fun security toys which all the cool kids have. Better yet, pair it up with an application fingerprinting function and direct inspection resources to those flows which look the most suspicious for some selective firewalling.
Phoenix of Security World
A network firewall has seen many recent challenges with both insertion and visibility, but its true next generation is about to rise again. It must evolve from examining every network connection under a DPI microscope toward ML-driven flow inference and tighter cooperation with its security siblings: CASB, endpoint clients, and application agents. Expect to hear a lot more product specific updates as this vision becomes a reality with Cisco Secure Firewall and Workload technologies paving way to a comprehensive new-normal firewall solution.
That is three strikes, and I’m out! Do share your thoughts in the comments, and happy firewalling!
We’d love to hear what you think. Ask a Question, Comment Below, and Stay Connected with Cisco Secure on social!
Cisco Secure Social Channels
Instagram
Facebook
Twitter
LinkedIn
Andrew,
Great write-up on the evolution of firewalls. I still reminisce at times about how super basic features such as ‘stateful firewalling’ were still becoming ubiquitous but you’ve done a fantastic job at summarizing the challenges of network-based security functions and how they continue to morph over time. I chuckled at the DevOps nervous laughs and the potential of bodily injury. I’ve certainly been part of some of those uncomfortable conversations. All in all, I really enjoyed this article. Thanks!
I really enjoyed how you told the whole story around the Firewall in a quick and easy way to read and how it might be in the near future. I love it.
I think the idea of Agent-On-Host-OS may work to let the firewall become more closer to the workload (Instead on doing the same on Hypervisor) but as you know if a Host-OS compromised by a potential attacker the Agent can easily stop or Manipulated like any other Agents/App/Services running on the Host OS.Also any compromised OS can easily hide the process from the Agent (Rootkit level process never seen by any AV/EDR/Debug-Tools).So i think that the Agent could potentially create some false data or simply ignore what is actually happening at the OS level. I am a true believer that “The NETWORK doesn’t lie” but not the same for any OS/Containers/Hypervisor as they could easily compromised by a potential attackers. The other thing that need improvement is the performance problem in modern networks. Firewalls are still struggling to keep up with the high speed/low latency network traffic. A TOR switch in just 1U can support Multi-Tbps for servers with highspeed NICs , but the Firewalls can only deal with just a fraction of those traffic .Even for L3/L4 stateful inspection (No DPI) the numbers are to still too low and that is the reason i like the way ACI handling the policy enforcement for Inter-EPG/Intra-EPG with old school stateless ACL.I think the idea of the bypassing/Offloading some trusted flow on-the-fly from Firewall to the Switch/Network-Fabric is the way to go as the Firewall only inspect some early Flows/Packets and then using API to program the Switch to handle those trusted/harmless flow entirely in the Hardware. For visibility it think that the better way to collect meta-data from traffic is by using the Switch/Router to deal with that like the Cisco FNF+AVC with capability to extract some Context/Meta-Data at line-rate for every flow , even for traffic the passing within the VLAN.
Andrew, great writeup covering the journey of firewall-ing in to the near future. Its a fact that firewalls are still very relevant irrespective how the users, hosts and services have changed. I really liked the reference to network firewalls actually talk about the insertion point of firewall rather than an independent entity., future indeed is ML driven firewalls working in tandem with their security siblings. Awesome read., thank you.
For someone not technical (like me) it was delightful to read this blog. It provided me a clear understanding of were are we going with Firewall.
Thank you!!
Well Done Andrew ! I like this article !
Andrew,
As usual amazing post with full of information.