Cisco Blogs
Share

Security Multi-Tenancy Part 1: Defining the Problem


July 17, 2018 - 10 Comments

Pre-Virtual Virtual Firewalls

Nowadays, everyone likes to talk about network function virtualization. Most security vendors build their firewall products to run on a few popular hypervisors. However, the “virtual firewall” term predates this virtualization craze. Many firewall administrators use this nomenclature to describe an ability to create multiple virtual partitions or contexts within a single physical security appliance. Each of these virtual firewalls has its own configuration, stateful connection table, and management capabilities. However, they may not be as independent or isolated as one would assume – more on this later. Even though Cisco Adaptive Security Appliance (ASA) software supported virtual firewalls with multiple-context mode for quite some time, we deliberately delayed similar functionality in our threat-centric Firepower Threat Defense (FTD) product in order to get it right. As any decent engineer would tell you, getting the right solution starts with fully understanding the problem. Namely, why do our security customers deploy virtual firewalls?

Understanding Use Cases

As it turns out, not all customers deploy multiple security contexts specifically for multi-tenancy. Some look for routing table separation, where each virtual firewall represents a separate Virtual Routing and Forwarding (VRF) domain. This functionality comes in handy especially when trying to protect several internal organizations with overlapping IP spaces. Other firewall administrators leverage multiple-context mode to separate and simplify policy management across different domains. Instead of looking at a single flat policy, they break it up into smaller chunks based on individual network segments. This may also involve management separation, where administering individual security contexts is delegated to other organizations. A common example here is a big college where several departments manage their own networks and configure individual virtual firewalls on a shared physical appliance at the extranet edge. Other customers go even deeper and require complete traffic processing separation between different tenants or network segments. For instance, one typically does not want their production applications to be affected by some traffic from a lab environment. As these requirements add up, it becomes clear how most existing firewall multi-tenancy solutions come apart at the seams.

Reality Check

There are several operational considerations that need to be taken into account when deploying virtual firewalls.  All security contexts on a single appliance run the same software image, so you cannot test upgrades on a limited number of tenants. Similarly, they all live or die together – rebooting just one is not possible. When it comes to features, you need to keep track of which are not supported in the virtual firewall mode. Often enough, these subtle nuances come up when you are already so far down the implementation path that turning back is either expensive or completely impossible. But wait, there is more!

While virtual firewalls can certainly be used for routing or policy domain separation, it comes with a lot of unnecessary complexity. One needs to create firewall contexts, assign different interfaces, configure them all independently, and then keep switching back and forth in order to manage policies and other relevant configuration. If you need a single access policy across all of your contexts, it must be independently programmed into each virtual firewall. Luckily, features like VRF help in avoiding multiple-context mode by enabling routing domain separation only. When it comes to policy simplification, some of my customers found managing multiple virtual firewalls too cumbersome, converged back into a single security context, and leveraged Security Group Tags (SGTs) to significantly reduce the rule set. Unless you indeed require complete separation between tenants, it makes very little sense to deploy virtual firewalls.

When it comes to management separation, multiple-context mode seems like a perfect fit. After all, each tenant gets their own firewall to play with, all without impacting anyone else. Or is that really true? Even though each virtual firewall has its own independent configuration, they all run within a single security application on a shared physical device. In most implementations, it means that the management plane is shared across all of the virtual contexts. If one tenant is pushing a lot of policy changes or constantly polling for connection information, this will inevitably impact every other virtual firewall that runs on the same device. However, the real problem lies within the shared data plane.

Despite the perceived separation, all virtual firewalls ultimately run on shared CPU, memory, and internal backplane resources. Even when assigning different physical interfaces to different security contexts, all of the traffic typically converges at the ingress classification function in the CPU. While one sometimes can configure maximum routing or connection table sizes on per-context basis, it still does not limit the amount of network traffic or CPU resources that each particular tenant consumes. In order to classify packets to a particular virtual firewall, the system must spend CPU cycles on processing them first. If a particular tenant is getting a lot of traffic from the network, it can consume a disproportionally large amount of system CPU resources even if this traffic is later dropped by a rate-limiter. As such, there is never a guarantee that one virtual firewall does not grow too big and impact every other security context on the same box. I have seen many cases where firewall administrators were caught completely unaware by this simple caveat. Not being a problem with any specific vendor, this is just how most virtual firewalls are implemented today.

Thinking Outside the Contexts

After looking at the use cases and analyzing challenges with existing virtual firewall implementations, I knew that our approach to implementing multi-tenancy in FTD must fundamentally change. An ideal solution would provide complete management and traffic processing separation across all tenants, so one virtual firewall truly cannot impact anyone else on the same box. This separation should extend to independent software upgrades and reloads. At the same time, all of the available FTD features should always be supported when implementing virtual firewalls. Not only must it simplify the experience for an end user, but also significantly cut down on both development and testing times.

While these may have seemed like impossible requirements, I had a really cool idea on how we can get there for our customers. This novel approach builds on the multi-service capabilities of our Firepower platforms as well as such developing trends as application containerization. With the solution scheduled to arrive in just a few months, check back for an update in one of my next blog posts!



In an effort to keep conversations fresh, Cisco Blogs closes comments after 60 days. Please visit the Cisco Blogs hub page for the latest content.

10 Comments

  1. Well summarized blog. Whether it is route-domains or contexts all these came through over-time and those vendors had proprietary hardware with ASICs could not take virtualization to the extent of reserving hardware resources (CPU/Memory) per domain or context. Cisco today is in a unique position to support both route-domains (Single OS instance supporting multiple virtual management and data planes and give its customers the choice of hardware virtualization per tenant instance. In my view this would be the right thing to do, so as to have all customer use-cases covered be it commercial, enterprise or Service provider environments.

  2. Hi Andrew, Thanks for the update on this topic. The lack of the ability to have independent routing tables within FTD is a show stopper for many of our customers, aside from the other issues mentioned. Will this new architecture facilitate that? Or is it designed to solve the other issues you described only (eg, resource sharing, inability to reboot one instance, inability to test new firmware on an instance, etc). These all sound like a fantastic improvement on how virtualisation is currently performed, however the ability to have different default routes per context/instance/tenant/etc is often required.

    • Steve, This solution will certainly allow for routing domain separation, but we are independently looking at much simpler ways to solve it; e.g. VRF Lite. Andrew

  3. Hi, in the multi-instance or container based approach how the IPS functionality going to work ? will each Firewall container will have a IPS process or IPS will be a different Container ? will the multi-container approach provide flexibility to configure both Routed and Transparent mode Firewalls ? Do we need dedicated Appliance for FW and IPS even in Container mode or single Appliance can do both ? specially when this will be available in CCW ?

    • Each instance will support a full FTD feature set. More on the actual implementation in my next blog.

  4. Andrew, Thanks for defining the issues that FTD will be addressing. I look forward to your next post.

  5. Hi Andrew, One use case that is immediately useful for multiple instances on aa single chassis is the use case where we want to inspect VPN traffic and handle SSL decrypt off box. Instance_1 --> Load balancer to SSL decrypt --> Instance_2 By sandwiching the LB between 2 NGFW instances, we could terminate the VPN traffic, decrypt the SSL payloads, then L7 inspect the traffic in clear text, before ultimately passing the traffic off to the destination servers. I'm wondering if another use-case could be where the VPN and SSL decrypt are handled by instance 1 and layer 7 handled by instance 2 (No LB in this mix). Could a hardware resource limitation be set on each instance so that SSL in instance 1 would never starve CPU cycles from the inspect engines in instance 2 or vise-versa?

    • Hi Aaron, I'm going to talk more about the actual implementation in the next blog, but hard resource separation will definitely be a part of the feature. Decryption as a service is something that we can consider in the future as well. Andrew

  6. This is great, looking forward to the next one!

  7. That is great.Even i and too many customers i worked with them are satisfied with current Multi-Context features on ASA , it will be interesting to see what will replacing the old but stable Multi-Context features on FTD . I can't wait to read the part 2 . Thanks