It was a Sunday afternoon when my manager called me. Two days prior, a large US city fell victim to a ransomware attack that disabled a sizable portion of the municipal network. I found myself on an airplane a few hours later.

Our first order of business was to quarantine potentially infected systems away from known-clean systems. In the interest of time, we installed a large Cisco ASA firewall into the datacenter distribution layer as a crude segmentation barrier. Once management connectivity was established, I was tasked with firewall administration, a job that is generally monotonous, thankless, and easy to scapegoat when things go wrong.

Long story short, the network was changing constantly. Applications were being moved between subnets, systems were being torn down and rebuilt, and I was regularly tasked with updating the firewall rules to permit or deny specific flows. After about 30 minutes, I decided to apply some basic automation. Managing firewall configurations using Python scripts is powerful but not particularly new or interesting, so I won’t focus on that aspect today.

Instead, consider the more difficult question. How do you know that your firewall is permitting and denying the correct flows? Often times, we measure the effectiveness of our firewall by whether the applications are working. This is a good measure of whether the firewall is permitting the desirable flows, but NOT a good measure of whether the firewall is denying the undesirable flows. Answering this question was critical to our incident response efforts at the customer site.

How would you solve this problem?

I decided to use the packet-tracer command built into the ASA command line. The command supports native xml formatting, which makes it easy to parse into Python data structures. Using Python parlance, the output contains a list of dictionaries, each one describing a phase in the ASA processing pipeline. Stages might include an ACL check, route lookup, NAT, QoS, and more. After the phase list, a summary dictionary indicates the final result.

ASA1#packet-tracer input inside tcp 50000 80 xml
  <subtype>Resolve Egress Interface</subtype>
  <extra>found next-hop using egress ifc management</extra>
  <config>Implicit Rule</config>
  <extra>deny all</extra>
  <drop-reason>(acl-drop) Flow is denied by configured rule</drop-reason>

Now, how to automate this? I decided to use a combination of Python packages:

  1. Nornir, a task execution framework with concurrency support
  2. Netmiko, a library for accessing network device command lines

The high-level logic is simple. On a per firewall basis, define a list of checks which contain all the inputs for a packet-tracer test. Each check also specifies a should key which helps answers our original business question; should this flow be allowed or dropped? This allows us to test both positive and negative cases explicitly. Checks can be TCP, UDP, ICMP, or any arbitrary IP protocol. The checks can also be defined in YAML or JSON format for each host. Here’s an example of a checks list for a specific firewall:

  - id: "DNS OUTBOUND"
    in_intf: "inside"
    proto: "udp"
    src_ip: ""
    src_port: 5000
    dst_ip: ""
    dst_port: 53
    should: "allow"
    in_intf: "inside"
    proto: "tcp"
    src_ip: ""
    src_port: 5000
    dst_ip: ""
    dst_port: 443
    should: "allow"
  - id: "SSH INBOUND"
    in_intf: "management"
    proto: "tcp"
    src_ip: "fc00:192:0:2::2"
    src_port: 5000
    dst_ip: "fc00:8:8:8::8"
    dst_port: 22
    should: "drop"
    in_intf: "inside"
    proto: "icmp"
    src_ip: ""
    icmp_type: 8
    icmp_code: 0
    dst_ip: ""
    should: "allow"
  - id: "L2TP OUTBOUND"
    in_intf: "inside"
    proto: 115
    src_ip: ""
    dst_ip: ""
    should: "drop"

Inside of a Nornir task, the code iterates over the list of checks, assembling the proper packet-tracer command from the check data, and issuing the command to the device using Netmiko. Note that this task runs concurrently on all firewalls in the Nornir inventory, making it a good fit for networks with distributed firewalls. Below is a simplified version of the Python code to illustrate the high-level logic.

def run_checks(task):
    # Iterate over all supplied checks 
    for chk in checks:
        # Build the string command from check details (not shown)
        cmd = get_cmd(chk)
        # Use netmiko to send the command and collect output

Behind the scenes, the code transforms this XML data returned by the ASA into Python objects. Here’s what that dictionary might look like. It contains two keys: Phase is a list of dictionaries representing each processing phase, and result is the final summarized result.

  "Phase": [
      "id": 1,
      "type": "ROUTE-LOOKUP",
      "subtype": "Resolve Egress Interface",
      "result": "ALLOW",
      "config": None,
      "extra": "found next-hop using egress ifc management"
      "id": 2,
      "type": "ACCESS-LIST",
      "subtype": None,
      "result": "DROP",
      "config": "Implicit Rule",
      "extra": "deny all"
  "result": {
    "input-interface": "UNKNOWN",
    "input-status": "up",
    "input-line-status": "up",
    "output-interface": "UNKNOWN",
    "output-status": "up",
    "output-line-status": "up",
    "action": "DROP",
    "drop-reason": "(acl-drop) Flow is denied by configured rule"

In the interest of brevity, I won’t cover the extensive unit/system tests, minor CLI arguments, and dry run process in this blog. Just know that the script will automatically output three different files. I used the new “processor” feature of Nornir to build this output. Rather than traversing the Nornir result structure after the tasks have completed, processors are event-based and will run at user-specified points in time, such as when a task starts, a task ends, a subtask starts, a subtask ends, etc.

One of the output formats is terse, human-readable text which contains the name of the check and the result. If a check should be allowed and it was allowed, or if a check should be dropped and it was dropped, is it considered successful. Any other combination of expected versus actual results indicates failure. Other formats include comma-separated value (CSV) files and JSON dumps that provide even more information from the packet-tracer result. Here’s the terse format when executed on two ASAs with hostnames ASAV1 and ASAV2:


For the visual learners, here’s a high-level diagram that summarizes the project’s architecture. After Nornir is initialized with the standard hosts.yaml and groups.yaml inventory files, the host-specific checks are loaded for each device. Then, Nornir uses Netmiko to iteratively issue each packet-tracer command to each device. The results are recorded in three different output files which aggregate the results for easy viewing and archival.

Firewall 2


Join me for a live webinar on April 2 “Is Your ASA Actually Doing What You Expect? —  I’ve written a Python-based tool using Nornir and Netmiko that can rapidly test distributed Cisco ASA deployments for policy compliance. This session explores how the tools work in-depth.

If you’d like to learn more about the project or deploy it in your environment, check it out here on the Cisco DevNet Code Exchange. As a final point, I’ve built Cisco FTD support into this tool as well, but it is experimental and needs more in-depth testing. Happy coding!

Related resources:


We’d love to hear what you think. Ask a question or leave a comment below.
And stay connected with Cisco DevNet on social!

Twitter @CiscoDevNet | Facebook | LinkedIn

Visit the new Developer Video Channel