It was a Sunday afternoon when my manager called me. Two days prior, a large US city fell victim to a ransomware attack that disabled a sizable portion of the municipal network. I found myself on an airplane a few hours later.
Our first order of business was to quarantine potentially infected systems away from known-clean systems. In the interest of time, we installed a large Cisco ASA firewall into the datacenter distribution layer as a crude segmentation barrier. Once management connectivity was established, I was tasked with firewall administration, a job that is generally monotonous, thankless, and easy to scapegoat when things go wrong.
Long story short, the network was changing constantly. Applications were being moved between subnets, systems were being torn down and rebuilt, and I was regularly tasked with updating the firewall rules to permit or deny specific flows. After about 30 minutes, I decided to apply some basic automation. Managing firewall configurations using Python scripts is powerful but not particularly new or interesting, so I won’t focus on that aspect today.
Instead, consider the more difficult question. How do you know that your firewall is permitting and denying the correct flows? Often times, we measure the effectiveness of our firewall by whether the applications are working. This is a good measure of whether the firewall is permitting the desirable flows, but NOT a good measure of whether the firewall is denying the undesirable flows. Answering this question was critical to our incident response efforts at the customer site.
How would you solve this problem?
I decided to use the packet-tracer command built into the ASA command line. The command supports native xml formatting, which makes it easy to parse into Python data structures. Using Python parlance, the output contains a list of dictionaries, each one describing a phase in the ASA processing pipeline. Stages might include an ACL check, route lookup, NAT, QoS, and more. After the phase list, a summary dictionary indicates the final result.
ASA1#packet-tracer input inside tcp 10.0.0.1 50000 192.168.0.1 80 xml <Phase> <id>1</id> <type>ROUTE-LOOKUP</type> <subtype>Resolve Egress Interface</subtype> <result>ALLOW</result> <config/> <extra>found next-hop 192.0.2.1 using egress ifc management</extra> </Phase> <Phase> <id>2</id> <type>ACCESS-LIST</type> <subtype/> <result>DROP</result> <config>Implicit Rule</config> <extra>deny all</extra> </Phase> <result> <input-interface>UNKNOWN</input-interface> <input-status>up</input-status> <input-line-status>up</input-line-status> <output-interface>UNKNOWN</output-interface> <output-status>up</output-status> <output-line-status>up</output-line-status> <action>DROP</action> <drop-reason>(acl-drop) Flow is denied by configured rule</drop-reason> </result>
Now, how to automate this? I decided to use a combination of Python packages:
- Nornir, a task execution framework with concurrency support
- Netmiko, a library for accessing network device command lines
The high-level logic is simple. On a per firewall basis, define a list of checks which contain all the inputs for a packet-tracer test. Each check also specifies a should key which helps answers our original business question; should this flow be allowed or dropped? This allows us to test both positive and negative cases explicitly. Checks can be TCP, UDP, ICMP, or any arbitrary IP protocol. The checks can also be defined in YAML or JSON format for each host. Here’s an example of a checks list for a specific firewall:
---
checks:
- id: "DNS OUTBOUND"
in_intf: "inside"
proto: "udp"
src_ip: "192.0.2.2"
src_port: 5000
dst_ip: "8.8.8.8"
dst_port: 53
should: "allow"
- id: "HTTPS OUTBOUND"
in_intf: "inside"
proto: "tcp"
src_ip: "192.0.2.2"
src_port: 5000
dst_ip: "20.0.0.1"
dst_port: 443
should: "allow"
- id: "SSH INBOUND"
in_intf: "management"
proto: "tcp"
src_ip: "fc00:192:0:2::2"
src_port: 5000
dst_ip: "fc00:8:8:8::8"
dst_port: 22
should: "drop"
- id: "PING OUTBOUND"
in_intf: "inside"
proto: "icmp"
src_ip: "192.0.2.2"
icmp_type: 8
icmp_code: 0
dst_ip: "8.8.8.8"
should: "allow"
- id: "L2TP OUTBOUND"
in_intf: "inside"
proto: 115
src_ip: "192.0.2.1"
dst_ip: "20.0.0.1"
should: "drop"
...
Inside of a Nornir task, the code iterates over the list of checks, assembling the proper packet-tracer command from the check data, and issuing the command to the device using Netmiko. Note that this task runs concurrently on all firewalls in the Nornir inventory, making it a good fit for networks with distributed firewalls. Below is a simplified version of the Python code to illustrate the high-level logic.
def run_checks(task):
# Iterate over all supplied checks
for chk in checks:
# Build the string command from check details (not shown)
cmd = get_cmd(chk)
# Use netmiko to send the command and collect output
task.run(
task=netmiko_send_command,
command_string=cmd
)
Behind the scenes, the code transforms this XML data returned by the ASA into Python objects. Here’s what that dictionary might look like. It contains two keys: Phase is a list of dictionaries representing each processing phase, and result is the final summarized result.
{
"Phase": [
{
"id": 1,
"type": "ROUTE-LOOKUP",
"subtype": "Resolve Egress Interface",
"result": "ALLOW",
"config": None,
"extra": "found next-hop 192.0.2.1 using egress ifc management"
},
{
"id": 2,
"type": "ACCESS-LIST",
"subtype": None,
"result": "DROP",
"config": "Implicit Rule",
"extra": "deny all"
}
],
"result": {
"input-interface": "UNKNOWN",
"input-status": "up",
"input-line-status": "up",
"output-interface": "UNKNOWN",
"output-status": "up",
"output-line-status": "up",
"action": "DROP",
"drop-reason": "(acl-drop) Flow is denied by configured rule"
}
}
In the interest of brevity, I won’t cover the extensive unit/system tests, minor CLI arguments, and dry run process in this blog. Just know that the script will automatically output three different files. I used the new “processor” feature of Nornir to build this output. Rather than traversing the Nornir result structure after the tasks have completed, processors are event-based and will run at user-specified points in time, such as when a task starts, a task ends, a subtask starts, a subtask ends, etc.
One of the output formats is terse, human-readable text which contains the name of the check and the result. If a check should be allowed and it was allowed, or if a check should be dropped and it was dropped, is it considered successful. Any other combination of expected versus actual results indicates failure. Other formats include comma-separated value (CSV) files and JSON dumps that provide even more information from the packet-tracer result. Here’s the terse format when executed on two ASAs with hostnames ASAV1 and ASAV2:
ASAV1 DNS OUTBOUND -> FAIL ASAV1 HTTPS OUTBOUND -> PASS ASAV1 SSH INBOUND -> PASS ASAV1 PING OUTBOUND -> PASS ASAV1 L2TP OUTBOUND -> PASS ASAV2 DNS OUTBOUND -> PASS ASAV2 HTTPS OUTBOUND -> FAIL ASAV2 SSH INBOUND -> PASS ASAV2 PING OUTBOUND -> PASS ASAV2 L2TP OUTBOUND -> PASS
For the visual learners, here’s a high-level diagram that summarizes the project’s architecture. After Nornir is initialized with the standard hosts.yaml and groups.yaml inventory files, the host-specific checks are loaded for each device. Then, Nornir uses Netmiko to iteratively issue each packet-tracer command to each device. The results are recorded in three different output files which aggregate the results for easy viewing and archival.
Join me for a live webinar on April 2 “Is Your ASA Actually Doing What You Expect? — I’ve written a Python-based tool using Nornir and Netmiko that can rapidly test distributed Cisco ASA deployments for policy compliance. This session explores how the tools work in-depth.
If you’d like to learn more about the project or deploy it in your environment, check it out here on the Cisco DevNet Code Exchange. As a final point, I’ve built Cisco FTD support into this tool as well, but it is experimental and needs more in-depth testing. Happy coding!
Related resources:
- Visit the DevNet Security Dev Center to learn more about Cisco Security support for third-party integration across its portfolio with 18+ open APIs and integration points.
- If you are new to network programmability in general:
- To explore Python-based network automation:
- If you want to test FTD:
- To learn basics of configuration management:
- To learn about Nornir (Cisco Live session):
- To learn about Netmiko and other Python tools (Cisco Live session):
- To learn more about DevNet professional certifications
We’d love to hear what you think. Ask a question or leave a comment below.
And stay connected with Cisco DevNet on social!
Twitter @CiscoDevNet | Facebook | LinkedIn
Visit the new Developer Video Channel
Nick,
Awesome job- I think this could really be useful for a lot of customers!
Nick, A great idea to verify if some traffic is blocked against a deny rule! 🙂
This could be extended by adding a manifest which would contain the approved or expected rules/flows to pass by. Then the manifest would be compared with the actual data, to automatically change FW rules on-demand fulfilling the least privilege principle. This tool can help to build the baseline manifest based on Netflow/Syslog/ASA Cfg. https://netai.co.
Thanks Piotr, maybe something to explore in the future!