How to Automate Testing Using Genie and pyATS

Imagine a sudden network outage after upgrading one of your core network devices. This is a scenario that we would like to avoid at all costs. Yet, we still don’t always utilize the right tools in order to avoid these scenarios and we still rely on manual testing after upgrading our devices. In this blog, we are going to show how we can use Genie and pyATS to do automated testing on your networks.

What is pyATS?

You must know it by now, but the first rule of pyATS is that it pronounced as py-A-T-S. It stands for Python Automated Test Systems and was originally developed for internal Cisco engineering use, but it was later made available for the public. Now, thousands of network engineers are using it worldwide and we have a large community contributing to it. The pyATS solution is modular like Lego blocks and it has multiple layers:

• The original pyATS framework, which serves as the foundation
• The Genie Library framework, which is a standard open-source, platform/vendor agnostic library system
• XPRESSO, a web dashboard to manage your test suites, testbeds, test results and their insights
• Multiple bindings to other frameworks in the form of libraries, extensions,
  plugins and APIs.

What is Genie?

Every network engineer must be familiar with the CLI. While it is easy to read for humans, it is unfortunately not so easy to read for computers. Instead, computers prefer formats such as JSON and Genie helps us accomplish that: it parses CLI output into JSON output, which allows us to easily compare one output with the other. In automated testing, it is essential that we can easily compare the output before and after a change.

Workflow of the use case

For this use case, we have three steps. In the first step, we take a snapshot of the topology, which will serve as our baseline. We take a snapshot by parsing a set of commands and storing the output in JSON format. In the environment variables, we can specify the commands that we would like to test, for example:

core_commands = [
    "show ip arp",
    "show ip interface brief",
    "show mac address-table",
    "show version",
    "show inventory",
    "show running-config",
    "show boot",
    "show ip route",
    "show ip ospf neighbor",
    "show ip bgp summary",
    "show ip bgp neighbors",
    "show ip ospf database"
]

In the second step, we perform the changes on our devices, e.g., upgrading the firmware or making a simple configuration change.

In the third and final step, we take another snapshot of our topology. Again, we will parse the commands and the output will be available in a JSON format, which allows us to easily compare the output with the baseline output. Using pyATS, we can define our own criteria for passing or failing the tests. In this use case, we have defined two testcases:

• In the first testcase, we wanted to make sure that the configuration before and after are identical. In case there are any differences, then the test fails and the user will be notified
• In the second testcase, we actually perform a change and we want to make sure afterwards that the change was successful. We can parse the output and validate that the desired state has been reached. In case it has not, then the test fails and the user will be notified

After each test, a report will be generated and this report can be viewed in the UI of pyATS. Moreover, the UI will have an archive of past tests, which allows you to easily compare different tests with each other.

What can we achieve with pyATS?

With this use case we have shown how to leverage pyATS for automated testing, which will reduce errors and bugs in production environments. Moreover, it will give us the following benefits:

• Consistent test coverage and insightful reports
• Give time back to the engineers: they can focus on the failed test cases, instead of performing manual testing
• Reduction in human errors: manual testing is very error-prone

Want to learn more?

• Check out the Github repository on DevNet Code Exchange!
• Visit the pyATS and Genie DevNet Dev Center

This PoV was conducted together with Systems Architect Charles van Iersel
and Customer Delivery Architect Joep Remkes

---

We’d love to hear what you think. Ask a question or leave a comment below.
And stay connected with Cisco DevNet on social!

Twitter @CiscoDevNet | Facebook | LinkedIn

Visit the new Developer Video Channel