pyATS Demos: Test Cases – NetDevOps Series, Part 16

And we are back for the final post on how to work with test cases in pyATS and Genie. Please remember you must have a reserved sandbox with your VIRL simulations running and be connected to it via VPN. Please see my previous posts for further information on how to complete the environment setup.

Working with Test Cases

Now that you know how to run some basic tests with pyATS and Genie, it is time to explore how we could give it a proper structure to build more complex tests. That’s what Test Cases are all about: a framework that allows you to build repeatable and more sophisticated testing processes.

Let’s take a look at this example:

Task-1: basic_example_script
|-- commonSetup 
| |-- sample_subsection_1 
| `-- sample_subsection_2 
|-- tc_one
| |-- prepare_testcase 
| |-- simple_test_1 
| |-- simple_test_2 
| `-- clean_testcase 
`-- commonCleanup 
`-- clean_everything

The sections are easy to understand:

You can define a number of tasks to run in your test case (in the example above we have just 1 task)
Then you will have some common setup to do, structured in subsections
After that, you would go into the real Test Case (tc), with 3 phases: preparation, execution and cleaning
Finally, as a good citizen, you would need to clean after yourself, everything you set up during the common setup phase

Let’s see it working in your own setup. In this case we will use the -alpine image because it has vi already included in it, and you will need it to edit some files during this demo. We will ask our pyATS container to provide a shell (ash for –alpine image) so we can work with it interactively.

$ docker run -it --rm \
  -v $PWD:/pyats/demos/ \
  ciscotestautomation/pyats:latest-alpine ash

Once inside the container shell you have access to its directory structure and tools. Inside the pyats directory you will find multiple examples and templates to use with pyATS. To get started let’s focus on a basic one.

(pyats) /pyats # cd examples/basic

There you will find the basic_example_script.py python script file that defines a very simple Test Case. It includes quite some python code for all the sections mentioned before, but actually not doing much (in fact only logging), so it is a good starting point as a template to develop your own test cases.

(pyats) /pyats/examples/basic# cat basic_example_script.py

This script will be executed from a job, defined in this file:

(pyats) /pyats/examples/basic# cat job/basic_example_job.py

You would run the job with:

(pyats) /pyats/examples/basic# pyats run job job/basic_example_job.py

You can see in the report shown at the end of the execution process that all tests in our task PASSED.

Let’s insert a simple verification test in our test case. Please edit the python script with vi basic_example_script.py, scroll down to the TESTCASES SECTION and look for the First test section. There you need to insert the required code as per the following:

    # First test section
    @ aetest.test
    def simple_test_1(self):
        """ Sample test section. Only print """
        log.info("First test section ")
        self.a = 1
        self.b = 2
        if self.a != self.b:
            self.failed("{} is not {}".format(self.a, self.b))

As you can see we are defining 2 simple variables with fixed values of 1 and 2, and then inserting a conditional statement that fails if they are different. So, obviously the test will now fail because 1 and 2 are different.

Save the file and try it.

(pyats) /pyats/examples/basic# pyats run job job/basic_example_job.py

Check the execution logs and you will find how a failed test looks like when executing a test case:

...
2019-04-04T08:32:09: %AETEST-INFO: Starting section simple_test_1
2019-04-04T08:32:09: %SCRIPT-INFO: First test section
2019-04-04T08:32:09: %AETEST-ERROR: Failed reason: 1 is not 2
2019-04-04T08:32:09: %AETEST-INFO: The result of section simple_test_1 is => FAILED
...
2019-04-04T08:32:09: %EASYPY-INFO: +------------------------------------------------------------------------------+
2019-04-04T08:32:09: %EASYPY-INFO: | Task Result Summary |
2019-04-04T08:32:09: %EASYPY-INFO: +------------------------------------------------------------------------------+
2019-04-04T08:32:09: %EASYPY-INFO: Task-1: basic_example_script.commonSetup PASSED
2019-04-04T08:32:09: %EASYPY-INFO: Task-1: basic_example_script.tc_one FAILED
2019-04-04T08:32:09: %EASYPY-INFO: Task-1: basic_example_script.commonCleanup PASSED
2019-04-04T08:32:09: %EASYPY-INFO: +------------------------------------------------------------------------------+
2019-04-04T08:32:09: %EASYPY-INFO: | Task Result Details |
2019-04-04T08:32:09: %EASYPY-INFO: +------------------------------------------------------------------------------+
2019-04-04T08:32:09: %EASYPY-INFO: Task-1: basic_example_script
2019-04-04T08:32:09: %EASYPY-INFO: |-- commonSetup PASSED
2019-04-04T08:32:09: %EASYPY-INFO: | |-- sample_subsection_1 PASSED
2019-04-04T08:32:09: %EASYPY-INFO: | `-- sample_subsection_2 PASSED
2019-04-04T08:32:09: %EASYPY-INFO: |-- tc_one FAILED
2019-04-04T08:32:09: %EASYPY-INFO: | |-- prepare_testcase PASSED
2019-04-04T08:32:09: %EASYPY-INFO: | |-- simple_test_1 FAILED
2019-04-04T08:32:09: %EASYPY-INFO: | |-- simple_test_2 PASSED
2019-04-04T08:32:09: %EASYPY-INFO: | `-- clean_testcase PASSED
2019-04-04T08:32:09: %EASYPY-INFO: `-- commonCleanup PASSED
2019-04-04T08:32:09: %EASYPY-INFO: `-- clean_everything PASSED

As you can see you don’t need to be a Python expert to use the test cases framework. You have templates readily available for you, where you can insert the specific tests you would like to run and execute them straight away.

Check all BGP neighbors are established

We will end up by exploring another example test case to help you check all BGP neighbors in the network are in the desired established state.

The test case structure includes the following sections:

Common setup: connect to all devices included in your testbed.
Test cases: learn about all BGP sessions in each device, check their status and build a table to represent that info. If there are neighbors not in a established state the test will fail and signal this condition in an error message.

In order to run it first you will need to install git on your pyATS container, clone a repo with additional examples, install a tool to create nice text tables (tabulate), go into the directory and execute the job:

(pyats) /pyats/examples/basic# cd ../..
(pyats) /pyats # apk add --no-cache git
(pyats) /pyats # git clone https://github.com/kecorbin/pyats-network-checks.git
(pyats) /pyats # pip3 install tabulate
(pyats) /pyats # cd pyats-network-checks/bgp_adjacencies
(pyats) /pyats/pyats-network-checks/bgp_adjacencies # pyats run job BGP_check_job.py --testbed-file /pyats/demos/default_testbed.yaml

As a result, you will find the following table in your logs, displaying all BGP neighbors in all your devices, and their current status:

2019-04-05T18:10:41: %SCRIPT-INFO: | Device | Peer | State | Pass/Fail |
2019-04-05T18:10:41: %SCRIPT-INFO: |------------+----------+-------------+-------------|
2019-04-05T18:10:41: %SCRIPT-INFO: | csr1000v-1 | 10.2.2.2 | established | Passed |
2019-04-05T18:10:41: %SCRIPT-INFO: | nx-osv-1 | 10.1.1.1 | established | Passed |

It was never this easy to make sure BGP neighbors across your network are properly established!

See you soon for some more NetDevOps awesomeness, stay tuned!

Any questions or comments please let me know in the comments section below. Or reach me on Twitter or LinkedIn.