This blog is authored by Rodolphe Trujillo,
Cisco Customer Delivery Engineering Technical Leader

Matching current and expected operational states

One of the challenging topics of network automation is to test the routers to see if their current operational state match the expected operational state. Or, in the case you are changing configuration, compare pre-operational states with post-operational states.

The traditional way for network engineers to do this is to use the CLI. We explore the data returned on the screen… retrieve the relevant states, counters… and put them in an Excel document. As there are hundreds of those states and counters, verifications become difficult if the operation has lots of checks. Moreover if you have all those checks multiplied by a large number of devices, it is impossible to cover without automation.

You have multiple ways to automate checks on your network. You can use custom scripts, open source solutions like Netmiko or Ansible. Alternatively, you can use pyATS, which is a very good testing framework from Cisco.

Automate checks with Network Services Orchestrator (NSO)

The aim of this post is to show a way to automate checks with NSO and create an Excel document for audit.
This is not THE way to do it, just A way, but I have applied it on thousands of devices and run millions of checks with a similar method as the one presented here.

Along with NSO I will also use the pyATS parser. Historically, I was using TextFSM homemade templates for doing the parsing part of my workflow. However, I tend to progressively migrate to the very rich parsers list of pyATS.

Note: The code provided only works with CLI NEDs and Cisco devices.

Prerequisites

On the Python installation running alongside your NSO let’s install some libraries I’ve prepared for this demonstration :

pip install pyats-parser

This library, as described in the documentation, bring the power of the huge PyATS parsers collection to every piece of python you can write, the downside is, because of dependencies, you will have to install all the PyATS libraries and not only the parsing part.

pip install nso-live-status

In fact, you can directly install this library because it depends on pyats-parser to run. So, pyats-parser will be installed automatically. NSO-live-status is a wrapper to the live status of NSO, and I wanted to have a convenient function:

run_live_status(root_nso_object, device, str_cli_command)

Not only will this function execute the live status, but if the command is recognized by pyATS it will be parsed and the object returned will contain a structured output.

pip install nso-restconf

This will be useful for manipulating NSO through the Restconf API. Finally, the popular openpyxl for generating excel documents:

pip install openpyxl

An example of check

For this example we will simply check the version of IOS-XR to see if it match a target version.

I’ve prepared a NSO Package which can be used as a “checks starting pack” at: https://github.com/rtrjl/check_device, so all you have to do is:

git clone git@github.com:rtrjl/check_device.git

in your NSO packages directory, and

cd check_device
cd src
make clean
make

in the NSO CLI session:

# packages reload

Once the package reload is done, we can try the action in the NSO CLI session:

# check_device check_version device Paris-75 target_version 7.5.1
check_status NOK
check_message [check_version] Current version 7.3.2 doesn't match the 
target version 7.5.1 on the device Paris-75

Industrialization of checks

You now have an action to check the version and you can call it with the CLI which is nice, but NSO bring a very powerful built-in feature with an auto-generated RESTConf API on top of all your actions and services.
Using this RESTConf API we can launch the checks on a set of devices:

import json
from nso_restconf.restconf import RestConf

from concurrent.futures import ThreadPoolExecutor, as_completed

nso_api = RestConf(address="http://127.0.0.1", port=8080,
                   username="admin", password="admin")
# for the example credentials are in the code :-)
devices_list = ["Kiev-47",
                "Madrid-40",
                "Napoli-50",
                "Paris-66",
                "Riga-46",
                "Stuttgard-64"]
#the numbers have no significations they are just the last bytes of the IP of the routers on the lab I use.

def nso_check_version(device, target_version):
    data = json.dumps({'input': {'device': f"{device}", "target_version": f"{target_version}"}})
    result = nso_api.action(data, "check_device/check_version")
    return result.json()

The script should print something like this:

{'check_device:output': {'device': 'Kiev-47', 'current_version': '7.3.2', 'operating_system': 'IOSXR', 'check_status': 'NOK', 'check_message': "[check_version] Current version 7.3.2 doesn't match the target version 7.5.1 on the device Kiev-47"}}
{'check_device:output': {'device': 'Madrid-40', 'current_version': '7.3.2', 'operating_system': 'IOSXR', 'check_status': 'NOK', 'check_message': "[check_version] Current version 7.3.2 doesn't match the target version 7.5.1 on the device Madrid-40"}}
{'check_device:output': {'device': 'Napoli-50', 'current_version': '7.3.2', 'operating_system': 'IOSXR', 'check_status': 'NOK', 'check_message': "[check_version] Current version 7.3.2 doesn't match the target version 7.5.1 on the device Napoli-50"}}
{'check_device:output': {'device': 'Paris-66', 'current_version': '7.3.2', 'operating_system': 'IOSXR', 'check_status': 'NOK', 'check_message': "[check_version] Current version 7.3.2 doesn't match the target version 7.5.1 on the device Paris-66"}}
{'check_device:output': {'device': 'Riga-46', 'current_version': '7.3.2', 'operating_system': 'IOSXR', 'check_status': 'NOK', 'check_message': "[check_version] Current version 7.3.2 doesn't match the target version 7.5.1 on the device Riga-46"}}
{'check_device:output': {'device': 'Stuttgard-64', 'current_version': '7.3.2', 'operating_system': 'IOSXR', 'check_status': 'NOK', 'check_message': "[check_version] Current version 7.3.2 doesn't match the target version 7.5.1 on the device Stuttgard-64"}}

In parallel

The script is running well. However, with this approach it can take a lot of time because it will connect to devices one after the other. It will be fine on a small network but on a bigger one…?

Let’s use threads to accelerate the process:

from concurrent.futures import ThreadPoolExecutor, as_completed

processes = []
with ThreadPoolExecutor(max_workers=10) as executor:
    for device in devices_list:
    processes.append(executor.submit(nso_check_version, device, "7.5.1"))

for task in as_completed(processes):
    print(f"{task.result()}")

The same output as the sequential one will be printed but in this version we leverage another interesting feature of NSO:
devices can be reached in parallel to run show commands, with limits , it will depend on the amount of RAM on the server on which NSO is installed, but we can run 10 workers without problems.

Now we have retrieved the checks we can put the results in a good old excel document:

checks_list = []
for task in as_completed(processes):
    result = task.result()["check_device:output"]
    checks_list.append([result["device"], result["operating_system"], result["current_version"], result["check_status"], result["check_message"]])

from openpyxl import Workbook

wb = Workbook()
ws = wb.active
headers = ["Device", "OS", "OS version", "Check status", "Check message"]
ws.append(headers)
for data in checks_list:
    ws.append(data)
wb.save("devices-version-audit.xlsx")

In this demo we have looked at different topics such as:

using pyATS parsers with NSO
doing NSO live status (show commands) in parallel leveraging the auto-generated RESTCONF API
writing a check using actions in NSO