CI/CD Pipelines for Infrastructure Configuration and Management, Part 3

Welcome back everyone to my third installment in this series of blogs on using CI/CD pipelines for infrastructure configuration and management.

Using Gitlab CE

In Part 1 of this series, we looked at how to use Gitlab CE as a foundational component for a CI/CD pipeline to manage and configure a simple CML simulated network. Then, at the end of the previous blog post about this topic, we left off at having a GitLab CE and runner installation running in Docker containers. And, we had created a new GitLab user called developer.

After creating the new user by following the “Register now” link, make sure you login in to your local installation of GitLab, with the root account, and approve the developer user under the Admin/Users/Pending approval tab. The new user would not be able to login without being approved into the platform by the root user.

Defining CI/CD pipeline components in GitLab

Let’s keep going in our journey and in this blog let’s go ahead and define the CI/CD pipeline components in GitLab. Make sure you are logged in as the developer user for the rest of this blog post. Create a new blank public project named cicd. In this new project let’s create a file named .gitlab-ci.yml. Make sure you name this YAML file exactly like that: .gitlab-ci.yml. This is important because this file will contain the CI/CD pipeline definition and GitLab will be looking for it in the root of the repository.

The main purpose of the CI/CD pipeline in our example is to configure OSPF on Cisco Open NX-OS. Ansible will be used to perform the configuration changes and our test networking environment will be a set of Cisco Nexus 9000v virtual switches running in CML. Both Ansible and CML will be covered in future blog entries as part of this series. You know by now that testing is a critically important stage in any CI/CD pipeline. In our case, we will use Cisco pyATS to take a snapshot of our environment before the OSPF configuration changes are applied as well as after. We will cover pyATS and how to build an OSPF specific snapshot also in a future blog entry. For now, the important thing to remember is that the testing stage of our CI/CD pipeline is done with pyATS.

In conclusion, our CI/CD pipeline for configuring OSPF on NX-OS will have three stages:

pyats-pre-snapshot
deploy-OSPF
pyats-post-snapshot

The pyATS pre and post snapshot stages will check for OSPF specific metrics and make sure that the changes performed in the deploy-OSPF stage have been applied successfully.

We will also need a test environment where we will have Ansible and pyATS installed and ready to run the OSPF configuration Ansible playbook and the pyATS snapshot scripts. I have already created a Docker container with all the pre-requisite tools installed and ready to go. The container can be found on Docker Hub at the following link: https://hub.docker.com/r/adrianii/cicd-en. Or if you so desire, you can build and run the container from scratch each time the pipeline is run. Just create a Dockerfile with the following definition:

FROM python:3.8
RUN apt-get update 
RUN apt-get install -y telnet
RUN pip install ansible==2.9.13
RUN pip install paramiko==2.7.2
RUN ansible-galaxy collection install cisco.nxos
RUN pip install 'pyats[full]'

Defining the stages of our CI/CD pipeline

We are now ready to define all the stages of our CI/CD pipeline in the .gitlab-ci.yml file. For a complete list of all the keywords, and what each of the ones below means, check https://docs.gitlab.com/ee/ci/yaml/.

default:
before_script:
- export ANSIBLE_HOST_KEY_CHECKING=False

image: 'adrianii/cicd-en:0.1.0'

stages:
- pyats-pre-snapshot
- deploy-OSPF
- pyats-post-snapshot

pyats-pre-snapshot:
stage: pyats-pre-snapshot
script: cd pyats && pyats run job job.py --testbed-file cml_test.yaml --trigger-datafile pre_trigger_datafile.yaml --html-logs pre_snapshots
artifacts:
paths:
- pyats/pre_snapshots/pre_ospf_dist-sw01.json
- pyats/pre_snapshots/pre_ospf_dist-sw02.json
when: always

deploy-OSPF:
stage: deploy-OSPF
script: ansible-playbook -i hosts actions/configure-ospf/configure-ospf.yml

pyats-post-snapshot:
stage: pyats-post-snapshot
script:
- cd pyats && pyats run job job.py --testbed-file cml_test.yaml --trigger-datafile post_trigger_datafile.yaml --html-logs post_snapshots
- pyats diff pre_snapshots/pre_ospf_dist-sw01.json post_snapshots/post_ospf_dist-sw01.json --output ospfdiff/
- pyats diff pre_snapshots/pre_ospf_dist-sw02.json post_snapshots/post_ospf_dist-sw02.json --output ospfdiff/
- cat ospfdiff/diff_pre_ospf_dist-sw01.json 
- cat ospfdiff/diff_pre_ospf_dist-sw02.json

artifacts:
paths:
- pyats/ospfdiff/diff_pre_ospf_dist-sw01.json
- pyats/ospfdiff/diff_pre_ospf_dist-sw02.json
when: always

The important thing to remember from this CI/CD pipeline definition for GitLab is that the Docker image adrianii/cicd-en:0.1.0 will be used to spin up a new container each time the pipeline is run. This container contains all the pre-required tools to perform the actions in each of the three stages defined. The pyats-pre-snapshot stage will run a script (cd pyats && pyats run job job.py –testbed-file cml_test.yaml –trigger-datafile pre_trigger_datafile.yaml –html-logs pre_snapshots) and save the artifacts from running that script for each of the Nexus distribution switches in the paths specified. The artifacts will also always be saved, whether the script ran successfully or failed. The deploy-OSPF stage of the pipeline will run an Ansible playbook that can be found at the location specified. And finally, the pyats-post-snapshot stage will run another pyATS script, save the artifacts and compare the outputs from the pre and post stages.

Up next: How YAML files are built for pyATS and Ansible

What each of these scripts contains and how the YAML files are built for both pyATS and Ansible will be discussed in the next blog posts as part of this series.