Avatar

As developers, it’s easy to get bogged down in our day-to-day work. Rarely is there time to reflect and retrospect. Well, here and today , let me take the opportunity to reflect on our journey of launching pyATS to our customers, and share our experiences and our learnings.

Time Flies

It’s hard to believe that we are already closing onto the 4th year since we’ve launched pyATS to developers through Cisco DevNet. To many of us, it still feels like yesterday.

From first 100 downloads, to the current 20,000+ downloads/month – the pyATS development team has gone a long way. As the sands of time slip by, we delivered new features, expanded the libraries, refactored old components, took opportunities when they arose, and took risks where permissible. The team takes a lot of pride in what we do – being one of the first to launch a once-internal tool and make it available to the open-source NetDevOps community at large, enabling people, empowering solutions…

The success of this team is tied to the quality and ingenuity we deliver.

We Are Engineers

As engineers, our job is to build quality products with infinite potentials. Since pyATS is a framework – as the builders, we aspire to build the most robust system that other developers can use, extend, and build upon.

We took careful steps designing the core system – where every step of the way we focused on enabling future possibilities. We felt like we built a system that can answer to everyone’s need, and paired it with thorough documentation that takes developers through the thought processes behind our designs.

“pyATS is beautiful to read.” – Raymond Hettinger [@raymondh, Python Core Developer]

We thought we were ready for anything.

When pyATS launched in DevNet, the initial reception was not as good as we had hoped. We wished for a bang – to reshape the NetDevOps test & validation domain. A lot of passion has gone into the making of this infrastructure. And we know that it is powerful… what happened?

Flashing back to our very first Cisco Live session in early 2018, throughout the presentation, we focused on the abstract nature of the infrastructure and talked about the possibilities that it unlocks. We joked about the thousands of pages of documentation that pours over the power and flexibility of the framework. We touted the words “sky is the limit.” But there was just a lot of puzzled eyes in the audience. We would say:

“pyATS’s libraries, Genie, comes with an abstracted library infrastructure. It enables you to build APIs in a structured fashion, abstracting out the differences between things such as OS and releases, and write single-source-scripts where the APIs systems loads the appropriate set of libraries based the device your script is currently running against.

Out of the box, we have thousands of libraries and APIs for you to build your scripts with.

You are in control of the business logic. Sky is the limit.”

We were proud of our designs, and in turn, hoped that other developers would see the light. We assumed that everyone would be interested in the inner-workings of the framework itself (e.g., extending it), and forgot the most important aspect: solve real-world problems with it. In other words, a user would have to see value through all that abstractness and map the pieces & features into a solution that solves their challenge at hand.

An analogy to this would be like offering a box of a few thousand pieces of Lego to someone and say – you can build anything with this. Well, in theory you could – but it will probably take a long time to get there.

What we lacked was that seeding recipe.

We did have examples. But our examples demonstrated the features of the infrastructure, how the logic blocks interface with one another. It did not help users understand why this might be interesting to them, or how to apply it to solve day-to-day networking problems.

Through time, and under the inspiration of pioneering users, and experts from Cisco DevNet, CX & SE organization, we graduated towards a solution-based approach:

Walk The Journey – Don’t Just Describe It

The success of a product is the combination of two critical components: a robust solution, paired with use cases that help users solve immediate problems.

Today we’d approach things vastly differently. Instead, we would say:

“pyATS comes out of the box, ready to use. You can start with simple things such as learn & compare your network operational states:

  • Learn how your network is currently behaving (configuration + various show commands for operational data such as interface status, routes, neighbour’s)
  • Make a change on your network (eg, change config, or wait for some network event)
  • Learn the new network state again, and do a comparison to a previous snapshot

The diff system immediately tells you, what changed in your network, helping you debug failure events, and pin-point whether a change-config yielded the intended effects, etc…”

Without jumping into the intricate details of how the system is implemented, we instead focus the target audience on usage scenarios that relates to their day-to-day work. This new approach immediately rhymes with everyone – and offers that initial seeding that broadens the horizon.

Take another example. In pyATS test script execution, we explicitly designed the runtime engine to support custom loop generators. Here’s what the design looks like:

Siming Yuan pyATS journey 1

And we would talk about this feature like this:

“In pyATS, testcases are typically run in a linear approach: in the order that they are defined. In addition to that, you can create testcases that are looped. For example, the same testcase/code can be iterated through and reused, with each iteration (an independent, isolated object instance) having its own set of input variables.

We built it so that it explicitly to support generators. You can create your own testcase generator and feed it into the loop engine, where you are in total control of what to run, and how to run it.”

In the above discussion – the value of this feature is not immediately obvious.

Today, we approach this feature very differently. Something more like:

“In pyATS, typically, test scripts follow a what you see is what you get approach: your testcases run in the order that you’ve defined them in the script. For example:

  1. Connect to Device
  2. Configure Interface
  3. Verify Ping

And you can build linear suites that run actions one after another, automating mundane tasks.

The system also allows you to take total control of how testcases are run, and instead, “generate” testcases on the fly.

For example, in a chaos-monkey/chaos-engineering model where you want to inject changes (chaos) into your network, you can build a testcase generator that first reads the current state of your network, make decisions on what damage it could cause, cause the damage (run testcase), and based on a combination of testcase result (eg, if the chaos was successfully injected), and whether the system reacted well, decided whether to automatically recover and report, or keep going!”

The feature itself didn’t change. What changed is that we now walk in the shoes of our users, and relate our features to what network engineers want to do with their network devices.

The Software Pyramid

Software development is like building a pyramid. You start with a solid foundation, build the libraries, add to it the documentation, and top it off with the use-cases. However, when developers consume your product, they start with the use-cases – where it relates to them the most. It gives users an immediate feedback on what problems it can help them solve – which itself is the key that opens other doors, leading to references in the documentation, leverage more APIs, and unlocking more potential, etc.

Siming Yuan pyATS journey 3

Good use cases are the catalysts that kickstart the adoption process.

Going back to our analogy, if the framework itself is like pieces of Lego blocks, the documentation would be the guide that tells you how the blocks fit together, and use cases are the booklets that instructs you on how to land on the moon.

Siming Yuan pyATS journey 4

Thank You

After nearly four years, our fundamental idealism on building abstract, powerful systems did not change. What changed is that we’ve gotten a lot better at telling our story.

With close to 1000 users in our WebEx teams room, you have taught us a lots. pyATS became the bridge between test engineering and customers. It brought back use cases, customer pain points, and is helping Cisco to better test our products. There is still a long road ahead, and we truly look forward to it.

Sky is the limit.

Related resources