When developing applications in a microservices architecture, one of the first obstacles a developer faces is determining the state of the underlying orchestration and container services. The developer has no real insight into how the underlying services and orchestration system(s) are performing. As part of the developer experience we built for this new application development methodology, Shipped we started building the Shipped Synthetic Monitor service to simulate an end-to-end experience. The service was designed by capturing the steps of a typical deployment scenario and then monitoring the services and orchestration system tasks. The reporting provides each step and task availability and assurances as shown in the figure below.
This is the overall status page which lists the overall status and then each task has an individual status. We track the following states:
- Operational
- Degraded Performance
- Partial Outage
- Major Outage
The Shipped Synthetic Monitor (aka Synmon) tests Shipped functionality, with optional reporting to a web-based dashboard, and with critical errors reporting to Spark (see image below). Synmon includes an end-to-end test of Shipped and the underlying Mantl platform components, creating a project and service, bootstrapping it, and deploying it to an environment. We are continuously improving the monitoring platform and have built out testing on other components of Shipped, including test on Bintray, creating a Docker repository, and more. In addition to the benefits we have received in enhanced visibility and trouble detection, this effort has really helped us with stability in both Shipped and Mantl.
The choosing of the monitoring platform was difficult. We initially decided to go with Stashboard, an open source tool by Twilio. Although it worked, we ended up switching due to an OAuth 2.0 upgrade that Google GAE upgraded to, and as a result, there was no longer support for Stashboard. We wanted to be sure to switch to a system that would have full support, and chose StatuspageIO. It was fairly easy to replace Stashboard since Synmon was designed with a micro service framework in mind. The switch has lowered cost, and created a more reliable status page, with support if needed.
In order to further enhance Synmon, we plan to integrate Performance Benchmarking, Perfkit soon as well, which will be the next step in fully understanding our users performance needs. Pefkit is an, “…open effort to define a canonical set of benchmarks to measure and compare cloud offerings. It’s designed to operate via vendor provided command line tools”, we look forward to integrating this platform and hope to help contribute to the goal of making a standardized benchmarking tool(Perkit Github).
With Synmon integration into Cisco Shipped, the result has been the identification of bugs, defects, and improved visibility on the key components of Shipped were failing the most. As we add more test cases to Cisco Shipped we are continuing to gain critical insights on the stress points of Cisco Shipped. We are continuously adding test case to Synthetic Monitor resulting in continuously stabilizing and improving the visibility of the underlying processes and system calls.
3rd Party Tools
If your interested in learning more about the tools we are working with, please click on the links below.
I would like to thank Nick Hayward on my team for his hard work and dedication to improving the performance and reliability of Cisco Shipped, developer experience with microservices, and his co-authoring of this blog post.
For more information about Cisco at OpenStack Summit, visit cisco.com/go/openstacksummit. If you are at the event, come by the Cisco Booth C11 to talk to the Cisco Shipped and Mantl teams.
CONNECT WITH US