Performing Syslog Event Analysis and Forwarding at the Edge. Inside a Container!

For many with an IT Operations background we know Syslog event messaging as a highly useful logging function. It is ubiquitous in Cisco hardware products and controllers, and most management software; it’s also prevalent in other IT. Syslog is used to inform about operational state, component failure, security incidences, and other informational items.

Our Cisco DNA Center and Cisco Secure Network Analytics (formerly Stealthwatch), along with common solutions like Splunk and Elasticsearch, receive syslog event data for analysis, reporting, alerting, and archiving.

Networks continue to grow to address the increased demands of mobile users and IoT. Since data producers and consumers can be distributed across locations, centralized logging can be inefficient with bandwidth usage. Logging is also used for diverse purposes – management/ops, security, accounting, and regulatory compliance. Different management tools may process specific log types and must actively filter to ignore others, so forwarding all messages, multiple times to different consumers is an inefficient use of bandwidth, processing, and storage.

We have an opportunity to address this through spare capacity with Edge computing in the AppHosting functions of the Catalyst 9000 Series Switches. You’ve probably heard of or used AppHosting (Docker containers) embedded in switches for ThousandEyes collectors or iPerf agents. However, consider the benefits of performing syslog event analysis and forwarding at the edge, inside a container. We can leverage more complex filtering and forwarding that optimizes our bandwidth usage and provides an option to maintain local switch-container copies of the event messages in case of connection loss or application failure.

To achieve this benefit, we will deploy Syslog-NG, a popular open-source solution that also has a commercial offer. We configure the switch hosting the Syslog-NG container-app to forward its syslog event messages back into the container. Other network devices, servers, applications and IoT endpoints supporting syslog can send their messages at the container’s hostname/IP address for processing.

A Syslog-NG configuration file defines the sources, filters, destinations, and logging combinations.

This GitHub repo has been created to explain the technical details, provide a Dockerfile and syslog-ng.conf configuration file. In it we suggest filtering against ACL violation message patterns. Feel free to expand them to suit your needs! We also suggest destinations of your Cisco Secure Network Analytics or DNA Center instances. You can easily define your own Splunk, Elasticsearch or other syslog receivers.

We also provide a template for container-local log archiving using a date-grouping model. Once the AppHosted Syslog-NG is running and the switch and other optional nodes are forwarding their syslog event messages into it, then the message forwarding flow may look like this.

For more advanced and bandwidth-frugal environments, it’s possible to deploy additional instances of Syslog-NG on remote site switches with their own AppHosted instances of Syslog-NG.

One of the first questions may be “Can it perform?” My own lab testing pumped 40,000 Syslog messages into the container in one minute with negligible increase of CPU on the container or the hosting switch. Additionally, we should recognize that the AppHosting environment is purposely engineered to not impact the switch’s main function – moving packets! If you have more than 40,000 syslog messages a minute, you may have other things to worry about than CPU usage. ?

We hope you find this use-case helpful, and it provides you some thoughts of other ways to use the AppHosting feature of the Catalyst 9000 series switches.