Last we spoke, it was about network device configuration management. Let’s move our focus up the stack to applications and management of their configuration. Whether enterprise or cloud-architected, running on physical servers, in virtual machines or in containers, how are you managing your applications?
Puppet, Chef, Ansible and Salt are popular answers to this question and leading contenders for initial provisioning and management of configuration drift of data center applications -- whether they be common off the shelf (COTS) or custom built applications. Two of these configuration management technologies, Puppet and Chef, are supported by Cisco Intelligent Automation for Cloud 4.1. The collection of features enveloping these two Ruby-based technologies within Cisco IAC is referred to as Application Configuration Management (ACM).
Approach to Agent Bootstrapping
Puppet and Chef are similar in nature -- in more ways that we’ll discuss in this post. An example of similarity being that both of these ACM technologies require an agent (Puppet) or client (Chef) installed on the server under management (node).
Agent Bootstrapping Methods
Both types of ACM technologies support client-only and client/server deployment models, referred to as agent/master for Puppet and client/server for Chef installations. Whether only using an agent-only (client-solo -- Chef) or using an agent/master deployment model, unless your virtual or physical server image has the agent preinstalled, you’ll need to go perform the prerequisite work of agent installation.
IAC performs this dirty work by bootstrapping the appropriate agent (or client) whether on initial server provisioning or on-demand on any existing server when a user assigns an application to a server. Mechanics used to perform agent installation varies. The mechanics used within IAC are listed in the “Agent Bootstrapping Methods” chart. Initially, IAC used WinRM as its mechanism to bootstrap agents on Windows severs until customer feedback drove use of an alternative mechanism -- psexec. We found that customer security teams were either uncomfortable with or had policy in place against the use of WinRM as a method to execute scripts remotely and made the switch to psexec, which “is a light-weight telnet-replacement that lets you execute processes on other systems, complete with full interactivity for console applications, without having to manually install client software”.
Part of the agent installation involves establishing a connection between the ACM server (Puppet Master or Chef Server) and the node (server with agent/client installed). IAC orchestrates the registration of the node with it’s respectively, assigned ACM server. This process is different depending on whether Puppet or Chef is used. In the case of Chef, IAC has the chef-client register with the Chef server using the private key assigned to the chef-validator, which IAC loads into the node during client installation. In the case of Puppet, IAC performs an initial puppet agent run, which lodges a certificate authorization request on the Master, which IAC subsequently orchestrates the signing of on the Master. With agent bootstrap complete and authorized, secure communication between the ACM server and client, attention is turned to the management of connections IAC may have established with n number of Puppet or Chef servers.
System Health -- ACM
Connection and System Health
In the case of client/server deployments, IAC will establish connection to one or more Puppet Masters and one or more Chef Servers. Each connection is treated with care as the health of each connection facilitates IAC’s ability to successfully orchestrate applications. Connections are established using a service account permissioned appropriately. The health of the connection between each ACM server is evaluated once every 30 minutes by default. Connection health is determined by performing connectivity, authentication and authorization tests. Details of these tests and a screenshot of the System Health console can be seen in the ”System Health -- ACM” chart.
CloudSync Finite State Machine
Cloud Object Model and CloudSync
Immediately after establishing a healthy connection, CloudSync runs. CloudSync is a synchronization process driven by a finite state machine whose responsibility is to not only perform initial object discovery and granular fingerprinting -- essentially a deep interrogation of cloud objects and their attributes -- but also, manage ongoing reconciliation of infrastructure changes with respect to their representation of the provider’s cloud infrastructure as modeled within the service catalog. Note the “CloudSync Finite State Machine” chart, which is laced with Extension Points, where cloud administrators may insert custom logic on state transition for any give object within the model. Once collected, this inventory (e.g. a Chef Role) is presented to the cloud administrator for the ACM server for use within their cloud. Cloud administrators may choose to register the discovered objects for use by end users.
Cisco IAC Cloud Object Model -- Chef
Cisco IAC Cloud Object Model -- Puppet
For example, the cloud administrator may choose to register a Puppet Role as being available for end users to assign to a server. Registration of this role may include assignment of additional metadata, including price of the role as a one-time or recurring charge for use of the application and assignment of tenant permissions (whether to make the role available to all tenants or only select tenant(s)).
It’s through the relationships derived within the Cloud Object Model and assignment of tenant permissions that the specific applications are presented to a given end user. Service Resource Containers are used as a logical construct owned by the cloud administrator wherein tenant-specific resources may be hosted. Applications delivered to tenants may be created in a virtual data center that is serviced by either a Puppet Master or Chef Server. See the Cisco IAC documentation for further details on other constructs within these and other models.
Manage Applications -- Node Classification
Approach to Node Classification
Once registered for use, applications become visible to end users, who may assign applications to their servers whether during initial server provisioning or to an existing server. Upon selection of application(s) by the end user, IAC classifies the node by writing a hiera file (Puppet) or by writing a run-list (Chef) on the respective ACM server and forces an immediate agent run to ensure application configuration is promptly enforced.
In this sense, IAC provides a common user experience for node classification irrespective of the underlying technology chosen by the cloud provider (the organization running and administering IAC). As the IAC product suite evolves, so has our approach in terms of classification via Puppet and the more programmatically effective use of a custom-written External Node Classifier, taking advantage of the ability for the node_terminus configuration to to interact with an ENC.
Application Configuration Management Highlights
- Integration with Puppet and Chef
- Connections to n number of servers
- System health checks for these servers
- Application infrastructure discovery (CloudSync)
- Bootstrapping of agents (green and brownfield)
- Financial Management
- Pricing of applications
- Showback for application orders
- Run rates including application consumption by user, org, tenant
Financial Management -- Application Pricing & Showback
- Tenant-specific application catalogs
- Tenant/application consumption dashboards
- Application provisioning for virtual machines, physical servers
- “My Applications” interface for application management
- Service Offering Elections
- 3-tiers of control on enable/disable application configuration management services at provider, tenant and organization levels
- Multi-Cloud Platform Support
- Support same services ubiquitously across all platforms
Financial Management -- Application Run Rates
- Application User Persona
- “My Application” interface for application management
- ACM Server and application usage dashboard
Cognizant of the plethora of application configuration management tools available to Cisco customers, including commercial, open source, and homegrown tools, we’re very interested to hear which ones you have found to be the best fit in your environment. Have you established revision control practices as you manage infrastructure as code?Having reviewed Cisco’s approach within its cloud management platform, IAC, whether you manage configuration of physical servers, virtual machines or use CM to build containers or hosts that run containers, how does your approach compare?
Tags: Chef, CIAC, cisco IAC, cloud-based applications, configuration management, enterprise applications, Puppet
Automation was a major theme at VMWorld 2014 in Barcelona this month and it was clear that IT professionals are discovering that they can’t keep pace with business using manual provisioning and de-provisioning processes. The problem is that with the physical boundaries of time and space, there simply isn’t enough staff to meet the need. As a result, there is a renewed interest in automation at all levels of the organization, to utilize automation allowing IT to scale beyond their current capabilities.
There are a lot of solutions on the market today aimed at this problem, but you have to be careful. Many of them are point solutions, have limited coverage or functional scope.
Cisco has been talking about automation for years and their approach is to deliver heterogeneous management – because your data center is heterogeneous. These tools allows your subject matter experts to transfer their knowledge into automated workflows allowing your business to respond to market changes faster than ever before. Meaningful automation that spans your enterprise with unified management allows you to manage your data center as an interrelated whole. Read More »
What is Shellshock?
Recently a bug called Shellshock was discovered that could potentially allow remote attackers access to Linux, Unix, and Mac OS X operating systems. This bug, also called Bash Bug because it exploits vulnerabilities in the bash shell of *nix based systems, is rated a 7.5 on the Common Vulnerability Scoring System, the highest score a bug may get. It also has a low difficulty rating making it a very serious vulnerability.
How Does it Affect Nexus 9000 Series Switches?
Cisco Nexus 9000 switches are not vulnerable out of the box. The NX-OS image includes a version of bash that is affected by the Shellshock bug. In order for this vulnerability to be exploited a user must successfully SSH into a switch and successfully log in. The Common Vulnerability and Exposure IDs for these vulnerabilities are:
Shellshock could potentially allow attackers to log in to the switch and send remote commands to attack the network or possibly set up attacks for later exploitation. However, as long TACACS or local authentication has been used to secure the login, the vulnerability is not exploitable. Nexus 9000 Series switches can run in NX-OS mode or ACI mode. The bug affects all current releases of the NX-OS mode image for the Nexus 9000 Series. The ACI image for the Nexus 9000 Series has been analyzed and is not vulnerable to Shellshock. Even though the ACI image is not affected, it does now contain a patched version of the bash shell as well.
Shellshock Fixed in Recent Patches
The Nexus 9000 NX-OS team was one of the first, if not the first, in the networking industry to fix these issues with hot patches. Hot patches that fix all six of the above listed CVEs have been released by Cisco for the existing NX-OS software and Guest Bash Shell. These patches are currently available for download from http://software.cisco.com/download/navigator.html
There’s a very helpful configuration guide posted on our website on how to perform NX-OS software maintenance upgrades (SMU) as well as using the downloadable RPM to patch the Guest Bash Shell.
Shellshock Fixes for Nexus 3000, 5000, 6000 and 7000.
A Shellshock fix has already been released for the Nexus 7000 Series in NX-OS release 6.2. For the remaining platforms, we will be providing fixes through upcoming NX-OS software releases by the end of this month.
Tags: Bash bug, Shellshock
The tenth OpenStack release codenamed Juno was released on October 16, 2014. This press release provides a good summary of what to expect in Juno. It also discusses important new capabilities included in the more than 340 new enhancements built in to Juno and highlights different usecases that showcase the diversity of workloads supported on OpenStack.
In the first part of the Cisco and Openstack Juno Release blog, I covered Cisco’s OpenStack team contributions to the Neutron project. Here I’ll provide details of our contributions to other OpenStack projects as well highlight our development efforts on StackForge. Cisco was the sixth top code reviewer for the Juno release across all projects in Juno release and is Foundation’s fifth largest company in terms of OpenStack membership.
This Nova blueprint was completed in Juno and provides support for configuration and provisioning of instances with SR-IOV port connectivity. The implementation generates SR-IOV specific libvirt domain and network configuration XML for the instances as well as includes the capability to schedule instances based on the compute nodes SR-IOV capabilities. One of the key use-cases for SR-IOV is Network Function Virtualization (NFV) that requires high performance traffic throughput in and out of a virtual machine providing network services (Virtual Network Function or VNF).
We proposed and implemented support for metering Network Services in Neutron using Ceilometer. This included new pollsters and notification handlers for Load Balancer as a Service (LBaaS), Firewall as a Service (FWaaS) and VPN as a Service (VPNaaS). The metrics are categorized into Provider or Service Level, providing different level of details. Provider level metrics help determine the type of implementation and its feature, whereas the Service level metrics provide more granular metric details on the service health and consumption. Separately, instance metrics were enhanced as part of this blueprint to support read and write metrics per instance disk device.
In the Cinder project, Fibre Channel Zone Manager allows FC SAN Zone/Access control management in conjunction with Fibre Channel block storage. It has a pluggable architecture and we contributed the Cisco FC Zoning plugin that automates creation, deletion and modification of zones in zonesets. Zones are configured automatically as part of the active zone set for the specified VSAN in the FC SAN to provide a more flexible and secure way of controlling access.
Enhancements to Horizon to enable configuration of IPv6 subnet modes is also part of the Juno release. This allows tenants to configure address and Route Advertisement (ra) mode for their subnets through the user dashboard. Neutron supports multiple IPv6 address configuration modes including SLAAC and DHCPv6 (both Stateful and Stateless modes).
The Cisco OpenStack team has been actively developing across different projects on StackForge as well. This provides an excellent platform for OpenStack related projects to make use of OpenStack project infrastructure and also continue to collaborate in the open.
OpenStack Services Puppet Modules -- One of challenges that we hear about from our OpenStack customers is how to make OpenStack more manageable and deployable. There are several different deployment options for OpenStack and we have tremendous experience with automating the underlying system and service configuration via Puppet. We work with customers, partners and the community to enhance Puppet modules for OpenStack services and integrate with Cisco infrastructure as well. We also recently announced, in collaboration with RedHat, Cisco UCS Integrated Infrastructure that combines Cisco’s server, switching and management technologies with Red Hat’s enterprise-grade OpenStack platform.
Group Based Policy (GBP)– Currently staged on StackForge, this project aims to provide policy abstractions that extend the current Neutron API resources and introduces a declarative policy driven connectivity model that presents application-oriented interfaces to the user. The Group Based Policy framework implementation provides the flexibility for new API resources – End Points, End Point Groups, Contracts and Classifiers – that can be mapped to existing Neutron resources or passed directly to a third party controller. In addition to a mapping driver that supports all existing Neutron plugins, Cisco will also be releasing a driver to directly integrate GBP with its Application Policy Infrastructure Controller.
Nova Solver Scheduler – For resolving complex constraints based on policies and business rules, we have been collaborating with the community to develop a smart Nova Scheduler driver that models compute placement as a supply and demand problem. The intent is for the Solver Scheduler to integrate with the Gantt project that is aiming to separate out the Nova scheduler as a standalone project.
Cisco’s OpenStack team contributions are across numerous projects in OpenStack. Our aim is to work with the community, with our customers and partners to enable more successful OpenStack User Stories, resulting in a win-win situation. We are going to be presenting several general sessions that were selected as part of the community voting process at the upcoming Kilo Summit in Paris. You can find more details in this blog post and we look forward to seeing you there!
You can also download OpenStack Cisco Validated Designs, White papers, and more at www.cisco.com/go/openstack
Tags: ACI, Cinder, Cisco, horizon, IPv6, NOVA, OpenStack, Puppet, StackForge
[Note: This is the last installment of a four-part series on the OpFlex protocol in Cisco ACI, how it enables an application-centric policy model, and why other SDN protocols do not. Part 1 | Part 2 | Part 3]
As noted earlier in this series, modern DevOps applications such as Puppet, Chef, and CFEngine have already moved toward the declarative model of IT automation, so there is already some obvious synergy between DevOps and the Cisco ACI policy model. DevOps automation products are also optimizing application delivery processes and are designed to automate critical IT tasks to make the organization more agile and efficient.
In an early 2014 blog post, Andi Mann, vice president of strategic solutions at CA Technologies, wrote about the evolution to DevOps and the synergy with the Cisco ACI policy model:
Though the DevOps approach of today—with its notable improvements to culture, process, and tools—certainly delivers many efficiencies, automation and orchestration of hardware infrastructure has still been limited by traditional data center devices, such as servers, network switches and storage devices. Adding a virtualization layer to server, network, and storage, IT was able to divide some of these infrastructure devices, and enable a bit more fluidity in compute resourcing, but this still comes with manual steps or custom scripting to prepare the end-to-end application infrastructure and its networking needs used in a DevOps approach.
The drag created by these traditional application infrastructures has been somewhat reduced by giving that problem to cloud providers, but in reality this drag never really went away until Cisco innovated application-centric programmability with Cisco ACI. This innovative new solution is now poised to greatly benefit the whole application economy, especially management of the DevOps application environment…
Read More »
Tags: CA Technologies, CFEngine, Chef, Cisco ACI, devops, OpenFlow, OpFlex, ovsdb, Puppet Labs, SDN