Workload Automation, Job Scheduling, Applications and the Move to Cloud
In an earlier part of my career I learned the extreme importance of Workload Automation, aka Job Scheduling. Workload automation is the oldest IT technology on the planet coming from the need to schedule jobs on an IBM Mainframe. Job Scheduling has evolved from driving JCL (Job Control Language) to Workload Automation where the Scheduler stitches together batch and real time activities across mainframes, proprietary OS systems, x86 systems, applications (both packages and commercial off the shelf such as SAP or Oracle or Informatica) and now web service enabled applications whether they be onsite or in the cloud. Walk into the operations center of any data driven company and you will see multiple screens where operations are monitoring the state of these jobs. Why are they so critical? Over 50% of all transactions that occur on this planet are batch in nature. They are scheduled based upon specific times or based upon dependencies being met. These workloads can be a complex and interrelated set of activities. Effectively these job streams are the business processes that drive modern enterprises.
Without these jobs companies don’t get information (and large amounts of it) in the right place at the right time. Most companies today could not close out their financial quarters without enterprise schedulers to move data from their disparate systems into a consolidate place for either the general ledger to close out or for a critical Business Intelligence report to run to drive placement of the correct product into the specific physical location to serve the global economy. Workload automation tools open and close stock exchanges and process all the transaction data from trades. They also drive compliance checks. This is important stuff for the global economy! This was my realization in touring key operations centers and realizing that half of the big monitors were covering the movement of batch data in the enterprise.
Hiccups in these job tasks from target systems running out of resources or other “ghosts in the IT machine” can create major issues. Company’s can miss SLAs with their customers. Data can be out of sync across systems. It can cause the delay in sales order and revenue recognition. This can be a billion dollar problem.
Workload Automation, as the technology is now called, has been transformed under the watchful eye of key Gartner, Forrester, IDC, and EMA analysts. This “ancient” technology is reinventing itself. Why is this important in the data center? Virtualization and Cloud have completely disrupted the legacy model of Workload Automation. We have customers using our Cisco Tidal Enterprise Scheduler product that integrate it with VMware. Why do they need this? Well the Scheduler is the one IT tool that understands and is aware of when very compute intensive jobs need to get executed. In one hour we will need 16 vCPU of compute resources to drive an ETL process, or pre-process and load data into a Hadoop cluster. The scheduler can proactively, at just the right time, take a Business Intelligence application from say SAP and increase the resource state to be ready just in time to receive and process the huge compute operation. When the execution is complete the application is then placed into a low resource state for normal operations. This is incredibly powerful for IT administrators. Workloads can be placed at the right resource with the right state at the exact time it is needed, resulting in less compute horsepower required when averaging over many workloads. Compute power is recovered and reused.
We are in the multi-year march from on premise applications deployments to the cloud. Workloads now need to be places across datacenters and into applications running in Service Provider clouds. We now have critical CRM, ERP and BI cloud service offers that need integration to. The old workhorse of the job scheduler has now morphed into cloud based Workload Automation. We have come a long way from Job Control Language to heterogeneous system and application scheduling. The original parents of the technology must be proud of how far their progeny have come. Talk to Derek Evan at Cisco, he is our rock star.
Learn more about this technology at Cisco at http://www.cisco.com/en/US/products/ps11092/index.html .