Understanding the Truth Behind Batch Processing in the Modern World

October 26, 2010 - 2 Comments

There is a myth that all data processing occurs in real time.  But the reality is that batch and event based processing are still very much alive and the majority of data processing is still done through batch processing.  Our average customer uses Cisco Tidal Enterprise Scheduler to execute ~50K jobs on a daily basis, but we also have large financial services companies automating the execution of over 100K jobs daily, not bad right?

So, with over 50 percent of all business processes leveraging batch operations, it is essential to keep your batch production running smoothly in order to keep your business running smoothly.  You cannot afford to have failed jobs.  Failure is not acceptable as it directly impacts the business and can impact revenue – such as the inability to process orders or to generate invoices.  What are we getting at here?

The goal of business process automation is to increase process speed, reliability, and control so you can provide better customer services, increase productivity, and capitalize on new revenue opportunities.  How do we achieve this goal?  You need to streamline the key business processes that drive value for customers.  You need to optimize, automate, and transforms these processes to separate your organization from the competition.

Your key business processes can be high-level and strategic, or they can be targeted and tactical.  They can cut across organizational boundaries or they can focus on functional areas such as the IT department.  All in all, the answer is the same, workload automation makes key business processes faster, more efficient, more management and secure, and less prone to errors and delays.

One key component of workload automation is enterprise job scheduling.  Automating job scheduling and other datacenter tasks is a requirement for the complex datacenter. But we all know that the integration between IT Systems using existing tools must be custom-coded or left undone, leaving the datacenter with increased security risk and a reliance on business users to monitor system health. A planned approach to job scheduling and datacenter automation should not only eliminate these risks and inefficiencies, but also reach across heterogeneous environments as a service platform for workload automation and infrastructure activities–and this is exactly what does.

Many business processes have complex logic and dependency on both external and internal systems and the need to manage the proper execution of these complex processes continue to fuel a demand for an enterprise job scheduler – consider how ordering data may need to be obtained from various partners, decrypted, consolidated, transformed and then fed into ERP systems.  IT groups must support many applications and servers across multiple platforms that frequently operate independently of each other.  Coordinating all these applications and networks to streamline job scheduling can increase productivity and reduce costs. 

In the end, if you think that by applying more staff, toolkits, and rudimentary scheduling software to cobble together automated batch processing solutions is the way to go, think again.   This is cost-prohibitive, inefficient, and error-prone.  Enterprise job scheduling products enable datacenters to solve this problem by simplifying both complex and routine tasks.

Whether you are consolidating datacenter operations, moving apps to more cost-effective platforms, or transitioning from customized to packaged applications, you need to simplify and lower the cost of batch management with a single interface to all batch processes across platforms and applications, and increase productivity of operations staff by automating all batch scheduling tasks.

Minimize the risk of failed jobs with intelligent scheduling.

In an effort to keep conversations fresh, Cisco Blogs closes comments after 60 days. Please visit the Cisco Blogs hub page for the latest content.


  1. What do you mean by “intelligent” scheduling? Can the scheduling change based on business input?

    • Marcio,

      “Intelligent” can be subjective. However, lets consider the realm of a typical OS based scheduler or a scheduler that is embeeded with an application.

      These schedulers often are purely “time based”; hence triggering an activity at a “point in time” without regard to whether it is appropriate to perform the activity. Often times, these schedulers are also bounded by their own ecosystem.

      I think we’d agree that the modern business process rarely is confined to a single system or a single application.

      An enterprise scheduler extends the reach of business process automation by reaching across the entire heterogeneous landscape that embodies different data sources, various systems and software application stacks.

      An enterprise scheduler can allow scheduling to be intelligent by incoporating the appropriate business logic required to determine when the next processing step should be and any customization of the execution details.

      As a simple example, consider the implications to running a BI process for report generation before the appropriate upstream processes were completed successfully or that all data was loaded and transformed. I think you’d agree that running purely based on a time schedule would not suffice and there needs to be a better more intelligent approach.

      Thanks for the question.