Cisco Blogs

Unanticipated Consequences of Virtualization

- March 31, 2009 - 4 Comments

Virtualization is a new tool in the data center efficiency tool box. When used properly, the virtualization tool rewards the data center owner with reduced costs and efficient data center operations. If the tool is used improperly, unanticipated consequences can occur. Like all tools, virtualization should come with a warning sticker and like most warning stickers, individuals rarely take the time to read or understand it. If the virtualization warning sticker is ignored, the IT operational penalty can be steep.Why should data centers virtualize in the first place? The simple answer is to improve data center processing efficiency. Through virtualization, 10:1 server efficiency gains are possible. This efficiency gain directly lowers CapX and OpX. From a micro perspective, the operational and energy benefits of virtualization are obvious. However, on the macro level, some surprising cause and effects begin to emerge.Once a company’s IT hardware is virtualized and CapX and OpX gains begin to accrue, the virtualization story doesn’t end. The data center can, in fact, become more unstable and costly as a direct result of virtualization. Data power and cooling infrastructure, if not managed properly in a virtualized environment, can become the wild card or Achilles heel.When a critical power system is engineered for a data center, it is designed to support a maximum load. Any load applied above that design limit will result in the tripping of protective devices such as circuit breakers and fuses. Most data center operation managers do not recognize that Uninterruptible Power Supply (UPS) systems (legacy UPS systems in particular) are also designed to support both maximum and minimum load limits. If virtualization drives the load below this lower limit, bad things begin to happen. The UPS can initiate unwanted shut downs, lose the ability to parallel and switch to battery an excessive amount of times thereby draining the batteries and reducing overall battery runtime.Virtualization best practice dictates that unused servers be removed from the data center. Removal of servers can create a very unbalanced load on the UPS distribution system. The amperage loads on each of the three phases, or legs, of the UPS distribution system need to be closely monitored. For example, the amperage load can be very high on one leg and very low on another leg of a three phase circuit. If the electrical distribution system is not rebalanced correctly when servers are consolidated, a small addition to the wrong leg (i.e. the leg with high amperage) can cause a data center outage. If the high leg is overloaded, it causes the circuit breaker to trip.A virtualized data center forces a UPS to operate at its lower performance range by reducing electrical load. Such a scenario significantly reduces the overall operating efficiency of the UPS. This low UPS efficiency is compounded if the electrical distribution system is 2N (dual electrical path system). Legacy design UPSs suffer the most when operated in this low efficiency state.Virtualization can also impact the operating performance of generators. When operating backup generators on a light load (reduced by virtualization), a phenomenon called “wet stacking” can occur. During wet stacking, un-burned generator fuel is deposited in the exhaust manifold and exhaust stack during operation. As a result, the generator emits large amounts of black smoke. This black smoke will very likely exceed EPA and Source Registration limits on pollution. This can result in a $10,000 per day fine. Generator exhaust systems subjected to wet stacking have also been known to catch fire, just like an unclean chimney can cause a chimney fire in a house.When virtualizing a data center, the number of backup generators required also needs to be reassessed. The Diesel engine on a backup generator needs to be maintained very close to operating temperature in order to ensure reliable starting. To maintain this temperature, generators use an internal, electrically operated water heater called a jacket water heater. A generator’s jacket water heater will consume far more electricity over the lifetime of the generator than the generator will ever produce supporting a typical load during emergencies. A re-evaluation of generators after virtualization may reveal that an excessive number of generator(s) with jacket water heaters are gobbling up power and adding to the electrical bill for no reason. Underutilization of a generator means that fuel will be consumed at a much lower rate, therefore, more fuel testing for water and biological infiltration within stagnant generators fuel tanks will be required. In addition, the data center may now have an excessive capacity of fuel that needs to be stored which will increase overall OpX.The issue of delivery of power from the utility must also be considered when virtualizing a data center. The virtualized customer will likely have an electrical agreement in place with the utility provider. This agreement will often include a clause that penalizes the data center owner if overall electrical consumption drops below a contracted monthly consumption amount.Some real-estate agreements include the cost of electricity as a flat rate, usually billed on a cost-per- square-foot basis. This real-estate agreement may need to be renegotiated for energy savings to be returned to the data center owner. If not, the savings will accrue to the real estate developer. If such a pass-through energy agreement exists, not much effort is made on the part of the real-estate owner/developer to increase electrical efficiency. In this case, the real-estate owner/developer enjoys additional markup profit from the data center’s reduced energy bill.Virtualization can also drastically impact the data center cooling system. Like power systems, the cooling system is designed to operate within an upper and lower limit (in this case, temperature limits as opposed to amperage limits). If the upper limit is exceeded, the system will not properly cool the data center and the data center may have to be shut down. If, through virtualization, the cooling load is reduced to below the lower design limit, the cooling system becomes unstable and very likely shuts down because of low head pressure on the compressors (this is a safety protection feature). As a result, compressors will also begin to short-cycle and thereby reduce their own expected lifespan by many years. In some cases, consistent operation below lower load limits can possibly void the manufacturer’s warranty.To address the issue of possible low level cooling system shutdown, a device called a hot gas bypass may have to be installed on the compressors. During a low level operation this device places an artificial load on the compressor to maintain operation and thereby prevents short cycling. This artificial load is designed to maintain operation within the lower level design parameters. However, this artificial load negates any virtualized savings to the electrical meter. All the good work done by virtualization is undone done by this bypass system. In some cooling system designs, hot gas bypass is always on, even during normal load operation, contributing to the cooling system inefficiency. The solution is to evaluate the data center cooling system to ensure it is designed to operate at the new lower cooling level and that hot gas bypass is switched off and in use only when needed.The virtualized data center owner should also re-evaluate all service contracts and remove any unused power and cooling equipment from the contract. Lack of attention to this detail almost ensures that the data center owner will continue to pay for service on equipment that was taken out of service.Unused power and cooling equipment, if it stays in the data center, needs to be periodically tested and operated. Once the virtualized data center reaches its new capacity, this equipment may very likely have to be placed back online.Virtualization also has financial ramifications. Equipment depreciation and budgets may be impacted. If equipment is taken out of service, sold or salvaged, the full remaining un-depreciated residual value may be due in the same year as the sale or salvage. This unanticipated cost can negatively impact your budget.In conclusion, if a data center is virtualized, all power, cooling, billing and maintenance systems need to be reanalyzed. The rewards of virtualization still far outweigh the risks. However, from a data center operations and financial point of view, those risks need to be understood and managed.For additional information, please contact Paul A. Marcoux, VP Green Engineering, Cisco Systems, DSSGEmail;

In an effort to keep conversations fresh, Cisco Blogs closes comments after 60 days. Please visit the Cisco Blogs hub page for the latest content.


  1. A really interesting green computer technology I found is Userful Multiplier. It's where multiple people can use the same computer at the same time each with their own monitor, mouse and keyboard. This saves a lot of electricity and e-waste. A company called Userful recently set a virtualization world record by delivering over 350,000 virtual desktops to schools in Brazil. They have a free 2-user version for home use too. Check it out:

  2. As new tool in the data center efficiency tool box Thank you for sharing

  3. That was an informative article. Thanks.

  4. I always enjoy your blog posts, very informative and handy too. Keep them coming!