The role of IT is to maximize business capacity, and refreshing your servers is an opportunity to increase capacity, reduce hardware and most importantly, lower costs for the business. It is reasonable to assume that packing more physical servers into less rack space helps you accomplish these goals.
Here is the problem: If you increase server density by repackaging it into a smaller physical chassis, it does not mean you get more servers into fewer racks. Even with moderately dense blade and rack solutions today, you run out of available power (and more critically) available cooling, long before you run out 
of physical space. You might say ‘we are building a new data center that is sure to offer higher density’ or ‘we are moving to a new co-location facility and this won’t be an issue.’ However, it takes expensive and complex technology to achieve higher average rack cooling density on the facility side, and critical trade-offs on the server side….this the fine print. Unfortunately, purchasing decisions are being made on the number of servers vendors can stuff into a rack unit without truly looking at the fine print.
Let’s look at real power and cooling limitations of modern data centers; not aging facilities that are long in the tooth, but facilities being built today to last another 10-15 years. What are the power and cooling design guidelines? To discuss these factors, we invited Vali Sorell, the Chief Critical Facilities HVAC Engineer at Syska Hennessy, one of the largest data center design firms in the world, to provide input:
Typical power per rack in enterprise-type data centers today rarely exceeds 8 kW. There are always some high power racks or cabinets in the typical data center, but overall averages are usually considerably LOWER than the IT load planners had anticipated. The fact that there are pockets of higher density should not affect the way in which racks and equipment are purchased. In those isolated cases, cooling provisions can be made to account for it. This brings up a few points that need to be considered when planning a data center:
With the exception of high performance computing applications, specifically ones in which cabling distances affect the speed to a solution, there is not a need to densify. Densifying when it is actually not called for creates situations in which high power servers are all grouped into a small number of slots. Without proper air flow management, providing a cooling solution for that layout can be complicated, and is often overlooked. Often, data center owners spend too much time “minding the gap” between adjacent cabinets, and miss the issue that blatant gaps inside cabinets are just as harmful to effective operation. Even if blanking panels are used to stop the resulting gaps, some degree of internal recirculation and bypass is promoted through the operation of the high density servers and their use of the higher power server fans. As a result, the use of higher density servers can result in lost efficiency, poor internal air flow, and higher entering IT equipment temperatures.
Densifying is not cost effective. To deliver cooling and power to a cabinet populated to 20 kW will cost more than 2X than doing the same for twice as many cabinets populated to 10 kW. The typical argument FOR densification is that it will use less floor area. That approach misses the bigger picture that the back-of-house spaces, which deliver cooling and power to the cabinets, are not affected by densification. The only determinant of back-of-house floor areas is the TOTAL power delivered to the data hall. For high density installations, that back-of-house floor area to data hall ratio could be upwards of 5 to 1; for lower densities it could be 2 or 3 to 1. Adding more complexity to that high density solution is that the overhead or underfloor spaces required for delivering the cooling air increases quickly as the loads per rack increase.
Additionally, with higher density, more modes of failure exist; and when a failure occurs, the response time to prevent a shutdown of a facility is significantly reduced. The net result: increase in density leads to a decrease in reliability.
The bottom line is that new data centers rarely average over 8kW per rack, and increasing average density above 12kW requires expensive supplemental cooling technology that adds complexity and affects overall data center reliability. Consider that a typical blade enclosure on average consumes between 270W-470W per rack unit (RU) depending on workload. This means a 42U rack today could easily consume over 20kW of power and cooling capacity! Vendors are creating servers today that put several nodes into a 2U package, but they aren’t all sharing the Fine Print that affects customer decisions. It would sound something like this:
Density isn’t they key to refreshing your data center servers – Efficiency is the Key. Cisco UCS brings the most power efficient platform to your data center by unifying the fabric and unifying management to maximize each and every rack. No other platform provides the rack efficiency both in power and in operation. Let’s look at your racks and the business efficiency you will gain with Cisco UCS!
Thanks to our Cisco Power & Cooling experts, Roy Zeighami & Jeffrey Metcalf
For More Information:
Nice 57 Chevy.
White space is the most expensive space per square foot for most organizations. I respectfully disagree with the theme of this post and there is an alternative viewpoint. The bottom-line is construction cost of a 25,000 square foot data center is cheaper than building a 50,000 data center at the same power density.
At the heart of this debate is the method, effectiveness and reliability of cooling high density. Airflow cooling with a raised floor, drop ceiling and containment as identified in this blog post is conventional thinking where data centers are designed the way they were 20 years ago. However, there are many “proven” alternative designs that challenge the status quo and support low- to high-density requirements more effectively.
In 1955, the conventional wisdom of the airline industry was commercial aircraft must have a propeller because naturally jet propulsion would cause undue stress on an aircraft causing it to break apart in flight. Enter the “jet age” when Boeing jet test pilot Alvin “Tex” Johnston challenged conventional wisdom by doing barrel rolls in the first prototype 707 in front of 250,000 people.
Thank you for your perspective. It is always nice to see comments from business leaders and CEO’s like yourself. The message is not an argument against technology like OptiCool’s rack door heat exchanger. The point is – available space, power and cooling define the amount of compute, storage and networking resources that can be deployed per rack. Simply packaging more servers into a smaller form factor does not automatically imply users can deploy more servers per rack. There is no such thing as a free lunch. An overwhelming majority of data centers in the world cannot support a rack density above 12 kW. To support high density racks, additional mechanical and electrical infrastructure equipment is required. This additional infrastructure adds cost and complexity. There are specific use cases that may justify the additional expense and added complexity. However, this is not the case for the vast majority of data center deployments. The bottom-line, customers must be aware that deploying servers packaged in a smaller form factor does not mean they can deploy more servers per rack.
Please allow me to chime in on this.
I do have experience with close coupled cooling. We identified years ago that the power density increases would lead to a day when forced-air cooling would no longer be practicable. I had several spaces where there just was no good solution to the heat load.
The rear-door evaporator (heat exchanger) captures the heat from the air before it leaves the cabinet so no hot air exists anywhere within the data hall. Since we no longer have to move air mass across the room to and from bulk evaporators, the cost of infrastructure to support higher heat density levels has been reduced to parity, even as low as 6kW. We calculate the comparative cost per unit power will be much less at heat loads above 6kW up to at least 30kW. While the forced air model may not even be able to achieve that density.
The server fan speed ramp up was troubling for us until we got away from the forced-air model. Because there is much less hot/cold air combination, most of our server fans are running slower, quieter and consuming less power. The only hot air leakage is within the cabinet but that has been reduced since the heat exchanger has a fan pulling the air from the cabinet.
It took us awhile to work through it all but now we think this system is actually much less complex than forced air. All the airflow problems have been eliminated thus operation is substantially simplified. As an added benefit our efficiency improved.
We were able to avoid many space related problems. We delayed or eliminated moving to larger buildings and space expansion projects. Even if these system cost much more to build, it would still be a relative bargain.
We have also worked to optimize our power distribution as well. Once we abandoned the status quo, we were able to find ways to deliver more power per cabinet for half the money, with less loss.
There is definitely no free lunch. I would say that if you double the size of your facility, you would have to nearly double the length of fiber and other cable runs, those are crazy expensive. You can double a list of other items too; super structure, fire suppression, taxes, insurance, roof to leak, cleaning, lighting, foot steps to perform maintenance, and rent.
I do encourage power densification as a cost savings strategy.
I second everything Mr. Doyle just expressed, especially his statement “Once we abandoned the status quo”. It seems that the status quo of datacenter designers is to simply apply the traditional methods of air and water based heat removal that they know and are comfortable with.
So in the interest of brevity, rather than pick apart the generalizations and assumptions expressed in this post, allow me to respectfully make a couple of observations. First, the message being delivered by the post seems confused. On one hand Mr. Egan makes arguments against densification, dubbed as “the fine print”. On the other hand, he makes a pitch for the benefits of UCS, itself a medium density platform. What is the post trying to say regarding adopting higher density platforms in the context of datacenters? The wide adoption of virtualization in general, and specifically the increasing density of blade/converged compute platforms such as UCS, Nutanix, Simplivity, and others, is undeniable. Should IT architects not take advantage of the efficiencies in denser compute, storage, and networking simply because the status quo of datacenter designers is apparently stuck in delivering low density, air-cooled datacenters?
Second, Mr. Sorrel takes the position “Densifying is not cost effective”, and then articulates the inherent limitations of CRAC based datacenter design in achieving higher per cabinet densities. On its face this is such a simplistic and self-reinforcing view, I can only presume that portion of the post was taken entirely out of context from some other writing and is just a victim of bad editing trying to keep the post really short.
In reality, there are many examples of datacenters, both new and retrofit, that have moved beyond the status quo and applied more progressive cooling and power delivery technologies that are far more efficient, achieve much higher densities, provide more flexibility, and cost much less to implement and to operate. Clearly, there are manifold economic and performance advantages to adopting higher density computing, networking, and storage. Until datacenter designers start taking into account ALL of the elements that make up the true capex and opex of a datacenter, not just power and cooling, status quo thinking will continue to deliver overpriced and underperforming datacenters that do not best serve the interests of organizations that are paying for them.