The typical set up for datacenters to deal with power outages is to have a backup generator take over during such an occasion. This is a simple and effective solution but with a small problem: A generator takes some time to start and reach its operational RPM, time enough for all of the IT equipment to go down. The typical solution for this is to have all of the IT equipment sustained during the generator’s start up by an array of batteries. Again, a very effective and simple solution.
“What if the generator doesn’t start?”
This question has developed an entire new field of facilities management in itself: IT Equipment backup. How long should it be? Long enough for a mobile generator to be brought to the site perhaps, in the event of a failure to start? Many schools of thought were also developed in thriving to achieve cost effective solutions. Low voltage DC current plants for instance require less power conversions than the more traditional AC UPS (uninterruptable Power Supply) since most IT equipment essentially consist of solid state machines operating with DC power. But the lower voltages means that higher electrical currents are required in order to provide the required power. Low voltages are safer to work with but they require more copper to handle the consequently larger currents. More copper means large cables, more complex cable tray structures and ultimately more cost. Higher voltage AC current on the other hand is less costly but yields more inefficiencies due to power being converted to DC so that it can be stored in batteries, and be again converted back to AC to supply the equipment.
Typical specified battery backup time for either type of system varies anywhere from 2 hours up to 8 hours and even longer in more geographically remote systems. A typical 2000Ah battery string will cost between $20k and $30k and require regular maintenance at a cost of about $3k/year. Also, the specified 10 year lifespan is more of a realistic 5 to 6 years due to cycle testing. Larger facilities will often require 10 to 20 of these battery strings.
A typical 2000Ah battery string will cost between $20k and $30k and require regular maintenance at a cost of about $3k/year.
One detail that has been somewhat neglected in all of this is that these IT Equipment battery backup systems do not backup the cooling systems. The reality is that in a typical datacenter, during a power outage where the equipment is supplied by batteries and the cooling system is not running, a critical room temperature of about 40 degrees Celcius will be exceeded in 30-45 minutes. So any battery backup system allowing more capacity than this is basically a wasted investment in initial cost, maintenance and floor space. The fact that initially, these systems were supposed to support the equipment only for a few minutes until the generator has sprung into action has been lost somewhere over time.
So why so much fear that the generator will fail? I had an interesting conversation a few years back with Mr. Jim Tucker who was the senior maintenance management specialist for the government of Nova Scotia. He told me that as the year 2000 approached and the Y2K craze was trending with regards to technology’s potential incapacity to handle it, he was put in charge of an initiative to test the government’s standby infrastructure. They were horrified when the results came in to find that 85% of the generators failed to start! That is enough to raise doubts as to a generator’s reliability. Further investigation however showed that the vast majority of the failures were due to start batteries being not up to par. Another much smaller percentage was due to fuel degradation over time. These accounted for almost all of the failures with a few exceptions that could still have been prevented with a bit of maintenance.
Mr Tucker was subsequently involved in developing the C282 CSA standard which aims at eliminating standby power failures through smart maintenance. The latest standard C282-15 has come a long way in ensuring generator reliability with a minimum amount of sustained testing and data gathering. These large diesel engines are designed to sustain an incredible beating in the field with typical MTBF of 10,000 hours. It is really the inactivity in a standby system that brings them to failure.
In light of all of this, perhaps it would be wiser to invest less in extensive IT equipment backup systems, and focus instead on ensuring the reliability of the initial true backup solution, which can sustain the entire facility including cooling, for as long as necessary. Reduce battery backup time to 10-15 minutes and divert these regained funds toward standby generator maintenance.
If the word “Maintenance” is a problem with upper management in their everlasting battle against operational expenditures, you can always “buy” the CSA C282 maintenance package with the purchase of the generator and pass it on as Capital Expenditures… In 2012, I got a quote from Caterpillar to do just that, which came in at around $18.5K per site per year. This may seem like considerable money. But this is less than one battery string, which typically lasts around 6 years, and which also requires yearly maintenance and testing). So for a site with 4 or 5 battery strings, reducing backup time from 2 hours to 15 minutes for instance would basically cover the generator’s maintenance. Just as significant of a bi-product of this is the floor space regained from the batteries that can now go toward more equipment racks to provide revenue.
Something to think about…
By Claude Morin, Facilities Management and Subject Matter Expert Missing Link Technologies Ltd.
Comments