Minimizing Downtime in Critical Power Infrastructure

In the data-driven world we live in today, fast and reliable access to information to and from your company’s business system is critical to the survival of your business. Without a reliable infrastructure, your data is at risk. The primary focus of this critical power system design is what will a power outage and the resulting loss of data cost?

Downtime costs money. Mistakes made in the design and implementation of your critical power system can significantly impact your business. Unfortunately, no one standard solution is right for every application; considerations such as reliability and total cost of ownership must be addressed with every new design.

The following information provides a general guideline in the design, implementation, and operation of your critical power system, specific to the uninterruptible power supply (UPS). We will look at four areas of the process: general UPS design, redundancy, design coordination, and system testing and maintenance.

Background

A UPS provides clean and uninterrupted power to your critical load, regardless of the state of the incoming power source. Any power anomaly from the source is filtered through the UPS so it is transparent to your critical load. This includes everything from a complete loss of input to the UPS to sags or spikes coming from the source. In all circumstances, the UPS will correct the anomaly and automatically provide clean, uninterrupted sinusoidal power to the critical load.

The most common failures of any UPS system can be broken down into the following main areas: a failure of the DC source (batteries), improper grounding systems, distribution system faults, poor maintenance practices, improper distribution coordination, or improper operation (human error). None of these failures are captured in the below mean-time-between-failure (MTBF) analysis since none are considered a failure of the UPS.

A typical single module UPS system has an MTBF of about 2.6M hours using the bypass source as an alternate supply for the UPS and about 100,000 hours without factoring in the bypass. The MTBF factors in the reliability of the UPS bypass in conjunction with the UPS normal operation. On average, the UPS will operate more than 10 years without transferring to bypass.

The above MTBF analysis considers only UPS internal failures. For example, a failure in an inverter IGBT gate signal would result in a fluctuation in the inverter output voltage. In this event, the UPS must transfer to bypass in order to maintain the system output voltage. Other examples of internal failures include improper UPS ventilation, resulting in an internal over-temperature fault, improper DC bus voltages, logic failures, etc. Although these types of failures are extremely uncommon, they can occur.

System Redundancy

A typical single module UPS (referred to as an “N” configuration, where N represents the size of the critical load) has an MTBF of 2.6M hours, taking into account the reliability of the internal bypass. In most industries and equipment, this would be acceptable. That is not the case in most critical data centers. Therefore, efforts have been made to provide a redundant UPS design to improve the reliability of the overall backup system (figure 1.)

Figure 1: N configuration for a typical UPS power system without redundancy

One of the more popular configurations with UPS power systems adds one more module than required to support the critical load (“N+1”). In an N+1 configuration, as shown in Figure 2, two or more UPS systems provide power to the critical parallel bus, which feeds the critical load. A failure in one UPS module would allow the UPS to remove itself from the critical bus, while the remaining module, or modules, supports the critical load.

Figure 2: N+1 configuration for a UPS with redundancy

The next step in redundancy utilizes two independent “N” systems to support an “A” side and a “B” side power source for the critical load (figure 3). In this case, a failure of the “A” side system would typically not affect the “B” system. This would be considered a “2N” type system. The critical load should either be a dual-corded power supply system or would need to incorporate a static transfer switch to benefit from both the “A” source and the “B” source.

Figure 3: Two independent “N” systems to support an “A” side and a “B” side power source

Another common configuration, the “2N+1” design, merges the “2N” and the “N+1” designs so that each side (“A” and “B”) has enough modules to support the load, plus one additional module per side as shown in Figure 4.

Figure 4: 2N+1 Design, merging the “2N” and the “N+1”

Other redundant configurations exist. An isolated redundant configuration (figure 5) involves a UPS feeding the critical load while a redundant UPS provides the bypass power to the primary UPS. Similarly, a catcher system, as shown in Figure 6, consists of a quantity of UPS systems feeding independent critical loads, while a redundant system provides a power path to multiple systems in the event one should fail.

Figure 5: An isolated redundant configuration

Figure 6: Catcher configuration

Every application can be unique. How the redundancy is provided, and the configuration used to achieve redundancy, depend on the specific application and the distribution system itself. In addition, each application has limitations and implications. Initial cost, space constraints, single points of failure, operating costs, and maintenance should be considered with each design.

Design Coordination

In general, the most popular specifications associated with UPS systems are efficiency, footprint, weight, communications, and cost. While these are all important specifications when considering a UPS, none address the reliability of the UPS. All of the above specifications relate to the total cost of ownership (TCO) of the UPS, but they certainly do not address generator compatibility, step load responses, harmonic content, or how the UPS responds to distribution faults.

To design a reliable system, it is important to look at the power system in more detail and truly understand the different components. Will the UPS interact correctly with your generator? Will the UPS continue to provide clean power to your distribution transformers during transformer startup? All of these critical questions must be addressed in the design phase. The following is a short list of specifications associated with UPS systems that should be used when designing a reliable system. While in no way a complete list, it offers a deeper insight into the UPS.

• Input current harmonics measured in total harmonic distortion (THD): Every power supply generates current harmonics as a natural response to the AC to DC conversion process. Over time, input current harmonics will cause damage to your distribution system and decrease the reliability of the system. UPS systems should be limited to 3 percent THD or less at 100 percent load, or 5 percent or less at 50 percent load.
• Input leading power factor: To mitigate harmonics, most UPS systems utilize input filters which consist of an inductor and a parallel capacitor bank. Unfortunately, as the UPS load decreases, the capacitors generate more kVAR in relation to the true kW of the system and will produce a leading power factor. If the generator realizes too much leading kVAR, it will shut down. UPS systems should mitigate the input leading power factor to less than 5 percent of the rating of the UPS, specifically at loads less than 10 percent.
• Inverter step load transient response: This specification shows the amount of voltage deviation from the inverter when a step load from 0 percent to as much as 150 percent is applied to the UPS inverter. An immediate in-rush of current from the UPS inverter, or removing the current, tends to cause the UPS inverter voltage to fluctuate. UPS specifications should be limited to +/- 2 percent during step load conditions.
• Dual input sources: Although it sounds simple, a dual input source provides increased reliability to the system and prevents a single breaker failure from disrupting both the UPS converter source and the bypass source.
• DC power supply: As the reliability of the UPS during a power outage directly relates to the reliability of the DC power supply, the specification and runtime for the battery, flywheel, or super capacitors is critical.
• If VRLA batteries are used, they should be configured in a parallel string configuration so an open cell does not eliminate the entire system.
• If batteries are used, the run-time should be based on the battery’s end of life. As the run-time versus battery life is not linear, an 80 percent battery life does not correlate to 80 percent run-time.

As mentioned earlier, the above list represents a small sample of specifications which should be considered when designing the UPS. Each could have a significant impact on the reliability and operation of the UPS in your overall critical power system. The common theme to the above specifications is how the UPS will interact with the other equipment in your electrical system.

Testing, Maintenance, and Training

After the considerable time spent designing and coordinating your critical power system, now comes the installation and implementation of the system. The process is only half over, if that. The design can be solid, but if the implementation fails, the system fails. It is now time to select equipment, verify, and validate the design through testing.

Do not underestimate the importance of testing, as it represents the most important part of implementing the system, and it starts at the manufacturer’s factory. Ideally, this testing would apply to all of the equipment in the critical power system and not just the UPS, including the cooling system, batteries, switchgear, and distribution system equipment. The first step should be a factory demonstration.

At the manufacturer’s facility, you can start to familiarize your team with the equipment and verify the equipment meets your specifications. Consider it a test drive. This is the first training opportunity for the equipment you will be implementing. Your team should also use this time to review the specifications and requirements. Factory witness testing is highly recommended for the following reasons.

1. It is important that you confirm the operating characteristics as defined by the specification for the equipment. These performance specifications were originally provided to ensure the operation of not only the UPS, but also it will perform correctly with the up and downstream equipment. Factory testing is an early test, in a controlled environment, to confirm these specifications and log them for future reference.
2. You can start to build your own performance history log for the equipment. Having good factory test data to use as a comparison in the future will prove priceless in expediting field issue resolutions, as you now have initial test data to use as benchmarks. This should include system settings, software versions, and test results.
3. Testing is the best time to become familiar with the equipment; it is your chance to understand the system and its controls in a controlled environment. This is also a perfect time to continue your operational training of the equipment. Learn how to operate the equipment in the company of the factory engineers and ask questions. Now is the time. Your chosen contractor should also attend the factory testing to confirm installation details and instructions.

After installation, a factory-authorized technician should perform a startup of the system to confirm the system has been installed properly and is ready to be tested at your facility. Startup provides a third data collection point and a third opportunity for equipment training.

After startup, a large portion of the factory testing should be repeated onsite (commissioning) so data from the controlled environment can be compared to data from your real-world application. Commissioning additionally provides a fourth data collection point and the fourth training opportunity for your system operators.

Finally, for the first time, you should test your entire critical power system together as one. All of the data you collected, training you have completed, and hard work come together in this final test, and generally this stage is where you find your problems. Confirm the UPS works with the generator. Confirm the grounding system is installed properly, and the breakers in your distribution system have been coordinated properly. This final test represents the fifth data collection point and the fifth training opportunity for your system operators.

The next step in the process, after you have turned the data center over to the IT department, is to implement the recommended maintenance plan and parts replacement schedule. All of the hard work in designing and testing your data center can be quickly disrupted without proper maintenance. Now you have an excellent recorded history for all of the equipment in your critical power system, continuing on with the maintenance and maintaining that history of your equipment only increases the reliability of the system. The following points demonstrate the importance of performing proper maintenance on your critical power system:

1. performing routine maintenance provides a history that can be trended over time to predict equipment end of life
2. performing routine maintenance helps identify equipment or parts that are not performing as desired before they become a major issue in your power system
3. the manufacturer’s recommended parts replacement schedule can be completed during routine maintenance to improve the reliability of your system
4. changes to the system can be documented and tracked over time
5. maintenance programs provide documentation and validation for any warranty claims

It is important to note the day-to-day operation of your equipment can be performed by your highly trained operators with support from the factory authorized service representatives, but all maintenance should be performed by the factory-authorized service representatives with support from your highly trained operators.

Summary

I mentioned earlier while the most popular specifications for a UPS (efficiency, footprint, weight, communications, and cost) have a large influence on the system total cost of ownership, they are not the only factors in the TCO analysis. Other factors that influence the TCO analysis include:

1. system efficiency
2. cooling requirements
3. footprint and weight
4. initial system investment
5. generator sizing and compatibility
6. distribution system coordination
7. factory support during testing, commissioning, and IST
8. recommended maintenance intervals and associated costs
9. recommended parts replacement programs and associated costs
10. spare parts recommendations
11. mean-time-to repair estimates
12. service support and response
13. the cost of downtime as a result of one failure

The question was previously asked: what will a power outage and the resulting loss of data cost?

The reliability of the overall system is the No. 1 reason you are building a critical power infrastructure. No one solution is right for every application, but there is a common approach which can address the items listed above and lead you to success. Design and coordinate the system, select equipment which best matches your operating requirements and TCO, test the system, and implement a proper maintenance routine. Although each design may differ, the road to success is consistent. Focus on the process, reliability, and consider the overall system total cost of ownership.

John Steele serves as an engineering manager for Mitsubishi Electric Power Products, Inc. in Warrendale, Pa. Steele has been affiliated with Mitsubishi since December 2007. He holds a bachelor’s degree in electrical engineering from West Virginia University.