Developing a good disaster recovery strategy is essential for both enterprises and the data centers which serve them. A widely acknowledged benefit of using third-party data centers is their investment in backup UPS power systems and generators. However, with the ultimate goal of helping customers avoid downtime, data center operators have also developed robust operational playbooks for handling natural disasters. A closer examination of those strategies can provide useful lessons for enterprises as well.
It’s important to note the many different kinds of natural disasters to which data centers can be and have been exposed. These can range from earthquakes and wildfires on the West Coast, to hurricanes or floods in the eastern and southern parts of the country.
The most recent example of a wide-scale natural disaster which threatened data center continuity took place in February of 2021 when a major blizzard and unprecedented cold weather descended upon Texas and other southern states. It affected millions of people and businesses with rolling power outages leading to a loss of electrical heating, frozen pipes, and ultimately flooded facilities and water shortages when warm weather returned. Travel became nearly impossible, jeopardizing the movement of critical staff and supplies like fuel, and major cities such as Dallas and Houston came to a virtual standstill.
The data center industry largely held its own. Major data center providers in the region maintained uninterrupted availability of customer IT operations and data throughout the event. This didn’t happen by accident. Data centers in the region had leveraged their systems and activated their playbooks ahead of time, and it paid off.
During the rolling blackouts, UPS/battery and generator systems activated as planned to maintain power until utility power could be restored. Anticipating travel issues, data center operators adjusted shift schedules, moved staff into place from other locations, and arranged for nearby hotel accommodations for staff and customers who needed to be on-site. Some also stockpiled additional food and water supplies in each facility. In addition to topping off fuel in all generators, data center operators arranged for standby refueling and monitored consumption in real-time, diverting deliveries based on usage and sometimes sourcing backup fuel from out-of-state providers.
The data center industry’s response to the Texas blizzard event of 2021 serves as a primer for the industry’s response to – and planning for – such disasters. Among the key elements of the playbook:
- Review the most recent preventive maintenance paperwork on all critical infrastructure.
- Help customers prepare to implement disaster recovery plans.
- Keep customers updated throughout, providing a detailed review of all building maintenance system reports and alarms.
- Monitor weather forecasts consistently.
- Put all operations and engineering staff on standby; re-order shift schedules.
- Test all generators and backup systems.
- Top off fuel tanks and schedule fuel deliveries for generators in advance based on burn rates, or have fuel tanks waiting at the facilities.
- Monitor fuel use in real-time, diverting deliveries based on usage.
- Source back-up fuel from out-of-state providers.
- Put contractors on alert (electricians, HVAC, and critical infrastructure).
- Book hotel rooms nearby for customers and staff who may need to be at the data center.
- Ensure facilities have food and drinks on-site in case travel home or to a hotel is not possible.
- Prepare to leverage other data centers in a metropolitan area as backup. In addition, carve out a “sister city” partnerships to establish a second set of eyes for BMS, security and ticketing systems for quicker reaction time.
- Consider bringing in staff from other facilities to assist.
- Match protocols to weather events: for snow events, alert plowing company and test personal snow blowers. For hurricane events, inspect roofs multiple times prior to the event and inspect ground-level drainage systems before and after the event.
Avoiding Downtime
The events of this past winter demonstrate advance planning for natural disasters and can translate into significant savings. It is estimated downtime costs small businesses $427 per minute and large enterprises as much as $9,000 per minute. Another statistic cited by Atlassian indicates Fortune 1,000 downtime can cost as much as $1 million per hour. Amazon, whose entire business model relies on uptime, estimated costs of downtime are around $13.22 million per hour. The Federal Emergency Management Agency estimates as many as 40 to 60 percent of small businesses never reopen after a disaster.
A data center playbook, as outlined, shows how disaster preparedness planning is critical to both the data center and the enterprise it serves. That playbook must be comprehensive because it must account for the operations, infrastructure, and the people themselves in a comprehensive manner which ensures security, geographic diversity, compliance, and perhaps most important, uptime.
