As shown by the once-in-a-generation winter storms which ravaged much of North America in December, winter weather is creating dangerous conditions for businesses of all types, including data centers and many others. Terms like “bomb cyclone,” “atmospheric river,” and “polar vortex” are now common in weather forecasts, bringing severe winter weather which includes frigid temperatures, high winds, heavy snowfall, flooding, icy roads, and power outages.
With the extreme winter weather often comes extremely low temperatures, power outages, water line breaks, and dangerous road conditions. At its worst, this can bring cities, residences, and businesses to not only a standstill, but result in potentially damaging, dangerous, and expensive situations which may take days or longer to resolve. That means it is more important than ever to ensure your data center has effective business continuity and disaster recovery plans. The data center and related edge computing sites play an important role in supporting operations even during the worst winter weather. Yet even with the best preparation a major blizzard or ice storm can force data center leaders to adapt quickly to address power outages, limited staffing, facility damage and more.
As with any organization, an effective plan starts with fundamentals. Establish a crisis management team and a damage assessment team trained in evaluating the impacts of winter weather. Identify triggers which determine how and when to respond to a winter weather event. Ensure a comprehensive – and well-practiced – site communication process. These are starting points for all types of facilities.
Data centers require some additional focus to ensure readiness before, during and after a winter event. Key measures include:
Create an equipment inventory which includes all servers, routers, network gear, telephony equipment and more at each site. In an emergency, having information at your fingertips is critical – but many organizations do not fully know the details about what equipment is available and where it is located. Include all the details. What is in each rack? What is the recovery strategy if equipment is damaged? What is the strategy if systems go offline? This detailed analysis will be indispensable to informing key decisions.
Ensure the facility is weathertight. A data center can be destroyed by something as simple to correct as a leaky roof. You must do everything possible to keep water out. Key steps include:
- Check seals on windows, doors, and exterior openings throughout the facility, and ensure they can be sealed against high winds.
- Make sure the roof is sound and can withstand a heavy load of snow and ice.
- Use protective coverings like pipe foam to wrap around exposed water-carrying pipes which may not receive radiant heat.
- Identify and repair any cracks in surrounding walls or windows.
- Ensure gutters and storm drains are clear of debris.
Prioritize critical edge computing sites which are often mission-critical for sustaining operations in today’s distributed networks. An enterprise data center may remain operational, but multiple edge sites could be impacted. Focus on the most important sites and ensure there is a plan to protect personnel and technology.
Establish a detailed plan for shifting data, potentially to other regions, to support continuity. Be sure to incorporate the challenges and opportunities offered by edge computing sites.
Protect sensitive equipment, including:
- Cover racks and critical equipment, if possible, to protect it from potential leaks from pipes or the ceiling/roof and wrap all pipes.
- Ensure you have adequate generator capacity to prevent systems from shutting down.
- Test uninterruptible power supplies (UPS) and batteries twice a month prior to severe weather seasons.
- Be certain no water or moisture from condensation can collect under a raised floor,
- If equipment is located at a co-location or managed data center, ask to review the data center’s severe weather procedures
Test all generators, as backup power is critical in the event of a power outage. This equipment requires maintenance to enable it to perform as expected in a crisis. That means ensuring they are filled with clean fuel, and the fuel line and air filters are contaminant-free.
Ensure adequate fuel supplies by engaging a minimum of three fuel vendors so you are covered in the event of a long power outage. There is heavy competition for fuel after a storm, so check your contract to make sure you have an adequate supply.
Develop a plan for a graceful shutdown of equipment, including a checklist of any equipment which may be impacted by a power outage. There is an order for shutting down equipment which needs to be documented so no steps are missed. For example, networks and switches should be at the bottom of the list for shutdown and at the top of the list for bringing devices back online. Devices and applications should be validated before handing over to the business.
Increase data backups. When you know winter weather is approaching, increase the frequency of routine data backups from weekly to several times a week. Winter storms can be unpredictable – some are slow moving but others arise quickly. Don’t be surprised!
Create “battle boxes” with materials needed to respond to the weather event. There are several types:
- Business recovery – includes everything needed to recover the business, including contact lists, business continuity and disaster recovery plans, damage assessment forms, winter weather checklists, emergency supplies and more.
- Individual “go bags” – provide supplies employees will need if they get stuck on the road or have to shelter in place.
- Technology – provide the essentials to help recover critical IT systems.
- Information security – materials to protect data. Bad actors often take advantage of crisis situations to probe for weak spots.
Battle boxes should be created and stored at the designated recovery location. Details on where the box is located and who has access to the box should be documented and stored on a local drive for easy access during a winter weather event.
By following a well-defined process to respond to a chaotic winter weather disaster, you can help your data center get back to work as quickly and efficiently as possible.