Preparing for natural disasters and climate change
By Jason Hood, Roxtec International AB
With recent reports indicating an increased threat of extreme weather-related events and rising sea levels due to climate change, critical infrastructure is expected to become more threatened by water ingress and flooding. For communication infrastructure, including data centers, it is critical to maintain operational reliability and reduce the risk of downtime. Specifically, data centers must remain operational as downtime caused by any failure is expensive and uptime iscritical.
Flooding and water ingress remain a threat in extreme weather-related events, but even in less extreme circumstances, water can be a nuisance and lead to longer-term failures or decreased operational reliability. Certain design considerations can be followed that will mitigate water ingress and reduce the risk of downtime. With projected climate change effects likely, adopting effective sealing designs of openings in data centers as the best practice could effectively mitigate risk and should be considered in data center hardeningstrategies.
From a 2014 data center industry survey conducted by the Uptime Institute, 7 percent of enterprise data centers experienced 5 or more outages within the previous year.
Downtime in data centers
Downtime is a common concern for owners and operators of data centers. Although many precautions are taken, downtime still happens. The 2014 Data Center Industry Survey, conducted by the Uptime Institute, reported that between 25 percent and 46 percent of global data center operators and IT practitioners had experienced an outage thatyear.
The same survey found that 3 to 7 percent had experienced an outage 5 times ormore.
Not only does downtime occur, it is also costly. The 2016 Cost of Data Center Outages, by the Ponemon Institute, reported that the average data center outage costs $740,357. This survey of 63 data centers from multiple sectors (e-commerce, financial, colocation, healthcare, etc.) provides a high estimate for downtime for an average-size (14,090 square feet) data center. Other surveys have reported an average hourly cost of downtime exceeding $300,000. Data center outages happen, and when they do, the costs are substantial. With increasing reliance on technology and data, the value of data will continue to increase, which translates into higher costs for downtime in thefuture.
Examples of water trees in power cable insulation. Water treeing is a micro-crack propagation of degraded insulation.
Causes of downtime
According to the National Survey on Data Center Outages by the Ponemon Institute, uninterruptible power supply (UPS) battery failure (65 percent) was reported to be the most common cause of data center outages, with three of the top four causes UPS-related. Human error was also reported as a common cause ofoutages.
More recently, the 2016 Cost of Data Center Outages by the Ponemon Institute also reported UPS failure as the most common cause of data center outages. The same report also shows that human error is a primarycause.
Both surveys show that human error is a common cause of data center outages, with a value as high as 51 percent. Other sources claim that human error is responsible for 70 percent of data center outages. From this data, it is clear that the most common causes of downtime are human error and mechanical issues. However, weather-related events are also cited as a cause of downtime. Although there are many types of weather-related events, in some areas such as the United States, 90 percent of natural disasters involve flooding and cause more economic damage and loss of life and property than any other naturalhazard.
Unsealed ducts quickly fill with water and debris.
While weather-related events might cause fewer outages, they do occur, and the effect can be significant. The damage caused by Hurricane Sandy in 2012 was extensive, with several data centers in lower Manhattan suffering from outages and subsequently pumping basements, generator rooms, and replacing damaged switchgear to restore power. During this event, internet downtime doubled and took almost four days to recover. Similarly, in 2015, the U.K. was battered with an extreme rainfall event that caused the river Aire to exceed its banks and reach a Vodafone facility in Leeds, causing an outage for several days. While these are extreme examples, flooding might be a more frequent occurrence than commonly believed. For example, in a 2015 survey conducted by Zenium, 60 percent of respondents stated that their data centers were located in low-risk areas, with 40 percent of facilities in the U.K. considered to be flood-resistant. However, this survey also discovered that 1 in 2 data centers had experienced disruption of service due to natural disasters, including seismic activity andflooding.
Even though data center downtime is most commonly associated with mechanical issues or human error, weather-related events and flooding also cause downtime, and appropriate planning should be taken to mitigate possiblerisks.
Site selection
During the site selection process, factors are evaluated such as environment, climate, reliability of power, fiber connectivity, labor pool, and financial impact (taxes, land incentives). While these are all important to evaluate, it is also important to evaluate the risk of natural disasters, considering risks such as seismic activity, extreme weather and flooding. Data center location is arguably the best defense against natural disasters, where avoidance of areas that are prone to natural disasters is the strategy. There are industry standards to help guide site selection to mitigate risk of flooding. For example, the TIA-942 standard series provides guidelines for four resiliencylevels.
Meanwhile, the Uptime Institute defines and describes four tier ratings based on the infrastructure required to sustain operations. Under this classification system, a higher tier level indicates higher site availability, thus more stringent recommendations are warranted for higher-tier-rated data centers. And the TIA-942 standard suggests the most-resilient data center facility be located more than 300 feet from the 100-year flood plain and greater than a half mile from coastal or inlandwaterways.
This graph produced in 2014 by the U.S. Global Change Research Program shows that in the Northeast, heavy rainfall events increased by 71 percent between the study-period years 1958-2012.
For environmental risks, historical data for tornadoes, hurricanes, earthquakes, and flooding can be analyzed to help identify areas prone to natural disasters. Resources such as FEMA, USS, NOAA, European Commission and European Environmental Agency provide helpful information that can be used for this purpose. Historical data, as well predicted trends, can help identify areas that are at high risk of natural disasters. This type of analysis can be critical for ensuring the right location is chosen to mitigate the most risk of naturaldisasters.
Site selection guidelines such as these are a common-sense approach to mitigate risk of natural disasters, but the risk of some level of flooding might still be possible if additional factors aren’t considered. Specifically, site-specific factors such as elevation, slope, and water table should be evaluated as water intrusion and flooding can still occur even outside of a flood zone and under moderate to heavy rainfall periods, where water table levels can become a concern. ANSI/BICSI 002 – 2014 provides good recommendations for mitigating risks of water intrusion due to water table levels. This standard provides several good suggestions for choosing a site with a low water table and points out potential issues with locating data centers in low-lying areas where water table levels and groundwater can become aconcern.
Important to note is that sections 5.7.1.6.3 (electrical) and 5.7.2.4.2 (communications) both recommend service entries to be underground. While this will mitigate risks of damage to overhead lines, underground distribution is not free from threats. The effects of heavy rainfall or a high water table can be exacerbated based on specific locations of vaults, ducts and electrical equipment. ANSI/BICSI 002 recognizes this and points out that utility ducts should be above the water table and to determine if utility maintenance holes can cause water ingress based on theirlocation.
These standards and guidelines highlight the importance of a water migration strategy for data centers. Placing data centers in low-risk areas is a great approach to risk mitigation due to natural disasters, but even under less onerous conditions, water can be a threat that data center owners, designers and operators must contend with to mitigate risk ofdowntime.
The real threat: Water
With proper site selection and design considerations, the risk of inundation by floodwaters can be minimized. However, water issues can exist even without extreme flooding conditions. Under moderate rainfall events, flooding of the data center might not occur, yet underground fiber and power distribution ducts and vaults can fill with water. Water inside vaults, while a nuisance (water must be pumped out for maintenance), can also lead to high humidity levels and pose a threat to power distribution systems. For example, common failure modes of switchgear include excessive temperature, partial discharge and humidity. Humidity can also increase partial discharge and lead to bushing failures and long-term insulation damage. IEEE Standard 493-1997 IEEE Recommended Practice for Design of Reliable Industrial and Commercial Power Systems, also documents the leading causes of switchgear failure from data collected through end-user surveys. Appendix E from this survey reports that the leading contributing cause to switchgear bus failure was exposure to moisture (30 percent) for insulatedbus.
While the exposure to moisture might not directly cause a failure, it facilitates deterioration of the insulation system, leading to a failure. This survey also shows that the “Exposure to Dust or Other Contaminants” was the second-leading contributor to insulated bus failure. This is important to note as humidity in the presence of contaminants can also increase partialdischarge.
Rooftop penetration protected against water ingress.
The most common sources of humidity in substations are ambient air with high humidity, water leaks into substations, and water in cable trenches. Water intrusion can cause instant issues such as short circuits. However, the effects of water and moisture can also be longer-term, where the result can be insulation damage, corrosion, cable and equipment failure. One of the most commonly documented examples of longer-term insulation damage from water on medium voltage cables is water treeing, which is a micro-crack propagation of degraded insulation. These micro-cracks grow from stress points in the presence of water and can eventually lead to cable failure. These stress points are usually caused from manufacturing, transportation, pulling cables or service ofcables.
While typical duct designs can ensure proper slope to direct water away from buildings and equipment, and vaults can be located above the water table, moisture can still be present inside vaults or ducts leading to and away from generators, switchgear, load banks and transformers. This moisture can cause long-term damage by facilitating insulation breakdown. Using cables optimized for these environments can reduce the risk, but not all cable failures are due to a breakdown in cable insulation. Splices, terminations and joints are also a potential weak link as poor workmanship can lead to water ingress. The effects of water might not be immediate, and longer term, water in and around power distribution ducts and equipment can lead to premature cablefailure.
Not only are power cables at risk; underground fiber-optic cables are also threatened by moisture. While locating fiber-optic cables underground removes inherent dangers of aerial placement, underground locations can be subjected to constant exposure to water inside ducts and vaults. The effects of moisture on fiber-optic cables are well documented. Some of the effects are signal attenuation due to water molecules embedding in micro-cracks, corrosion of connectors, signal loss and mechanical damage due to freezing. The effects of water can be minimized by using the correct type of cable. Typically, outdoor cables are designed to minimize water penetration with the use of gel-filled tubes or water-swellable materials and are durable for harsh environments. As is the case with power cables, even with proper cable selection, the connections can be the weakest link. Using the right materials can minimize damage from moisture, but preventing water ingress and minimizing moisture where possible will provide the optimum protection of critical fiber-opticinfrastructure.
Climate change
Even if proper site selection is achieved, will it be enough for the future? Climate change seems to be changing the rules and could potentially be the biggest threat to infrastructure. For example, in the U.K., increased frequency of coastal, fluvial or pluvial flooding is expected to damage key ICT assets such as cables, masts, pylons, data centers, telephone exchanges, base stations or switching centers and increased flooding from all sources is the most significant risk to infrastructure. One of the most striking assessments highlighting the potential impact of climate change in the U.K. is a report by AEA Technology compiled for DEFRA, the U.K.’s Department of Environment, Food and Rural Affairs. The report, Adapting the ICT Sectors to the Impacts of Climate Change, provides information related to underground and above-ground infrastructure. The following points summarize some of the potential impacts offlooding.
- Elements of infrastructure below ground are vulnerable to flooding, rising water tables, wateringress.
- Above-ground risks include precipitation, unstable ground andhumidity.
- Flooding is a risk in conduits; low-lying areas are at increased risk of flooding, as are access holes and undergroundfacilities.
In the U.S., the report Climate Change Impacts in the United States: The Third National Climate Assessment, the U.S. Global Change Research Program reports similarinformation.
Here are some of the key findings from thereport.
- Infrastructure is being damaged by sea-level rise, heavy downpours and extreme heat; damages are projected to increase with continued climatechange.
- Over the past century, global average sea level has risen by 8inches.
- Since 1992, sea-level rise has been almost twice the rate observed over the lastcentury.
- Sea-level rise combined with coastal storms has increased the risk of erosion, storm-surge damage, and flooding for coastalcommunities.
- Sea level is expected to increase 1 to 4 feet over thiscentury.
- Coastal communities are at the greatest risk of storm surges and rising sea levels. Combined tide levels and rising sea levels is already contributing to chronic flooding in many U.S. cities. Although it’s uncertain to know exactly how much sea level will rise in the future, moderate projections for sea rise predict nearly 490 communities in the U.S. will face chronic inundation by the end of the century. Additional scenarios show the number as 668 communitiesaffected.
Although most of the communities are small, larger cities will be affected including Boston, New York, Miami, San Mateo and Newark. Of course, data centers are already located in these cities and undoubtedly, some could be at risk offlooding.
A recent paper, Lights Out: Climate Change Risk to Internet Infrastructure predicts that in 2030, with a 1-foot rise in sea level, 235 data centers will be affected, as well as 771 points of presence (PoP), 53 landing stations, and 42 internet exchange points (IXP).
Following are some key points from thispaper.
- Sea-level rise is projected to be 1 to 6 feet by the year2100.
- Under the most modest projection, 4,067 miles of fiber conduit will be underwater.
- Internet infrastructure is designed to be weather- and water-resistant. They are not designed to be surrounded by or underwater.
- Risks include physical damage at landing stations, physical damage via tidal inundations, and corrosion leading to signalloss.
- Buried conduits will become
submerged. - Much of the infrastructure was deployed over the past 20 years and is aging, meaning that all seals and claddings are likely to be vulnerable to damage if they are underwater.
For new data centers, choosing to build outside of coastal areas seems like a safer option, but this option is not risk-free. Heavy rainfall events are increasing, which can also increase the risk of flooding. In the U.S., heavy downpours have increased in certain areas by as much as 71percent.
Part of the reason for an increase in heavy downpours can be attributed to a warmer climate. For every 1 degree Celsius, there is 7 percent more moisture in the air. Based on the predicted global temperature rise of 3 to 5 degrees, heavy downpours could become even more frequent in the future, causing localized flooding and changing floodboundaries.
Even with reports such as these, in a recent survey of 867 data center operators and IT practitioners, only 14 percent reported they were taking climate change into consideration and “re-valuating site selection based on higher temperatures, increased flooding, or water scarcity.” In the same survey, only 11 percent reported they are taking steps to mitigate increased flood risk. Although these numbers are low, they show that the threat of climate change and flooding is starting to be recognized and that some data center operators will implement a proactive water mitigationstrategy.
Water mitigation
Standards, such as ANSI/BICSI 002-2014 and the TIA-942 standards series, do provide some guidelines for mitigating the risk of water ingress. TIA-942 recommends a floor drain be placed in areas where risk of water ingress exists. It also states, “The data center and all support equipment should be located above the highest expected floodwater levels. No critical electronic, mechanical or electrical equipment should be located in basement levels.” In practical terms, this isn’t always the case. Even if equipment is at ground level, feeders often enter substations below ground and can become a pathway for water and humidity. In these areas, it is common to use pumps and dehumidifiers to remove water and humidity. These are all good design practices, but an often-overlooked area is the utility ducts and distribution vaults as a water and humiditysource.
Commonly, specifications will point to some type of sealant for ducts, but many times, there is nothing installed and ducts are leftopen.
In this scenario, ducts, vaults and maintenance holes fill with water and debris, where cables and equipment can be exposed to high levels of water and humidity. This is the highest-risk scenario and is very common when construction specifications do not include details for sealing ducts and buildingentries.
In many cases, after a problem with water ingress has been discovered, some type of maintenance procedure is established to solve theproblem.
On-site remedies vary, but most commonly, foam or silicone is used. While these remedies will provide some level of protection, they are not effective long-term at stopping water pressure that builds up behind theseal.
The Ponemon Institute’s National Survey on Data Center Outages cites UPS failures as top causes of data center outages. Human error, water incursion, heat-related causes, and PDU/circuit breaker failure also made the list.
Mechanical sealing solutions that are purpose-built for sealing underground ducts provide the highest level of protection. These solutions are typically a compressed rubber solution with tight tolerances that, when compressed, provide a water-tight seal that will contain a high level of water pressure for long-term reliability. These solutions are designed for life-of-building duration or maintenanceschedule.
Sealing underground power and fiber ducts can be one of the most effective and least costly methods for protecting critical infrastructure from water and humidity. Without solutions that are purpose-designed for stopping water, ducts can become a pathway for water ingress during floodingevents.
Even in less-severe weather conditions, ducts can be a source of humidity, which can affect short- and long-term operational reliability of fiber and powersystems.
Other areas that are potential points of water ingress include cable and pipe penetrations for data center cooling/rooftop penetrations, power systems (generator/load bank, transformer/switchgear) exterior wall, fiber/network room. These are all common points of water ingress that need to be protected. However, all penetrations through the building envelope should be evaluated as a potential leak path and appropriate solution applied to mitigate water ingress into the datacenter.
Downtime in data centers happens. Recent surveys show that downtime is still a concern for owners and that the reasons for downtime vary. Although most downtime occurs due to human error or mechanical issues, water-related issues occur, causing short- and long-term consequences. With proper design and site selection, water-related issues in data centers can be minimized. However, with climate change projections, data center fiber and power infrastructure will become more threatened by flooding and wateringress.
Building away from coastal areas and outside of flood zones makes sense, but might not always be possible. Based on a risk assessment of existing data centers, appropriate hardening measures can be taken to mitigate water ingress and protect power and fiber to ensure long-term operational reliability and uptime. For future data centers, designing and implementing purpose-designed sealing solutions can offer the best long-term protection against the threat of water.u
Jason Hood is global segment manager, infrastructure, for Roxtec International AB.