Fri, 19 Apr 2013
data centres, colocation
A series of data centre fires in the US and Canada over the past few years has focused attention on building design and fire suppression systems. However, reading through press clippings surrounding incidents in Michigan and Wisconsin, the running theme in each report is one of ‘disaster recovery planning’ on the part of the data centre, or rather the lack of it.
Whilst disaster recovery is indeed a pertinent question for any data centre, in the world of commercial data centres any off-site duplication is really down to the technical and budgetary allowances of individual clients. The only thing a data centre service provider may do is advise and provide details of off-site options either within it’s own portfolio or as a part of the client’s office-data centre deployment.
Off-site duplication is really down to the technical and budgetary allowances of individual data centre clients.
Most commercial data centre clients will (indeed, all should) maintain periodic backups. Those with real high-availability requirements can be offered multi-site replication or live-live systems, such as ConnetU multi-site layer 2 connected colocation, dedicated server and cloud services. However, it’s fair to say that the expectation for any client of a ‘Tier 3’ designed data centre, is that the DC will not go down. Full stop.
In reality, relying on the primary data centre to maintain it’s uptime by avoiding catastrophic incidents such as fire is not beyond the realms of unfairness for most SMEs. Whilst critical data should always be replicated on a daily basis in some way, it is usually the failure of IT hardware itself and not the data centre environment that is the main concern of IT managers. Which makes the fires in Michigan and Wisconsin all the more interesting subjects for examination.
Most recently, Datacenterknowladge.com reported on a fire in the basement of a 1930’s local government building in Michigan that seems to have spread to take out electrical and telecoms infrastructure. It is not clear whether there were sufficient fire suppression systems within the upper floor data centre itself; in fact, precious little data on the incident has been provided to the press and public. What is apparent is that the building was wholly unfit for purpose; due to its historic nature it had been exempt from having to install any modern fire suppression systems, not even a simple sprinkler system.
Michigan County building dating from the 1930’s had no fire suppression systems installed
A similar complete absence of fire suppression systems led to the loss of 75 devices (servers, switches, routers), in a small commercial data centre in Wisconsin in 2008. Whilst the owner pointed towards the good functioning of his alarm systems, he failed to comment of the fact that there was no automatic fire suppression apparatus installed to quell the outbreak at an earlier stage.
What is common and striking through all these sorry tales is not so much the fact that disaster recovery planning was poor or non-existent but rather that there was no purpose designed fire suppression system able to take care of the primary data hall and / or the building environment as a whole; this latter point being most relevant in the cases of Michigan where having the correct suppression systems in the data halls was immaterial as the building it itself was the primary vulnerability.
Having the correct fire suppression systems in data halls is immaterial when the building it itself is vulnerable.
Although these are extreme cases, they underpin the need to consider suppression systems as a fundamental element in dater centre design and build. If it is possible to control and contain an outbreak of fire and to avoid damage from traditional sprinkler systems, then the percentage chances of requiring a major off-site redeployment are significantly reduced.
This two sided necessity has been a part of data centre designers’ thinking for many years. The answer to the problems is two-fold: firstly, make sure the data hall itself is not placed in a vulnerable building or in any way exposed to traditional fire suppression systems such as sprinklers; in other words, that the building actually has a fire system (unlike Michigan) and can take care of itself in the event of an outbreak, and also that those (e.g. water sprinkler) systems will not adversely affect the data hall when activated.
Secondly, within the data hall, ensure that professional fire suppression systems are installed, including alarm and trigger sensors. Commonly this is Very Early Smoke Detection Apparatus (VESDA) which capable of detecting a single smoke particle at hundreds of monitoring locations on the data floor. This is then coupled with a double (or triple) knock fire alarm which does not automatically engage the fire suppression until multiple sensors are triggered, avoiding a false alarm.
Planners have relied for many years on the use of gas based systems for fighting outbreaks of fire whilst minimising collateral damage to sensitive electrical equipment. These gas based systems are what are commonly found in Tier III facilities such as those used by ConnetU and until recently have tended to utilise heat reduction gasses such as FM-200.
Storage of gaseous fire fighting chemicals such as nitrogen/argon mix or FM-200 (heptaflouropropane)
Gas suppression agents such as FM-200 manufactured by Dupont include heptaflouropropane which works to absorb heat, weakening the fire. However, at high temperatures it also produces an unfriendly mix of hydrogen fluoride, carbon dioxide and carbon monoxide – dangerous to humans. So whilst highly efficient, the downside is that a data hall cannot be safely re-entered for several hours after deployment.
A newer but simpler concept is creating a low oxygen (hypoxic) environment. These mainly use injections of safe inert gases (e.g. nitrogen) to starve the fire of oxygen by reducing to level of oxygen to between 15% and 12.5% in the data hall. The benefits of this mix include the fact there is no chemical residue left after deployment. Furthermore, with oxygen at these levels it is safe for humans to enter the hall at any time to work on damaged or disconnected systems. In fact, some data centres will run a continuous level of low oxygen as a precautionary measure - this prevents a fire from ever starting, which reduces the overall complexity and points of failure of the fire system.
Other more recent alternatives to enter the fire suppression market include water mist or ‘Hi-Fog’ systems which function by creating a mist so fine that it avoids damage to electrical circuits whilst at the same time damping and starving fire of oxygen.
The release fine particle water-mist, or ‘Hi-Fog’
Water-mist systems also have a number of other benefits such as being harmless to humans, being lower cost (as they only use filtered water, not using chemicals which are costly and need to be carefully stored and managed by trained staff) and being able to be deployed locally, to fight a fire in one corner of the data hall without discharging throughout the whole room. All the same, the notion if using water in the data hall feels somewhat counter-intuitive.
ConnetU’s colocation, cloud and dedicated server facility in Byfleet is a good example of a data centre maintaining an independent purpose built structure with an advanced VESDA fire alarm and FM-200 gas in the data hall. Also, ConnetU services running from the new London Bridge facility also provide a similar balance of building independence and advanced VESDA fire systems, including both nitrogen hypoxic in the data hall and FM-200 in the power room.
Fire alarm management panel in the Byfleet data centre
It seems fairly clear that no amount of data hall fire suppression would help the Michigan data hall as is was vulnerable to the state of the building in was were located. Wisconsin, on the other hand, although touting state of the art connected fire alarms, also failed to consider the damage a fire could cause before the fire engines arrived from the local station.
Correct building design and appropriate fire suppression systems are as a critical a part of data centre design as any other aspect, such as power distribution, connectivity or security. ConnetU performs rigorous technical due diligence on any colocation data centre prior to becoming a tenant, and we encourage all data centre customers to do the same.