Dear Cloud Riders,
As you probably know, we experienced a major issue on Sunday, June 2, when a fire started in the Vert Galant Business Park in Saint-Ouen-l'Aumône (business park with nearly 350 companies) at the premises of a company near our DC5 datacenter.
The situation remained critical for many hours. Today, fortunately, the impact on our datacenter is relatively minor.
The origin of the fire is not yet known, but one thing is certain, the outcome could have been totally different.
It was in the late morning when a massive explosion was heard, followed by a fire setting off to the headquarters of Cogetrad Industries, an industrial waste treatment company also located in the Business Park. The incineration of solvents and other products processed by the company has led to a rapid increase to hell threatening the surrounding companies, including DC5.
It is important to understand that datacenter operation is relatively noisy and requires reliable energy supplies. For obvious reasons, they are generally confined to business parks or industrial areas properly served by electricity networks, where the PLU (Local Urban Development Plan) is favourable to this type of activity and where land reserves are high.
Enterprises in these areas are therefore exposed to the risks generated by the activities of other businesses located in the same geographical sector.
A notable exception is DC3, located between a residential area and a protected nature park.
We've already had a similar experience with DC2 several years ago, when a neighbouring company, specialised in paper recycling, suffered a fairly similar fire. An incident in which we have gained a lot of experience and capitalized it on documentary procedures in the form of REX (Return of EXperience).
DC5 is atypical: the cooling of the datacenter is based on innovative technologies that massively uses outside air, unlike the usual hot air recycling techniques. This specificity makes the site much more sensitive than any other datacenter to external pollutants and smoke risks.
Built to target hyperscalers wishing to "implement hosting regions in France", DC5 extends over 12,000m2, the available electrical power reaches 22MW. Securing such a site by all possible measures was therefore a matter of course for Arnaud de Bermingham, CEO of Scaleway.
An important attention was therefore paid, from the design stage, to the datacenter security by using the experience accumulated with the DC2 REX to safeguard against these risks, which are increased by the cooling technique used.
One can say that these anticipatory measures played their role well when you consider that the firewalls lasted nearly ten hours.
During the incident, Laurent Uber, in charge of design and maintenance of our datacenters, was directly contacted by the director of DC5, himself informed by the SSIAP* security agent who sounded the alert. While on a mission in Sweden, he coordinated the actions synchronized with Arnaud, who arrived on-site shortly afterwards.
"Our priority has been to secure people, assets and equipments. Fortunately, our SSIAP security agent was on-site. Very focused on the instructions to be given, we, together with Arnaud, ordered the implementation of the safety protocol: shutdown of air treatment units, closure of fire dampers, inhibition of free-cooling modes on air conditioning equipment, forcing of adiabatic mode in computer rooms to cool and wash away air potentially polluted by smoke, forced opening of the entrance gates for fire engines, indication of fire hydrants on site and closure of the retention tank outlet valve to contain incendiary water. A decision was then taken, in coordination with the fire brigade, to secure the generators and fuel pipes, located only a few metres from the fire."
The close collaboration of our teams and the fire brigade commander made it possible to maintain the operational continuity of the site and avoid the interruption of the high voltage, thus making DC5 failsafe.
Whilst informing our customers in real time of the situation with regular photos of the situation via Twitter, the employees on site did everything possible to facilitate access to the fire brigade and inform them of the characteristics of the datacenter and fire suppression systems.
"The most life-saving elements were the more than 3-metre high firewall built on the property line last summer to contain the fire in our neighbours premises, as well as the heavy firefighters's lane and fire hydrants designed for DC5. This engineering decision was of paramount importance in preserving the site. The firefighters were able to attack for many hours in the heart of the blaze, protected from flames and smoke by our firewalls. The different degraded operating modes of our HVAC installations, the direction of ventilation, considered at the time of construction for an easy handling of the operational teams in case of emergency (recycling of hot air and adiabatic cooling), also made it possible to repel toxic fumes trying to enter the datacenter. The whole system has been very effective."
The main risk that still persists today are the chemicals in the smoke. This risk is particularly present when it comes from chlorinated solvents "We sent a specialized company the next morning to take air samples throughout the building to determine if toxic fumes have infiltrated the building at the start of the fire. The risk is relatively low because we blocked all air inlets relatively early." reassures Laurent Uber, Director of the Datacenter Division.
Following this incident, DC5 employees were put off work, due to the hazard related to the toxicity of smoke persisting. Thus, local technical support and physical operations were suspended for a few days. Assessments are still in progress to measure the real impact of the accident.
Coverage: source Grand Paris
*SSIAP: The SSIAP (Service de Sécurité Incendie et d'Assistance à Personnes) is a compulsory diploma course in France. The SSIAP includes various training courses concerning establishments receiving the public (ERP) and high-rise buildings (IGH).