Last week, we were confronted with an unprecedented heatwave. While we had already experienced significant temperature spikes at the beginning of July, an exceptional heatwave episode required the cooling equipment on two of our Datacenters, DC2 and DC3 to run on its maximum capacity. In addition to the technological and mechanical resources at our disposal, we ensured the smooth running of our infrastructures by rallying an exceptional number of employees.
☀️Nous allons battre ce Mercredi et Jeudi les records de température des 20 dernières années, avec 42C voir plus (sur Dc3).
— 📡 Arnaud de Bermingham (@a_bermingham) July 22, 2019
Nous seront en limite de fonctionnement de nos équipements, mes équipes sont mobilisées avec un dispositif exceptionnel toute la semaine.
A completely transparent communication
With the risk of an incident related to the 42 °C announced by Météo France last week, we decided to provide our customers with a full visibility on the situation.
As a result, Arnaud de Bermingham implemented a real-time communication which was followed up on our Twitter accounts.
Besides, we also encouraged all our customers to carry out, as a preventive measure, an exceptional backup of their data in order to protect them from a potential data loss. In the unlikely event of an incident, that could have ensured a complete data recovery of their activity.
☀️📈Nous recommandons à tous nos clients de s'assurer du bon fonctionnement de leurs sauvegardes et PRA, d'éteindre les serveurs non utilisés ou non provisionnés. Nos équipes sont entièrement mobilisés et des moyens conséquents sont en place. 2/2
— 📡 Arnaud de Bermingham (@a_bermingham) July 23, 2019
Mobilizing employees and securing our Datacenters
In order to control the situation better, several measures were implemented. Indeed, we launched our "high temperature" scheme, consisting of four major actions:
- Activation of our emergency installations to rebalance the operating loads on our refrigeration units.
- Installation of additional automatic and manual sprinkler systems in conjunction with external heat exchangers.
- Implementation of an internal procedure to reduce the load on our Datacenters and ensure the sustainability of our customers' activities.
- Mobilization of the management as well as additional human resources on site.
Furthermore, a detailed monitoring was implemented using our regular monitoring equipment and thermal cameras. That way, we were able to monitor in real time the status of our infrastructures.
☀️ The situation remains stable at the moment. The heatwave is expected to peak around 4.30pm. To follow live information on the heatwave and the state of our services, go to https://t.co/Fhq7NxX4aV. Updates will be published every 30 minutes as of 4.00pm.
— Scaleway (@Scaleway) July 24, 2019
Thursday, 3:30pm, 45 °C at DC3: No customers impacted
After several hours under high temperature, we benefited from the particular design of our Datacenters which includes several cooling units.
Indeed, while noticing the first signs of a performance decrease on the GF3 cooling unit (the unit we relied on since the beginning of the heatwave), we voluntarily switched to GF2 in a totally secure way, ensuring that our infrastructures were maintained in operational conditions.
The close monitoring continued overnight, with regular status updates. Our customers were always informed of our actions, whether preventive or curative, and thanks to the combination of all these efforts - and the collaboration of our customers who followed the situation, there were no incidents neither on the infrastructure nor on the quality of our services.
