Hello friends, I hope you are well. It is Sunday, March 28, 2021. It has now been 18 days since an event took place that affected not only its main victim, OVHCloud, but also the direct and indirect clients and partners who were using or operating these infrastructures.
For those who cannot face reading it all, I have also recorded this article as an audio podcast here: https://spreaker.com/episode/44088041
It is also available on YouTube:
Now that things are a little calmer, I am taking the opportunity to write a short post about what happened, and give you a broad account of how I experienced this incident, both operationally and emotionally. I am trying to be as transparent as I possibly can, while respecting my contractual obligations to clients, partners and suppliers.
You may notice a one-hour difference in the « copy/pastes », because I live in Ireland. I have also erased private information from the screenshots, such as anything that could identify a client, etc.
Alerts mainly come through 3 channels: Pushover on our phones, email, and a dedicated alert channel on Telegram.
01:12: The first alert comes in. It concerns a client of MIS, the service for OVHCloud managed infrastructure and servers for which Digital Network is an integrated partner and, in that capacity, handles the commercial and technical side.
It is just a machine that no longer responds to pings. Usually the root cause is something mundane: an electrical, network, or server hardware incident. Nothing serious here: this is a high-availability setup. The client's services remain available, with no service interruption. The process is simple -> quick analysis, email sent to the client to warn him of the incident, a more detailed check on the API side: nothing suggesting a suspension for non-payment, no response from the KVM, the machine does not want to reboot, the on-call technician tries to reach Enterprise support, but in vain.
01:40: A second alert comes in, from the shinken server we inherited, which monitors the « Legacy » MIS infrastructure. That means it only monitors infrastructure labelled « obsolete » that clients have not yet been able or willing to migrate, for all sorts of reasons of their own (I will come back to managing obsolescence another day).
Again, nothing too dramatic: this has been happening repeatedly for a few weeks when OVH's automatic backup of the PCI instances runs on the machine. A ticket is already open about it. We know that the machine comes back afterward as soon as the backup is finished, and the critical legacy infrastructure also has separate external monitoring through pingdom.
02:11: An alert comes in for a DN cloud client (whose infrastructures are built at OVH on bare-metal servers and spread evenly across the Roubaix, Gravelines and Strasbourg sites). I am still up and working, so I tell support I will check on this client.
I notice that the machine is being backed up (vzdump in progress from Proxmox VE to an NFS backup server), and the backup seems to be struggling. Apparently the connection to the storage server is in trouble. I decide to stop the task. The machine does not look healthy, so I unlock it and restart it. A quick check: at 02:16 it is up again.
A few minutes later, another DN cloud client goes down, followed by a report that the corresponding backup infrastructure is completely down:
I do a few manual checks and I notice that none of our storage servers on SBG2 is accessible anymore. I decide to stop all backups running at hypervisor level on SBG. (The backup system on the machines themselves, which uses rdiff-backup, does not affect machine performance or availability when the backup destination is unreachable).
02:19: it is starting to get serious. Alerts are starting to pour in by the hundreds.
The OVH maintenance site does show an incident affecting services in 2 rooms at SBG, which matches our problem. Again, no reason to get too worried: I am thinking of a major network or power incident. But we are in the middle of the night, the client impact remains negligible, I stay zen.
Meanwhile, I inform clients of this wider incident through the various dedicated channels.
With nothing more I can do for now, I decide to go to bed. I keep my phone nearby so I can find out when things return to normal and run a few precautionary checks… time passes. It is past 05:00 and still nothing. I decide to run a few checks from bed on my phone and tablet.
And that is when, while I am running diagnostics from outside and from other datacenters, over both the public network and the vRack, the person on call posts this information in a Telegram channel:
Now, for many years I have been used to handling just about any crisis with a level of impassivity that drives other people mad. But usually I am handling other people's crises. They rarely affect me directly, although similar situations have happened to me 3 or 4 times over the past 22 years.
Without a second thought, I jump out of bed in one leap, and I run to my office, where 4 workstations are on 24/7. I sit down in front of my main workstation, and I click first on the NextInpact link. First I look at the date: could it be April 1st? Then, before reading the article, I go straight to the end to check whether this is an early April Fool's joke… Then I go back up and I see the photo of the building, reduced to a carcass that vaguely resembles a plastic toy thrown into a fireplace.
At that precise instant, for perhaps 7 or 8 minutes, I felt myself falter. At 05:36 I write on the channel « No, seriously, this is the end«. To be totally honest, looking at the photos already circulating, I had tears in my eyes. OVH is not my company, but it is a longstanding partner. I had my first server right at the start of OVH. As far as I know, it is one of the 5 servers you can see at the bottom left of this photo. That server would later be shipped to me, but unfortunately I did not keep it.
I go and wake my darling wife, who has also been my business partner since 1999. With slightly misty eyes, I tell her: a datacenter has burned down. At first she thinks I mean a simple « major problem » at a datacenter, so I spell it out: no, I mean it really burned down, there is nothing left but a pile of ashes!
3 or 4 minutes pass, it is time to steady myself, pull myself together and get back into the fight.
Several sources tell me that SBG1 and 2 have reportedly been destroyed (we will discover later that only part of SBG1 was destroyed), that SBG3 was not damaged, but nobody knows what condition the infrastructure inside will be in, given the enormous heat generated by such a fire…
As this photo and others you may have seen online show, the fire was quite something…
At 07:00, in view of the scale of the damage, I decide to activate, without waiting for client approval, all of the PRAs (business recovery plans, even if here it is in reality more a matter of PSI, a PRA extending to all of the company's business activities) for all those who are impacted by the incident and who have this option… For the others, who had services simple enough to benefit from a request-distribution system with an active fallback: they did not even notice the incident…
A few minutes later, the services concerned are up, although one particularly critical client, whose SaaS application is used by tens of thousands of people at banking institutions, will have to take a few additional steps to complete the restoration of service. Excerpt from the reports:
We quickly take stock across Digital Network, ScalarX, and MIS:
– ScalarX side : all the clients concerned had redundant infrastructures, and no backup was lost. ScalarX's management infrastructure is in the London DC, so everything is 100% operational
– DN side: excluding infrastructure covered by a PRA option, about 33% of client production infrastructure is down. For 85% of these 33% backups are available (some of them because of / thanks to a coding error I made in December; I will come back to that). Our internal infrastructures are all operational, none of them was located at SBG anymore.
– MIS side: it is a mixed picture. Some are partially affected, but a few clients have lost both their infrastructure and their backups. Internally, backups have been lost, as has the Legacy infrastructure hosting older clients (temporarily or permanently, I do not yet know; as I write this, we expect an answer within a few days). (But we have up-to-date backups of all data and databases).
– The teams prioritize what can be prioritized. We coordinate with OVH to replace the destroyed infrastructure wherever possible in other datacenters, but the supply chain will soon be in a state of DDOS, faced with the enormous wave of requests.
Tickets and calls rain down from every direction. I take as many DN tickets as I can and deal with them using a simple copy / paste -> this is a widespread incident; it has to be handled as a whole, not individually. I deal directly over Telegram with the more « special » clients, those who need individual attention, while the teams work to stem the wave of requests.
On the DN side, I find myself mainly with two cases:
– The clients for whom we have a backup but who prefer to wait for the restart.
– The clients for whom we have a backup and do not wish to wait.
Something extraordinary is worth mentioning: some clients asked to be « at the bottom of the list«, because they felt their infrastructure was not vital compared with some other clients' activities. That kind of selflessness commands respect.
People outside the trade think you just need a backup and « it all starts again ». In practice, things are more complicated, especially when the client has root access and does things as they please. This is an area I know inside out. I have migrated entire server rooms and moved systems between very different environments: I know how to work very fast.
I leave DN support to keep handling standard tickets, because there are still 2 DCs running, and I switch into flow mode. I quickly regenerate all the infrastructures, then the clients' environments (Thanks bash). For clients whose environments will change, I quickly write bash « drivers » to translate all the mappings and make everything work.
The idea is that the client should have almost nothing to do. When the old access credentials cannot be recovered, the new ones are automatically updated in the code of the clients' sites and applications. Of course, there are technical limits: the incidents mean network connections are not necessarily optimal, and (except for the clients who have dedicated backups), when you request dozens or hundreds of backups from a server at once, it will not deliver the data as fast as when you request just one (as happens under « normal » operating conditions).
In short, everything that can be restarted is restarted. Meanwhile, a few particularly difficult / critical MIS cases are escalated to me: we have backups, but the infrastructure needed to reinstall the clients is unavailable. Some of them will have to find a solution elsewhere if we cannot provide one. I decide to use DN infrastructure to get these clients running.
Small problem: there are fundamental differences between the DN and MIS infrastructures. Software versions aside, the directory structure, permissions, etc. are different. Nothing is the same: the two approaches start from different paradigms. I warn the clients concerned and explain what I can do for them, and what the implications or limitations will be -> they agree.
From there, the same work as above, all thanks to the BASH God: having understood how their systems are organised, and so that clients will not have to learn this new environment, I go back into flow mode : writing systems that regenerate all the client environments. The profiles vary enormously, from 1 site to more than 1500 vhosts with just as many SSL certificates to generate.
Some have the data and database dumps, but have lost all the vhost configurations -> after talking it through, we find solutions for each case: retrieving the vhost name from the database, for example, etc. This gets those clients running until their new or old infrastructure is available again (for the moment, at the time I am writing, they are still in this mode).
(For a moment I thought about recording all this and seeing how I could make a video out of it. But 90% of the time I was handling data I could not show, because it all contained private client information, etc. There is not much point in showing a screen that is permanently blurred.)
After that I decided to devote my time to helping do the same thing for people who were not clients. Making temporary infrastructures available free of charge, helping those who think they have lost everything by giving them recovery leads like those I mentioned in my previous livestream; for those who are interested, it is available here:
I cannot describe every one of those days: that would take a book. There was, in the meantime, this major electrical incident at Gravelines, affecting an entire room, which knocked out clients of mine who were already running on their recovery infrastructure, etc. The biggest problems rarely come alone… In short, for each problem a new solution appears and is put in place.
In parallel OVHcloud starts delivering infrastructure to me. I use it to rebuild infrastructure + backups ready to receive what remains on SBG3 as soon as it powers up. I made that decision fairly quickly: I have been through major datacenter disasters before (electrical incidents, floods, etc.). Although none had ever had this kind of impact (particularly the physical destruction of all the equipment, with no possibility of recovery), experience tells me that after an incident like this, the risks of restarting the infrastructure, plus the network rebuilding problems, the time needed to restore multi-DC vracks, etc., add up to a nightmare in the making. I tell all the clients they will not be restarted on SBG3 when it becomes available again. They will be migrated to new infrastructure at RBX before being brought back up.
It turned out to be the right choice: all the public IPs in my ranges are routed through the vRack, and it still had no external connectivity the last time I checked. I migrate all the clients who are still down, then bring them back up -> moments of great joy, or relief for some: no production was lost.
What I am going to tell you now is what will probably astonish you the most… Apart from the financial and technical consequences for clients: these moments are why I do this job, and I love these moments. So of course I prefer it when I handle them externally, as a provider, but I still took immense pleasure in it. These moments of crisis give you experience that nobody else can acquire without going through the same thing: these are the moments when you surpass yourself, and when you become better, and these are the moments when, yes, I have to admit it: I have the most fun!
So, granted, I spent more than 15 days working 18 to 20 hours a day, once you subtract 2 hours 30 minutes of sleep a day, time moving around the house between kitchen, bathroom and office, time on the phone, etc. At over 45, with a fairly disabling autoimmune disease, I look back on these past 15 days with a smile and a certain pride.
One additional note: I had my biologic treatment on the morning of March 11, 24 hours after the incident, and it usually knocks me out for a few days. I only stopped working for the journey (during which I was still answering clients on Telegram), then got straight back to work when I arrived at the hospital and again as soon as I got home, without flinching. It feels good to know you can still rise to a challenge like this, even one you never agreed to!
Right, I will stop this little account here. I had told myself to stay under 2500 words to keep it digestible, and I see I have passed 2900. See you all soon!
Christophe Casalegno
You can follow me on: Twitter | Facebook | Linkedin | Telegram | Youtube | Twitch | Discord
Leave a comment
Add a comment