PROD, PRA: everything is down!

Friday, March 21, 2025, an afternoon like any other at ScalarX… or almost.
– A client with a dual-datacenter PROD / PRA infrastructure
– Solid servers, dual power supply, dual network feeds, RAID, replication, backup,
then suddenly… nothing.

1) An SSD in the production system RAID has just died.
2) The production failover IP switches over to the PRA in a second datacenter
3) A ticket is opened at OVH to replace the failed SSD
4) OVHcloud proceeds with the replacement
5) The second SSD gives up the ghost
6) The server can no longer boot, no more OS.

At the same time:
1) The PRA stops responding
2) Errors reported by the kernel indicate a hardware problem
3) A hardware analysis is carried out by support: a RAM stick and the CPU are dead
4) A replacement is carried out
5) Corruption caused by the incident requires restoring an earlier database backup, since replication is no longer available.
6) The platform was down for several hours.

But, through an extraordinary combination of circumstances, that is fortunately not what happened. A look back at this story, which ended this morning:

John Benett has been CTO of MYAGA (Make Your Application Great Again), the agency he cofounded, for more than 17 years. Among his clients, Neotypa is a B2B company specializing in selling components used to manufacture and repair drones, robots and connected devices.

John takes care of his clients: they are all hosted on dedicated platforms, cloud and baremetal, each tailored to their specific needs.

These platforms run on StackX, managed by ScalarX, 24/7.

For this client, John rented 2 servers from OVHcloud in prod/pra mode, in 2 different datacenters, each equipped with Intel(R) Xeon(R) Gold 6242R processors, 384 GB of RAM, 2 x SSDs dedicated to the system in RAID 1 and 2 x 1.9 TB NVMe SSDs, also in RAID 1, dedicated to the datastore.

A dedicated IP range is announced in a vRack, which allows the use of a failover IP with no external interaction such as an API call to the hosting provider or anything similar.

In case of an incident, the services can be failed over to the PRA with no particular modifications, without changing IP address and with no dependency on a load-balancer.

There is also an external backup in a 3rd datacenter.

A few days ago, our monitoring detected a hardware incident in production: an SSD in the system RAID 1 had failed.

The client’s service is uninterrupted, and we consider several scenarios together.

The following procedure is approved by the MYAGA teams and the end client:

1) Failover of production onto the PRA
2) Replacement of the defective NVMe by OVHcloud
3) Return of production to the original server after tests and validation.

The failover operation is scheduled for a weekend.

Except that… the work is going to be postponed, and fortunately so!

1) Yesterday (Friday), the server used for the PRA stops responding.
2) Error messages are visible via the KVM on the console:

kernel: CMCI storm detected: switching to poll mode
kernel: mce: [Hardware Error]: Machine check events logged

Generally a sign that hardware is beginning to wear out…

3) A reboot is carried out and the server is available again.
4) SQL replication is dead
5) Replication cannot be repaired and needs to be rebuilt
6) While the database is being restored, another crash…
7) A ticket is opened at OVHcloud,
8) OVH confirms problems with the RAM reported via IPMI
9) From experience with this kind of error, we ask for a complete hardware check.
10) Support schedules the work.
11) We generate another emergency backup of production
12) A few hours later, OVH replaces a RAM stick and the CPU
13) The machine is started up again
14) We restart the replication rebuild, then let it catch up.
15) Everything proceeds without incident and everything is operational again

The client had no interruption of services.

What saved prod?

1) It was not the technical specs
2) It was not the expertise of our teams
3) It was not our 24/7 service, available in less than 12 minutes

What saved prod was:

1) A slight delay
2) A good dose of luck

My experience of more than 25 years in this trade: whatever the level of redundancy of your infrastructures and the quality of your processes, there are always situations that, however unlikely, can still happen. I have seen plenty of them. You have to accept that, and also accept that sometimes chance, luck or even intuition also have a role to play.

Resilience, sometimes, is also knowing not to fail over too fast.

Leave a comment