Revising ScalarCloud’s backup strategies

Hello everyone, over the last 5 years at ScalarX, we have had the opportunity to experiment with different backup strategies for our ScalarCloud cloud infrastructures, each with its advantages and drawbacks.

The current strategy (apart from clients with dedicated, bespoke backup infrastructure) is as follows:

1) Databases are backed up locally. Those backups, along with all files on the system, are then backed up to dedicated storage servers using rdiff-backup, at the frequency and retention period chosen by the client (by default, once every 24 hours with a rolling 30-day retention period)

2) Incremental backups are carried out daily at the hypervisor level via Proxmox Backup Server

For this analysis, we also need to distinguish between two main types of clients: those with a redundant disaster recovery solution (PRA), and those with a single server.

This strategy has proved particularly effective and has met our backup needs perfectly so far.

There are, however, 2 points to note:

1) Moving hypervisor backups to PBS has a side effect: although backups run 10 times faster on average, restoring an entire machine takes between 3 and 10 times as long as with the old vzdump system.

2) When a request to restore files or a database concerns a backup older than the latest one available, the restore time is again considerably longer.

In practice, this has not caused any particular problems so far: 90% of requests involve restoring a small set of files or directories from the latest backup, which is directly accessible, and incidents requiring a full restore are rare.

However, we have more and more clients with profiles involving tens of millions of files on their sites and applications.

As we do regularly (twice a year on average), we tested this strategy against several types of incidents, ranging from a time-delayed cryptobomb to the destruction of a datacenter to determine how long large-scale recovery would take.

The conclusion is that, depending on the nature and scope of the incident, these restore times, especially for all the complete machines in their existing state, can be very (too) long for our liking.

We are therefore working on 2 additional approaches:

An option for clients who want their rdiff-backup backups to use production nvme storage, making restores from backups older than the latest one much faster than they are today.

Reintroducing vzdump backups alongside our current backup procedures to cover the possibility of a major but rare incident. This increases the backup storage required per client.

We are studying the commercial implications of these measures, but it seems we cannot simply fold them into the existing offerings. They will therefore probably be offered as options (nothing has been decided yet; we are still assessing it).

In any case, it will always remain possible to set up backups to and from your own infrastructure / offices at no recurring cost, as some clients already do.

Leave a comment