SxFilesBkp (ScalarX File Backup)
SxFilesBkp is a GPL-licensed Bash script designed to make incremental, versioned file backups with rdiff-backup, to a remote destination over SSH. It is designed for production use on Linux servers (particularly the StackX fleet), with a pragmatic approach: robust, traceable and operable.
It is the “natural” evolution of sxbackup and brainbackup, which I wrote during a series of live coding sessions a few years ago on my YouTube channel. I took database backups out to put them in a separate module, which I will also publish.
The script uses rdiff-backup as a reliable “engine”: it backs up a list of directories, maintains an incremental history, then applies a retention policy (30D by default). It adds the basics expected in production: protection against concurrent runs, timestamped logs, input validation and load-limiting options (nice/ionice).
Aim and philosophy
SxFilesBkp aims for a simple, effective balance:
Local source : an explicit list of directories to back up (.dirs file).
Remote destination : a backup server accessible over SSH.
History : increments retained, browsable and purged according to the retention policy.
Operations : systematic logging, strict locking, explicit errors.
The destination format is deliberately stable and structured:
${REMOTE_USER}@${REMOTE_HOST}::${REMOTE_ROOT}/${REMOTE_TAG}/${REL_PATH}
This makes it possible to neatly separate backups by server (REMOTE_TAG) and directory (REL_PATH).
Main features
Safe execution in production:
Running as root is mandatory.
Protection against concurrent runs reinforced through lockfile + flock (checks for symlinks, type and root ownership, umask 077).
Logs systematically written to /var/log/sxfilesbkp/ with stdout/stderr mirroring (tee) and timestamps.
Remote incremental backup with rdiff-backup:
– Automatically supports “modern” rdiff-backup syntax (detects actions through “rdiff-backup backup –help”).
– Falls back to compatibility with the older CLI (“sxbackup-like” mode) if necessary.
– Optional support for –remote-schema to force the way rdiff invokes the remote server.
Example: “ssh -C {h} rdiff-backup –new server”
Retention and purging:
– Configurable retention policy: RETENTION_REMOTE (e.g. 30D, 12W, 6M, 1Y).
– Automatically purges increments after backup (can be disabled).
– Mode: purge-only.
– Pragmatic handling of certain non-fatal return codes (e.g. rc=2 is treated as a warning when removing increments in some cases).
Exclusions and housekeeping:
– Other filesystems can be excluded (–exclude-other-filesystems).
– Special files can be excluded (–exclude-special-files).
– Optional exclusion of symlinks.
– Optional glob-based exclusions from a file (rdiff-backup patterns).
– Automatically excludes directories containing a .nosxbackup marker (configurable).
Load control:
– Optional wrapping with nice (configurable level).
– Optional wrapping with ionice (configurable class/level).
Technical architecture
1) Declarative configuration
– A sxfilesbkp.cfg file in “KEY:VALUE” format (with trimming and comment removal).
– A sxfilesbkp.dirs file listing source directories (one per line).
– Optional: a glob-based exclusions file (rdiff-backup).
2) Path normalisation
Each source directory is converted to a stable relative path (REL_PATH).
“/” is protected by default (ALLOW_SOURCE_ROOT:no) to prevent a costly operational mistake.
3) Stable remote destination
The destination is built in this form:
user@host::root/tag/rel
This structure (ROOT/TAG/REL) makes it possible to:
– share a datastore on the backup server,
– separate backups by machine,
– avoid collisions.
4) Execution
– For each directory: incremental backup.
– Then (if enabled): purge increments older than RETENTION_REMOTE.
Usage
Typical directory structure:
/etc/sx/sxfilesbkp/
sxfilesbkp.cfg
sxfilesbkp.dirs
sxfilesbkp.excludes (optionnel)
Initialisation (backup + purge according to retention):
sxfilesbkp
Dry-run mode:
sxfilesbkp --dry-run
Disable purging:
sxfilesbkp --no-purge
Purge only (without backup):
sxfilesbkp --purge-only
Test remote compatibility (before backup):
sxfilesbkp --test
Change configuration paths:
sxfilesbkp -c /etc/sx/sxfilesbkp/sxfilesbkp.cfg -l /etc/sx/sxfilesbkp/sxfilesbkp.dirs
Logs:
ls -latr /var/log/sxfilesbkp/
tail -f /var/log/sxfilesbkp/sxfilesbkp-YYYYMMDD-HHMMSS.log
Note: the script must be run with root privileges, because root is life.
Configuration (example)
sxfilesbkp.cfg (KEY:VALUE format):
REMOTE_HOST:backup_server
REMOTE_USER:server_username
REMOTE_ROOT:/datastore/servers
REMOTE_TAG:server_dirname
RETENTION_REMOTE:30D
VERBOSITY:3
PRINT_STATS:yes
CREATE_FULL_PATH:yes
EXCLUDE_OTHER_FILESYSTEMS:yes
EXCLUDE_SPECIAL_FILES:yes
EXCLUDE_SYMLINKS:no
#EXCLUDE_GLOBS_FILE:/etc/sx/sxfilesbkp/sxfilesbkp.excludes
EXCLUDE_IF_PRESENT:.nosxbackup
USE_NICE:yes
NICE_LEVEL:10
USE_IONICE:yes
IONICE_CLASS:2
IONICE_LEVEL:7
ALLOW_SOURCE_ROOT:no
sxfilesbkp.dirs :
/boot
/etc
/home
/opt
/root
/run
/srv
/usr
/var
sxfilesbkp.excludes (optional, rdiff-backup globbing):
/cache/
/.cache/
/node_modules/
/.npm/
/.composer/cache/
/tmp/
Design notes
SxFilesBkp is deliberately “boring”: little magic and plenty of safeguards.
– Configuration parsing avoids eval and rejects invalid keys.
– The lock is treated as an attack surface (symlink, owner, type, permissions).
– The directory-list approach avoids surprises and makes auditing easier.
– The REMOTE_SCHEMA option caters for heterogeneous environments (old/new rdiff-backup, wrapped remote server, etc.).
Todo list
– Strict, explicit validation of the remote configuration (sanity check: user/host/root/tag).
– Optional “single dir” mode (one-off override of the list without editing the .dirs file).
– More readable statistics at the end of a run (summary by directory).
– Encryption-at-rest option on the datastore side (if the backend supports it), with accompanying operational documentation.
Resources
The script + example configuration files: sxfilesbkp.tar.gz
—
Christophe Casalegno (find all my social networks at: all.bo)