SMS (StackX Monitoring System)
SMS is a GPL-licensed Bash script designed to provide a simple, fast, readable local system-monitoring page for the StackX servers at ScalarX.
SMS, short for StackX Monitoring System, generates a static HTML page on the fly, directly from the server, to quickly check the state of essential components: processes, network, DNS, filesystems, storage, memory, system load, databases, cache, SMTP, backups and system updates.
The principle is deliberately simple: no heavy stack, no complex agent, no unnecessary dependency. A Bash script, a few configuration files, local checks, and HTML output immediately usable from an internal page or a protected endpoint.
Main features
– Generates a local HTML monitoring page.
– Clear display of OK, WARNING and ERROR states.
– Logs runs in /home/sx/log/sms/.
– Automatically falls back to /tmp/sx-sms if the main log directory is not accessible.
– Collects detected errors in a dedicated list.
– Reads a StackX configuration file.
– Automatically reads the processes to monitor from /home/sx/data/process2monitor.txt.
– Supports simple integration into a PHP page.
System checks:
– Critical server processes.
– Internet connectivity.
– Local network.
– DNS resolution.
– Mounted filesystems.
– Disk space.
– Inodes.
– Memory.
– load average.
– Latest APT updates.
Service checks:
– MySQL / MariaDB.
– MySQL / MariaDB replication.
– PostgreSQL.
– PostgreSQL replication.
– Memcached.
– Redis.
– Postfix SMTP.
– Elasticsearch / OpenSearch.
– Secured OpenSearch.
– Software RAID.
– ZFS.
Backup checks:
– Recency of database backup logs.
– Presence of SQL / PostgreSQL dumps.
– Detection of the backup completion marker.
– Detection of a backup in progress.
– Checks file backups made with sxfilesbkp / rdiff-backup.
– Distinguishes between a missing module, an unconfigured module, a running backup, a recent backup and a failed backup.
Technical architecture
SMS takes a very direct approach that is compatible with decentralised monitoring:
– Pure Bash.
– Local checks.
– Standard system commands.
– Minimal HTML output.
– Text-file configuration.
– No daemon.
– No database.
– No heavy application dependency.
In particular, the script reads:
/home/sx/.sx
/home/sx/data/process2monitor.txt
/home/sx/data/fs_list.txt
The /home/sx/.sx file contains the variables needed for certain checks, such as SQL or PostgreSQL credentials.
The process2monitor.txt file contains the list of processes to monitor.
Example:
apache2
fail2ban-server
memcached
php-fpm8.2
pure-ftpd
sshd
The fs_list.txt file contains the list of expected mount points.
Example:
/
/datastore
/datadrop/cd101
Usage
SMS can be run directly:
./sms.sh
Or integrated into a protected PHP page:
<?php
function checkall()
{
$homecheck = "/home/sx/data";
$checkscript = "sms.sh";
$check = "$homecheck/$checkscript";
system("$check");
}
checkall();
?>
Important note: the script must be placed in a directory that is not directly readable by the web server. The PHP page calls the script on the server side, but the script itself must not be publicly exposed.
Configuration example
Minimal configuration file example:
loginsxmysql:sx
passsxmysql:xxxxxxxx
For PostgreSQL, the script can use:
postgresql_database:sx
postgresql_user:sx
postgresql_password:xxxxxxxx
postgresql_port:5432
For Redis:
redis_password:xxxxxxxx
For OpenSearch:
opensearch_password:xxxxxxxx
opensearch_security_enabled:1
Available checks
Server processes:
SMS uses pgrep to check that every process listed in /home/sx/data/process2monitor.txt is running.
all_process_check
check_process
Network:
SMS can check connectivity to specified IPs, such as 1.1.1.1, 8.8.8.8 or 127.0.0.1.
network_check "Internet" 1.1.1.1 8.8.8.8
network_check "localhost network" 127.0.0.1
DNS:
SMS checks DNS resolution for reference domains.
dns_check google.com cloudflare.com
Filesystems:
SMS checks that the expected mount points are present.
fs_check yes
Storage:
SMS checks disk space and inodes against WARNING / ERROR thresholds.
storage_check 95 90 95 90
In this example:
– Disk space error at 95%.
– Disk space warning at 90%.
– Inode error at 95%.
– Inode warning at 90%.
Memory:
SMS checks memory usage.
mem_check 95 90
Load average:
SMS automatically calculates the number of CPU threads and applies a threshold based on:
THREADS * 4
If the load exceeds the threshold, the script takes a second measurement after a short pause to distinguish high load that is still rising from high load that is falling.
load_check
System updates:
SMS checks the date of the last APT update from /var/log/apt/history.log*.
check_update 10
In this example, a warning is displayed if the last update was more than 10 days ago.
Database checks
MySQL / MariaDB:
SMS checks the SQL connection, access to the sx database, and the ability to list tables.
mysql_check $LOGINSXMYSQL $PASSSXMYSQL 0
If a replication threshold other than 0 is supplied, SMS also checks:
– Last_Errno.
– Seconds_Behind_Master.
– Slave_IO_Running.
– Slave_SQL_Running.
PostgreSQL:
SMS can check:
– Availability through pg_isready.
– Login through psql.
– The node’s role: primary or standby.
postgresql_check 127.0.0.1 5432
PostgreSQL replication:
SMS checks replication according to the node’s role:
– On a standby: WAL receiver status.
– On a primary: the number of streaming standby connections.
– Replication lag if available.
postgresql_replication_check 9600
Application service checks
Memcached:
memcached_check 127.0.0.1 11211
Redis:
SMS first tries redis-cli if available, then falls back to a minimal RESP request through /dev/tcp if necessary.
redis_check 127.0.0.1 6379
The script can distinguish between:
– PONG.
– Authentication required.
– Connection failure.
Postfix SMTP:
smtp_check 127.0.0.1 25
Elasticsearch / OpenSearch:
SMS can check an Elasticsearch / OpenSearch endpoint over HTTP, then read the cluster state.
elastic_open_check 127.0.0.1 9200
States handled:
– HTTP 2xx/3xx : OK.
– HTTP 401/403: WARNING, authentication required.
– Green/yellow cluster: OK.
– Red cluster: ERROR.
– Unknown response: WARNING.
Secured OpenSearch:
SMS also supports OpenSearch with admin authentication and TLS.
opensearch_check 127.0.0.1 9200
If security is enabled but the password is missing, the script reports a WARNING rather than a false ERROR.
Advanced storage checks
Software RAID:
SMS can check /proc/mdstat and detect a degraded RAID array.
raid_check
ZFS:
SMS can check ZFS pool status through zpool status -x.
zfs_check
If zpool is missing, the check returns an error when the module is explicitly enabled.
Backup checks
SMS includes specific checks for StackX backups.
Database backup:
Depending on the database type, SMS checks:
– sxbkpmdb for MySQL / MariaDB.
– sxbkppgdb for PostgreSQL.
The script checks:
– Recency of the latest log.
– Presence of the [INF] Done marker.
– Presence of a recent dump.
– Detection of a backup still in progress.
– Distinction between pending, running, missing and error.
File backup:
SMS checks sxfilesbkp when the module is installed and configured.
In particular, it checks:
– Presence of logs.
– Recency of logs.
– Presence of the [INF] Done marker.
– Presence of the marker for a running backup.
– Valid remote configuration.
Usage:
backup_check 30 54
In this example:
– Warning if the last backup is more than 30 hours old.
– Error if the last backup is more than 54 hours old.
Complete call example
Example sequence used at the bottom of the script:
startpage "StackX Local monitoring"
title "StackX Monitoring System"
table_init
titlecheck "Server processes"
all_process_check
network_check "Internet" 1.1.1.1 8.8.8.8
network_check "localhost network" 127.0.0.1
dns_check google.com cloudflare.com
fs_check yes
storage_check 95 90 95 90
mem_check 95 90
load_check
center_column
mysql_check $LOGINSXMYSQL $PASSSXMYSQL 0
memcached_check 127.0.0.1 11211
smtp_check 127.0.0.1 25
backup_check 30 54
check_update 10
table_end
endpage
log
Some checks are present but disabled by default in the final example, so they can be enabled only when appropriate for the server profile:
#elastic_open_check 127.0.0.1 9200
#opensearch_check 127.0.0.1 9200
#redis_check 127.0.0.1 6379
#postgresql_check 127.0.0.1 5432
#postgresql_replication_check 9600
#raid_check
#zfs_check
Design notes
SMS is designed for the operational needs of StackX : quickly checking a server’s real state, locally, without an unnecessary layer.
The aim is to have a reliable local view that is immediately readable, suited to StackX environments and usable in a decentralised way. Any system in the world, whether built in-house or from a third party (for example Pingdom / Solarwind), can therefore query the output and send an alert according to the returned states (WARNING, ERROR…).
The script favours:
– Simple execution.
– Readability.
– No heavy dependencies.
– Compatibility with heterogeneous servers.
– The ability to work even in constrained environments.
– Easy integration into ScalarX’s internal tools.
SMS takes a pragmatic approach: if a critical service has stopped, if an expected filesystem is not mounted, if a backup has not run, if replication is broken, if memory or storage exceeds the thresholds, that information must appear immediately.
Known limitations
– The HTML output is deliberately minimal and old-school.
– Some checks depend on system commands available locally.
– SQL/PostgreSQL/OpenSearch credentials must be correctly entered in the configuration.
– The script must be protected on the web side and must not be exposed directly.
– The logic is StackX-oriented and is not intended to become a universal, generic monitoring agent.
Todo list
– Add a text CLI mode alongside the HTML output.
– Add a –quick mode to restrict checks to critical elements.
– Add a –full mode to enable all available checks.
– Add finer-grained timeout handling.
– Add concise output usable by cron or an external agent.
– Add stackx-smsd without touching sms.sh, to provide a standalone server and avoid the effects of system updates.
Resources
Script: sms.sh
—
Christophe Casalegno (find all my social networks at: all.bo)