What to monitor and alert on

The rest of this section tells you how to check things by hand. This page is what to watch when nobody is looking.

Every failure that has cost an operator money on this platform was quiet. The site kept serving, every page loaded, the admin panel stayed green — and the cron app was dead, or Redis had gone, or the licence heartbeat had been firewalled off three days earlier. None of those announce themselves. All of them are one HTTP request or one shell command away from being an alert.

Start with the four signals at the top of the next section. Everything after them is refinement.

The four that matter

The first three need no session and no permission key. The fourth does, which is why it is last.

The external probe: GET /api/health

This is the endpoint to point a load balancer, an uptime monitor or a container orchestrator at. It is unauthenticated, it is cheap enough to poll every few seconds forever, and — unlike every other endpoint on the platform — its status code alone is the answer, so a monitor never has to parse a body to know what to do.

A 200 body carries the detail behind the verdict:

{
  "status": "ok",
  "uptime": 8241,
  "timestamp": "2026-08-12T12:25:47.738Z",
  "checks": {
    "database": { "status": "up", "latency": 1 },
    "redis": { "status": "up", "latency": 1 }
  }
}

• status is ok or degraded. degraded still answers 200, and it means exactly one thing today: Redis is unreachable. That stops scheduled jobs and cross-process cache invalidation, but it does not stop this process serving requests — and since every node shares one Redis, failing the probe on it would drain your whole fleet out of the load balancer for a fault that removing nodes cannot fix. Alert on degraded; do not depower on it.
• The database is the only fatal check. With MySQL gone essentially every other endpoint 500s, so a 503 here is the honest answer rather than a node that stays in rotation serving errors.
• Every probe is bounded (2s for the database, 1s for Redis). The endpoint answers even when a dependency is wedged, which is the case where a naive health check hangs and your monitor blames the network.
• uptime is whole seconds for this process. A number that keeps resetting is a crash loop, which pm2 jlist's restart count confirms.

Cache-Control: no-store is set on every response, so a proxy in front of the platform cannot serve you a stale verdict about liveness.

The internal probe: GET /api/admin/system/health/batch

Permission access.admin. This is what draws the health card on the admin dashboard at /admin, and it is the one endpoint that folds the whole install into a number.

It returns overall: { score, status } plus a services array. Each service is { name, status, message, latency?, critical? } where status is one of up, down, warning or unconfigured.

How the score is built

Start at 100, then for each service:

status is then critical if any critical service is down or the score is under 50, warning under 80, and healthy otherwise.

Two consequences worth building your alert around:

• A single critical service down takes the score to 60 — still above the 50 floor, but criticalDown forces critical anyway. Alert on overall.status, not on the number.
• unconfigured is free. An install with no exchange provider and no mail configured can read 100. The score answers "is anything broken", not "is anything missing".

The per-service probe, and why not to poll it

GET /api/admin/system/health takes a service query parameter and really calls the third party: Stripe's balance endpoint, TransFi's /v3/balance, the SMS provider's own healthCheck(), every configured fiat-rate provider, the EVM RPC and WebSocket endpoints for each chain, and ScyllaDB when ecosystem is enabled. An unknown or missing service returns an empty object.

PM2

production.config.js defines three apps: backend, frontend and cron. pm2 jlist is the machine-readable list; three fields matter.

pm2_env.status is not online

Any of stopped, errored or launching for more than a few seconds is an incident. stopped on backend or cron is the one to page on, because PM2 will not bring it back on its own.

A climbing restart_time on cron

The cron app carries max_memory_restart: "2G" — far below the web backend's ceiling, deliberately, because that is the entire point of running the scheduler in its own process. PM2 measures RSS, so it recycles the app before V8 reaches its heap limit and starts producing multi-second garbage-collection pauses.

One restart is the ceiling working. A restart count that climbs every few hours means a job is leaking, and the platform is losing whatever was mid-flight each time. Alert on the rate, not the value: a delta over a day is the useful signal.

Exit code 78, which no restart can fix

78 is EX_CONFIG from sysexits.h, and every production.*.config.js lists it in stop_exit_codes. PM2 therefore stops the app instead of restart-looping it, leaving the explanation on screen instead of scrolling it away behind sixteen retries.

Three things raise it:

An app in stopped with a non-zero exit code is not a transient failure. It is waiting for a person.

pm2 jlist | node -e 'JSON.parse(require("fs").readFileSync(0)).forEach(p=>console.log(p.name, p.pm2_env.status, p.pm2_env.restart_time))'
pm2 logs cron --lines 100

Scheduler liveness

A dead scheduler is invisible from the website. Every page loads, every other probe is green, and no withdrawal is processed, no price is written, no investment settles, no binary contract expires.

The scheduling process — whichever one registered the jobs — rewrites a single Redis key cron:scheduler every 15 seconds with a 90-second TTL. SCHEDULER_STALE_MS is 60 seconds: a beat older than that is treated as no beat, deliberately shorter than the TTL so the message can say how long it has been silent.

status is one of:

duplicate is reported ahead of staleness because it is worse than having no scheduler at all: BullMQ hands a repeatable job to whichever worker takes it and the single-flight guard is per-process, so every money-moving job runs twice over the same rows.

redis-cli EXISTS cron:scheduler     # 0 = nothing has scheduled in 90 seconds
redis-cli TTL cron:scheduler        # seconds left before it disappears

The scheduler console is the screen behind all of this, and Scheduled jobs lists every job with its cadence.

Queue depth and backlog age

The notification queue

health returns components.emailQueue with waiting, active, completed, failed, delayed and the Redis cache hit rate. queue/stats returns the same counts plus a computed health.status, which flips to degraded when failed > completed × 0.1, and a failureRate percentage.

What to alert on:

• waiting growing and not draining. A queue that is deep but moving is a busy platform; one that is deep and static means the workers are gone — which usually means the same thing as a missing scheduler heartbeat.
• failed climbing at all. These are notifications customers did not get: withdrawal confirmations, KYC decisions, password resets.
• metrics per channel. A successRate collapsing on EMAIL alone is a provider problem, not a platform one.

All three are gated on access.notification.settings, not access.admin, so a role that can read the health card cannot necessarily read these.

Backlog ages, and where the numbers come from

The health probe already computes backlog age against one file, backend/src/utils/sla.ts, which is the single definition of "late" for the whole product:

frontend/config/sla.ts mirrors the file so the queue screens badge rows with the same thresholds the dashboard warns on. If you change one, change both.

The practical alert is the Withdrawal Queue line going to warning: it means real customer money has been sitting in PENDING for a week. See The withdrawal queue.

Host metrics

Three of these are not visible from inside the product at all, and two of them have taken installs down.

Disk

Two directories grow without bound.

• PM2 logs. Nothing in this product installs or configures pm2-logrotate, so PM2 writes to ~/.pm2/logs forever. On a busy install the cron log alone is the largest file on the box within a couple of months. Install the module (pm2 install pm2-logrotate) or rotate the directory yourself.
• frontend/public/uploads. KYC documents, dispute evidence, ticket attachments, avatars and product images. It never shrinks, and it is also the directory your backups have to carry — see Backups.

Alert at 80% of the filesystem. A full disk stops MySQL writes, which stops everything.

MySQL connections

Each backend process opens its own pool: DB_POOL_MAX, default 25, with DB_POOL_MIN 2 and a 30-second acquire timeout. A standard deployment is three apps, two of which are backends (backend and cron), so a default install can hold 50 connections before anything else — against MySQL's own default max_connections of 151.

That is comfortable until you run a second web process, raise DB_POOL_MAX, or point a second host at one database. Watch Threads_connected against max_connections; alert at 80%. Connection exhaustion surfaces as SequelizeConnectionAcquireTimeoutError and looks, from the outside, exactly like a slow site.

Redis

Redis is a hard boot dependency, not a cache. The backend refuses to start without it (exit 78) and it carries the distributed locks that stop a binary order settling twice, the scheduler heartbeat, sessions, rate limits, the cross-process cache invalidation bus and the BullMQ job queues.

Watch two things:

• used_memory against maxmemory. If Redis is configured with a maxmemory and an eviction policy, it will start discarding keys under pressure — and the keys it discards include locks and the heartbeat. Alert on evicted_keys being anything other than zero.
• Availability. Redis disappearing mid-run does not stop the backend: it logs one critical line, repeats once a minute while the outage lasts, and logs again when it clears. pm2 logs backend | grep -i redis is where that lands.

redis-cli INFO stats | grep evicted_keys
redis-cli INFO memory | grep -E 'used_memory_human|maxmemory_human'

Licence reachability

The licence heartbeat runs every 6 hours and the offline grace period is 72 hours. Those two numbers together produce the nastiest delayed failure on the platform: a firewall change that blocks outbound 443 to updates.mashdiv.com produces an install that works perfectly for three days and then starts refusing licensed addons — long after anybody connects the two events.

Probe the egress, not the symptom:

curl -sS -o /dev/null -w '%{http_code}\n' https://updates.mashdiv.com

Alert on anything that is not an HTTP response. Full detail, including what the heartbeat sends and how to recover an install that has already fallen out of grace, is on Licensing.

A minimal alerting recipe

If you have no monitoring stack, this is enough to catch every failure named on this page. It needs curl, redis-cli, pm2 and a mail command.

#!/usr/bin/env bash
# Run from cron every 5 minutes. Prints nothing when healthy.
set -u
ALERT_TO="ops@example.com"
API="http://127.0.0.1:4000/api/settings"
APPS="backend frontend cron"
problems=""

# 1. Is the API answering? 503 is maintenance, not an outage.
code=$(curl -sS -o /dev/null -w '%{http_code}' --max-time 10 "$API" || echo 000)
case "$code" in
  200) ;;
  503) ;;  # maintenance mode - remove this line to be told about updates
  *)   problems="${problems}API returned ${code}\n" ;;
esac

# 2. Is every PM2 app online?
for app in $APPS; do
  state=$(pm2 jlist | node -e "
    const l=JSON.parse(require('fs').readFileSync(0));
    const p=l.find(x=>x.name==='$app');
    console.log(p?p.pm2_env.status:'absent');")
  [ "$state" = "online" ] || problems="${problems}PM2 ${app} is ${state}\n"
done

# 3. Is anything scheduling? The key has a 90-second TTL.
beat=$(redis-cli EXISTS cron:scheduler 2>/dev/null | tr -d ' \r')
[ "$beat" = "1" ] || \
  problems="${problems}NO scheduler heartbeat - nothing scheduled is running\n"

# 4. Disk.
use=$(df -P / | awk 'NR==2 {gsub("%","",$5); print $5}')
[ "$use" -lt 85 ] || problems="${problems}Disk at ${use}%\n"

# 5. Licence egress.
curl -fsS -o /dev/null --max-time 15 https://updates.mashdiv.com || \
  problems="${problems}updates.mashdiv.com unreachable - 72h grace is running\n"

if [ -n "$problems" ]; then
  printf "%b" "$problems" | mail -s "Bicrypto alert on $(hostname)" "$ALERT_TO"
fi

chmod +x /usr/local/bin/bicrypto-watch.sh
crontab -e
# */5 * * * * /usr/local/bin/bicrypto-watch.sh

When an alert fires