Operations

Where the matching engine runs, the five cron jobs the addon registers, the repair scripts for a divergent order book, what your backups do not cover, and what disabling the extension actually stops.

5 min readUpdated 3 August 2026operations, cron, backup, matching-engine, scripts

Running Ecosystem is running an exchange. The parts that need ongoing attention are the engine's placement, five scheduled jobs, gas balances, and backups the platform does not take for you.

Where the matching engine runs

The engine is not a separate service. It boots inside the backend process, and exactly one process in the deployment may own it.

Ownership is decided by a lease keyed ecosystem-matching, held in the engine_lease table and renewed through Redis. A process that loses the race runs a read-only view of the engine — it can price and value, it cannot match.

The placement rule is structural rather than a race, and for a good reason. In the default production layout, PM2 starts backend with CRON_MODE=off alongside a separate cron app with CRON_MODE=only. Order placement is served over HTTP, and placing an order puts it in the serving process's in-memory queue. If the cron process won the lease, the web process could no longer match anything and every order users placed would rest forever. So a cron-only process is refused the lease before any store is consulted — which also means the split stays safe when Redis or MySQL blink at exactly the wrong moment.

Two consequences worth internalising:

The AI Market Maker must run in the same process as the engine. Its bots enqueue into the matcher's in-memory queue, and an order enqueued anywhere else is simply refused. Its placement is derived from the matcher's, not configured separately.
Worker threads of one process share a lease by design. A threaded backend is one holder, not several, so a threaded install does not stop matching the moment the database hiccups.

If the ecosystem extension is disabled while ai_market_maker or trading_bot is enabled, the scheduler logs a refusal banner naming the jobs that will decline to run and why. That combination cannot work in any cron layout, because both engines trade ecosystem markets.

Scheduled jobs

Enabling the extension registers five jobs, visible and controllable at Admin → System → Cron.

Job	Every	What it does
`verifyPendingEcoDeposits`	1 min	Checks confirmation depth on pending deposits in Redis and credits them. The only job that credits a deposit.
`backgroundDepositScanner`	1 min tick	Supervises the rate-limited background sweep of recently-active deposit addresses (72h TTL, per-chain token bucket).
`btcDepositScanner`	1 min	Scans Bitcoin wallets through the configured provider chain.
`ecosystemWithdrawRecon`	5 min	Re-enqueues orphaned `PENDING` withdrawals whose rows outlived the in-memory queue.
`processPendingEcoWithdrawals`	30 min	The legacy pass, running the same recovery on a longer cadence.

A sixth recovery runs once at boot, before the scheduler starts taking new work: it sweeps every PENDING withdrawal with no age filter, so rows orphaned by a previous process lifetime are picked up in order.

The two withdrawal jobs coalesce onto one in-flight pass, because on every 30-minute boundary both fire at once and would otherwise issue duplicate explorer probes against the same stale rows.

The BTC scanner arms its own 60-second interval when it starts, so removing the supervisor tick would not stop it. Both scanners register a teardown that is invoked when the extension is disabled. If you disable Ecosystem and still see deposit crediting in the logs, the process did not pick up the change — restart it.

Repair scripts

Six scripts ship for the situations the product cannot fix from a screen. All of them are dry-run by default; each needs an explicit flag to write.

# Order book divergence — ghost, missing and mismatched price levels
pnpm rebuild:eco-orderbook                 # report
node backend/scripts/rebuild-eco-orderbook.mjs ETH/USDT --execute

# Funds locked in `inOrder` with no matching open order
pnpm reconcile:eco-inorder                 # report
node backend/scripts/reconcile-eco-inorder.mjs --apply

# Orders whose funds were never properly locked
pnpm fix:eco-orders

# Duplicate or discontinuous candles, ecosystem and futures
pnpm fix:eco-candles

# The open-orders index against the order ledger
pnpm eco:index:check                       # verify, non-zero exit on drift
pnpm eco:index:repair                      # backfill and prune

# Accumulated AI market-maker orders resting in the book
pnpm eco:mm:orders                         # survey
node backend/scripts/eco-mm-orders.mjs --apply --keep=200

Three rules that come from the scripts themselves.

Restart the backend after any of them apply changes. The engine holds the order book and the open-order queue in process memory; a repaired table and a stale process disagree immediately.

Run eco-mm-orders with the backend stopped. Cancelling underneath a live engine races its settlement — the engine can be filling an order between the script reading it and cancelling it.

reconcile-eco-inorder only ever releases. It never debits a balance and never raises a hold, so a user with genuinely open orders cannot be over-credited. An under-locked wallet is reported, not silently fixed — that is a different script's job.

eco:index:check is designed to be used as a deployment gate: it exits non-zero when the index and the ledger disagree, which answers "is it right?" with a number rather than an opinion.

Backups

The built-in database backup covers MySQL. So does mysqldump. Orders, candles, the order book, the trade tape, the open-orders index and stop orders live in ScyllaDB and have no backup path in the product. If you run Ecosystem, you own Scylla's backups — nodetool snapshot plus an offsite copy, on a schedule you test.

What each store actually holds, so you can size the risk:

Store	Holds	Losing it costs
MySQL	Wallets, balances, addresses, tokens, markets, ledger, UTXO set, transactions	Everything. This is the money.
`.env`	The encrypted vault key	Every private key on the install, permanently
ScyllaDB	Orders, book, candles, trades	Trading history and resting orders — balances survive
Redis	Pending deposits, caches, locks	Pending deposits in flight; they are re-found on the next address scan

The .env row is the one people get wrong. A database backup without the file that decrypts it is not a backup of a single wallet. Store them separately, and store the passphrase separately again.

Restart semantics

There is no configuration reload. These all require pm2 restart backend:

any <CHAIN>_* variable — provider instances are constructed at module load;
SCYLLA_*, REDIS_*, and the vault variables;
permission grants, because the route gate is held in memory;
anything a repair script changed underneath a running engine.

Custom EVM chains are the exception: their variables are hydrated from the database into process.env at boot and re-applied when the chain registry reloads, so creating or editing one takes effect immediately.

Disabling the extension

Turning Ecosystem off in Admin → System → Extensions is a real stop, not a UI toggle. The matching engine does not boot, the five cron jobs deregister, the deposit scanners tear down, and the addon's routes and admin nav disappear.

What it does not do is move money. Balances stay, addresses stay, resting orders stay in ScyllaDB, and pending withdrawals stay PENDING — they resume when the extension is enabled again, through the same boot-time recovery.

If you are turning it off to stop a runaway, stop the process rather than only disabling the extension, and confirm the logs go quiet before you walk away.

A routine worth having

Daily — open /admin/ecosystem. Look at the coverage counts, the stuck withdrawal count, and any chain flagged degraded.
Weekly — check master and custodial gas balances against the chains that actually see withdrawals; run pnpm eco:index:check.
After every deploy — confirm the vault is unlocked if you do not set the passphrase, and confirm the engine took the lease rather than falling back to read-only.
Before adding a chain — run its diagnostics, and read the readiness rows rather than the green ticks.

Admin console — the screens these numbers appear on
Deposit wallets — the flows these jobs serve
Troubleshooting — symptom-first diagnosis