Reboots, backups and recovering server-run bots

What a server reboot does to supervised Hummingbot processes, why PM2 has to be told to come back, what to back up beyond a database dump, and how to prove your bots are quoting again.

10 min readUpdated 6 August 2026reboot, backup, supervisor, recovery, instances, restore

A reboot is the one event that takes out every layer of this addon at once — the Python processes, PM2, the backend and the supervisor inside it. None of those layers restarts the one above it. PM2 brings the backend back, and only the backend's supervisor brings the bots back, so a missing link at the bottom means your market makers stay down and nothing in the panel will say why, because the panel is down too.

Your users' bots are unaffected by any of this — they run on their own machines and reconnect when your API answers again. This page is about the instances you run yourself under Admin → Hummingbot → Bot Instances.

What a reboot actually kills

Layer	What happens	What brings it back
The Hummingbot processes	Killed with the box. Each runs in its own process group (`detached`), so nothing about them survives a power cycle	The supervisor, from `desiredStatus`
PM2 itself	Killed	The boot hook installed by `pm2 startup`, replaying the list recorded by `pm2 save`
The backend	Killed	PM2
The instance supervisor	Runs inside the backend	Backend boot, main thread only, `hummingbot` extension active
Resting orders on your own book	Nothing cancels them	You do — Command Center, or the bot itself when it comes back

Pressing Stop sends SIGTERM and gives Hummingbot 20 seconds to pull its resting orders off the book before escalating. A reboot gives it whatever the operating system's shutdown timeout happens to be, and a power cut gives it nothing.

So assume a hard restart leaves live quotes on your own market with nothing managing them. If the bots are not going to come back promptly, cancel them from /admin/hb/command — but disable the instance's key or switch on read-only mode first, or a bot that does come back re-quotes over your cancellation within seconds.

The chain, in order

PM2's boot hook fires and resurrects the saved process list. If the list was never saved, this step produces an empty PM2 and the platform stays down.
The backend boots. Redis has to be reachable — it is a hard dependency and the process exits 78 without it, which PM2 does not restart-loop.
The supervisor starts, on the main thread only, and only when the hummingbot extension is active. Its failure is deliberately non-fatal.
The reconciler ticks every two seconds, reads every hb_instance row and converges each one towards its desiredStatus.
Each bot is spawned from scratch — the doctor runs, the controller and script YAML are regenerated from the linked preset, Hummingbot's own credential store is filled in where a Bicrypto connector file is missing, the config password is decrypted, and the market is preflighted against the exchange's list before the process is launched.

The installer runs pm2 startup, which installs the boot hook. It never runs pm2 save, so PM2 comes back after a reboot with nothing to resurrect — no backend, no supervisor, no bots, and no admin panel to notice from.

Run it once, after pnpm start has brought everything up and pm2 list looks right:

pm2 save

Re-run it any time you change what is running. Full detail, including why there is no systemd unit, is in Processes and ports.

Desired state is what recovers a bot

hb_instance.desiredStatus is a durable column holding RUNNING or STOPPED. The Start, Stop and Restart buttons write the row and return; none of them spawns anything. That is what makes recovery automatic — the intent outlives the process, the backend and the box.

Only Stop writes nothing but desiredStatus. Start also clears the crash latch — status goes from CRASHED back to STOPPED — and blanks lastError, which is why an operator-initiated start proceeds even with auto-restart off. Restart does the same and stamps restartRequestedAt, a timestamp rather than a flag, so a request that lands while a fresh process is already coming up does not immediately kill it again. Both Start and Restart refuse with 422 before writing anything if the stored config password can no longer be decrypted.

After a reboot the pid recorded on the row is dead, so the supervisor spawns a fresh process for every row whose desiredStatus is RUNNING.

Two things stop a bot coming back, and both are deliberate:

Situation	Why it stays down	Fix
The row was `CRASHED` when the box went down and Auto-restart is off	The reconciler leaves a crashed instance alone until an operator looks at it, rather than restarting into the same failure forever	Read the card's cause, then press Start
The instance has no strategy preset — its preset was deleted, which sets `presetId` to NULL	Hummingbot launched with no strategy starts happily, burns ~500 MB and never places an order. Refusing is better	Edit the instance, pick a profile and a market

The second refusal reads: "this instance has no strategy preset, so there would be nothing for Hummingbot to run."

A backend restart is different from a reboot

A backend restart — a deploy, pnpm restart, pnpm updator — does not kill the bots. Each child was spawned into its own process group, so it keeps trading while the backend is down. On the next boot the supervisor re-adopts it rather than starting a second copy alongside it:

The pid on the row is checked for life, and on Linux its /proc/<pid>/cmdline must still contain the instance's installPath before it is adopted. Pids are recycled; adopting blindly would let the panel report — and later kill — an unrelated program.
On a non-Linux host the command line cannot be read, so the row's pid is trusted as-is.
The card shows a Re-attached badge, and the log records Re-attached to running process pid <n> after a backend restart.

The supervisor owns the row and can signal the process, but it does not own the pipes it was never handed. Live log capture only resumes after a restart — until then the instance log holds supervisor lines but no new bot output.

After an update, reinstall the connector into every checkout

pnpm updator stops the platform, installs dependencies, migrates, runs pnpm seed, rebuilds the frontend and starts again. It updates the connector packages that ship inside the platform. It does not touch the copies already sitting inside your Hummingbot checkouts, and a checkout keeps running last release's connector until someone replaces it.

Two ways to do that:

python hummingbot/install_connectors.py /path/to/hummingbot

Or from the panel, on the instance card — Reinstall connector:

Runs the shipped installer against this instance's checkout

The installer path is resolved from the platform's own layout and can never be pointed at another script; the checkout path and interpreter are the ones already validated on the instance.
It refuses with 409 while the instance is RUNNING or STARTING — rewriting the package under a live bot would leave it running half-replaced code. Stop it first.
clean wipes the target directory before copying, which is the fix for a nested bicrypto/bicrypto/ package.
It times out after 3 minutes, and every line of installer output is appended to the instance log tagged [installer].

pnpm updator also re-runs the strategy preset seeder — see The Strategy Studio for exactly what that does to presets you have edited or deleted.

The memory watchdog only enforces on Linux

Each instance carries a memoryLimitMb ceiling, default 1536. The watchdog reads a process's resident size from /proc/<pid>/statm, and that path exists only on Linux — everywhere else the reading is null, and null is never read as "within limits".

So on a Windows or macOS host the cap is recorded and never applied. The host memory panel at the top of /admin/hb/instances says so directly, beside the committed/free/total figures it draws from the real host.

The doctor's Memory headroom check is a warn, not a fail: less free memory than the cap allows does not block a start. On a box that also runs MySQL, Redis and — with Ecosystem installed — ScyllaDB, an out-of-memory event can take the exchange down rather than the bot, which is the whole reason the ceiling exists.

What to back up

Restoring MySQL onto a fresh box gives you hb_instance rows describing bots that point at filesystem paths which do not exist, holding a config password that no longer decrypts if .env did not come with it, against a Hummingbot checkout whose encrypted connector credentials are gone.

The rows are the registration. The bot lives in three other places.

What	Where	Why it matters
`hb_strategy_preset`	MySQL	Every preset — family, pair, connectors, the config JSON, status and version
`hb_instance`	MySQL	Paths, interpreter, linked preset and key, market, base URL, the encrypted config password, desired status, restart counters, last exit
`api_key`	MySQL	The keys your bots sign with, including the signing secrets. Nothing can re-issue a secret in place — a lost one means rotation, and rotation means re-provisioning the checkout
`api_key_audit_log`	MySQL	Creation, rotation, permission changes, kill-switch events and every authentication rejection
`.env`	Filesystem	`APP_ACCESS_TOKEN_SECRET` is what `hb_instance.configPassword` is encrypted under
The checkout's `conf/`	Filesystem, on the bot host	See below

`.env` is part of this backup

The Hummingbot config password is encrypted at rest with AES-256-GCM under a key derived from APP_ACCESS_TOKEN_SECRET (PBKDF2, domain-separated so the ciphertext is not interchangeable with anything else that secret protects). It deliberately does not use the ecosystem vault's key — that key only exists once the vault has been unlocked, which would mean your market makers could not start after a reboot until someone logged in and unlocked it.

Rotate or lose APP_ACCESS_TOKEN_SECRET and every stored password becomes undecryptable. The panel detects that and asks you to re-enter it, rather than failing the bot with an obscure error:

stored config password could not be decrypted (APP_ACCESS_TOKEN_SECRET may have
been rotated) — re-enter it on the instance

The checkout's `conf/` directory

File	Regenerated?	Consequence of losing it
`conf/.password_verification`	No	A known word sealed with the config password. It is the only way to detect a wrong password, and it is never overwritten on a mismatch — rewriting it would silently make every saved connector key undecryptable
`conf/connectors/*.yml`	Only the `bicrypto` ones, and only when missing	Encrypted API credentials. At start the supervisor writes a Bicrypto connector file only if it does not already exist — anything already configured is left strictly alone. So a lost file is refilled from the linked key on the next start, but rotating that key changes nothing until you delete the stale `conf/connectors/bicrypto.yml` and start again. An XEMM taker venue — Binance, MEXC — is your own exchange account* and must be re-added with `connect <venue>` inside Hummingbot
`conf/controllers/instance-<id>.yml`	Yes, on every start	Nothing. Regenerated from the preset
`conf/scripts/instance-<id>.yml`	Yes, on every start	Nothing. Regenerated, and it says so in its own header

The checkout itself is not a backup target — re-clone the source and rebuild (./install && conda activate hummingbot && ./compile), then reinstall the connector.

Instance logs live under storage/hb/instances/<instance-id>.log, relative to the backend process's working directory, rotating at 5 MB with one previous file kept. They are crash evidence, not a record you need to preserve, and deleting an instance deletes them.

The platform-wide picture — MySQL, ScyllaDB, .env, uploads and what Redis is not — is in Backup and restore.

Verifying after a reboot

Do these in order. The first three answer "is the exchange back", the rest answer "is my liquidity back".

PM2 has the apps.
```
pm2 list
```
backend and frontend should be online, plus cron unless you run with the inline scheduler opt-in. An empty list means pm2 save was never run — start the platform and run it now.
The addon's surface answers. No authentication needed.
```
curl https://your-domain.com/api/hb/ping
curl https://your-domain.com/api/hb/time
```
ping returns {}. time returns {"serverTime": …} in unix milliseconds — compare it with a known-good clock. Signed requests are refused outside ±10 seconds, so a box that came back with a wrong clock rejects every one of your customers' bots with Request outside recvWindow, and records a rejection on each key's audit trail. Confirm NTP is running before you look at anything else.
The instance cards read Running at /admin/hb/instances. A Re-attached badge means that process outlived the backend rather than being restarted — expected after a deploy, not after a reboot.
The bot is actually quoting. Open the instance terminal at /admin/hb/instances/<id> and confirm a ladder on the book. This is the check that matters: a market maker can start, stay alive, report RUNNING and never place an order — that is what a pair the exchange does not list looks like, and it is why the market is preflighted at start.
Cross-check in the Command Center at /admin/hb/command. An instance key belongs to the admin who minted it, so that account should now appear with open orders against the market it quotes.

If the bots did not come back

pm2 logs backend --lines 200 | grep HB

Line	Means
`[HB] Hummingbot supervisor not started: <reason>`	The supervisor failed to start. Deliberately non-fatal — the exchange boots and serves without it, and nothing else will report the absence
`[HB] Hummingbot instance supervisor started`	It is running; the problem is per-instance. Read the card and press Diagnose
Nothing at all	Either the `hummingbot` extension is not active, or this process is not the main thread. The supervisor is main-thread-only by design

Then, per instance: the card names a probable cause from the exit itself rather than "exited with code 1" — it reads that from the last 200 lines the supervisor keeps in memory for the current run, which is the only thing that 200 bounds. Diagnose tests the real environment right now. The instance log is the whole run, not a tail: the file keeps every line until it rotates at 5 MB, the panel loads the last 300 by default and holds up to 2000 as it streams, and [supervisor]-tagged lines are the panel's own. Symptom-by-symptom fixes are in Troubleshooting.

Never delete a running instance to "clean up"

Deleting an instance whose process is alive is refused with 409:

This instance is still running. Stop it first, or delete with force=true to terminate it — deleting a live bot without stopping it would leave a process nothing can manage.

The panel offers Stop and delete, which sends SIGTERM on the way out. That is best-effort only: once the row is gone there is nothing left to escalate on, no SIGKILL after the grace window, and no record that the process ever existed. A bot with no row is a bot nothing can ever stop, still quoting with real money on your own book.

Stop it, watch the card reach Stopped, then delete.

Where to go next

Bot Instances — registering, provisioning, the doctor, logs and the supervisor's safety model.
The Strategy Studio — what the seeder does on every update, and why editing a published preset changes a running bot.
Troubleshooting — instances that go straight to Crashed, and bots that start but never quote.
Backup and restore — the platform-wide procedure this page's tables plug into.