Running the desk across more than one process

Bicrypto's default production layout is three PM2 apps: backend, frontend and cron. That is already more than one backend process, and this addon cares about that more than any other extension does, because its tick engine holds a single provider socket and its risk engine closes real positions.

The one-line version — exactly one process runs the desk — is in the install guide. This page answers the three questions that follow it: which process is running my engine, what are the others doing, and what happens when the wrong one dies.

The lease

The desk is arbitrated by an engine lease keyed forex-trading. It has two arbiters:

The order matters and it is deliberate. Redis stores only an opaque holder tag, so a lease left behind by a SIGKILLed process is indistinguishable from a live one and the restart would be refused for a full TTL — twice, since nothing polls for promotion. The row carries hostname and pid, so a holder that died on this machine is recognised as dead and reclaimed instantly.

Arbitration is a single conditional UPDATE, never a read-then-write, so MySQL evaluates it under a row lock and exactly one of two racing processes can win. The lease TTL is 20 seconds and the holder renews at a third of that, so the only recurring cost is one UPDATE and one Redis SET roughly every seven seconds, on the leader alone.

What the lease owns

Everything below runs on the lease holder and nowhere else:

Everything not in that list — admin lists, the deals ledger, order and position history, settings — is served from the database and works identically on any process.

What a non-holder can still do

A process that lost the claim keeps serving HTTP. It just cannot quote.

• getSymbolState() returns QUOTING_HALTED for every symbol it has never seen, so opens and closes are refused with "Pricing is temporarily unavailable for this instrument — please retry shortly".
• A withdrawal from an account with open positions is refused with "Pricing is currently unavailable for one of your open positions", because live metrics come back stale.
• The markets rail has nothing to stream.

That is why raising instances on the backend app is not a way to scale this addon. production.config.js ships exec_mode: "fork" with instances: 1; put the backend into PM2 cluster mode with more than one instance and every request that lands on a non-holding instance behaves as above, intermittently, for a fraction of your customers.

Granularity is processes, not JS realms

Every worker thread of one process is treated as the same lease holder. The Redis value is hostname:pid and the database row is refreshable by any realm of the owning process, precisely so threads cannot stand each other down one renewal at a time.

So production.thread.config.js — the opt-in threaded web tier — is unaffected by the lease. It is separately protected: server.ts gates the whole forex subsystem on isMainThread, so worker threads never boot the engine at all.

Failure direction

The fail-open case prints a boxed MULTI-PROCESS BACKEND WITH NO WORKING ARBITER banner, but only where a sibling realm provably exists. A single-process install hears nothing from it.

The cron split

production.config.js defines both halves of the split in one file so they cannot disagree: backend gets CRON_MODE=off, cron gets CRON_MODE=only on port 4001. There is no .env variable to set. The one escape hatch is CRON_MODE=inline, which that file reads and honours by dropping the cron app entirely.

The cron process is refused the lease structurally, before any store is consulted. That is not a race it loses; it is a decision made in code, because order placement is served over HTTP by the web process and the desk has to stay where the orders arrive.

Two forex jobs are still declined on the scheduler, and for a reason unrelated to quotes: runFxExecutionReconciler and runFxHedgeMonitor drive an execution provider's broker connection, and two desks over one broker account replay the same fills and lose syncCursor updates. The lease holder runs both itself every 60 seconds instead. Both are silent no-ops on a pure B-book desk, so the refusal only engages once an execution provider is enabled or in-flight external state exists.

The quote mirror

This is the part of the split that costs money if you get it wrong.

FxTickEngine.getLastTick() is a synchronous read of an in-process Map, and it sits at the bottom of every currency conversion in the addon. Two scheduled jobs depend on it:

• processFxSwaps converts each rollover into the account currency.
• reconcileFxAccounts marks equity and used margin the same way.

On a process with no stream that Map is empty, every conversion returns null, and the rollover is simply never charged — then retried hourly, and then expired uncharged after seven days. Silent revenue loss, with nothing failing.

How it works

The leader publishes; the follower polls. Neither half arms itself — the decision is made once, next to the lease.

Both keys carry a 15-minute TTL. The snapshot is capped at 5,000 symbols and truncation is logged loudly, because a dropped symbol is exactly the conversion leg a rollover may be waiting on.

The pin channel exists for one case, and it is a money case. Swap conversion routes through the USD hub, so a EUR-denominated account holding GBP/JPY needs USD/JPY and EUR/USD. On a single process the swap job pins those itself and the next tick settles; on a follower that call would only write to a local set no stream reads, so the request is handed to the leader instead.

Nothing here runs on a single-process install: it holds the lease, it has the stream, and the publisher is armed only where a follower can exist.

The staleness bounds

Every entry carries the leader's receivedAt and the snapshot carries the leader's publishedAt, so age is measured entirely on the leader's clock and skew between two machines cancels out. A tick that cannot be dated is never stored.

The open-session bound is deliberately not fxTradingQuoteStaleSecondsFx. That setting decides whether a price is tradable; this decides whether a mid is good enough to convert a fee, which is a far weaker question. Reusing the trading threshold would refuse a correct charge over a 30-second gap on a quiet cross.

When the mirror is down

The follower raises a critical quote-mirror-unavailable alert — but only when the desk actually has open positions or resting orders, because on an idle desk the claim would be untrue. The two causes it names are the two real ones:

1. The web process is not running with CRON_MODE=off, which is the only signal that arms the publisher.
2. The two processes are not sharing the same Redis, so the follower is reading a snapshot nobody wrote.

A second alert, quote-mirror-stale, names the individual symbols refused for age while their session was open.

There is no automatic promotion

A process that loses the claim stays a follower for its entire lifetime. It does not poll, and there is no failover.

That is a decision, not an omission. The forex lease is configured with no stand-down handler, because stopping the tick engine would close the quote stream that the terminal, the order gate and the margin sweep all read — and a demoted-but-still-serving web process would then refuse every open and close for everyone connected to it. Losing a lease mid-flight is logged at error level and an operator has to restart the loser.

If the leader dies, the row lapses after its 20-second TTL, or is reclaimed immediately if the replacement comes up on the same host and the old pid is gone. An ordinary pm2 restart, deploy or graceful stop hands the lease back on the way out, so the replacement takes it cleanly.

Reading the role off a running box

There is no admin screen for the lease. Three places tell you.

The lease row. This is the authoritative answer:

SELECT `id`, `hostname`, `pid`, `expiresAt`
FROM `engine_lease`
WHERE `id` = 'forex-trading';

The Redis key, which should agree with it:

redis-cli GET engine:lease:forex-trading      # -> hostname:pid
redis-cli TTL engine:lease:forex-trading      # -> up to 20

The logs. Everything the engine says is tagged FX; the cron banner is tagged CRON.

Symptoms and what they mean

Checklist for a split or clustered box

• pm2 list shows both backend and cron running. If cron is stopped, nothing scheduled is running at all
• The backend app is instances: 1 — one backend process
• Both processes point at the same Redis and the same MySQL
• engine_lease has a forex-trading row whose hostname/pid is the web process, not the scheduler
• The web process logs fx tick engine started and fx quote mirror publishing every 5s
• The scheduler logs fx quote mirror following the lease holder's snapshot
• No quote-mirror-unavailable alert in the execution alert inbox

Next

• Install and enable — the boot order and the cron jobs
• External (A-book) execution — the venue connection that cannot be shared between processes
• Troubleshooting — symptom-first diagnosis