Ecosystem v6.4.0

Release Date: August 12, 2026 Tags: TRADING, MATCHING-ENGINE, ORDER-BOOK, MARKET-ORDERS, ORDERS, CANCELLATION, SETTLEMENT, MARKET-MAKER, RELIABILITY, BUG-FIXES

Overview

A trading release, and the largest part of it is depth that was not real. An order the ledger had already cancelled or deleted could go on being displayed as resting liquidity nobody could remove — missing from the customer's Open Orders, untouched by Cancel All, and back again after every restart. Separately, the unfilled part of a market order had nowhere to go and stayed in the book at its funding price, on the wrong side of the price it came from: best bid equal to best ask, spread 0.000000, funds locked, and that stale price becoming the reference the next market order was priced from.

Cancelling was the other half. On a deployment where the matching engine runs in a different process from the web server, every cancellation was refused; Cancel All stopped at the first five thousand orders; and an order that failed to cancel was left where no cancel could reach it again.

Read Upgrade Notes before updating — one check tells you whether your markets are affected, and one repair only works in a particular order. Requires Core v6.6.4.

Update Instructions

pnpm updator

Update Core first. It carries the connection, settings and rate-limit repairs these changes sit on.

Restart the backend afterwards. There is no database migration. The restart is what clears stranded order-book depth, because the index repair runs during startup, and it is when a market order left resting by an earlier version is retired and its funds returned. Run the read-only check in Upgrade Notes after the restart — it is the only thing that can confirm a market is clean.

Upgrade Notes

An order book showing depth that nobody can cancel — check yours, and repair in this order

This is the one to read if a customer has ever reported an order they could not get rid of. The symptom is exact: the book shows a price level, the customer's Open Orders does not list it, Cancel All reports success and removes everything except that one, and it is still there after a restart.

What is behind it is not an order. The matching engine loads the orders it works with from its own index of open orders rather than from the ledger, and a row left in that index after its order was cancelled, filled or deleted still describes an open order — so every restart loaded it as live, funded, resting liquidity, and the routine that rebuilds the displayed book then made the price level agree with it. Meanwhile every list a customer sees is built from the ledger, which correctly shows nothing. So the depth was visible to everyone, cancellable by no one, and the periodic sweep that exists to remove unbacked levels was the thing keeping this one alive.

The platform now compares the two at every start — before it loads any order — and rebuilds any market where the index holds rows the ledger has disowned, then drops those orders from its working set so the level loses its backing and is swept in the same startup. It repeats the comparison for one market an hour while it runs, so a row that appears mid-session no longer waits for a restart. Nothing else needs doing on an ordinary install.

Verify it, on a live install, at any time:

pnpm eco:index:check

It reads both tables and writes nothing, and it exits non-zero if they disagree, so it also works as a deployment gate. A market reported with STALE= is one that was carrying this. Clean installs print that the index matches the ledger.

If a market still shows unbacked depth after updating, repair it in this order:

pnpm eco:index:repair                                         # 1. the order index
# 2. restart the backend
node backend/scripts/rebuild-eco-orderbook.mjs BTC/USDT       # 3. report only
node backend/scripts/rebuild-eco-orderbook.mjs BTC/USDT --execute

• Fixed cancelled, filled and deleted orders being reloaded from the order index at every restart as live resting liquidity, and the displayed book being rebuilt to match them.
• The order matters, and reversing it looks like the repair failed. The book-rebuild tool recomputes levels from the ledger, which is the right authority — but the running engine rebuilds the book from the index it loaded at startup, so repairing the book first is undone at the next sweep. Repair the index, restart, then rebuild the book only if something is still wrong.
• If you have ever run pnpm fix:eco-orders on this install, run pnpm eco:index:check even if nobody has complained — see below.

A market showing a spread of exactly 0.000000

Check any market where the best bid and the best ask have been the same price, or where the spread has read 0.000000 and not moved. That is not a thin book. It is a market order that could not fill and was left resting in the aggregated book at a price no order can be honoured at.

A market order is an instruction to trade at whatever the book is now. It carries a price only so the platform knows how much money to hold against it: for a sell, the best bid it expects to hit. That figure is a funding cap, not a quote — and it was being written into the book as one. So a market sell whose counterparty disappeared before it could fill was published as an ask at the bid, which crosses the book by definition, and the engine then re-armed it to keep sweeping a book that had nothing left in it. It never left. Its funds stayed held, and the level it created was read as real liquidity by the next market order to arrive.

From this release the unfilled part of a market order is cancelled and refunded at the end of the cycle it could not fill in, which is what every exchange does with a market order. Orders left resting by an earlier version are retired the same way on the first cycle after you restart.

• Fixed the unfilled remainder of a market order resting in the book for ever at its funding price, holding the customer's money and crossing the book.
• Fixed the next market order on that market being priced from the stale level the stuck one created.
• Go and look at the customers involved on any market that has been sitting at a zero spread. Their funds are returned automatically when the stuck order is retired, but the orders they placed while the book was crossed were priced against a price that was not real — a market buy in particular would have had too little held against it and may have stopped half-filled.

Markets with an idle AI market maker are now swept, after two quiet rounds

Until now, a market whose market-maker row said ACTIVE had the ghost-level sweep switched off entirely whenever none of its price levels were recognisably the market maker's own. That is a statement about a database row, not about what the engine is doing: a maker that is paused in memory, idle, that failed to start, or whose worker died reads ACTIVE indefinitely while quoting nothing. So on those markets a real customer's unbacked level could never be removed by anything, which is exactly the state the first entry above is about.

The engine now waits for evidence instead. A maker that is quoting rewrites its quotes continuously and they are marked as its own, so a market that shows none of them across two consecutive sweeps — ten minutes — is not being quoted, and its levels are treated as customer depth and reconciled. A maker that starts quoting again resets that count on its first quote, and one that resumes later clears its book and re-quotes from scratch, so nothing it needs is lost. Levels that ARE recognisably the maker's are protected in every case, as before.

• Changed an idle-but-ACTIVE market maker no longer disables ghost-level cleanup for its whole market; two quiet sweeps authorise it, and the transition is stated in the log once rather than every five minutes.
• If you keep a market maker configured ACTIVE while deliberately not quoting — a paused strategy, a market you have stopped making — its old levels will now be cleaned up like any other unbacked depth. Stop the market maker for that market if you want its ladder preserved, or leave it running.
• pnpm rebuild:eco-orderbook is unchanged and still skips AI markets unless you pass --include-ai; that is a manual tool making a whole-market decision, and it cannot see which levels the maker wrote.

pnpm fix:eco-orders no longer writes when you run it

The repair tool for orders whose funds were never locked properly now reports by default and applies only with --execute, which is how every other writing script in the platform behaves. If you call it from a runbook or a scheduled job, add the flag.

That change is not tidying. The tool was judging every Copy Trading order against the wrong wallet, so it could report perfectly funded orders as faulty and retire them — releasing nothing, because the wallet it looked at held nothing, and stranding the real hold with no way left to return it. On the install this was found on, all three legitimate orders on a market were reported as faulty. Its write half also left the aggregated book untouched on every order it retired, marked orders with a status no other part of the platform recognises, and left exactly the index rows described above — while printing [OK].

• Fixed the tool checking the ECO wallet for orders whose funds are held in a Copy Trading wallet, and reporting funded orders as underfunded.
• Fixed the order-book level never being found, so a retired order's depth stayed on the book while the run reported success.
• Fixed retired orders being written with a status the platform treats as neither open nor closed, leaving them out of every list while nothing regarded them as settled.
• Changed the tool to a dry run by default; pass --execute to apply.
• If you ran it before this release, check the affected accounts: run pnpm reconcile:eco-inorder to report holds with no order behind them, and pnpm eco:index:check for index rows it left.

Cancelling an order on a multi-process install

If your platform runs the matching engine in one process and serves HTTP from another — a dedicated engine or cron host, or a web process whose matcher stood down while it kept serving pages — then every cancellation on that process was refused, with a message about a matcher lease. Placing an order worked, because placement hands the order to whichever process owns the matcher; cancelling had the same machinery available and was not using it.

Both cancel doors now hand the work over the same way. The customer is told their cancellation is being completed rather than shown an error, and it lands within a second or so.

• Fixed single-order cancel and Cancel All being refused on any process that does not own the matching engine.
• Nothing to configure. If you have been telling customers to retry, or restarting processes to move the lease, you can stop.

Cancel All stopped at the first five thousand orders

The bulk cancel read one page of a customer's order history — five thousand rows — and cancelled the open orders in it. Any open order older than that was left untouched: funded, resting in the book, and unreachable by the only bulk exit the product offers. It affects exactly the accounts with the most at stake, since a page of history is reached fastest by market makers and bot users.

• Fixed Cancel All only considering the newest five thousand orders. It now walks the customer's whole partition, keeping only their open orders, and says in the log if a history is so large that it stopped early.
• Fixed the same cap on the all-markets Orders list, which is where a customer is sent to find an order whose market they cannot remember. Open orders older than a page of history were missing from the one list that would have shown them. History itself is still capped at a page — it is newest-first, so that is the part anybody scrolls.
• If a customer has reported orders that would not go away, run Cancel All once more after updating — the ones that were out of reach are now in it.

Changed

Cancel All is offered whenever anything is open, and states how many

The button lived on the FILTERED list, so it disappeared whenever the current filters matched nothing — and "hide other pairs" is on by default, which means a customer whose only open orders were on another market had no bulk cancel at all. When it was shown it also understated itself: the endpoint cancels every market, so pressing it while looking at one pair cancelled the others too, silently.

• Changed the button to appear whenever the account has any open order, and to carry the count it will act on — Cancel all (7) — so its scope is visible before it is pressed.

Cancel All now says how many orders it could not cancel

The endpoint deliberately keeps going when one order fails, so that a single bad row cannot block the rest. It reported only the successes, so a customer whose order was left open was told everything had been cancelled — and the trade panel took the same reply as permission to clear every row from their list, hiding an order that was still open, still funded and still in the book.

• Changed the cancel-all response to report failures alongside successes.
• Changed the trade panel to refresh the order list from the server after a bulk cancel rather than marking every row cancelled locally.

An order placed with a lower-case pair is stored on the real market

Market lookup is case-insensitive, so a client posting mo / usdt found the MO/USDT market and placed successfully — and the order was then stored against the pair as it was typed. Everything that keys on the pair is case-sensitive, so the order landed on a market that does not exist: absent from that market's order list even though the customer had just placed it there, its depth displayed on no market at all, and matchable only against other lower-case orders. Normal clients send the pair as it appears in the market list, so this needed an API caller or a hand-built request to reach.

• Changed the pair to be canonicalised at placement, for orders and for stop orders, so what is stored always matches the market list.
• Orders already stored against a lower-case pair keep it. They are visible in Orders with the market filter cleared — cancel them from there.

Fixed

A market order's fill was taken out of somebody else's liquidity

A price level in the book is shared — one row holding the total of every participant's resting size at that price. Because a market order was being written into the book at a price derived from the opposite side, its fills were then subtracted from a level that belonged to other people entirely, and the reduction was written back as an absolute figure rather than a difference. A fill large enough to reach zero deleted a level that was fully funded and perfectly matchable.

Market orders no longer join the aggregated book at all, on any of the six paths that add to a level, take from one, or rebuild one — and because they never add, they can never subtract.

• Fixed a market order's fill decrementing, and in the worst case deleting, another participant's funded depth at the same price.
• Fixed market orders being published as aggregated depth, which advertised an offer nobody had made and put it on the wrong side of the book.

An order was priced from a level with no order behind it

The displayed book and the orders the engine can actually fill are kept in step by writes that can be lost — a cancel whose adjustment missed, a process that died mid-fill, an index row that outlived its order. So a level can outlive every order behind it, and until now placement could not tell the difference: it read the best price off the book and priced and funded against it. Nothing could ever fill there, which is how a market order came to rest and cross the book in the first place.

Placement now asks the matching engine what it is actually holding, skips any level no resting order backs, and uses the next one down. Where the engine has no opinion — a market it never loaded, or a process that does not run the matcher — every level is used exactly as before, so this can only make a price more real, never refuse a placement.

• Fixed a market or crossing limit order being priced from a displayed level that no order backed.
• Fixed a market buy's funding being capped too low when a level displayed more than was resting behind it, which stopped its sweep half-done and left the rest of the hold stranded.

Crossing visible depth did nothing if you held a resting order of your own

The engine refuses to match an order against another of the same person's, which is correct — but it then had to decide which of the two carries on looking for a real counterparty, and it chose the wrong one. It passed over the newer order: the one that had just arrived into a book already containing the other, and therefore the only one of the two with anywhere left to go.

The effect is invisible on a shallow book and silent on a deep one. A customer who held any resting order placed an order that crossed real, funded depth belonging to somebody else, and it was skipped past every counterparty behind their own order. The book showed liquidity, they crossed it, and nothing happened.

• Fixed a self-match retiring the arriving order instead of the resting one, so a customer holding one order was refused fills against everybody behind it.

An order that failed to cancel was left where no cancel could reach it again

Cancel All takes each order out of the matching engine first, so that no fill can land while its funds are being released. If anything then went wrong for that one order — most reachably, a missing wallet for the currency it had to be refunded in — it was never put back. The order stayed open in the ledger, holding the customer's money and resting in the book, while being invisible to the engine: it could no longer be filled, and it could not be claimed for cancellation again by any later attempt, including the customer trying the button a second time.

• Fixed an order being left out of the matching engine after a failed cancellation, which made it permanently unfillable and permanently uncancellable.
• Fixed a missing refund wallet being treated as a reason to skip an order silently rather than as the failure it is.

A cancelled order could keep its place in the matching engine's index

Every path that retires an order is supposed to remove it from the index the engine loads open orders from. Three did not always manage it: a placement rolled back after a failed fund hold removed the order but kept the index row whenever the lookup it needed failed; the sweeper for corrupted order rows deleted them without touching the index; and cancelling an order that was already cancelled reported success without clearing the row. Each one left the state described in Upgrade Notes — depth on the book with nothing behind it.

• Fixed a rolled-back order placement leaving its row in the order index.
• Fixed the corrupted-order sweeper leaving index rows behind for the rows it deleted.
• Fixed cancelling an already-cancelled order reporting success without removing its index row, which was the one door that could have cleared it.
• Added a startup and hourly comparison of the order index against the ledger, in both directions, with an automatic rebuild of any market that disagrees.

A cross-process cancellation could be given up on

On a deployment where the matching engine runs in a different process from the web server, a cancellation is handed to the engine's process through a queue. A request that failed was retried five times and then dropped — leaving the order open, still holding the customer's funds, with nothing left that would ever try again. That was tolerable while the queue only carried background work. It is not now that customer-facing cancellations use it.

The retry limit existed for a real reason: one measured install had a thousand orders whose owning customer had been deleted, and each retry wrote two lines to the log every second. Both ends are fixed rather than traded off.

• Fixed a queued cancellation being abandoned. It now backs off — one second, two, four, up to once every five minutes — and stays queued until it succeeds, so a database blip or a redeploy delays a cancellation instead of losing it.
• Fixed the case that produced that thousand-order backlog: an order whose customer no longer has a wallet can now be cancelled, releasing nothing, instead of failing for ever. There is no hold to release for such an order.
• Changed repeated failures to be reported as one line per five minutes with a count, the age of the oldest, and the distinct errors, instead of one line per order per second.

A cancellation could report money it never returned

The refund figure a cancellation reported was the amount it intended to release, and it was returned even on the paths where nothing moved — a missing wallet, or a wallet that could not be opened at all. Copy trading frees a follower's allocation by that number, so a figure that never moved over-freed their budget while the real hold stayed locked.

• Fixed the reported refund to be what was actually released. A cancellation that could not release anything now says so.

The order book only moved when the two-second poll noticed

Every fill, cancel and book repair is announced by the matching engine. Those announcements were addressed to a channel no panel had subscribed to — the panels register with a depth and the engine's message carried none, and the two have to match exactly — so not one of them was ever delivered. The ladder appeared to work because a separate two-second poll re-read the book anyway, which is why the symptom was lag rather than a dead panel.

• Fixed the engine's order-book updates being sent on a channel nobody listens to. The ladder now moves as fills happen, and the two-second poll remains as a backstop and for deployments that run the engine in a separate process.
• Changed the depth a panel asked for to be honoured on every update rather than only on the polled snapshot, so a 25-deep and a 50-deep panel on one market no longer overwrite each other's registration.

A market whose book had been emptied never got its levels back

The routine that keeps the displayed book in step with the resting orders could delete a level with nothing behind it, but it stopped before it could ever create one — so a market whose aggregated levels were lost, whether to a repair run in the wrong order or to the sweep itself, displayed nothing while funded orders went on resting. It now rebuilds the missing levels from the orders it holds.

• Fixed the order-book reconciliation skipping any market whose book was empty, leaving real resting orders with no depth displayed for them.
• Changed the two cases where a market's book is deliberately not swept to say so in the log, at most once an hour per market, instead of passing over silently.

An order-book repair could report success while changing nothing

A write that matches no row succeeds in ScyllaDB — there is no affected-row count — so the five-minute reconciliation counted every repair it issued as a repair it made. A level that cannot be addressed by the price key the sweep derives is therefore immortal AND reported as fixed, which is the worst pairing: the log says "Fixed 3 orderbook discrepancies" and the depth is still there.

• Fixed each repair being counted before it was verified. Every write is now read back, and only a level that actually changed is counted.
• Added a separate line for repairs that did not take, naming the offline rebuild, because those will fail identically on every sweep until someone runs it.

One failed database read stopped the order-book sweep for every market

The reconciliation asks which markets have an AI market maker before it deletes anything — correctly, since those levels have no order behind them by design. If that one read failed, the whole pass was abandoned: every market, every five minutes, for as long as the read kept failing. A connection-pool blip during a busy period switched off the only generic ghost-level sweep on the platform.

• Fixed a failed read abandoning the sweep. The last successful answer is reused for up to half an hour — those rows are edited by hand, so it is almost certainly still correct — and only a staler failure stops the pass.

A restart that failed to start the matching engine broke it until the next one

The engine is started once and the result is remembered. That included a failure: if the first attempt threw — ScyllaDB unreachable during the boot window, a market list that did not load, a lease call that errored — the failure itself was remembered, and every later attempt in that process got the same one back without anything being retried. The result was a process with no matcher, no order-book reconciliation and no stop-order monitor, refusing every placement and every cancellation, for as long as it ran. Restarting fixed it, which is why it read as bad luck rather than as a fault.

• Fixed a failed engine start being cached for the life of the process. The next request starts a fresh attempt, so a transient database problem during boot costs one failed request instead of an outage that only a restart clears.

A permit withdrawal skipped its own balance check

The withdrawal path that spends a token allowance checks the holder's on-chain balance before it signs anything — except the check was never waited for. The result was that the path walked straight past it into the transfer, and a refusal for insufficient funds surfaced, if at all, as an unrelated error somewhere else in the process.

• Fixed the permit withdrawal path proceeding without waiting for its on-chain balance check.

The deposit monitor stopped when any deposit tab closed

The chain monitor that watches a deposit address, and the timeout behind the pending-deposit prompt, were both torn down whenever a connection closed rather than the last one. Closing one deposit tab therefore stopped watching the address the customer was still staring at in another.

• Fixed the deposit address monitor being torn down while the customer still had the deposit screen open elsewhere.