The exchange provider is degraded — the silent half-failure

The exchange provider is the account this platform holds at Binance, KuCoin or XT. Spot prices, markets and charts come from its public API; deposit addresses, deposit and withdraw networks, and balances come from its authenticated one.

When the credentials fail, the platform does not stop. It logs one warning and falls back to an unauthenticated instance so the read-only half keeps working. That is a deliberate trade — a half-dead provider is better than a dead site — but it means the failure is invisible from the outside: charts render, the market list populates, prices tick, and every spot deposit and withdrawal quietly stops.

Symptoms

Order placement is not in the working half, even though the quote that precedes it is. POST /api/exchange/order reads the ticker from the public API, then calls createOrder and fetchOrder — both authenticated — so on the fallback instance ccxt throws and the route answers 500 Unable to process order: <message>. The customer sees a book they can watch and cannot trade on.

Customers see it as a deposit screen that errors rather than showing an address, and a withdrawal that is debited and then refunded. The API answers a 503 with:

Unable to load currency data from binance. The exchange API credentials may be
missing or invalid.

That message is now explicit. It used to read "Currency not found", because ccxt returns an empty map — not an error — from fetchCurrencies() when the provider serves it from an authenticated endpoint and no usable credentials are present. Binance's sapi/capital/config/getall is the common case, and an empty map is exactly what the unauthenticated fallback instance produces.

Where the fallback happens

In backend/src/utils/exchange.ts, during initialisation:

1. Credentials are read from the environment by name, built at runtime from the provider chosen in the admin: APP_${PROVIDER}_API_KEY, APP_${PROVIDER}_API_SECRET, plus APP_${PROVIDER}_API_PASSPHRASE where the provider needs one. Missing or empty either of the first two logs API credentials for <provider> are missing. and gives up for this attempt.
2. checkRequiredCredentials() runs. If it fails, the instance is closed and immediately replaced with a credential-less one — before markets are even loaded — logging API credentials for <provider> are invalid.
3. Markets are loaded. If loadMarkets() throws for a reason that is not a rate limit or a clock skew, the instance is closed and replaced with a credential-less one, which is then primed with its own loadMarkets() so public data is actually usable. The warning is:

Falling back to an unauthenticated binance instance — authenticated data
(deposit/withdraw networks, balances) will be unavailable until valid API
credentials are configured.

That line is the whole diagnosis. Grep for it first:

pm2 logs backend --lines 500 | grep -iE "EXCHANGE|CURRENCY"

The hub screen

Admin → Finance → Trading Infrastructure → Exchange Providers (/admin/finance/exchange), gated on access.exchange.

It shows the one active provider — enabling a provider deactivates every other one, so there is exactly one at a time — with three badges (version, Active / Inactive, Licensed / Unlicensed) and three tabs:

Two buttons live in the header: Verify Credentials, and Activate License when the provider is unlicensed. The result of a verification is rendered as a full-width alert below the tabs, and on failure it prints the exact environment variable names for this provider — built literally from the provider name, because they are read verbatim by the backend.

Four screens with no menu entry

None of these appear in the navigation. They are reachable only from the Quick actions grid at the bottom of the hub page.

Those keys are seeded separately from the hub's access.exchange, so a role that can open /admin/finance/exchange does not automatically get any of the four. On Markets the gate is the table's own access.exchange.market / view.exchange.market plus the list route's view.exchange.market — the page itself carries no route-level permission entry.

Balances and Fees are greyed out on the hub with "Please verify credentials first" whenever the most recent verification failed — so if those two cards are disabled, you already have your answer.

The balance route fetches a live balance and maps it to asset, available, inOrder and total, dropping assets with nothing in them. Two of its three failures are typed, which is the useful part:

Only ccxt.AuthenticationError and ccxt.NetworkError get their own status. The "no exchange" case is raised inside the route's own try as Exchange or provider not available, and because that is an ordinary Error the trailing else catches it and rewrites it — so that string never reaches you. A bare 500 here means "look at the log", not "the credentials are fine".

Verifying credentials

Verify Credentials calls POST /api/admin/finance/exchange/provider/{productId}/verify (edit.exchange). It builds a fresh instance — it does not consult the cached one — and does two things in order: loadMarkets(), then fetchBalance(). Only a successful balance fetch counts as valid.

It is deliberately excluded from the admin audit trail: it round-trips a saved key to see whether the exchange still accepts it, and saving the key is the audited action.

Read the message it returns, not just the red or green:

Missing and invalid are different faults. The first never left your server; the second is a conversation the exchange refused. Only the first is fixed by editing .env.

Clock skew is retried automatically up to twice with a fresh time sync before it is reported, so seeing this message means two syncs failed.

Proxy configuration

The Settings tab takes a proxy URL in http://, https://, socks4:// or socks5:// form (with optional user:pass@). Leave it empty to disable. Without one the platform forces an IPv4 agent — some exchanges cannot whitelist IPv6.

The page's own guidance, and it is right: test before you save. The test uses a separate connection, so a proxy that fails the test will not have been committed.

Invalid proxy URL: <the url>

Saving the proxy writes to the provider row (PUT /api/admin/finance/exchange/provider/{productId}), which also clears that provider from the manager's cache — so a proxy change, unlike a credential change, does take effect on the next call.

The health check will not tell you

The Exchange Provider entry in GET /api/admin/system/health/batch checks the database for an active provider row and then checks that the two environment variables are present. It never talks to the exchange.

Every one of those is critical: false, so none of them turns the system health screen red, and up means only "two strings exist in the environment". A key that is present and revoked reports up forever. Treat this entry as a configuration check, never as a liveness one.

The currency cache

cacheExchangeCurrencies runs hourly and warms the Redis exchangeCurrencies key with a 120-second TTL, so the key is expired for most of the hour and GET /api/exchange/currency reads the database instead. That is by design: the blob carries price, which processCurrenciesPrices rewrites every two minutes, and a longer TTL would let a cached price disagree with the row.

Two consequences worth knowing:

• The cached blob is the platform's own list of enabled currencies, from the exchangeCurrency table. A stale one is at most two minutes behind, and the endpoints fall back to the database on a miss but never re-populate the key themselves. This job stopping does not break the currency list.
• The deposit screen's networks do not come from this cache. They come from a live authenticated fetchCurrencies() call against the provider, which is precisely the call that returns {} on a degraded instance. So a currency can be listed and enabled everywhere and still offer no depositable network.

If the deposit screen shows a currency but no chain to deposit on, you are in this runbook, not looking at a cache problem.

Fix it

Things that are not this fault

• Blank charts with working deposits. That is the candle cache or ScyllaDB, not credentials. See the Charts are empty entry in Troubleshooting.
• Everything on-chain failing while spot is fine. Ecosystem withdrawals do not touch the exchange at all — see Ecosystem payouts have stopped.
• Nothing scheduled happening anywhere. Check the scheduler console before assuming a provider problem.
• NEXT_PUBLIC_EXCHANGE is a frontend variable — the first three letters of the provider alias, read by the TradingView chart component and the market-data socket. It is inlined into the browser bundle at build time, so changing it requires pnpm build:frontend, and it has no effect on the backend's provider choice, which comes from the active exchange row.