A chain node fell behind — RPC and confirmation diagnosis

On-chain deposits stop confirming and on-chain withdrawals stop broadcasting for one chain while everything else on the platform behaves perfectly. Nothing on the dashboard says "the Ethereum node is behind", because from the platform's point of view the node answered — it just answered with yesterday's block.

The product has a screen built for exactly this. Extensions → Ecosystem → Blockchains → Requirements (/admin/ecosystem/blockchain) is a per-chain configuration report with a live probe attached to each chain, and it is where this runbook starts and mostly ends.

If payouts have stopped on every chain at once, this is the wrong runbook — check the scheduler console first, because a stopped cron process produces the same symptom everywhere simultaneously. If only token withdrawals fail on one chain and native ones still work, read Ecosystem payouts have stopped instead: that is a gas problem, not an RPC one.

The screen

The page opens on a summary band — Ready, Needs attention, Not tested — then a Platform prerequisites card, then four chain families: EVM, UTXO, extended (Solana, Tron, TON, Monero) and custom EVM chains you added yourself.

Each chain collapses to one line: the symbol, the network badge read from {CHAIN}_NETWORK, a status pill and one sentence naming the single most important thing wrong with it. The sentence is chosen in a fixed order — a missing required variable beats an invalid network, which beats a failed check, which beats a warning — so the line is the next thing to fix, not a summary.

Expanding a chain gives four blocks:

Will it work? is the part to read. A green RPC check with a red Withdrawals line means the endpoint is fine and something else — a locked vault, a missing master wallet, a node that will not accept eth_sendRawTransaction — is stopping payouts.

Running the test

Three ways to trigger it: Test all chains in the page header (which queues PLATFORM first and then every chain, three at a time), Test platform on the prerequisites card, and the per-chain button inside an expanded card.

It only contacts endpoints you have already configured in .env or the custom chain table — the request cannot point the server at an arbitrary URL, which is why it needs only view.ecosystem.blockchain rather than an edit permission. Nothing is written on-chain and no secret is returned.

What an EVM test actually checks

The nine methods are eth_getBalance, eth_call, eth_gasPrice, eth_estimateGas, eth_getTransactionCount, eth_getLogs, eth_getBlockByNumber, eth_getTransactionReceipt and eth_sendRawTransaction. A method that exists but rejects the deliberately minimal parameters counts as supported; only "not implemented" is a failure. This is the check that catches a cut-price RPC provider whose free tier omits eth_getLogs — token deposits stop, native ones keep working, and nothing else in the product explains why.

A chain-id mismatch is reported in words rather than as a bare failure: RPC reports chain id 137, but this chain is configured as 1. That is the signature of a copy-pasted endpoint pointing at the wrong network, and it is worth checking before you accuse a provider of being down.

The two probes, and why HTTP and WebSocket are asked separately

The platform's own health checks use a matched pair, deliberately built on raw fetch and ws rather than on an ethers provider:

• probeEvmRpcBlockNumber() POSTs {"method":"eth_blockNumber"} to the HTTP(S) endpoint and parses the hex result. Any non-2xx, any JSON-RPC error, or a missing block number throws.
• probeEvmWssBlockNumber() opens a WebSocket to the WSS endpoint, sends the same request, reads the reply and terminates the socket on every path — answer, error or timeout.

Two endpoints, one question. That is what makes the pair diagnostic: a chain whose HTTP RPC answers and whose WebSocket does not is a different fault from one where both are dead. The first costs you realtime deposit detection and nothing else — the monitors fall back to HTTP polling, which the requirements page reports as a warning on the Realtime watching line, not a failure. The second is a dead chain.

The system health entry states it in one line, and it is worth grepping for:

pm2 logs backend --lines 500 | grep -iE "HEALTH|WSS connection error"

Connected to Ethereum (mainnet) via HTTP (latest block: 20913882), but WebSocket
connection failed

Proving the node is behind

This is the step nothing in the product does for you.

A stale node is invisible everywhere else: confirmations are counted against its idea of the tip, so a deposit that the chain confirmed twenty minutes ago is still "unconfirmed" as far as the platform is concerned, forever.

The transaction provider is not the node

An EVM chain has two independent data paths, and they fail independently:

• The node ({CHAIN}_{NETWORK}_RPC) — balances, gas, nonces, broadcast, and the eth_getLogs scan that finds token deposits.
• The tx-history providers — the explorer-class indexers that find native coin deposits and populate transaction history.

tx-provider-probe.ts probes the second set separately, and it keeps no list of its own: it asks the runtime dispatcher for the plan it would build for this chain and probes exactly that. Same per-chain order, same key resolution, same support rules, same circuit-breaker state — so the page cannot disagree with what a real request does.

There are seven providers — etherscan, blockscout, routescan, ankr, moralis, covalent, nodereal — and the order is decided per chain. ETH, Polygon, Arbitrum and Celo lead with Etherscan; Base and Optimism lead with Blockscout, which needs no key at all; BSC leads with NodeReal; Fantom with Ankr. Whatever the order, every keyless provider that can serve the chain is appended to the end of it.

The result is a row of pills, and each state means something specific:

A chain with no keys configured anywhere is no longer automatically degraded. Where a hosted Blockscout or Routescan instance exists, a keyless provider goes Serving and the check passes with nothing in .env. The check reports skipped — with the explanation that native-deposit detection falls back to scanning blocks over the RPC — only when nothing keyless can serve the chain either. That is a degraded state you can run in; a configured-but-broken provider is a real failure and is reported as one.

Where this still bites: Etherscan's free tier is paid-only for BSC (56 and 97), OP Mainnet (10 and 11155420), Base (8453 and 84532) and Avalanche, with Gnosis joining them on 2026-09-01, and Fantom, Cronos and HECO are not on Etherscan V2 at all. Base and Optimism are covered keylessly and need nothing; so is Avalanche, by Routescan rather than Blockscout. BSC, Fantom, Cronos, HECO and Polygon Amoy have no keyless option at all, and they get their own section below. Rootstock no longer has its old quirk: its wallet transaction history goes through the dispatcher like every other EVM chain, and is served keylessly on both networks — chain id 30 from rootstock.blockscout.com, 31 from rootstock-testnet.blockscout.com.

Explorer provider failover

Three mechanisms keep a chain's explorer up, and each one changes what you see on the page.

Order is per chain. The Providers, in the order they're tried line on an expanded chain is that chain's real order, left to right. It comes from TRANSACTION_PROVIDERS_{CHAIN} if you set one, otherwise TRANSACTION_PROVIDERS, otherwise a built-in default whose first entry is a provider that is actually free for that chain. The page prints which of the three it used on an Order from line under the pills, so if the order surprises you, read it there.

Keys are lists. Every provider key variable may hold several comma-separated keys, and a provider is only failed after each of them has been tried. ETHERSCAN_API_KEY="key_a,key_b" is valid, and so is a chain-scoped BSC_NODEREAL_API_KEY — the chain-scoped list is tried first and the global one stands behind it as a spare. A value that looks like a URL, or a placeholder like <your-key-here>, is ignored and reads as no key rather than as a broken one.

Failures are remembered. A provider that fails is skipped for a cooldown on that chain and network: ten minutes for a rejected key, thirty for a plan rejection, sixty seconds for a rate limit, an hour for "does not index this chain", thirty seconds for a network or unclassified error. Those are base figures — the three transient kinds (rate limit, network, unknown) double on each consecutive failure, capped at five minutes; the deterministic ones do not grow. A key that comes back invalid or rate-limited is quarantined across every chain at once, because that is a property of the key. If every provider for a chain is cooling down the dispatcher tries them anyway, so the breaker can never be the sole cause of an outage.

pm2 restart backend
pm2 restart cron

Ankr rejects a key that is perfectly valid

Ankr projects carry endpoint security restrictions, and they are the one provider failure that is not fixed in .env at all. The pill reports what Ankr said, followed by the change that clears it:

All four are fixed in the Ankr dashboard under project → Settings → Security, not here. For a server-side install, clear the domain restriction and restrict by the server's outbound IP instead; a domain allowlist is meant for browser applications and locks the backend out by design.

The first two are treated as a bad key and quarantine it across every chain for ten minutes, because a domain or IP rule rejects every chain identically. The third is treated as a plan rejection instead, so a key that serves your other chains is not quarantined along with it.

Forcing a provider on one chain

Pin the order for a single chain and leave every other chain on its default:

TRANSACTION_PROVIDERS_BSC="nodereal,covalent"
TRANSACTION_PROVIDERS_ETH="blockscout,etherscan"

Names outside the seven are dropped with a warning in the backend log rather than failing the boot, so a typo shows up as a shorter pill row, not an error. A keyless provider that can serve the chain is still appended to whatever you write — usually what you want; TRANSACTION_PROVIDERS_STRICT="true" turns that off everywhere.

Restart backend and cron, re-run the chain's test, and confirm the pill row is the order you asked for.

Chains with no keyless fallback

Five chains have neither a hosted Blockscout instance nor Routescan coverage: BSC (56 and 97), Fantom (250 and 4002), Cronos (25), HECO (128 and 256) and Polygon Amoy (80002). Polygon mainnet (137) is fine — the gap is Amoy only. Nothing keyless is appended to their order, so unless you self-host an instance and name it in <CHAIN>_BLOCKSCOUT_HOST, only a keyed provider can serve them — and on Cronos and HECO no provider indexes the chain at all, keyed or not.

BSC is the one where a keyed provider is the normal answer for a production install: Etherscan's free tier stopped covering 56 and 97 in November 2025. Without one of the following, native-deposit detection on BSC runs on the RPC block scanner alone.

The testnet split is worth knowing because the two halves of the system disagree about it. The probe resolves every provider by chain id, so with a key present it reports Ankr and NodeReal as failed on 97, correctly. The dispatcher's own support check sees only the chain symbol, so it lists them as runnable and discovers the truth at request time — one wasted attempt, after which the breaker skips that provider for an hour.

Which variable points at which endpoint

The panel reads .env and reports what it finds. It cannot write it. There is no field on this screen that changes an endpoint, and there never will be — the same rule as everywhere else in the platform (see The admin panel, What the panel cannot do).

Two traps the report calls out by name because they read as working:

• The concrete key name depends on the network. Set ETH_NETWORK=mainnet and the RPC key is ETH_MAINNET_RPC; the page shows the generic pattern with a {NETWORK} placeholder until the network resolves, and tells you to set the network first.
• Polygon's mainnet network name is matic, not mainnet. An invalid value fails provider initialisation with "Chain ID not found".

pm2 restart backend
pm2 restart cron

What stops, and in what order

A dead or stale endpoint on one chain produces this sequence. Knowing it is what stops you from looking in the wrong place.

The customer-visible shape of all this is: deposits that "never arrive" and withdrawals that sit in the queue. Neither points at an RPC endpoint on its own.

UTXO chains fail differently

Bitcoin, Litecoin, Dogecoin and Dash have no block-height check on this page, because they have no eth_blockNumber. Their test probes the provider the factory would actually pick:

• Esplora API reachable — walks the mempool.space / litecoinspace mirror list in order and reports the one that answered plus its tip height. Failing over past a dead mirror is a warning, not a failure.
• Fee rate tier resolves — confirms BTC_FEE_RATE_PRIORITY names a tier the provider actually returns.
• BlockCypher API — proves reachability and the token, and prints the chain height it reports.
• Bitcoin Core RPC (only when BTC_NODE=node) — getblockchaininfo, and it fails outright if the node's chain does not match BTC_NETWORK, or warns if blocks is behind headers. That is the one UTXO check that names a sync gap in so many words: Node not synced: 812004/812391 blocks.

The factory's silent fallbacks are mirrored on the page rather than hidden: node is BTC-only and falls back to BlockCypher elsewhere; Dogecoin and Dash fall back to BlockCypher whatever you set; an unrecognised value falls back to the chain default. The Provider selection row prints selected: x → effective: y so you can see which one is really in use.

Where the symptom shows up is different too. On a UTXO chain a degraded provider looks like unspent outputs that never become spendable, not a stale block height. Extensions → Ecosystem → Wallets → UTXO (/admin/ecosystem/utxo, view.ecosystem.utxo) lists them, one row per outpoint:

A pile of recent rows on one chain still Unspent in the view dialog while withdrawals fail means the provider is answering for reads and failing on broadcast or fee estimation. A chain with no new rows at all while customers insist they have sent coins means detection itself is down.

Turning a chain off

Sometimes the right answer is to stop taking deposits on a chain until its endpoint is fixed. What that lever is depends on the family, and it is not one switch.

Disable when the endpoint is broken and the fix is not minutes away, when the node is on a fork or the wrong network (a chain-id mismatch is exactly this), or when the provider is rate-limiting you badly enough that detection is unreliable. Do not disable for a WebSocket-only failure — that costs realtime detection and nothing else.

The fix, end to end

Related

• Ecosystem payouts have stopped — the same symptom caused by an empty gas payer instead of a bad endpoint.
• A customer paid and the balance did not move — which of the four deposit pipelines you are in, and what each leaves behind.
• The scheduler and Scheduled jobs reference — the jobs named on this page, their cadence and what stops when they stop.
• Troubleshooting — the platform-wide checks.