Ecosystem payouts have stopped — master wallet balance and gas

Every ecosystem (on-chain) withdrawal is paid by a master wallet you have to keep funded with native coin. The platform holds the key; the coin is yours to top up. When a chain's master wallet runs dry, withdrawals fail one at a time with a gas error buried in the log, the customer is refunded, and there is no banner on the dashboard that says the chain is out of gas.

This runbook is the diagnosis and the fix. If payouts have stopped across every chain at once, this is probably not your fault line — check the scheduler console first, because a stopped cron app produces the same symptom on all chains simultaneously.

Symptom

• On-chain withdrawals on one chain fail while every other chain is fine.
• Customers report the withdrawal was "cancelled" and see the money back in their balance. The transaction row is FAILED with a description that ends Transaction failed: Failed to estimate gas.
• pm2 logs cron and pm2 logs backend carry WITHDRAW and WALLET lines around each attempt.
• Token withdrawals on that chain fail; native-coin withdrawals may still work.

That last point is the tell. The master wallet is the gas payer: it signs and pays for the transfer of a token held in a customer's custodial wallet. A chain whose only active assets are native never touches it.

Where the balance is shown

The refresher also caches per wallet in Redis under wallet:{id}:balance for five minutes, so a top-up you just made can take that long to appear even when it does get written.

The two endpoints with no screen

Both of these exist and answer; neither is called by any admin page. Reach them with an API key or from the server.

The transactions endpoint is the fastest way to prove a top-up landed, and to see what the gas payer has actually been spending, without leaving the platform.

Two funding actions, also API-only

A custodial wallet row can be funded directly. Both routes sign with the master wallet's key, so both fail for the same reason a payout does when there is no gas.

How a payout actually fails

The chain of events for a token withdrawal on an EVM chain:

1. The withdrawal queue picks the row up and flips it PENDING → PROCESSING (guarded, so two workers cannot both own it).
2. getGasPayer(chain, provider) loads the chain's master wallet, decrypts its private key and builds a signer. All three ways it can fail — no master wallet for the chain, a wallet row with no data, or a decryption failure — sit inside one try whose catch re-throws the same generic 500 "Withdrawal failed - please try again later". Which one it was is only in the log, as Master wallet not found for chain: X or Master wallet data not found for chain: X under WALLET. Do not go looking for those strings in an API response; they never leave the server.
3. getAdjustedGasPrice(provider) reads the network fee data and pads it by 20%. On failure it logs Failed to adjust gas price and throws "Failed to adjust gas price".
4. estimateGas(transaction, provider) simulates the transfer and pads the estimate by 20%. This is the one that fires when the wallet is dry. It logs Failed to estimate gas for transaction, logs the revert reason separately under GAS when the node returns one, and throws "Failed to estimate gas".
5. The queue catches it, marks the row FAILED and refunds the customer.

Every one of those is a generic 500 by design, so the customer-facing failure email says nothing about your gas balance. The distinguishing detail is in the log:

pm2 logs cron --lines 300 | grep -iE "WITHDRAW|WALLET|GAS|ECOSYSTEM"

An estimate failure is usually insufficient native balance, but not always — a paused token contract, a blocked address or a bad allowance produce the same message. Read the revert-reason line under GAS before you conclude it is gas, and confirm the balance on the explorer.

What the queue does with a failure

The recovery logic is careful, and knowing which branch you are in tells you whether the customer has their money back.

The refund path itself is guarded twice. Before writing FAILED it re-reads the row and refuses outright if the status is already COMPLETED or a trxId is present; then the FAILED write itself carries where: { trxId: null }, so a withdrawal that broadcast between the read and the write is still protected. After the write it re-reads once more and skips the refund if a hash appeared. The refund is idempotent on its own key (eco_refund_<id>), so a duplicate attempt throws and is logged rather than paying twice.

What recovers an orphaned row

Two things sweep for withdrawals that were debited and then orphaned:

• recoverEcoWithdrawalsAtBoot() runs once at startup, on the scheduling process only, before the cron workers begin. It sweeps every pending ecosystem withdrawal with no age filter, so rows orphaned by a previous process lifetime are picked up in order. It is bounded: past a deadline the server carries on starting and the sweep finishes in the background, so an unreachable explorer cannot hold the boot. Gated on the ecosystem extension being enabled, so disabling the addon really does stop it.
• ecosystemWithdrawRecon (every 5 minutes) re-enqueues rows created more than three minutes ago that the in-memory queue is not tracking — the crash between the wallet debit and the queue draining. processPendingEcoWithdrawals (every 30 minutes) runs the same sweep as a second net; the two share one in-flight pass so they cannot double-probe the same rows.

A stale PROCESSING row that already carries a broadcast hash (trxId) is promoted straight to COMPLETED with no re-broadcast and no chain lookup — the hash only ever gets written after a successful broadcast, so it is itself the proof that the coins left — and its UTXO inputs are settled.

A PROCESSING row with no hash is left alone until it is five minutes old (STALE_PROCESSING_MS) — before that a live worker may still own it. Past that, it is handled by chain, and the two cases are not alike:

• UTXO chains (BTC, LTC, DOGE, DASH). These handlers persist a pre-broadcast intent in txHashPending, so the sweep has something to ask the chain about. If that transaction is found, the row is promoted to COMPLETED with no re-broadcast. If it is not found, the sweep waits 30 minutes before concluding it was never broadcast — provider lag is not evidence — and only then reverts it to PENDING, guarded on trxId: null. A UTXO row that never got as far as writing an intent has nothing to check and is reverted straight away.
• Every other chain (EVM, SOL, TON, TRON, XMR) — the chains this runbook is about. There is no pre-broadcast intent to check, and the hash is written only after the network accepts the transaction, so a crash in that window leaves a row that may already be on-chain. The sweep never reverts it. It logs Cannot verify broadcast on-chain for this chain — NOT auto-reverting (double-broadcast risk). Manual review required. and moves on, leaving the row PROCESSING for the review queue below. Nothing automated will ever clear it.

UTXO chains behave differently

BTC, LTC, DOGE and DASH withdrawals are batched: several withdrawals ride one on-chain transaction. A lone withdrawal is parked for a batching window so companions can join it, and a same-chain parked companion is released to ride with it. hasQueuedUtxoCompanion() is the check that decides whether to park or go now.

Failure isolation, in order:

1. One member fails identifiably → that member alone is failed and refunded, and the rest are retried as a batch.
2. The whole batch fails with more than one member → every member is retried individually, and each result is settled per member. A member that already carries a broadcast hash is skipped and left for the recovery sweep.
3. A single-member failure is classified exactly as a non-batched one.

A partial batch failure is reported per member, not rolled back. So on a UTXO chain you can legitimately see one withdrawal FAILED and refunded while three others from the same attempt completed. Post-broadcast side effects (the confirmation email and the platform-fee record) are run outside every retry boundary and never throw — a failed email after a successful broadcast must not re-enter a path that would re-broadcast.

Fix it

The review queue

The Ecosystem overview counts payouts that need a human and ranks them so the rows nothing will ever resolve on its own come first:

stranded and unknown-outcome are the two the page counts as needing a person, and they are ranked to the top of the queue for that reason. Read stranded as "nothing will ever move this row but you" — it is not a row waiting on a retry.

A stale UTXO row is deliberately not labelled stranded; it stays in-flight, because on those chains the recovery sweep can verify it against the chain and resolve it without help.

The 5-minute stranding threshold mirrors the withdrawal queue's own STALE_PROCESSING_MS, so the page and the watchdog agree about which rows are abandoned.

The decisions themselves are taken on Admin → Finance → Withdrawals (/admin/finance/withdraw/log), which accepts an Approve or Reject on rows in PENDING and PROCESSING — a payout stuck mid-flight is exactly the row that needs resolving.

Set a floor so "empty" is not the alarm

The gasEmpty flag compares the stored balance against a floor that defaults to zero. Zero is a fact; anything else is a guess, and a good guess is per-chain — 0.01 is generous on Polygon and nothing on Ethereum. The result is that by default you are told a chain is out of gas only once it is completely out, which is one failed withdrawal too late.

Set your own floor in .env:

The value in force is echoed back in the dashboard payload's thresholds.gasFloor, so the page can say which floor it is applying.

The daily check

1. Open Admin → Ecosystem. Read the caption under Chains Not Ready: if it says "N with an empty gas payer", that N is today's work. The tile's own number counts blocked and degraded chains and can be zero while N is not.
2. Scan the chain table's Gas column for Out of gas markers, and for any figure below one working day of payouts on that chain.
3. Check the payouts-waiting-on-a-person count. Nothing should age past a day.
4. Once a week, read a chain's master wallet transaction history through the endpoint above and confirm the burn rate matches your expectation. A sudden rise is either volume or a fee spike; both change how often you top up.

Key management, and what it cannot do

The master wallet's private key is stored encrypted. The vault is unlocked by supplying the passphrase, which decrypts the key held in ENCRYPTED_ENCRYPTION_KEY using ENCRYPTION_KEY_PASSPHRASE-derived material and holds the result in memory for the life of the process.

When both .env values are present the vault unlocks itself at boot. When they are not, the Ecosystem overview shows a destructive The vault is locked alert with an Initiate vault button, and nothing signs on any chain — no withdrawal completes and no new deposit address is derived.