Troubleshooting

Organised by what you see, not by what is wrong. The first three entries account for most real incidents, and all three come down to the same thing: the quote stream.

Nothing quotes and every order is refused

Symptom. The markets rail renders instruments but prices do not move, or show a state badge instead of a number. Every order returns "Pricing is temporarily unavailable for this instrument — please retry shortly".

Cause, almost always. The provider's WebSocket is not connected. The tick stream is the only writer of a symbol's QUOTING state, and every open and close is refused while a symbol is not QUOTING. There is no REST polling fallback.

Only some symbols quote

Symptom. EUR/USD is live; gold or a stock never updates.

Three separate causes, in order of likelihood.

No symbol mapping for the active provider. Run Import Instrument Catalog and read the response. symbolsUnmatched lists catalog rows this provider cannot serve; strandedActive is worse — those are instruments already live for customers with no mapping for the feed that is now running. Import never removes another provider's mapping and never changes a status, so switching providers strands markets silently unless you re-import.

The plan does not cover that asset class. TraderMade sells no stocks. Polygon's entry plans have no real-time forex. Finnhub's free WebSocket carries US equities only and answers 403 on forex candles.

The session is closed. A US equity outside 09:30–16:00 New York is correctly SESSION_CLOSED. Check the instrument's symbol group and its calendar.

Symbols keep halting mid-session

Symptom. Instruments flip between quoting and halted, and customers get intermittent refusals.

The feed is going stale past the configured threshold — 10 seconds for FX and commodities, 30 for stocks. Genuinely illiquid instruments do this at quiet hours; a whole catalog doing it is a provider or network problem.

Raising fxTradingQuoteStaleSecondsFx does not fix a dying feed, it just makes you fill against older prices. Diagnose the feed first. Only widen the threshold if you have decided that a thin instrument's real update rate is acceptable to trade against.

Repeated provider errors in the log

The manager cools a failing provider off rather than hammering it, and logs each distinct state once rather than once per symbol per interval.

Log says	Meaning	Cool-off
rate-limited	You have exceeded the vendor's quota	The vendor's hint, clamped to 1–15 minutes, whole provider
rejected the API key	Wrong or revoked credential	1 hour, whole provider
does not serve this data on the current plan	The endpoint is not sold to you	1 hour, history only — the live feed keeps running

The last row is a design decision worth knowing: a plan refusal on one chart timeframe must not be able to pause the price feed that fills are priced against.

Auth and plan errors are configuration states. No amount of waiting clears them — change the key or the plan, then restart.

An instrument will not activate

"No active market-data provider" — activate one first.

"Instrument has no {provider} symbol mapping (only: ...)" — it carries a mapping for a different, inactive vendor. Run Import Instrument Catalog against the currently active provider, or set the mapping by hand.

"Instrument has no provider symbol mapping" — it has never been linked. Run the import.

"Illegal transition X -> Y" — the lifecycle is deliberately restrictive. See the transition table in Instruments, groups and sessions. The usual case is trying to go straight from ACTIVE to DELISTED; route through CLOSE_ONLY.

"N open position(s) exist" — you cannot take a market away from customers holding it. CLOSE_ONLY, wait, then DELISTED.

Customers cannot place live orders

Work down this list; each produces a distinct message.

Charts are flat or hours out of date while live prices work

History and live quoting are deliberately decoupled. The chart back-fill tries the active provider first, then up to two other providers holding credentials — because a bar is the same bar whoever serves it, while the live marks that fills are priced against must come from one source.

So a flat chart with working prices means every history source in that chain failed or is cooling off. Check the log for history paused lines, and consider configuring a second provider purely for history: Polygon and Twelve Data have deep history, and their key can sit in .env unused by the live feed.

Swaps are not being charged

A "ledger drift" alert arrived

The reconciler verifies every five minutes that the sum of an account's deals equals its balance. Drift means something wrote the balance outside the ledger.

Do not "fix" the balance with SQL. Open the deals ledger for that account, find where the running balance and the deal amounts diverge, and correct it with a reason-coded adjustment, which writes a proper ADJUSTMENT deal. The most common cause is a direct database edit made during an earlier incident.

Admin pages return 403

Every screen is permission-gated and roles ship with no grants. Assign the relevant view.forex_trading.* and edit.forex_trading.* keys under Users → Roles & Permissions — the full list is in the settings reference. Super Admin bypasses the check entirely, which is why it works for you and not for your staff.

The one key that stays refused regardless is fxTradingExternalRoutingEnabled on the extension settings endpoint. That is by design; write it from the core system settings page as a Super Admin.

A-book routing is not engaging

Orders that should hedge are filling internally instead. Use the dry-run tester (Execution → Routing Rules → test an order) — it runs the real resolver and returns a rule-by-rule trace. Then check, in this order:

1. fxTradingExternalRoutingEnabled is on. It is Super Admin only and lives on the core system settings page.
2. The provider is enabled, not merely configured.
3. The provider's environment matches the account type. DEMO and LIVE are separated end to end.
4. The rule's dimensions actually match. A null dimension is a wildcard; a populated one must match exactly.
5. The instrument has a venue symbol mapping for that provider. Rules pointing at a provider with mapping gaps save with a warning and then fall through the fallback ladder.
6. The provider is not degraded or auto-suspended. A hedge sync older than staleSyncAlertSec suspends new routing and raises an alert.

A provider will not disable

Disabling an execution provider is refused with a 409 while open external positions, in-flight routing orders or pending closes still reference it. That guard exists so you cannot orphan a live hedge with a toggle. Follow the drain procedure in External execution.

Quotes stopped after a deployment or restart

Exactly one process runs the desk, arbitrated by a Redis lease named forex-trading. The lease holder streams quotes, evaluates stop-out, routes externally and reconciles; every other process serves forex data from the database only.

If the leader dies or loses the lease mid-flight, there is no automatic promotion — tearing down the quote stream on a process that is still serving requests would make it refuse every open and close. The loss is logged loudly. An operator has to restart the loser.

The dedicated cron process is the one deliberate exception: it mirrors the leader's published quote snapshot rather than opening a second provider stream, because its jobs price money.

Getting more detail

Backend logs for this extension are tagged FX (the engine and the customer paths) and ADMIN_FX (admin actions). Every admin write is also recorded in the core audit trail.

For A-book incidents, the Execution Desk serves the operator runbook, which carries a playbook for every failure mode that ends in an operator action.