DRL Agent Deployment: What Breaks Between Training and Live

Most DRL agent deployment failures are not model failures — they are feature parity failures, and they produce no error message. The DRL agent described here converged cleanly: reward curves trending upward across all 5 seeds, clean backtest Sharpe on two years of EURUSD M15 data. Live on a paper account, it traded like it had learned nothing.

The debug session lasted four hours. The cause: Z-score normalization parameters had been fit on the entire historical dataset during training, but the production bridge was computing them fresh from a live rolling window with a different anchor date. Different mean, different standard deviation, same feature names — but the agent was receiving observations it had never seen during training. The model was fine. The bridge was wrong.

The Architecture Decision Is Simpler Than It Looks

Before getting to parity, the architecture question: Python MT5 API, native ONNX inside MQL5, or ZeroMQ messaging bridge. This decision gets over-engineered. For most DRL trading strategies, the right starting architecture is the Python API.

The Python MT5 library connects a Python script directly to the MetaTrader 5 terminal. The agent runs in Python, where all its training dependencies — PyTorch, Stable Baselines3, NumPy — are already available. Data comes in via `copy_rates_from()`, trades go out via `order_send()`. The terminal handles execution. This is the simplest deployment path, and for H1 and H4 strategies, or even M15 strategies, there is no measurable latency problem. The Python-to-terminal IPC overhead is single-digit milliseconds — irrelevant at bar-level decision frequency.

ONNX becomes justified when the architecture is genuinely latency-constrained: tick-level strategies where milliseconds matter, or when deploying to a machine where Python cannot run. ONNX moves inference inside the MT5 process — the trained policy exports as a `.onnx` file, loads via `OnnxCreate()`, and runs via `OnnxRun()` in the EA. Inference happens natively without IPC. But ONNX carries a cost: every preprocessing step that Python did in training must now be replicated in MQL5. That burden is where most ONNX deployments fail.

ZeroMQ is a different use case entirely. It exists primarily for MT4 compatibility, where no native Python API is available. For MT5, choose between Python API and ONNX based on latency requirements. Do not introduce ZeroMQ complexity unless MT4 is a hard constraint.

The decision tree for most practitioners: start with the Python API. Validate the bridge. If latency proves to be a measurable problem at the strategy’s decision frequency, then — and only then — migrate to ONNX.

Feature Parity: The Failure Mode the Backtest Never Shows

At barmenteros FX, most DRL rescue projects arrive with the same diagnosis: the model is not the problem. The bridge is. The degradation is immediate and silent — no error, no warning.

Three parity failures I encounter repeatedly:

Normalization parameter drift. The training pipeline computed Z-score parameters over the full historical dataset. Those parameters were never saved. The production bridge computes a rolling Z-score anchored to whenever the live system started. The means and standard deviations differ by enough to shift every normalized feature value outside its training distribution. The agent has never seen these input magnitudes during training, and the policy produces near-random actions.

Position state encoding mismatch. Training encoded position state as `{0, 1, 2}` (flat, long, short). The MQL5 bridge encoded it as `{-1, 0, 1}`. The feature dimension matches — the model accepts the observation without error — but a “long” position in production has the numeric signature of “flat” in training, and “short” has the signature of “long.” The policy makes directionally inverted position management decisions with complete confidence.

Timezone offset in session encoding. The Python MT5 API’s `copy_rates_from()` returns bar timestamps in UTC. The Python observation builder used `datetime.now()` without a timezone specification, defaulting to the operating system’s local timezone. The session indicator flags — London open, New York session, Asian session — were computed 6 hours off. The agent’s session-conditional policies activated in the wrong sessions throughout every live trading day.

All three cases: model runs, produces orders, Sharpe is worse than random. No errors in the log.

What Must Be Identical Between Training and Production

The parity checklist is not about reproducing the full training pipeline. It is about the six specific computation decisions that determine whether the agent’s observations in production match the distribution it trained on.

The log return formula is worth emphasizing. `np.log(close / close.shift(1))` and `(close – close_prev) / close_prev` are mathematically close for small moves but diverge on large ones. For a 1% move, the difference is approximately 0.005%. For a 5% move during a major news event, it is 0.12%. That divergence shifts the input outside the training distribution at exactly the moments when correct agent behavior matters most.

For normalization: the frozen μ and σ parameters must be saved during training and loaded by the production bridge at initialization. In MQL5, this means reading them from a file at `OnInit()` and storing them in global variables. Recomputing them at runtime from live data defeats the purpose of normalization consistency.

The Bar-Boundary Timing Problem in Live Execution

In training, every observation is built from completed bars. The dataset is static — bar $t$ has its final close, its final high and low, its final volume. The agent never encounters a partially-formed bar during training.

In production with the Python API, the script runs continuously. In an ONNX-based EA, the model gets called inside `OnTick()`. Both scenarios create the same risk: the observation builder queries the current bar before it closes.

In MQL5, `Close[0]` is the current bar’s close — a value that changes on every tick until the bar closes. `Close[1]` is the last fully completed bar. If the observation vector is built using `Close[0]` for any price-derived feature — log returns, ATR, RSI — it is using a price that was never present in any training observation. The agent is receiving partially-formed bar data and treating it as equivalent to the closed-bar data it trained on.

The failure is subtle. At the start of a new bar, `Close[0]` equals the open and `Close[1]` is the previous bar’s final close. The observation query early in the bar is close to correct. Mid-bar, `Close[0]` has moved. Near bar close, it is close to the final value but not identical. The distribution of `Close[0]` over a full bar is fundamentally different from the distribution of a completed close — it has higher intra-day volatility and is systematically biased toward the open in low-volatility regimes.

The fix is straightforward. Build observations only from completed bars. In the Python API script, check that the timestamp of the most recent completed bar has changed before querying the model — track `last_bar_time` and skip the model inference step if no new bar has closed. In an MQL5 EA, implement `OnNewBar()` logic using a static variable:

static datetime lastBarTime = 0;
datetime currentBarTime = iTime(Symbol(), Period(), 0);
if(currentBarTime == lastBarTime) return;
lastBarTime = currentBarTime;
// Build observation and query model here — only on bar close

This single change removes an entire class of out-of-distribution inputs that the backtest, running on static historical data, can never surface.

Validating the Bridge Before Real Money

The bridge validation sequence has three steps. None of them require live trading.

Step 1: Historical replay parity test. Select a 500-bar segment of historical data from the test period — data the model never trained on but where you know the correct training-pipeline output. Run the same data through the production bridge and log the full observation vector at every bar. Compare element-by-element against the training pipeline output for the same bars. The maximum absolute difference across all features and all bars should be less than `1e-6` for floating-point implementation equivalence. Any value above that threshold is a parity failure. Fix it before proceeding.

If the training pipeline and production bridge produce the same observation vectors on historical data, the parity problem is solved. If they differ, the source of the difference is findable — compare feature by feature, bar by bar, until the divergence appears. It will appear in one of the six items in the parity checklist above.

Step 2: Random agent test in production. Before connecting the trained policy, deploy the bridge with a policy that selects actions uniformly at random — buy, sell, or flat with equal probability, random lot sizes within defined bounds. Run this on a paper account for one week with real MT5 data. Log cumulative reward.

The expected result: cumulative reward close to zero, with a small negative offset from transaction costs (spread plus commission). If cumulative reward is substantially negative beyond what transaction costs explain, the reward function or observation has a systematic bias that the random agent is triggering. The policy is not the cause — the environment is. Fix it before loading the trained model.

Step 3: Canary account. After Steps 1 and 2 pass, deploy the trained policy on a real account funded at 5–10% of the intended trading capital, with minimum lot sizes (0.01 lots). Run for two weeks. Compare Sharpe and drawdown against paper trading baseline. If performance diverges meaningfully — better or worse — investigate before scaling. Real execution adds slippage, spread variation, and partial fill behavior that paper trading does not fully replicate.

Risk Guardrails Live in MQL5, Not in the Model

The DRL policy cannot be trusted to manage its own catastrophic risk. This is not a criticism of DRL — it is the correct system boundary. The policy is a function that maps observations to actions. It has no concept of “the Python process just crashed” or “this is an unusually illiquid session at 3 AM.” Those are operational realities that belong in the MQL5 EA layer.

Hard stop-loss on every position. If the agent sends an `order_send()` instruction without a stop-loss level, the EA applies a default stop automatically — sized as a fixed percentage of account equity or a multiple of current ATR. The agent never opens an unprotected position. This rule is not negotiable and is not configurable by the agent.

Maximum position size enforcement. The EA enforces a hard ceiling on total exposure regardless of what the agent instructs. If the agent sends a 1.00 lot instruction but the ceiling is 0.20 lots, the EA executes 0.20 lots and logs the override. The agent’s lot sizing suggestions are treated as requests, not commands.

Daily loss circuit breaker. If the account’s intraday drawdown exceeds a defined threshold — typically 2–5% of equity — the EA stops accepting agent instructions for the remainder of the trading day, closes any positions that can be closed at market, and resumes at the next session open. The agent does not control when this threshold is reached or what the response is.

Heartbeat watchdog timer. When using the Python API architecture, the EA expects a heartbeat signal from the Python process at a regular interval — every 30 seconds is a reasonable default. If the heartbeat is missed for two consecutive intervals, the EA treats the Python process as unavailable: it stops opening new positions and, if configured, closes all open positions at market. An unresponsive Python process with open positions is an unmanaged live account.

These four guardrails together mean the system fails safe. The DRL agent may produce suboptimal decisions in unexpected market conditions — that is manageable. An unmanaged open position during a connectivity failure is not.

Deploying a DRL trading agent is an engineering problem, not a machine learning problem. The model is typically the least broken part. The bridge — the feature computation layer between raw market data and the policy — is where real deployments fail. Get the parity right, validate it against historical data before paper trading, and enforce risk guardrails in the execution layer independently of the model. Those three practices cover the majority of deployment failures I encounter.

The next engineering problem, once the bridge is validated, is keeping it validated as market microstructure changes: normalization parameters that were fit in 2020 gradually diverge from current market conditions, session volatility profiles shift after major macroeconomic regime changes, and spreads at different brokers affect the reward signal differently than during training. That is a live monitoring and retraining problem — a different article.