Which DRL Algorithm for FX Trading — and Why the Max Drawdown Number Tells You Everything

In a 2025 benchmark on petroleum futures, DQN and PPO finished with almost the same annualized return and almost the same Sharpe ratio. Their maximum drawdowns were 22 times apart. That last comparison is the only one that matters for a live account: a drawdown is what triggers a margin call or ends a prop firm evaluation, and neither the return nor the Sharpe ratio told the two agents apart.

The algorithm comparison question is not “which one backtests best?” It is “which one survives contact with a real account?” Most deep reinforcement learning (DRL) research papers optimise for the first question. This article answers the second.

The Benchmark Number Most Traders Ignore

Wang and Liu (2025) compared four DRL agents on daily petroleum futures — Brent crude, plus RBOB and gasoil crack and time spreads — with a test period starting in January 2023 (Journal of Risk and Financial Management 18(7):347 (opens in a new tab), Table 2). Their results, as printed:

The paper prints these figures without units, so we quote them as printed rather than as percentages. The comparison does not depend on the unit, because it sets the agents against each other within one column.

On return and Sharpe ratio, DQN and PPO are indistinguishable: 1.476 against 1.450, and 2.028 against 2.041. The drawdown column is where they split — 16.594 against 0.752, a 22-fold gap. The Calmar ratio, which sets return against maximum drawdown, says the same in one number: 1.977 for PPO, 0.088 for DQN.

Prop firm evaluations impose 5–10% maximum drawdown limits. Two agents with the same return and the same Sharpe ratio can sit on opposite sides of that line, and only the drawdown column tells you which side. A drawdown that breaches it does not end a strategy — it ends an account.

A2C had the second-lowest drawdown (1.729, 2.3 times PPO’s), but a lower Sharpe ratio (1.413) and the lowest return of the four (0.933). A2C uses synchronous parallel rollouts without PPO’s clipped update — in a high-volatility batch, the policy can revise sharply.

The decision-relevant columns in any DRL algorithm comparison are the risk columns, max drawdown above all. Annualized return tells you what happened during the best sequences in a controlled backtest. Drawdown tells you what happens on the weeks that go wrong — and in FX, there are always weeks that go wrong. One caution: this benchmark is petroleum futures, not FX, over one test period. Read it as an illustration of why the drawdown column decides, not as a ranking to carry over to EURUSD.

Why DQN Fails as a Live FX Algorithm

DQN’s weakness for live FX trading is structural. It follows from two specific design choices.

The discrete action space. DQN learns a Q-function — an estimate of expected future reward for each available action in a given state. “Available actions” in a standard DQN implementation are fixed discrete choices: buy, sell, hold. The algorithm decides whether to trade but cannot decide how much to trade.

For a 15-minute EURUSD bar strategy, this is a fundamental limitation. Position sizing is one of the primary variables a sophisticated trading agent should optimise. An agent constrained to a fixed lot size on every signal cannot adjust its risk exposure to changing market conditions. It trades identically during a quiet Tuesday session and during an NFP release.

Overestimation bias. The Bellman equation at the core of DQN training includes a maximisation step — at each update, the algorithm selects the action with the highest estimated Q-value. In noisy environments, this causes the algorithm to systematically overestimate the quality of certain actions. The agent learns to chase reward spikes that are artefacts of price noise, not genuine signal.

FX markets are high-noise by design. The signal-to-noise ratio in any 15-minute price series is low. DQN’s overestimation bias means it is structurally inclined to be misled by that noise during training — and to carry those learned misbehaviours into the test period.

Architectural variants address both problems partially. Double DQN decouples action selection from action evaluation, which reduces overestimation. Dueling DQN separates state value estimation from action advantage estimation. But neither variant solves the discrete action space problem for bar-based FX trading. If the strategy requires “buy 0.07 lots based on current volatility,” DQN cannot express that decision in any form.

Why PPO Is the Starting Algorithm on Every DRL Project

PPO’s defining characteristic is a limit on how far the policy moves in one update. Its clipped surrogate objective removes the incentive to change any action’s probability by more than about 20% relative to the previous policy — the clip range ε = 0.2 of the original PPO paper (opens in a new tab). A single bad batch of training data is therefore unlikely to cause a catastrophic revision.

Why does this matter more in FX than in other domains? Because FX training data is full of regime segments — trending periods, ranges, and volatility shocks — that produce sharply different reward signals. Without clipping, a week of NFP-driven volatility could look to the algorithm like evidence that the entire previous policy was wrong — and trigger an update that overwrites months of stable learning.

With clipping, the policy improves incrementally. The agent is unlikely to unlearn a working strategy in one bad batch. That is the behaviour a live account needs. It is consistent with PPO’s small drawdown in the benchmark above, although the paper does not test clipping as its cause.

There is a second argument for PPO that is less obvious but matters at the early stages of any DRL project: input quality has more leverage than algorithm choice. Kili et al. (2025) report that a PPO agent trained on Kalman-filtered gold prices (XAU/USD) reached a 27.1% CAGR, against 3.46% for PPO on the raw prices (IJACSA 16(11) (opens in a new tab)). The algorithm did not change — only the noise characteristics of the input changed. Improvement came from filtering, not from switching to a more complex algorithm.

This points directly at how to sequence the work. While you are still validating the reward function and feature pipeline, use PPO. It is simple, stable, and — critically — it fails diagnostically. When PPO training curves oscillate without converging, the signal is clear: the reward function has a scaling problem, or a feature has lookahead bias. SAC’s additional complexity makes those signals harder to interpret. PPO gives you the clearest picture of whether the environment is sound before you add architectural sophistication.

We start every DRL trading project with PPO. Not because it is theoretically optimal, but because it is the most informative instrument for diagnosing environment design problems.

When to Move from PPO to SAC — and What Changes

SAC adds one thing PPO does not have: an entropy term in the objective. SAC simultaneously maximises expected cumulative reward and the randomness of the policy. The agent is penalised for becoming too deterministic — for always selecting the same action in a given state.

This regularisation targets a specific failure mode: a policy that has overfit a single market regime. A PPO agent trained on 2017–2019 data may learn patterns that worked perfectly in that period and fail entirely when the volatility regime shifts. The policy has converged to a deterministic mapping it cannot adapt out of.

SAC’s entropy term is designed to prevent premature convergence: its stochastic policy keeps exploring alternative action distributions rather than committing to patterns that have stopped working.

The practical question is when this matters for your specific project. The threshold we use:

1. Run PPO across 5 different random seeds
2. Count how many produce positive out-of-sample Sharpe
3. If 3 or more pass: PPO is adequate — do not touch the algorithm
4. If fewer than 3 pass: investigate the reward function and feature pipeline first

This sequence matters because PPO failure below the 3-seed threshold is almost always caused by a flawed reward function or feature engineering problems, not algorithm limitations. Switching to SAC will not fix lookahead bias in the feature pipeline. It will just make the lookahead bias harder to diagnose.

If you have validated the environment through PPO — 3+ seeds producing positive out-of-sample Sharpe — and there is genuine evidence of regime overfitting (strong in-sample performance that degrades sharply out-of-sample), then SAC is the appropriate upgrade. SAC is also the right choice when continuous position sizing is a genuine requirement and higher sample efficiency matters — for example, when historical data is limited or the strategy requires fine-grained lot sizing across multiple instruments.

DDPG and TD3: No Recommended Use Case for New FX Projects

In the benchmark above, DDPG had the lowest Sharpe ratio of the four (0.723), though the paper does not say why. The structural concern is this: DDPG’s deterministic policy latches onto short-term patterns in training data with no entropy-based exploration to prevent it. The policy overfits a specific volatility regime, and when that regime ends, the agent either takes no meaningful action or takes the same action it learned in training regardless of current conditions.

TD3 addresses DDPG’s worst instabilities. Twin critics reduce overestimation bias. Delayed policy updates ensure value estimates stabilise before the strategy is revised. On raw performance metrics, TD3 can be competitive with SAC. The problem is that TD3 still produces a deterministic policy — which inherits DDPG’s brittleness in markets where the exact sequence of price observations never repeats.

When a DRL system built on DDPG fails, a rescue review should look first for a policy that performed well during one specific volatility regime and went functionally dormant when that regime changed. That agent has learned a lookup table for one market, not a generalised policy. Switching to PPO or SAC at that point usually requires a full environment redesign — the DDPG training regime produced a policy too regime-specific to be fine-tuned.

For new FX projects there is no scenario where DDPG is the right starting algorithm. TD3 is a reasonable choice if you are implementing a continuous-action agent without a library dependency and need a simpler codebase — but SAC is available in every major RL library, and its stochastic policy avoids the brittleness described above.

The Algorithm Decision Sequence

The practical sequence for a new FX DRL project:

Step 1 — Start with PPO. It handles both discrete and continuous action spaces, tolerates hyperparameter variation, and fails diagnostically when the environment is misconfigured.

Step 2 — Run 5 seeds before drawing any conclusions. If fewer than 3 produce positive out-of-sample Sharpe, the problem is in the reward function or feature pipeline. Do not change the algorithm until the environment is validated.

Step 3 — Once PPO validates the environment, evaluate whether continuous position sizing is genuinely required by the strategy. If yes, benchmark SAC against the PPO baseline. If PPO with continuous actions meets performance targets, stay with it — the simpler architecture is easier to maintain and monitor in production.

Step 4 — HFT and limit order book applications: DRQN (Deep Recurrent Q-Network). The LSTM layer processes sequences of order book events that feedforward networks cannot capture. This is the one scenario where a DQN variant is the correct choice — and it requires tick-level data, not bar data.

Step 5 — DDPG and TD3: No recommended use case for new bar-trading FX projects.

Research papers often headline annualized return. Live accounts are constrained by drawdown limits, margin calls, and prop firm evaluation rules. Those constraints, not backtest return figures, determine which algorithm survives into production — and that is the only comparison that matters.

For full methodology coverage — reward function design, seed stability testing, state space engineering, and MetaTrader deployment architecture — see Deep Reinforcement Learning for Trading.