What Happens When the Same Backtest Produces Different Broker Results?

A backtest can change when you switch brokers because its results depend on the data, trading costs, fill assumptions, session rules, and order handling behind it. Treat a strategy result as a test of a specific setup, not a portable promise.

On Sunday night, that distinction is easy to miss. You load the same indicator settings, use the same entry and exit rules, and run the same date range. Broker A shows a small edge. Broker B shows a losing sequence, or a drawdown large enough to make the strategy untradeable for your account.

Nothing in the rule set changed. The environment did.

In 1999, NASA lost the Mars Climate Orbiter after navigation work used incompatible measurement units. The spacecraft reached Mars, but the planned orbit insertion did not produce the intended result. Arthur G. Stephenson’s NASA investigation board documented how a mismatch between pound-seconds and newton-seconds contributed to the failure.

The lesson is not that a trading backtest has spacecraft-level stakes. It is that a calculation only means something inside the system it was built for. Change the units, inputs, or rules around it, and the output can change with them.

The broker is part of the backtest

A strategy does not trade an abstract chart. It trades prices from a particular feed, under a particular broker’s execution rules.

One broker may show a different bid-ask spread around the open. Another may define a daily candle at a different time. A crypto exchange can have a different overnight price path from another exchange, even when both list the same asset. For a strategy that enters after a breakout or exits on a stop, those differences can decide whether a trade exists at all.

This matters most when a backtest has a thin margin. If the strategy earns a little more than its estimated costs, a wider spread or a few worse fills can remove that edge. If it depends on tight stops, a slightly different intraday high or low can turn a winning trade into a stopped trade.

The useful question is not, “Which broker gave me the better result?” Ask: “What assumptions created this result, and can I actually trade under them?”

Check the data before you trust the equity curve

Start by writing down what each test uses:

• The instrument name and venue.
• The date range and timeframe.
• The timezone or session boundary used for daily bars.
• Whether prices are bid, ask, midpoint, or last trade.
• The commission, spread, slippage, and financing assumptions.
• The fill rule for stops, limits, and market orders.

A backtest that leaves these blank is hard to audit later. It can also invite a dangerous shortcut: choosing the result that looks best and treating it as evidence.

Run the strategy on both brokers without changing anything. Then compare the trade list, not only the final return. Find the first trade where the results diverge. Was the entry price different? Did one feed print the trigger and the other miss it? Did the stop fill at a different level? That one trade often reveals the real dependency.

If costs erase the result, that is useful information, not a failed test. What Happens When Trading Costs Erase a Profitable Backtest? explores the same issue from the cost side.

A weaker result can be the more useful one

Broker B may produce the result you do not want to see. That does not automatically make it wrong.

A strategy that survives different feeds, realistic costs, and modest slippage has given you more evidence than one polished result from a single environment. A strategy that fails under those changes has exposed a condition you need to understand before risking money.

Look at max drawdown in each version. The better-looking test might hide a sequence of losses that your account size or discipline cannot tolerate. The worse-looking test may show that the method only works when fills are unusually favorable.

This is where position sizing belongs in the process. Decide how much of your account a single loss represents only after you have tested less favorable execution assumptions. A plan that needs every stop to fill perfectly is fragile by design.

Educational content, not financial advice.

Carry the uncertainty into Monday

NASA’s Mars Climate Orbiter did not fail because the mathematics were useless. The calculation depended on consistent inputs across the system. Your backtest deserves the same discipline.

Before Monday, save both broker reports and label the assumptions. Pick the more conservative version when planning risk. Then paper trade or observe the strategy in the broker environment you would actually use. Record expected entry, actual fill, spread, and exit for each setup.

That journal gives you a visible record of the gap between the backtest and live conditions. If an AI assistant queues a trade idea, the approval step is where you can compare its plan with the current broker price, your position size, and the conditions the test assumed. The final decision stays with you.