Shadow-Mode Optimization: Testing Before Touching Live Operations

Every optimization vendor arrives with a backtest, and every backtest looks good. This is close to a law of nature: nobody shows up with the runs that failed. The question for a portfolio owner is how to evaluate an optimizer using evidence that could actually have come out badly.

Shadow mode is that evidence.

What shadow mode is

A shadow-mode test runs the candidate decision system in parallel with real operations. It receives the same data your live process receives, at the same time, under the same deadlines. It produces its decisions: nominations it would have submitted, intraday trades it would have placed, setpoints it would have sent. Every decision is logged with a timestamp. Nothing is executed.

After a defined period, the logged decisions are valued against what the market actually did, and compared with the results of the decisions that were actually taken. The comparison is denominated in money, ideally as a difference in realized value per MWh against an agreed baseline.

In practice the test has two stages. A historical replay comes first: the system is run over past periods to check basic plausibility. Then the live shadow run, which is the stage that matters, because it is the only one where the future was genuinely unknown to everyone involved.

Why the historical stage alone is never enough

Historical backtests fail quietly in one specific way: information that did not exist at decision time leaks into the decisions. The commonest source is data revision. Production measurements, prices and forecasts all get corrected after the fact, and most databases silently overwrite the original values. A backtest run on today’s version of last year’s data is a test on data nobody had.

The effect is large, and it flatters the optimizer. We have repeatedly seen candidate systems whose apparent edge on revised historical data was several times larger than the edge they could demonstrate once restricted to strictly as-of inputs. The system was not cheating deliberately. The database was.

A live shadow run closes this hole by construction. Whatever the system knew, it knew before the outcome existed. There is no way to leak the future into a decision that was logged yesterday.

There is a second benefit that owners tend to underestimate: a shadow run exercises the entire data pipeline under real conditions. Feeds that arrive late, units that turn out to be km/h where m/s was assumed, timestamps labeled for the wrong end of the interval. These problems surface in shadow mode, where they cost nothing, instead of in live operations, where they cost money and trust.

How to benchmark an optimizer in shadow mode

A checklist that has held up in practice:

Agree the KPI before the run starts. The cleanest choice is the change in realized value per MWh versus the actual decisions taken over the same period, with the channel list and denominator written down. If the KPI is negotiated after the results exist, the results will win the negotiation.

Freeze the decision schedule. Specify exactly when each decision type must be produced and what data is allowed to exist at that moment. A day-ahead nomination decided at 11:30 may use nothing published after 11:30.

Log raw, log everything. Store the inputs the system saw, the outputs it produced, and the timestamps of both, in their original form. Disputes about a shadow run are settled by the logs or not at all.

Run long enough to cover more than one regime. Two calm weeks prove very little. The run should include at least some volatile days, some negative-price hours if the market produces them, and enough decisions that the result is distinguishable from luck. Define the evaluation window before the run, and resist the temptation to end it on a good day.

Value the decisions honestly. Shadow trades are valued at realized market prices, which assumes the trades themselves would not have moved the market. For small and mid-size portfolios this is a reasonable assumption; for large positions in thin intraday hours it is not, and the valuation should say so.

Watch for the soft leaks. Retroactive backfilling of the decision log, quiet curation of which days count, and evaluation windows chosen after the fact all reintroduce the backtest problem through the side door. The defense is procedural: rules fixed in advance, logs append-only.

Decide the promotion criteria in advance. What result, over what period, leads to a limited live deployment? Writing this down before the run turns the shadow test into a real decision gate instead of a demo.

What a good shadow run buys you

If the optimizer fails in shadow mode, you have lost some integration effort and gained the knowledge before any operational exposure. If it succeeds, you hold something rare: a money-denominated, timestamped, leak-resistant record of what the system would have done with your portfolio, in your market, under conditions nobody could rehearse.

That record is the correct basis for a commercial decision. A slide deck of backtests is a substitute for it, and a poor one.

When a vendor shows you a backtest, one question sorts most of them: which of these inputs existed, in exactly this form, at the moment each decision was supposedly made?