Overfitting is tuning a strategy until it fits the past so closely that it has learned the noise rather than anything that will repeat. Its backtest looks excellent. Its live results don't.
It happens to everyone who iterates, because iterating is the cause. Each time you change a parameter, add a filter or try another asset and keep the better backtest, you're selecting on the same history. Run enough variations and one will look great by chance alone.
Warning signs
- Many parameters, few trades. Five tuned numbers fitted to forty trades can explain almost anything.
- Sharp optima. A 47-day lookback is brilliant, but 45 and 50 are mediocre. Real effects are rarely that precise.
- Rules with no reason. “Skip Tuesdays in March” is what fitting noise looks like.
- All the return in one period. The strategy mostly encodes one market episode.
- You lost count of the attempts. The best of 200 tries is not the same evidence as a single test.
Defence 1: data you never tuned on
Split history in two. Develop and tune on the first part, the in-sample period. Then run the finished strategy, once, on the second part, the out-of-sample or held-out period. If performance collapses there, the in-sample result was mostly fit.
The discipline is in “once”. If you look at the held-out result, tweak, and look again, the held-out period has become part of your training data, and it can no longer warn you.
Defence 2: walk-forward testing
A single split depends on where you cut. Walk-forward testing repeats the split through time: tune on, say, five years, test on the next one, move both windows forward a year, and repeat. Stitching the test years together gives an out-of-sample record across many market conditions, and shows whether the parameters you would have chosen at each point kept working.
Defence 3: robustness checks
- Neighbouring parameters. Plot results across a range of values. You want a broad plateau, not a spike.
- Other markets. If a trend rule works on one stock index, does it do something sensible on others, or on currencies?
- Higher costs. Double the fees and slippage. An edge that disappears was probably too thin to trade.
- Simpler versions. Remove each rule in turn. If the result barely changes, the rule was decoration.
Defence 4: count your trials
The more variations you tried, the higher a Sharpe ratio has to be before it's convincing. Researchers have formalised this, for example the deflated Sharpe ratio of Bailey and López de Prado, which discounts a result by the number of trials behind it. Even without the formula, write down how many versions you tested and judge the winner with that number in mind.
Clean data is a precondition, not a cure
Overfitting is separate from look-ahead bias. Point-in-time data makes sure each test is honest; it can't stop you running a thousand honest tests and keeping the lucky one. You need both.
How MarketsCoder's agents handle it
AI agents on MarketsCoder iterate on a strategy toward a goal you set, which is exactly the kind of repeated search that overfits. So an agent tunes only on its development dates, and its final result is a single test on dates it never saw while working. The report shows both, so you can see how much of the improvement held up.