When I setup Walk-Forward Analysis in MultiCharts 12.0.16374.700 x64 (RC1) to use Number Of Runs: 1 and run WFA optimization, it instantly ends and shows window containing "No Data" text:
I expect these settings to produce a single backtest with intervals: IS: 80%, OOS: 20%.
Looks like Walk-Forward Analysis engine does not calculate correctly IS/OOS interval dates. Older MultiCharts versions produce the same behavior with Walk-Forward Analysis not working with some settings. But now it is very easy to spot the issue after WFA engine has been redesigned in MultiCharts 12. Just set "Number Of Runs: 1" and WFA fails.
The issue does not depend on the datafeed, resolution, signals applied e.t.c. Should be very easy to reproduce.
--
Also, looks like WFA does not work with "Round Type: Days" completely (on intraday chart). When I set "Round Type: Bars" WFA still fails with "Number Of Runs: 1", but start to work with "Number Of Runs: 3".
BUG: Walk-Forward Analysis does not work with Number Of Runs: 1
Re: BUG: Walk-Forward Analysis does not work with Number Of Runs: 1
UPD: I found data2 series settings (daily bars) can change WFA with "Number Of Runs: 1" behaviour. If I setup:
data1: 500 days back (intraday)
data2: 500 days back (daily bars)
Then WFA with "Number Of Runs: 1" fails and just prints "No data" as described above.
But if I change data2 to longer period:
data1: 500 days back (intraday)
data2: 1000 days back (daily bars)
Then WFA with "Number Of Runs: 1" starts an optimization process.
So the issues has something to do with MaxBarsBack and data2 series. Looks like data2 series has to be longer than data1 series for WFA with "Number Of Runs: 1" to succeed.
data1: 500 days back (intraday)
data2: 500 days back (daily bars)
Then WFA with "Number Of Runs: 1" fails and just prints "No data" as described above.
But if I change data2 to longer period:
data1: 500 days back (intraday)
data2: 1000 days back (daily bars)
Then WFA with "Number Of Runs: 1" starts an optimization process.
So the issues has something to do with MaxBarsBack and data2 series. Looks like data2 series has to be longer than data1 series for WFA with "Number Of Runs: 1" to succeed.
- TJ
- Posts: 7742
- Joined: 29 Aug 2006
- Location: Global Citizen
- Has thanked: 1033 times
- Been thanked: 2222 times
Re: BUG: Walk-Forward Analysis does not work with Number Of Runs: 1
What is your MaxBarsBack setting?
Re: BUG: Walk-Forward Analysis does not work with Number Of Runs: 1
In this particular case:
data1: 500 days back (intraday, 30 min)
data2: 500 days back (daily bars) - WFA optimization FAILS
MaxBarsBack is set to 200, so the optimizer has at least 300 daily bars to do the WFA optimization. But "Number Of Runs: 1, OOS, % of the first run: 20%" optimization fails with this settings. It only works if I set 685 days back for the data2 series:
data1: 500 days back (intraday, 30 min)
data2: 685 days back (daily bars) - WFA optimization OK
And with these settings "Number Of Runs: 1, OOS, % of the first run: 20%" optimization fails too:
data1: 500 days back (intraday, 30 min)
data2: 680 days back (daily bars) - WFA optimization FAILS (but 680 days back is close to the boundary where it works)
(I expected data2 has to have at least 700 bars backs for the WFA nRuns=1 optimization to work, i.e. 500+MaxBarsBack == 700, and to fail with data2 699 bars back. But it seems WFA engine calculations are a bit more complicated than that).
Anyway, what should be clear from my description is that:
1. BUG1: data2 series has to have more daily bars than data1 series for an WFA optimization to work with low number of nRuns (when data1 is intraday chart and data2 is daily chart). And this behavior is very counter-intuitive and should be investigated.
2. BUG2: WFA optimization in this setup does not work at all if I set Round Type: Days, regardless of data2 series length or Number of Runs. This is another bug in WFA engine.
P.S. Here is how I setup Instruments to help to reproduce the issue: It does not matter what signal is applied for the chart as long as MaxBarsBack is 200 to match my setup.
data1: 500 days back (intraday, 30 min)
data2: 500 days back (daily bars) - WFA optimization FAILS
MaxBarsBack is set to 200, so the optimizer has at least 300 daily bars to do the WFA optimization. But "Number Of Runs: 1, OOS, % of the first run: 20%" optimization fails with this settings. It only works if I set 685 days back for the data2 series:
data1: 500 days back (intraday, 30 min)
data2: 685 days back (daily bars) - WFA optimization OK
And with these settings "Number Of Runs: 1, OOS, % of the first run: 20%" optimization fails too:
data1: 500 days back (intraday, 30 min)
data2: 680 days back (daily bars) - WFA optimization FAILS (but 680 days back is close to the boundary where it works)
(I expected data2 has to have at least 700 bars backs for the WFA nRuns=1 optimization to work, i.e. 500+MaxBarsBack == 700, and to fail with data2 699 bars back. But it seems WFA engine calculations are a bit more complicated than that).
Anyway, what should be clear from my description is that:
1. BUG1: data2 series has to have more daily bars than data1 series for an WFA optimization to work with low number of nRuns (when data1 is intraday chart and data2 is daily chart). And this behavior is very counter-intuitive and should be investigated.
2. BUG2: WFA optimization in this setup does not work at all if I set Round Type: Days, regardless of data2 series length or Number of Runs. This is another bug in WFA engine.
P.S. Here is how I setup Instruments to help to reproduce the issue: It does not matter what signal is applied for the chart as long as MaxBarsBack is 200 to match my setup.
- Anna MultiCharts
- Posts: 560
- Joined: 14 Jul 2017
- Has thanked: 42 times
- Been thanked: 141 times
Re: BUG: Walk-Forward Analysis does not work with Number Of Runs: 1
Hello, Sten!
Thanks for bringing this to our attention. We reproduced this on our end. The issue will be investigated by our engineers and fixed ASAP.
Thanks for bringing this to our attention. We reproduced this on our end. The issue will be investigated by our engineers and fixed ASAP.
- Anna MultiCharts
- Posts: 560
- Joined: 14 Jul 2017
- Has thanked: 42 times
- Been thanked: 141 times
Re: BUG: Walk-Forward Analysis does not work with Number Of Runs: 1
Sten,
After further analysis it turned out that his feature is still in development and leaked into MultiCharts 12 Release-Candidate by mistake. These controls will be removed from MultiCharts 12 Release and will be introduced back after this functionality is ready and fully tested.
After further analysis it turned out that his feature is still in development and leaked into MultiCharts 12 Release-Candidate by mistake. These controls will be removed from MultiCharts 12 Release and will be introduced back after this functionality is ready and fully tested.
Re: BUG: Walk-Forward Analysis does not work with Number Of Runs: 1
Anna, I am sorry to hear this - the feature is great! Exactly what I need for some of my workflows. Thanks anyway!
Just to clarify: do you mean development team is going to remove "Number Of Runs:" and "OOS, % of the first run:" controls from the optimization settings dialog? And leave the new design of the WFA / Matrix Optimization dialogs?
As for me, I prefer "Number Of Runs:" and "OOS, % of the first run:" to be present in the release in it's current form rather than being removed. But that's definitely up to you to decide.
Just to clarify: do you mean development team is going to remove "Number Of Runs:" and "OOS, % of the first run:" controls from the optimization settings dialog? And leave the new design of the WFA / Matrix Optimization dialogs?
As for me, I prefer "Number Of Runs:" and "OOS, % of the first run:" to be present in the release in it's current form rather than being removed. But that's definitely up to you to decide.
- Anna MultiCharts
- Posts: 560
- Joined: 14 Jul 2017
- Has thanked: 42 times
- Been thanked: 141 times
Re: BUG: Walk-Forward Analysis does not work with Number Of Runs: 1
Sten,
we’ll remove the Round Type: Bars, Days controls. The rounding will use Bars type.
we’ll remove the Round Type: Bars, Days controls. The rounding will use Bars type.