BUG: Walk-Forward Analysis does not work with Number Of Runs: 1  [SOLVED]

Questions about MultiCharts and user contributed studies.
Sten

BUG: Walk-Forward Analysis does not work with Number Of Runs: 1

Postby Sten » 22 Jul 2018

When I setup Walk-Forward Analysis in MultiCharts 12.0.16374.700 x64 (RC1) to use Number Of Runs: 1 and run WFA optimization, it instantly ends and shows window containing "No Data" text:
WFA_nRuns_1.png
WFA_nRuns_1.png (24.88 KiB) Viewed 587 times
WFA_nRuns_no_data.png
WFA_nRuns_no_data.png (7.73 KiB) Viewed 587 times
I expect these settings to produce a single backtest with intervals: IS: 80%, OOS: 20%.


Looks like Walk-Forward Analysis engine does not calculate correctly IS/OOS interval dates. Older MultiCharts versions produce the same behavior with Walk-Forward Analysis not working with some settings. But now it is very easy to spot the issue after WFA engine has been redesigned in MultiCharts 12. Just set "Number Of Runs: 1" and WFA fails.

The issue does not depend on the datafeed, resolution, signals applied e.t.c. Should be very easy to reproduce.

--
Also, looks like WFA does not work with "Round Type: Days" completely (on intraday chart). When I set "Round Type: Bars" WFA still fails with "Number Of Runs: 1", but start to work with "Number Of Runs: 3".

Sten

Re: BUG: Walk-Forward Analysis does not work with Number Of Runs: 1

Postby Sten » 22 Jul 2018

UPD: I found data2 series settings (daily bars) can change WFA with "Number Of Runs: 1" behaviour. If I setup:

data1: 500 days back (intraday)
data2: 500 days back (daily bars)

Then WFA with "Number Of Runs: 1" fails and just prints "No data" as described above.
But if I change data2 to longer period:

data1: 500 days back (intraday)
data2: 1000 days back (daily bars)

Then WFA with "Number Of Runs: 1" starts an optimization process.

So the issues has something to do with MaxBarsBack and data2 series. Looks like data2 series has to be longer than data1 series for WFA with "Number Of Runs: 1" to succeed.

User avatar
TJ
Posts: 7151
Joined: 29 Aug 2006
Location: Global Citizen
Has thanked: 988 times
Been thanked: 2041 times

Re: BUG: Walk-Forward Analysis does not work with Number Of Runs: 1

Postby TJ » 22 Jul 2018

What is your MaxBarsBack setting?

Sten

Re: BUG: Walk-Forward Analysis does not work with Number Of Runs: 1

Postby Sten » 23 Jul 2018

In this particular case:

data1: 500 days back (intraday, 30 min)
data2: 500 days back (daily bars) - WFA optimization FAILS

MaxBarsBack is set to 200, so the optimizer has at least 300 daily bars to do the WFA optimization. But "Number Of Runs: 1, OOS, % of the first run: 20%" optimization fails with this settings. It only works if I set 685 days back for the data2 series:

data1: 500 days back (intraday, 30 min)
data2: 685 days back (daily bars) - WFA optimization OK

And with these settings "Number Of Runs: 1, OOS, % of the first run: 20%" optimization fails too:

data1: 500 days back (intraday, 30 min)
data2: 680 days back (daily bars) - WFA optimization FAILS (but 680 days back is close to the boundary where it works)

(I expected data2 has to have at least 700 bars backs for the WFA nRuns=1 optimization to work, i.e. 500+MaxBarsBack == 700, and to fail with data2 699 bars back. But it seems WFA engine calculations are a bit more complicated than that).

Anyway, what should be clear from my description is that:

1. BUG1: data2 series has to have more daily bars than data1 series for an WFA optimization to work with low number of nRuns (when data1 is intraday chart and data2 is daily chart). And this behavior is very counter-intuitive and should be investigated.

2. BUG2: WFA optimization in this setup does not work at all if I set Round Type: Days, regardless of data2 series length or Number of Runs. This is another bug in WFA engine.
WFA_RoundType_Days.png
WFA_RoundType_Days.png (23.11 KiB) Viewed 551 times

P.S. Here is how I setup Instruments to help to reproduce the issue:
Instruments_Setup.png
Instruments_Setup.png (6.08 KiB) Viewed 551 times
It does not matter what signal is applied for the chart as long as MaxBarsBack is 200 to match my setup.

User avatar
Anna MultiCharts
Posts: 405
Joined: 14 Jul 2017
Has thanked: 28 times
Been thanked: 102 times

Re: BUG: Walk-Forward Analysis does not work with Number Of Runs: 1

Postby Anna MultiCharts » 23 Jul 2018

Hello, Sten!

Thanks for bringing this to our attention. We reproduced this on our end. The issue will be investigated by our engineers and fixed ASAP.

User avatar
Anna MultiCharts
Posts: 405
Joined: 14 Jul 2017
Has thanked: 28 times
Been thanked: 102 times

Re: BUG: Walk-Forward Analysis does not work with Number Of Runs: 1

Postby Anna MultiCharts » 23 Jul 2018

Sten,

After further analysis it turned out that his feature is still in development and leaked into MultiCharts 12 Release-Candidate by mistake. These controls will be removed from MultiCharts 12 Release and will be introduced back after this functionality is ready and fully tested.

Sten

Re: BUG: Walk-Forward Analysis does not work with Number Of Runs: 1

Postby Sten » 23 Jul 2018

Anna, I am sorry to hear this - the feature is great! Exactly what I need for some of my workflows. Thanks anyway!

Just to clarify: do you mean development team is going to remove "Number Of Runs:" and "OOS, % of the first run:" controls from the optimization settings dialog? And leave the new design of the WFA / Matrix Optimization dialogs?

As for me, I prefer "Number Of Runs:" and "OOS, % of the first run:" to be present in the release in it's current form rather than being removed. But that's definitely up to you to decide.

User avatar
Anna MultiCharts
Posts: 405
Joined: 14 Jul 2017
Has thanked: 28 times
Been thanked: 102 times

Re: BUG: Walk-Forward Analysis does not work with Number Of Runs: 1  [SOLVED]

Postby Anna MultiCharts » 24 Jul 2018

Sten,

we’ll remove the Round Type: Bars, Days controls. The rounding will use Bars type.


Return to “MultiCharts”