Walk Forward Analysis - what parameters to use?

Questions about MultiCharts and user contributed studies.
Trader M
Posts: 8
Joined: 06 Jan 2017
Has thanked: 4 times

Walk Forward Analysis - what parameters to use?

Postby Trader M » 26 Jan 2017

Hello,

I want to pick up a topic that has been raised already before but I still don’t know how to handle it. It is the interpretation and use of Walk-Forward-Results. I’d like to discuss that using an example.

For simplicity reasons let’s use a system with only 1 parameter, the ATR length and full calendar years as OOS time frame.
Assume there are the following WFA results and I found that many systems fall into that category:

(sorry, I didn't know how else to format a table than using "Code Review", it's of course not PL Code)

Code: Select all

IS Net Profit OOS Net Profit OOS Parameters:
15000 2000 (year 2012) ATR = 13
13000 5000 (year 2013) ATR = 13
20000 9000 (year 2014) ATR = 17
25000 1500 (year 2015) ATR = 11
16000 1000 (year 2016) ATR = 14

So, the total result of the OOS period would be 18500 USD.

In addition to the WFA an optimization is executed that covers the time frame from 2012 to 2016 with the following result:
Net Profit: 35000 USD
Parameter: ATR = 15.

So, my questions are the following ones:

1) The ratio of OOS results compared to the net profit of the optimization result for the same OOS period is 18500/35000 = 52,8%.  Who of you considers this as a relevant KPI and how do you interpret it – for example is your pass criteria > 50%?

2) If the system is traded live what parameter value is to be taken and this is the question the other thread didn’t fully answer.

Option a) take ATR = 14 because it is the most recent one that the WFA used. Argumentation: The WFA proved that the system works “even” with optimized parameters, so this one can be taken. It is also the one that might fit best to the recent market conditions (which I doubt because adapting to market conditions still is profitable but doesn’t bring extra profit).
Option b) take ATR = 17 because that is the value that generated the best OOS results (9000 USD) in calendar year 2014.
Option c) take ATR = 15 because this value generated the best results over the whole 5 years.
Option d) take any random ATR value within the range.

I have doubts especially between Option a and Option c/d. My interpretation is the following:
The system passed the Walk-Forward test. The test demonstrated that the system can handle even parameter values that are optimized, therefore the system can be used for live trading. However, does it make sense to use optimal value of recent IS periods. Isn’t it much wiser to use the ATR = 15 or any value within the ATR range (let’s say between 10 and 17) that is especially not an optimized one? Isn’t the probability of outperformance much higher in this case? Yes, the OOS results are positive but show a strong underperformance compared to optimization over the whole OOS time frame.
Of course if option c is taken (ATR = 15), this can be considered as an IS period itself of which ATR = 15 would be the optimized value but obviously it brought much better results than the OOS of WFA.
If the WFA OOS results were > 100% of the optimization result or much better than the average of the optimization result for the other ATR values (which I have not listed in my example) it would be obvious take option a). But what to do in the case of our example?

I hope the issue is understandable.

Thanks in advance!
Trader M

MAZINGUER
Posts: 75
Joined: 06 Jan 2017
Has thanked: 9 times
Been thanked: 25 times

Re: Walk Forward Analysis - what parameters to use?

Postby MAZINGUER » 26 Jan 2017

Hi Trader M,

-The first thing I would say is that you never optimize an automatic system for maximum benefit. It is best to optimize by some ratio risk / benefit or similar
-Then it is important to perform the optimization in an ample historical so that the sample is representative (have periods bullish, bearish, gaps, etc.)
-Any system that has more than four or five parameters is at risk of being over-optimized
- When the robust zone is well bounded, the maximum performance combinations tend to work better in OS.
- The key to correctly delimiting the robust zone is to make the IS assessment a long historical one and to unabashedly pick up the market variability: Bullish, bearish and lateral, along with high and low volatility.
- When a system is not over-optimized there is a high probability that the best parametric combinations are the ones that project better results in OS.
- The choice of parametric combinations near or below the median consistently produces worse results in OS than the maximum performance combinations.

Trader M
Posts: 8
Joined: 06 Jan 2017
Has thanked: 4 times

Re: Walk Forward Analysis - what parameters to use?

Postby Trader M » 28 Jan 2017

Hi Mazinguer,

thanks for your reply.
If I read your statements correctly you are saying that a good system performs very well in OOS periods. The OOS results should be exceptionally good BECAUSE they are based on recent optimization (the IS) and not being only quite good DESPITE recent optimization.
In my example above - if the option of changing the system was not given - would you discard the system entirely or would trade it? And if yes with what parameters?

Cheers
Trader M

MAZINGUER
Posts: 75
Joined: 06 Jan 2017
Has thanked: 9 times
Been thanked: 25 times

Re: Walk Forward Analysis - what parameters to use?

Postby MAZINGUER » 28 Jan 2017

Hi, TraderM,
As I wrote in my previous post, there are many factors to take into account to perform a good optimization.
From the results of the system you have put, I would not trust because you have done your optimization based on maximum benefit (NetProfit) and I told you that is a mistake. Always try to optimize using a fitness function based on some risk / benefit ratio or similar.
Other points are also important: 1) consider a good track record, 2) the less inputs to optimize the better, etc.
In general, for a correct optimization:
1-Dimension the parameters with a logical value.
2-Perform backtest tests. The system must be robust even before optimizing.
3-Optimize the parameters. Better to use the method of genetic optimization than exhaustive, because with genetic optimization it looks for stable zones and not statistical coincidences. Observe the stable zones for each parameter. Note that the system must not only work with a value determined for each input. It should work fine, with a few dimensions of each input. For example, if you optimized the ATR in a range from 10 to 30, it is best to get optimal values ​​for the ATR of eg 22,23,24, 25 and 26, so that the robust zone for the ATR Would be from 22 to 26.
4- Once the robust zones have been obtained for all the inputs, the system must work well with any of the parameters that you choose and that are within the robust zones.
5-Perform several optimization tests by changing the area dedicated to OOS
6- Perform other tests to verify the robustness of the system such as the calculation of the SQN (System Quality Number)
6-Never put the system to work in real until you have tested its behavior in a simulated account
regards


Return to “MultiCharts”