I want to pick up a topic that has been raised already before but I still don’t know how to handle it. It is the interpretation and use of Walk-Forward-Results. I’d like to discuss that using an example.
For simplicity reasons let’s use a system with only 1 parameter, the ATR length and full calendar years as OOS time frame.
Assume there are the following WFA results and I found that many systems fall into that category:
(sorry, I didn't know how else to format a table than using "Code Review", it's of course not PL Code)
Code: Select all
IS Net Profit OOS Net Profit OOS Parameters:
15000 2000 (year 2012) ATR = 13
13000 5000 (year 2013) ATR = 13
20000 9000 (year 2014) ATR = 17
25000 1500 (year 2015) ATR = 11
16000 1000 (year 2016) ATR = 14
So, the total result of the OOS period would be 18500 USD.
In addition to the WFA an optimization is executed that covers the time frame from 2012 to 2016 with the following result:
Net Profit: 35000 USD
Parameter: ATR = 15.
So, my questions are the following ones:
1) The ratio of OOS results compared to the net profit of the optimization result for the same OOS period is 18500/35000 = 52,8%. Who of you considers this as a relevant KPI and how do you interpret it – for example is your pass criteria > 50%?
2) If the system is traded live what parameter value is to be taken and this is the question the other thread didn’t fully answer.
Option a) take ATR = 14 because it is the most recent one that the WFA used. Argumentation: The WFA proved that the system works “even” with optimized parameters, so this one can be taken. It is also the one that might fit best to the recent market conditions (which I doubt because adapting to market conditions still is profitable but doesn’t bring extra profit).
Option b) take ATR = 17 because that is the value that generated the best OOS results (9000 USD) in calendar year 2014.
Option c) take ATR = 15 because this value generated the best results over the whole 5 years.
Option d) take any random ATR value within the range.
I have doubts especially between Option a and Option c/d. My interpretation is the following:
The system passed the Walk-Forward test. The test demonstrated that the system can handle even parameter values that are optimized, therefore the system can be used for live trading. However, does it make sense to use optimal value of recent IS periods. Isn’t it much wiser to use the ATR = 15 or any value within the ATR range (let’s say between 10 and 17) that is especially not an optimized one? Isn’t the probability of outperformance much higher in this case? Yes, the OOS results are positive but show a strong underperformance compared to optimization over the whole OOS time frame.
Of course if option c is taken (ATR = 15), this can be considered as an IS period itself of which ATR = 15 would be the optimized value but obviously it brought much better results than the OOS of WFA.
If the WFA OOS results were > 100% of the optimization result or much better than the average of the optimization result for the other ATR values (which I have not listed in my example) it would be obvious take option a). But what to do in the case of our example?
I hope the issue is understandable.
Thanks in advance!
Trader M