using walk forward test results

Questions about MultiCharts and user contributed studies.
iceman
Posts: 37
Joined: 23 Oct 2009
Has thanked: 16 times

using walk forward test results

Postby iceman » 12 Dec 2011

When you run a walk forward test, you may get back 50 different time frame results. With different OOS input settings. Which results are considered most valid to use going forward? The last line because it is most current. Or are the best OOS results out of 50 considered valid regardless of where they were in the walk forward test?

User avatar
Andrew MultiCharts
Posts: 1587
Joined: 11 Oct 2011
Has thanked: 931 times
Been thanked: 558 times

Re: using walk forward test results

Postby Andrew MultiCharts » 14 Dec 2011

Hello. Would you mind to make it clearer for me? Are you interested in details of how walk-forward optimization works?

iceman
Posts: 37
Joined: 23 Oct 2009
Has thanked: 16 times

Re: using walk forward test results

Postby iceman » 14 Dec 2011

For example say you are using a trialing stop, stop loss, profit target etc with a strategy.
If you run a walk forward test and you get back 30 lines of test results. Each line may have different settings for the trailing stop, profit target etc. So my question is which line is the best to use, the best results may have occurred with data from 1 or 2 years ago. So is this really relevant now, for a short term interday strategy. Or just more current test results?

User avatar
furytrader
Posts: 353
Joined: 30 Jul 2010
Location: Chicago, IL
Has thanked: 155 times
Been thanked: 217 times

Re: using walk forward test results

Postby furytrader » 15 Dec 2011

If someone could write a book or article entitled "Walk-Forward Optimization For Dummies," it would be very much appreciated.

I have yet to read a clear, concise explanation of how to use it when developing a mechanical system or trading one.

In other words, not just focusing on how to execute a walk-forward optimization using MultiCharts but what the results means and how the information generated can be used to improve one's system development efforts.
These users thanked the author furytrader for the post:
iceman

User avatar
TJ
Posts: 7151
Joined: 29 Aug 2006
Location: Global Citizen
Has thanked: 988 times
Been thanked: 2041 times

Re: using walk forward test results

Postby TJ » 15 Dec 2011

super short, super high level, one liner note:

conceptually, in optimization, you want to develop a 3D model with a single distinct peak...
(preferably with the peak in the profit territory).

Image
Attachments
220px-MaximumParaboloid.png
220px-MaximumParaboloid.png (44.13 KiB) Viewed 4207 times
These users thanked the author TJ for the post:
iceman

User avatar
furytrader
Posts: 353
Joined: 30 Jul 2010
Location: Chicago, IL
Has thanked: 155 times
Been thanked: 217 times

Re: using walk forward test results

Postby furytrader » 15 Dec 2011

But what does that data represent? Are you looking for the one parameter setting that is consistently profitable across all of the time periods?

Or are you periodically re-optimizing your results and then showing what those re-optimized results did in the subsequent period?

That's what I don't understand.

Is walk-forward optimization a single test to find robust parameter settings across all of the past history - or is it a process of "re-training" your models over time and examining how that re-training process itself performed historically?

User avatar
Andrew MultiCharts
Posts: 1587
Joined: 11 Oct 2011
Has thanked: 931 times
Been thanked: 558 times

Re: using walk forward test results

Postby Andrew MultiCharts » 19 Dec 2011

Here is the scheme illustrating how not anchored walk-forwad optimization works.
Image
gif image hosting

This image below shows the way anchored walk-forwad optimization works.
Image
png image hosting

Each period (IS+OOS) indicates the results in optimization report with specific inputs of your signal. You can find the inputs under last column in the report. I believe, that depending on your profit during each period you can decide which inputs are the best.
Image
image hosting

User avatar
furytrader
Posts: 353
Joined: 30 Jul 2010
Location: Chicago, IL
Has thanked: 155 times
Been thanked: 217 times

Re: using walk forward test results

Postby furytrader » 21 Dec 2011

In the current issue of Technical Analysis of Stocks and Commodities magazine, there is an overview of the walk forward optimizer in TS 9.0 - I noted that it, in Figure 5 in the article, they provide a run-down of how to evaluate the results of a walk forward test on a "pass/fail" basis. I'm not suggesting that MC copy this - but it points to the fact that walk-forward test results are not always self-explanatory.
These users thanked the author furytrader for the post:
JoshM

momentum
Posts: 321
Joined: 23 Nov 2005
Has thanked: 9 times
Been thanked: 14 times

Re: using walk forward test results

Postby momentum » 25 Dec 2011

WFO is one of the most important aspects of automated trading. Are there plans to further develp both the reporting and functionality of WFO in MC? MC is absolutely killing TS except in WFA. One of the features needed early on is the ability to click on a selected INPUTS report and have it appear in the Strategy inputs, much like a plain OPTIMIZATION. Developing the WFO reporting along the lines of TS and beyond would be a good route to take unless you already have some better ideas.
These users thanked the author momentum for the post (total 2):
JoshMwaveslider

User avatar
JoshM
Posts: 2099
Joined: 20 May 2011
Location: The Netherlands
Has thanked: 1527 times
Been thanked: 1502 times
Contact:

Re: using walk forward test results

Postby JoshM » 26 Dec 2011

momentum wrote:WFO is one of the most important aspects of automated trading. Are there plans to further develp both the reporting and functionality of WFO in MC? MC is absolutely killing TS except in WFA. One of the features needed early on is the ability to click on a selected INPUTS report and have it appear in the Strategy inputs, much like a plain OPTIMIZATION. Developing the WFO reporting along the lines of TS and beyond would be a good route to take unless you already have some better ideas.

Very much agreed. The current WFA output, a simple HTML file, is not "MultiCharts worthy".

Regards,
Josh

User avatar
furytrader
Posts: 353
Joined: 30 Jul 2010
Location: Chicago, IL
Has thanked: 155 times
Been thanked: 217 times

Re: using walk forward test results

Postby furytrader » 26 Dec 2011

It would be great if we could get some people here who are familiar with using walk forward optimization to indicate how they use it and what they would like to see in an enhanced WFO module. How could it be made better??

sptrader
Posts: 710
Joined: 09 Apr 2010
Location: Colorado
Has thanked: 468 times
Been thanked: 261 times
Contact:

Re: using walk forward test results

Postby sptrader » 26 Dec 2011

furytrader wrote:It would be great if we could get some people here who are familiar with using walk forward optimization to indicate how they use it and what they would like to see in an enhanced WFO module. How could it be made better??

****************************************************************************
* I would like to see the "Custom Criteria" for optimization, be done with Easylanguage, rather than traders having to HIRE someone to code simple criteria.. HUGE Problem now..(most traders are NOT professional programmers)

*I'd like to see walk forward testing compare the WFO equity curve to a perfect 45 degree angle equity curve and be able to optimize for best fit on that..

momentum
Posts: 321
Joined: 23 Nov 2005
Has thanked: 9 times
Been thanked: 14 times

Re: using walk forward test results

Postby momentum » 27 Dec 2011

An easy first dtep would be to be able to do a WFO and then flip through equity curves showing the IS result and the OOS result on the same chart. Being able to quickly flip through results by a chart and seeing both IS and OOS on the same chart allows the user to easily see an acceptable set of paramters.

Then, allowing the user to click a button to put that set of inputs into the strategy much like a regular optimization.

WFO is probably the most important part of strategy development as both entry rules and trade management (exits, sizing etc) can be addressed and a robust model developed. Visual equity curve with IS and OOS is quicker and easier to understand.
These users thanked the author momentum for the post (total 2):
TJJoshM

familytrading
Posts: 33
Joined: 24 Sep 2009
Location: Monterey, CA
Been thanked: 3 times
Contact:

Re: using walk forward test results

Postby familytrading » 29 Jun 2012

I have just started WF analysis in 8.0 and I have a couple of suggestions.

1. Allow an equity curve to be plotted for each OS of the report as you flip through the tests.
2. Highlight what the system thinks is the "best" value depending upon the fitness used. So if Net Profit is used, what run and subsequent parameters result in the "best" fitness. Right now you sort in Excel on the "best" NP for the OS I think.
3. One thing I want to do is to change the IS/OS parameters and then check on the validity of graph. Kind of a 3D graph that has NP on the Z access and IS period on one access and OS on another. I can build this with the excel output but it is a pain. Any point on this curve will then give me a parameter set and I can then determine what best parameter set is at a maximum without being on a "local" maximum.

There are hundred's of ways to interpret WF but without the detail runs to review, it can't be done.

Thanks for listening
These users thanked the author familytrading for the post (total 2):
furytraderJoshM

User avatar
furytrader
Posts: 353
Joined: 30 Jul 2010
Location: Chicago, IL
Has thanked: 155 times
Been thanked: 217 times

Re: using walk forward test results

Postby furytrader » 05 Jan 2013

Is it correct to say that walk-forward optimization answers the following question:

"If you have a trading system that you plan on periodically re-optimizing, how effective is that periodic re-optimization process?"

In other words, are the system rules robust enough so that parameters discovered through periodic re-optimization will generate profits out of sample?

Is that what walk-forward testing shows us?

quantarb
Posts: 51
Joined: 14 Apr 2012
Has thanked: 9 times
Been thanked: 32 times

Re: using walk forward test results

Postby quantarb » 05 Jan 2013

furytrader wrote:Is it correct to say that walk-forward optimization answers the following question:

"If you have a trading system that you plan on periodically re-optimizing, how effective is that periodic re-optimization process?"

In other words, are the system rules robust enough so that parameters discovered through periodic re-optimization will generate profits out of sample?

Is that what walk-forward testing shows us?


That is exactly what walk forward analysis is supposed to do. I would be careful about looking at the out of sample performance from Multicharts. Suppose your in sample data is 300 days and your out of sample data is 100 days. If you use something like 50 day moving average your out of sample data will only contain 50 trading days since it will wait 50 days to calculate the moving average. This approach makes no sense in live trading. If you run a simple moving average crossover strategy and found the optimal parameter was 40 days. Would you wait 40 days and then start trading your strategy? This will result in poor out of sample performance even though your strategy might be robust.

There is a simple solution to fixing this problem. You run walk forward analysis to obtain the optimal parameter value for each out of sample period. Then you set the parameter value for each period and run it as a single back test. The end result is you get a nice detail back test to see how your strategy would perform out of sample from periodic re optimization.


if date > OOSDate1 and date <= OOSDate2 then
begin
parameter1 = 23;
parameter2 = 11;
end;

if date > OOSDate2 and date <= OOSDate3 then
begin
parameter1 = 13;
parameter2 = 42;
end;

if date > OOSDate3 and date <= OOSDate4 then
begin
parameter1 = 34;
parameter2 = 60;
end;




etc
These users thanked the author quantarb for the post (total 2):
furytraderJoshM

User avatar
furytrader
Posts: 353
Joined: 30 Jul 2010
Location: Chicago, IL
Has thanked: 155 times
Been thanked: 217 times

Re: using walk forward test results

Postby furytrader » 05 Jan 2013

Thank you for this insight - I hope the developers see this and figure out how to rectify this issue.
These users thanked the author furytrader for the post:
JoshM

User avatar
MC_Prog
Posts: 330
Joined: 28 Feb 2007
Has thanked: 64 times
Been thanked: 34 times

Re: Walk forward analysis for Multicharts

Postby MC_Prog » 06 Jan 2013

MC deserves praise for having Walk-Forward Analysis (WFA) present in the platform. This ensures that everyone is at a minimum exposed to the concept to stimulate their desire for more knowledge about it.

However it is also true what some have said above that WFA is a deep subject deserving of in-depth treatment that is not currently part of the MC platform and (perhaps) never will be, considering all the other competing priorities for the platform itself.

There is excellent 3rd party WFA software available now, fully compatible with MC 32b and 64b versions, that offers very powerful capabilities for the configuration, display, and evaluation of WFA testing and results.

IMO, MC should arguably concentrate their future development efforts in other areas, as the need for in-depth WFA software is quite well covered by the addon software, which is very reasonably priced for what it does.

Meecc
Posts: 40
Joined: 23 Jun 2011
Has thanked: 20 times
Been thanked: 7 times

Re: using walk forward test results

Postby Meecc » 07 Jan 2013

It would be nice to be able to view as to Quantarb's comment
the " aligned single backtest result using the optimized parameter on each OOS sections "
in one click right after the completion of WFO.

Quote of Quantarb
" There is a simple solution to fixing this problem. You run walk forward analysis to obtain the optimal parameter value for each out of sample period. Then you set the parameter value for each period and run it as " a single back test " . The end result is you get a nice detail back test to see how your strategy would perform out of sample from periodic re optimization."

Thank you

User avatar
furytrader
Posts: 353
Joined: 30 Jul 2010
Location: Chicago, IL
Has thanked: 155 times
Been thanked: 217 times

Re: using walk forward test results

Postby furytrader » 07 Jan 2013

MC_Prog, what other programs would you recommend that handle walk-forward analysis fairly well?

quantarb
Posts: 51
Joined: 14 Apr 2012
Has thanked: 9 times
Been thanked: 32 times

Re: Walk forward analysis for Multicharts

Postby quantarb » 07 Jan 2013

MC_Prog wrote:MC deserves praise for having Walk-Forward Analysis (WFA) present in the platform. This ensures that everyone is at a minimum exposed to the concept to stimulate their desire for more knowledge about it.

However it is also true what some have said above that WFA is a deep subject deserving of in-depth treatment that is not currently part of the MC platform and (perhaps) never will be, considering all the other competing priorities for the platform itself.

There is excellent 3rd party WFA software available now, fully compatible with MC 32b and 64b versions, that offers very powerful capabilities for the configuration, display, and evaluation of WFA testing and results.

IMO, MC should arguably concentrate their future development efforts in other areas, as the need for in-depth WFA software is quite well covered by the addon software, which is very reasonably priced for what it does.


What 3rd party software you talking? I don’t find the current auto walk forward analysis to be very useful by itself. There are some few major flaws in its current implementation. It is for this reason that I no longer brother optimizing lookback periods.

Improving the walk forward analysis should be a major priority for Multicharts, because we the end users currently cannot change or improve it ourselves. Walk forward analysis right now is a closed black box solution.

We still don’t have portfolio walk forward analysis. I don’t think are any 3rd party add-on can do portfolio walk forward analysis either.
I just wish Multicharts would at least give the tools to modify and create our custom walk forward methodology.
These users thanked the author quantarb for the post:
JoshM

User avatar
MC_Prog
Posts: 330
Joined: 28 Feb 2007
Has thanked: 64 times
Been thanked: 34 times

Re: Walk forward analysis for Multicharts

Postby MC_Prog » 07 Jan 2013

quantarb wrote:We still don’t have portfolio walk forward analysis. I don’t think are any 3rd party add-on can do portfolio walk forward analysis either.
OK, fair point. My comments above apply to single-symbol WFA. I don't personally consider that much of a limitation because I don't have the need to trade more than a relatively small number of symbols. However, those who are explicitly diversifying across a wide-universe portfolio could be feeling that need, I admit.

I'm happy to provide a personal recommendation if anyone wants to PM me, otherwise Google works pretty well for finding WFA software.

waveslider
Posts: 206
Joined: 16 Oct 2011
Has thanked: 63 times
Been thanked: 20 times

Re: using walk forward test results

Postby waveslider » 19 Apr 2013

The Walk-Forward output in MC is very primitive and needs improvement.

There need to be some metrics regarding the efficiency of the Walk forward testing and its effect/efficiency.

bomberone1
Posts: 188
Joined: 02 Nov 2010
Has thanked: 14 times
Been thanked: 13 times

Re: using walk forward test results

Postby bomberone1 » 19 Jan 2017

Do anybody of top coders YOU try to follow the http://www.clayburg.com/ or http://www.usingeasylanguage.com/trades ... ndicators/ codes?
Do anybody of top researchers try code by Adaptive System with integrate WFA or adaptive input that change when condition of market changes?
My best.

RockyLee
Posts: 1
Joined: 12 Sep 2017
Has thanked: 1 time
Been thanked: 1 time

Re: using walk forward test results

Postby RockyLee » 12 Sep 2017

Hi everyone,

I asked a new feature regarding the WalkForward, I think it could help us getting a better idea if our performance criteria is good or not.
Without adding much computing time!

Please upvote it so the developpement team get it done :)

https://www.multicharts.com/pm/public/m ... es/MC-2324
These users thanked the author RockyLee for the post:
iceman


Return to “MultiCharts”