MC walk forward optimization with GA

Questions about MultiCharts and user contributed studies.
kernel
Posts: 91
Joined: 19 Feb 2013
Has thanked: 21 times
Been thanked: 4 times

MC walk forward optimization with GA

Postby kernel » 21 May 2013

Hi there,

I encountered problem when using MC's walk forward optimization with genetic algorithm. It worked for a while on different securities, then problem occurred when I noticed it took much less time for the same computing load performed before, saying 40 min for 1500 combinations of parameters before (and the output appeared fine), but 1 min for the same 1500 combinations now. The error revealed itself by not popping up the report window at all after the completion of computing.

To remove the degree of variables introduced into the procedure, I went back to re-do the optimization with ALL settings to be the same as when I successfully did it before...but my luck ran out this time, nothing happened after completion and the report window refused to show up.

A quick search on the forum surprised me a little:
viewtopic.php?f=1&t=7642

It turned out I wasn't the only one running into the issue. I wouldn't stretch that far by drawing a conclusion that It was memory issue (however, I did get memory low warning suggesting I reduce the number of bars being processed.....2G memory on a 2-year 1min bar chart, I think it's sensical, suggesting memory leak issue?)

What surprised me was, this very issue rose two years ago and someone reported it back then, and after two years, it's still lingering around. (there wasn't any follow-ups in that thread, not sure what went on after that requested live chat.)

Another issue I have with MC's WFO implementation is, during each intermediate test run, MC simply picks the peak value of your selected target criterion, which is inevitably prone to curve-fitting (WFA was devised to avoid curve-fitting in the first place,for peter's sake, see the logic here?:-)). I believe many would agree with me that the "peak values" are not neccessarily the "best values", which involves the definition of the fitness function in GA implementation. In addition, it'd be much more helpful if the fitness function could be made accessible to end users, in stead of one criterion, combination of multiple target criteria (net profit, profit factor, max drawdown etc) could be selected by end users.

Finally, like many, I wasn't quite sure how the WF optimization should be employed correctly and appropriately in general. Intuitively, one select the parameter set from most recent time slice, if so, why would you need those from previous slices, realizing that the parameter set changes from slice to slice. Do those earlier time slices just serve as a proof or disproof that your strategy was sound/flawed in general?

cheers,

k

User avatar
Henry MultiСharts
Posts: 9165
Joined: 25 Aug 2011
Has thanked: 1264 times
Been thanked: 2957 times

Re: MC walk forward optimization with GA

Postby Henry MultiСharts » 21 May 2013

Hello Kernel,

Which version and build number of MultiCharts do you use ?
I would recommend you to update to MultiCharts 8.7 beta 1 and run your tests, this version contains all of the latest WFO improvements/fixes as well as better indication of why optimization has failed.
I believe many would agree with me that the "peak values" are not neccessarily the "best values", which involves the definition of the fitness function in GA implementation
When there is one criteria selected-peak value is the best value for this criteria.
it'd be much more helpful if the fitness function could be made accessible to end users, in stead of one criterion, combination of multiple target criteria (net profit, profit factor, max drawdown etc) could be selected by end users.
You can do that with Custom Criteria for optimization.
Finally, like many, I wasn't quite sure how the WF optimization should be employed correctly and appropriately in general.
Have you checked our Wiki articles for optimization?

kernel
Posts: 91
Joined: 19 Feb 2013
Has thanked: 21 times
Been thanked: 4 times

Re: MC walk forward optimization with GA

Postby kernel » 21 May 2013

Hi Henry, thanks for the prompt reply.
Hello Kernel,

Which version and build number of MultiCharts do you use ?
I would recommend you to update to MultiCharts 8.7 beta 1 and run your tests, this version contains all of the latest WFO improvements/fixes as well as better indication of why optimization has failed.
The version I use is 7.4, build 6776. Were there improvements/fixes as far as this particular issue (WFO failure) is concerned and necessity of upgrading my 7.4 to 8.7?
When there is one criteria selected-peak value is the best value for this criteria.
I beg to differ on this. Even with a single criteria, let's say you have a sharp spike and a round top with only 80% of the peak value of the spike, the latter would be a better choice for stability consideration....the problem is, MC shields the human decision-making involvement, which can be both good and bad. The pro, in general, handlings such as this simplify the optimization procedure and relieve the end users from having to get involved. However, the con is, when it comes to WFO, whether it's an optimal or sub-optimal parameter set is largely subjective, without leaving a leeway to human-involvement in the intermediate steps, the drawback negates the benefit and leaves the whole procedure pretty much useless....to workaround, one might go thru the whole optimization process manually and hand-pick the optimal set, but, it's a workaround, not a ideal solution.
You can do that with Custom Creteria for optimization.
Good to know one may add criteria as target, I noticed the work-forward efficiency is not among the standard target list, can we add that on?

However, the real problem is not about ability of adding criteria or not to the target list, it's about the aforementioned "best para set" selection.
Have you checked our Wiki articles for optimization?
Yes, I did. Thanks.

User avatar
Henry MultiСharts
Posts: 9165
Joined: 25 Aug 2011
Has thanked: 1264 times
Been thanked: 2957 times

Re: MC walk forward optimization with GA

Postby Henry MultiСharts » 23 May 2013

The version I use is 7.4, build 6776. Were there improvements/fixes as far as this particular issue (WFO failure) is concerned and necessity of upgrading my 7.4 to 8.7?
As I wrote before:
I would recommend you to update to MultiCharts 8.7 beta 1 and run your tests, this version contains all of the latest WFO improvements/fixes as well as better indication of why optimization has failed.
Please upgrade your MultiChartsto ver. 8.7 beta 1.
I beg to differ on this. Even with a single criteria, let's say you have a sharp spike and a round top with only 80% of the peak value of the spike, the latter would be a better choice for stability consideration....the problem is, MC shields the human decision-making involvement, which can be both good and bad. The pro, in general, handlings such as this simplify the optimization procedure and relieve the end users from having to get involved. However, the con is, when it comes to WFO, whether it's an optimal or sub-optimal parameter set is largely subjective, without leaving a leeway to human-involvement in the intermediate steps, the drawback negates the benefit and leaves the whole procedure pretty much useless....to workaround, one might go thru the whole optimization process manually and hand-pick the optimal set, but, it's a workaround, not a ideal solution.
MultiCharts optimization algorithms already address your concerns regarding strategies prone to curve-fitting. The best value can be at the top of the sharp spike or at the top of the round top. Genetic algorithms pick the optimal and sub-optimal parameters and optimize them. As the final result you can have the best result from a sharp spike or from the round top, it depends on your strategy. Please refer to Wikipedia description for more details: Walk forward optimization
Good to know one may add criteria as target, I noticed the work-forward efficiency is not among the standard target list, can we add that on?
How do you calculate the "work-forward efficiency" and then optimize it?
Can you suggest an algorithm of such functionality?

kernel
Posts: 91
Joined: 19 Feb 2013
Has thanked: 21 times
Been thanked: 4 times

Re: MC walk forward optimization with GA

Postby kernel » 23 May 2013

Please upgrade your MultiChartsto ver. 8.7 beta 1.
Hi Henry, I upgraded to ver8.7beta1 as you suggested, nice to see there were more work done after 7, good job by MC!

As far as the WFO module, I consistently ran into errors. The WFO successfully performed the first round of optimization of one parameter set, but popped up an error window report as follows:

Image

the error description reads "passing invalid parameters to service or function", basically the MSVCR90.dll does not like the parameters being passed in and quit.

What I tried to optimized was the beginning time and ending time between which trading is allowed. Nothing complicated here.

To narrow down the problematic area, after the WFO with GA failure, exhaustive optimization was tried under the same condition for the same target parameters, it performed successfully, no grunts. After that, the same optimization was also performed successfully in ver 7's WFO with GA (same condition, same para). In short, seems like the WFO/GA module (potentially the MSVCR90.dll) in ver 8.7 and 7.4 do not agree with each other, what appeared OK to the latter raised a red flag in the former.

How do you calculate the "work-forward efficiency" and then optimize it?
Can you suggest an algorithm of such functionality?
Sure, normalize earch OOS projected net-profit with the IS length, have that value divided by the IS net-profit, that gives you WFE. Which measures whether or not your strategy performs equally well in OOS as in IS.

At present, I do this by hand in spreadsheet after I copy-paste MC's output, what MC needs to do is adding a column named "WFE" as that in mine:

Image
Attachments
WFE.jpg
(46.8 KiB) Downloaded 1259 times
error.jpg
(59.02 KiB) Downloaded 1291 times

User avatar
Henry MultiСharts
Posts: 9165
Joined: 25 Aug 2011
Has thanked: 1264 times
Been thanked: 2957 times

Re: MC walk forward optimization with GA

Postby Henry MultiСharts » 24 May 2013

Image
kernel, when "Application error" window appears with Exception or Assert information please click Terminate button and manually collect the logs. Please go to Start --> Run
Paste the following line %USERPROFILE%\Local Settings\Application Data\TS Support\MultiCharts64\
Press enter. Compress to zip all folders except Cache folder.

Please send me the following information for further investigation:
- workspace you are using
- in QuoteManager select the symbol you are using, make a right click on it->Export data->Export instrument. Send me the Qmd export file for analysis.
- in Power Language editor->File->Export->export with dependent functions the studies you are using the workspaces you are providing. Send me the study export file.
- Please attach screenshots demonstrating the optimization inputs and parameters you are using for replicating the issue.

Once you zip all of this information - please send it to me for analysis (support@multicharts.com).

User avatar
Henry MultiСharts
Posts: 9165
Joined: 25 Aug 2011
Has thanked: 1264 times
Been thanked: 2957 times

Re: MC walk forward optimization with GA

Postby Henry MultiСharts » 24 May 2013

Sure, normalize earch OOS projected net-profit with the IS length, have that value divided by the IS net-profit, that gives you WFE. Which measures whether or not your strategy performs equally well in OOS as in IS.

At present, I do this by hand in spreadsheet after I copy-paste MC's output, what MC needs to do is adding a column named "WFE" as that in mine:

Image
Right, you do that after you have calculation/optimization results. But how will you optimize by this value while you are running optimization if this is not the current strategy parameter but a result of some custom calculation based on the optimization results?

kernel
Posts: 91
Joined: 19 Feb 2013
Has thanked: 21 times
Been thanked: 4 times

Re: MC walk forward optimization with GA

Postby kernel » 25 May 2013

Henry,

WFE is not one of those criteria in the fitness function based on which the parameters are to be optimized, but rather a critical measure whether or not the historical performace will carry on. It'd be very helpful to leave a column for WFE just like "net profit" or "profit factor" (currently MC does not include WFE in its output report).

By definition, WFE refers to the projected OOS data, which is "unseen" during optimization thus is NOT supposed to be one based on which optimization is performed, otherwise it'd be like a dog chasing his own tail, and it'd invalidate the whole purpose of WFO in the first place. :-)

Sorry for the confusion, hope this helps clarify.

-k
p.s. I'll get back to you when I get chance to collect those data when the error repeats itself again.

User avatar
Henry MultiСharts
Posts: 9165
Joined: 25 Aug 2011
Has thanked: 1264 times
Been thanked: 2957 times

Re: MC walk forward optimization with GA

Postby Henry MultiСharts » 27 Jun 2013

WFE is not one of those criteria in the fitness function based on which the parameters are to be optimized, but rather a critical measure whether or not the historical performace will carry on. It'd be very helpful to leave a column for WFE just like "net profit" or "profit factor" (currently MC does not include WFE in its output report).

By definition, WFE refers to the projected OOS data, which is "unseen" during optimization thus is NOT supposed to be one based on which optimization is performed, otherwise it'd be like a dog chasing his own tail, and it'd invalidate the whole purpose of WFO in the first place. :-)
We have added WFE to our to-do list. There is no exact ETA when this functionality will be implemented.


Return to “MultiCharts”