fitness function and predictive power

Questions about MultiCharts and user contributed studies.
fringe
Posts: 20
Joined: 17 Jan 2011
Been thanked: 2 times

fitness function and predictive power

Postby fringe » 07 Oct 2011

Hi everyone,

I am currently experimenting with different fitness functions that I found in this forum (see http://www.multicharts.com/discussion/v ... f=5&t=8838). As I was doing this, I thought: how does the choice of a particular fitness function influence the predictive power of a trading system?

Let's say I would like to increase the average trade of a trading system, while minimizing the drawdown and improving the profit factor. In this case, I could use a fitness function such as: (Net Profit / Max Drawdown) * Avg Trade * Profit Factor. My question is: am I curve-fitting the results, by trying to artificially make a trading system tradable (high enough avg trade and profit factor) that otherwise wouldn't be? Should I stick with more traditional fitness functions such as the CPC (Sunny Harris), Expectancy (Van K. Tharp), PROM (Robert Pardo), etc.?

With the fitness function above, my system walks forward correctly (IS vs OOS). I also ran a full walk-forward anaylysis and the WFE was good. Still my question remains: is it so because I artificially inflated the avg trade with my fitness function?

I hope I was able to explain myself... I am curious to know what you guys think about this matter.

Thank you!

User avatar
furytrader
Posts: 354
Joined: 30 Jul 2010
Location: Chicago, IL
Has thanked: 155 times
Been thanked: 217 times

Re: fitness function and predictive power

Postby furytrader » 07 Oct 2011

Avoiding the traps of curve fitting is probably one of the hardest parts of system development, and it's something I wrestle with all of the time, especially as I use tools like genetic algorithms and similar approaches to system development. I can't speak specifically to the fitness function you've created, but in terms of determining whether something is curve-fit, I focus on the following:

1) Do I have enough trades to provide confidence in the statistics? Am I generating 30 trades or 300?

2) Are a handful of trades generating the bulk of my results? (This won't be captured by using the average trade statistic or the profit factor) Looking at the equity curve is a good, quick way of determining this.

3) Is the market data over which these statistics were generated varied enough, showing high volatility and low volatility conditions, bull markets, bear markets, sideways markets, etc?

4) If the model has symmetric rules to get both long and short, is one side (long trades, for example) making all of the money?

5) Is the model based on a sound market principle or is it just "whatever seems to work"?

These are just some ideas - however, they underscore the fact that, at least in my opinion, one single number isn't going to tell you whether a particular system is robust. There are myriad ways to fool one's self into thinking that a aystem is robust when it is, in fact, curve fit.

Good luck!

fringe
Posts: 20
Joined: 17 Jan 2011
Been thanked: 2 times

Re: fitness function and predictive power

Postby fringe » 07 Oct 2011

Hi furytrader,

and thank you for your input. Luckily, my trading system passes all of the tests you mentioned.

Do you also run a Montecarlo simulation on your systems? Do you think that it would signal, a part from a clearer picture of the potential max DD, a trading system that is curve-fitted? If so, which software do you use for the Montecarlo simulation?

Thanks.

User avatar
furytrader
Posts: 354
Joined: 30 Jul 2010
Location: Chicago, IL
Has thanked: 155 times
Been thanked: 217 times

Re: fitness function and predictive power

Postby furytrader » 07 Oct 2011

I've not experimented with Monte Carlo simulations, as I don't understand how they would reveal whether a system is curve-fit. Aren't they based on statistics that may already be generated by a curve-fit system? If you could point me to some information that explains the value of Monte Carlo, I'd really appreciate it.

As for software, one piece of software that is helpful is called Market System Analyzer http://www.adaptrade.com/MSA/index.htm. It does things like Monte Carlo Simulation as well as significance testing and auto-correlation (i.e., run) studies. It also can be used to test different sizing methodologies to see which ones generate the best risk/return profiles, even across a portfolio of systems.

It's not terribly expensive ($349). It's worth checking out (I believe they let you use a trial version for 30 days).

fringe
Posts: 20
Joined: 17 Jan 2011
Been thanked: 2 times

Re: fitness function and predictive power

Postby fringe » 07 Oct 2011

Hey Furytrader,

Market System Analyzer is indeed the software I was looking into buying! :-) It seems very powerful and not that expensive.

I want to read more about Montecarlo Analysis myself. John Ehlers has a chapter on it in his excellent "Cybernetic Analysis for Stocks and Futures". It's by reading that chapter that I started wondering about the effectiveness of my fitness function...

User avatar
furytrader
Posts: 354
Joined: 30 Jul 2010
Location: Chicago, IL
Has thanked: 155 times
Been thanked: 217 times

Re: fitness function and predictive power

Postby furytrader » 07 Oct 2011

Yes, if you're looking to develop a sizing methodology for a system (or systems) that trade across different markets, the MSA will save you many hours of time. For that alone, it's worth the price.

khalaad
Posts: 323
Joined: 07 Jan 2007
Location: Lahore, Pakistan
Has thanked: 64 times
Been thanked: 57 times

Re: fitness function and predictive power

Postby khalaad » 07 Oct 2011

Gentlemen,

It is not my intention to be rude, but understanding the following might just save the $349 the software costs:
Ten rules for position sizing:

1. Bet high enough to make meaningful profits when you win.

2. Bet low enough so you are ok financially and psychologically when you lose.

3. If (1) and (2) don't overlap, don't trade.

4. Don't go adding a bunch of rules that don't work, just so you have ten rules.
Good luck.

Khalid


Return to “MultiCharts”