Backtesting bug throughout Multicharts ?  [SOLVED]

Questions about MultiCharts and user contributed studies.
wilkinsw
Posts: 662
Joined: 21 Apr 2013
Has thanked: 154 times
Been thanked: 104 times

Backtesting bug throughout Multicharts ?

Postby wilkinsw » 10 Apr 2014

Hi,

Before I go into further details can I ask anyone reading this post to carry out the following and report your findings:

Open a new workspace and chart - Any timeframe any symbol (I used 30min CL from 02/06/2003 to 26/02/2014).

Load a signal / strategy - Any settings (I used BM=OFF and 16k maxbarsback with >800 trades)

Note the results - for simplicity I note the return on max drawdown

Now copy that chart and paste 5 times.

Do all charts have the same return on max drawdown figure?


I find in the majority of situations when I carry out the above anywhere between 1 and 3 of the charts will show different results.

I also find the same happens in the portfolio backtester. Sometimes I have to repeat a large portfolio backtest about 10 times before the constituent signals show results that match results in the Multicharts backtester.

FYI I'm always in offline mode when I'm running backtests.

I don't have a clue where this inconsistency comes from. Would love to know. Routine system reoptimisations are taking me about 3x what they should due to this bug.

Also this bug has come on 3 different computers I've used and with 3 completely different strategies.

Thanks in advance for sharing!

User avatar
JoshM
Posts: 2195
Joined: 20 May 2011
Location: The Netherlands
Has thanked: 1544 times
Been thanked: 1565 times
Contact:

Re: Backtesting bug throughout Multicharts ?

Postby JoshM » 10 Apr 2014

(...)
Open a new workspace and chart - Any timeframe any symbol (I used 30min CL from 02/06/2003 to 26/02/2014).

Load a signal / strategy - Any settings (I used BM=OFF and 16k maxbarsback with >800 trades)

Note the results - for simplicity I note the return on max drawdown

Now copy that chart and paste 5 times.

Do all charts have the same return on max drawdown figure?
Yes, all have charts display the same drawdown here on MultiCharts64 Version 8.8 Release (Build 8967):

Code: Select all

Total trades: 181.00 Profit: -103.20 MaxDrawdown: -472.30 Initial capital: 100000.00

Total trades: 181.00 Profit: -103.20 MaxDrawdown: -472.30 Initial capital: 100000.00

Total trades: 181.00 Profit: -103.20 MaxDrawdown: -472.30 Initial capital: 100000.00

Total trades: 181.00 Profit: -103.20 MaxDrawdown: -472.30 Initial capital: 100000.00

Total trades: 181.00 Profit: -103.20 MaxDrawdown: -472.30 Initial capital: 100000.00
Now that I performed this test, as requested, can you report more details? Otherwise I feel we have to prove to you that there is no backtesting bug, while I think it should be the other way around. :]

For example, what strategies exhibit this behaviour? Can you share the code of these? Which data (or data feed) are you using? Did you use the same QuoteManager database on all three computers? Were the results different when you removed (i.e. cleaned) the cache? Was only the MaxDrawDown different, or were other metrics (e.g., total trades, net profit) also off? How much was the MaxDrawDown different: rounding error different or completely off?

Btw, a MaxBarsBack setting of 16,000 (sixteen thousand), is that a typo?

wilkinsw
Posts: 662
Joined: 21 Apr 2013
Has thanked: 154 times
Been thanked: 104 times

Re: Backtesting bug throughout Multicharts ?

Postby wilkinsw » 10 Apr 2014

No typo I'm afraid re maxbarsback.

More details will follow.

For now.....

I use imported minute ascii files to create my backtest symbols.

My backtests rarely have less than 600 trades.

I may wait to see if I can get a couple more replies first. Otherwise there is an awful lot of details and conditions to add. I was just hoping to gauge the prevalence of my issue first. If it's low then it'll be worth the 1-2 hours needed to setup a test environment (minus proprietary info) that can replicate the issue.

User avatar
JoshM
Posts: 2195
Joined: 20 May 2011
Location: The Netherlands
Has thanked: 1544 times
Been thanked: 1565 times
Contact:

Re: Backtesting bug throughout Multicharts ?

Postby JoshM » 10 Apr 2014

No typo I'm afraid re maxbarsback.
Why do you need such a high MaxBarsBack value?

If your strategy is so complex and elaborative that it needs 16,000 bars to perform its calculations, I'm inclined to think that what you term a backtesting bug might very well be a coding issue in your complicated strategy.

Of course I might be mistaken, but if you have a complex strategy, we probably first need to rule that out as causing this behaviour.

User avatar
TJ
Posts: 7740
Joined: 29 Aug 2006
Location: Global Citizen
Has thanked: 1033 times
Been thanked: 2221 times

Re: Backtesting bug throughout Multicharts ?

Postby TJ » 10 Apr 2014

maxbarsback is a misnomer,
it should really be called MinBarsForward.

:-)

wilkinsw
Posts: 662
Joined: 21 Apr 2013
Has thanked: 154 times
Been thanked: 104 times

Re: Backtesting bug throughout Multicharts ?

Postby wilkinsw » 10 Apr 2014

It's not that complex. Just has to reference bars from long ago. Calculation speed doesn't really improve using an array to store these values. It is a simple strategy.

I will wait to here from others and start putting together a test environment.

I can't for certain rule out the code so will go and look at unoverlapping code that creates the same issues. But nothing springs to mind (in the code) that would explain a copied chart generating different results....no randomising code or references to computer time or anything.

But I do notice from time to time with my live charts, it will load and generate completely false historical entries. I will then reload the chart and these strange entries dissappear. Happened today....first time for a couple of months.

User avatar
TJ
Posts: 7740
Joined: 29 Aug 2006
Location: Global Citizen
Has thanked: 1033 times
Been thanked: 2221 times

Re: Backtesting bug throughout Multicharts ?

Postby TJ » 10 Apr 2014

It's not that complex. Just has to reference bars from long ago.
::
You might have misunderstood the definition of Maxbarsback.

wilkinsw
Posts: 662
Joined: 21 Apr 2013
Has thanked: 154 times
Been thanked: 104 times

Re: Backtesting bug throughout Multicharts ?

Postby wilkinsw » 10 Apr 2014

You might have misunderstood the definition of Maxbarsback
Don't want to sidetrack the thread. There are two variables when multiplied mean that I need x mbb set. it's a highest high, lowest low xnb thing. That's all. 16kMBB is far in excess of what I need. I use this so that when I add everything to the portfolio backtester I can share the same data start state and have all intruments, timeframes and signals have their first trades at the same approx date.

Are we focusing on the mbb setting because that could be a setting that causes the described bug????????????????????????(or not)

User avatar
TJ
Posts: 7740
Joined: 29 Aug 2006
Location: Global Citizen
Has thanked: 1033 times
Been thanked: 2221 times

Re: Backtesting bug throughout Multicharts ?

Postby TJ » 10 Apr 2014

You might have misunderstood the definition of Maxbarsback
Don't want to sidetrack the thread. There are two variables when multiplied mean that I need x mbb set. it's a highest high, lowest low xnb thing. That's all. 16kMBB is far in excess of what I need. I use this so that when I add everything to the portfolio backtester I can share the same data start state and have all intruments, timeframes and signals have their first trades at the same approx date.

Are we focusing on the mbb setting because that could be a setting that causes the described bug????????????????????????(or not)
You have misunderstood the definition of Maxbarsback.

Please look it up in the wiki.

wilkinsw
Posts: 662
Joined: 21 Apr 2013
Has thanked: 154 times
Been thanked: 104 times

Re: Backtesting bug throughout Multicharts ?

Postby wilkinsw » 10 Apr 2014

TJ mate, there is a near infinite amount of things I don't know, fact.

One thing I do know is that,

you have misunderstood the purpose of this thread.

Also you have added no value to this thread.

Unless I have missed something of course.

If you can prove otherwise then be my guest.

User avatar
MAtricks
Posts: 789
Joined: 09 Apr 2012
Has thanked: 286 times
Been thanked: 288 times

Re: Backtesting bug throughout Multicharts ?

Postby MAtricks » 10 Apr 2014

I don't see how Maxbarsback is relevant to receiving different performance while using copy and paste.

Wlkinsw, do you have IOG turned on? I suspect that there's something off in the code.

wilkinsw
Posts: 662
Joined: 21 Apr 2013
Has thanked: 154 times
Been thanked: 104 times

Re: Backtesting bug throughout Multicharts ?

Postby wilkinsw » 10 Apr 2014

No IOG.

I think it's only fair I go away and do some trial and error research.

I'll be back!

(Sorry TJ for being short.... long day).

User avatar
Andrew MultiCharts
Posts: 1587
Joined: 11 Oct 2011
Has thanked: 931 times
Been thanked: 559 times

Re: Backtesting bug throughout Multicharts ?

Postby Andrew MultiCharts » 11 Apr 2014

Before I go into further details can I ask anyone reading this post to carry out the following and report your findings:

Open a new workspace and chart - Any timeframe any symbol (I used 30min CL from 02/06/2003 to 26/02/2014).

Load a signal / strategy - Any settings (I used BM=OFF and 16k maxbarsback with >800 trades)

Note the results - for simplicity I note the return on max drawdown

Now copy that chart and paste 5 times.

Do all charts have the same return on max drawdown figure?
Yes, Sir:

Image

Image

Image

Image

Image

Image

wilkinsw
Posts: 662
Joined: 21 Apr 2013
Has thanked: 154 times
Been thanked: 104 times

Re: Backtesting bug throughout Multicharts ?

Postby wilkinsw » 11 Apr 2014

Thanks Andrew.

I will come back with further info, I promise.

beck donald
Posts: 199
Joined: 25 Jan 2008
Has thanked: 2 times
Been thanked: 7 times

Re: Backtesting bug throughout Multicharts ?

Postby beck donald » 11 Apr 2014

Using limit and stop orders in the enhanced mode is likely to impact results the way you say see it showing up. Classic mode with orders based on close, I would guess do not push the envelope enough to cause the issue.

wilkinsw
Posts: 662
Joined: 21 Apr 2013
Has thanked: 154 times
Been thanked: 104 times

Re: Backtesting bug throughout Multicharts ?

Postby wilkinsw » 17 Apr 2014

I promised I would come back with more info and haven't. I will not have time to either for another week or so.

Feel free to close this thread and I can restart it at another time when I have gathered the required info

Thanks for your replies.

wilkinsw
Posts: 662
Joined: 21 Apr 2013
Has thanked: 154 times
Been thanked: 104 times

Re: Backtesting bug throughout Multicharts ?  [SOLVED]

Postby wilkinsw » 22 May 2014

Ok, I found the problem.

Apologies but it was a rookie mistake, the solution to which, at least I hope will help the newbies.

Basically there was an inappropriate use of a series function. I should have been using a simple function instead.

On one line of code I was using Highestfc( , ) instead of Highest( , ). The use of the former was creating variance in backtesting results. Strangely, not all the time.

Changing this means that backtest results are now always the same. The backtest results are the same with Highest as the majority of Highestfc backtests. I can't get my head around why Highestfc will cause a varying but not peristent difference in backtest results.

Thanks for the replies.


Return to “MultiCharts”