Hi,
Before I go into further details can I ask anyone reading this post to carry out the following and report your findings:
Open a new workspace and chart - Any timeframe any symbol (I used 30min CL from 02/06/2003 to 26/02/2014).
Load a signal / strategy - Any settings (I used BM=OFF and 16k maxbarsback with >800 trades)
Note the results - for simplicity I note the return on max drawdown
Now copy that chart and paste 5 times.
Do all charts have the same return on max drawdown figure?
I find in the majority of situations when I carry out the above anywhere between 1 and 3 of the charts will show different results.
I also find the same happens in the portfolio backtester. Sometimes I have to repeat a large portfolio backtest about 10 times before the constituent signals show results that match results in the Multicharts backtester.
FYI I'm always in offline mode when I'm running backtests.
I don't have a clue where this inconsistency comes from. Would love to know. Routine system reoptimisations are taking me about 3x what they should due to this bug.
Also this bug has come on 3 different computers I've used and with 3 completely different strategies.
Thanks in advance for sharing!
Backtesting bug throughout Multicharts ? [SOLVED]
- JoshM
- Posts: 2195
- Joined: 20 May 2011
- Location: The Netherlands
- Has thanked: 1544 times
- Been thanked: 1565 times
- Contact:
Re: Backtesting bug throughout Multicharts ?
Yes, all have charts display the same drawdown here on MultiCharts64 Version 8.8 Release (Build 8967):(...)
Open a new workspace and chart - Any timeframe any symbol (I used 30min CL from 02/06/2003 to 26/02/2014).
Load a signal / strategy - Any settings (I used BM=OFF and 16k maxbarsback with >800 trades)
Note the results - for simplicity I note the return on max drawdown
Now copy that chart and paste 5 times.
Do all charts have the same return on max drawdown figure?
Code: Select all
Total trades: 181.00 Profit: -103.20 MaxDrawdown: -472.30 Initial capital: 100000.00
Total trades: 181.00 Profit: -103.20 MaxDrawdown: -472.30 Initial capital: 100000.00
Total trades: 181.00 Profit: -103.20 MaxDrawdown: -472.30 Initial capital: 100000.00
Total trades: 181.00 Profit: -103.20 MaxDrawdown: -472.30 Initial capital: 100000.00
Total trades: 181.00 Profit: -103.20 MaxDrawdown: -472.30 Initial capital: 100000.00
For example, what strategies exhibit this behaviour? Can you share the code of these? Which data (or data feed) are you using? Did you use the same QuoteManager database on all three computers? Were the results different when you removed (i.e. cleaned) the cache? Was only the MaxDrawDown different, or were other metrics (e.g., total trades, net profit) also off? How much was the MaxDrawDown different: rounding error different or completely off?
Btw, a MaxBarsBack setting of 16,000 (sixteen thousand), is that a typo?
Re: Backtesting bug throughout Multicharts ?
No typo I'm afraid re maxbarsback.
More details will follow.
For now.....
I use imported minute ascii files to create my backtest symbols.
My backtests rarely have less than 600 trades.
I may wait to see if I can get a couple more replies first. Otherwise there is an awful lot of details and conditions to add. I was just hoping to gauge the prevalence of my issue first. If it's low then it'll be worth the 1-2 hours needed to setup a test environment (minus proprietary info) that can replicate the issue.
More details will follow.
For now.....
I use imported minute ascii files to create my backtest symbols.
My backtests rarely have less than 600 trades.
I may wait to see if I can get a couple more replies first. Otherwise there is an awful lot of details and conditions to add. I was just hoping to gauge the prevalence of my issue first. If it's low then it'll be worth the 1-2 hours needed to setup a test environment (minus proprietary info) that can replicate the issue.
- JoshM
- Posts: 2195
- Joined: 20 May 2011
- Location: The Netherlands
- Has thanked: 1544 times
- Been thanked: 1565 times
- Contact:
Re: Backtesting bug throughout Multicharts ?
Why do you need such a high MaxBarsBack value?No typo I'm afraid re maxbarsback.
If your strategy is so complex and elaborative that it needs 16,000 bars to perform its calculations, I'm inclined to think that what you term a backtesting bug might very well be a coding issue in your complicated strategy.
Of course I might be mistaken, but if you have a complex strategy, we probably first need to rule that out as causing this behaviour.
- TJ
- Posts: 7742
- Joined: 29 Aug 2006
- Location: Global Citizen
- Has thanked: 1033 times
- Been thanked: 2222 times
Re: Backtesting bug throughout Multicharts ?
maxbarsback is a misnomer,
it should really be called MinBarsForward.
it should really be called MinBarsForward.
Re: Backtesting bug throughout Multicharts ?
It's not that complex. Just has to reference bars from long ago. Calculation speed doesn't really improve using an array to store these values. It is a simple strategy.
I will wait to here from others and start putting together a test environment.
I can't for certain rule out the code so will go and look at unoverlapping code that creates the same issues. But nothing springs to mind (in the code) that would explain a copied chart generating different results....no randomising code or references to computer time or anything.
But I do notice from time to time with my live charts, it will load and generate completely false historical entries. I will then reload the chart and these strange entries dissappear. Happened today....first time for a couple of months.
I will wait to here from others and start putting together a test environment.
I can't for certain rule out the code so will go and look at unoverlapping code that creates the same issues. But nothing springs to mind (in the code) that would explain a copied chart generating different results....no randomising code or references to computer time or anything.
But I do notice from time to time with my live charts, it will load and generate completely false historical entries. I will then reload the chart and these strange entries dissappear. Happened today....first time for a couple of months.
- TJ
- Posts: 7742
- Joined: 29 Aug 2006
- Location: Global Citizen
- Has thanked: 1033 times
- Been thanked: 2222 times
Re: Backtesting bug throughout Multicharts ?
You might have misunderstood the definition of Maxbarsback.It's not that complex. Just has to reference bars from long ago.
::
Re: Backtesting bug throughout Multicharts ?
Don't want to sidetrack the thread. There are two variables when multiplied mean that I need x mbb set. it's a highest high, lowest low xnb thing. That's all. 16kMBB is far in excess of what I need. I use this so that when I add everything to the portfolio backtester I can share the same data start state and have all intruments, timeframes and signals have their first trades at the same approx date.You might have misunderstood the definition of Maxbarsback
Are we focusing on the mbb setting because that could be a setting that causes the described bug????????????????????????(or not)
- TJ
- Posts: 7742
- Joined: 29 Aug 2006
- Location: Global Citizen
- Has thanked: 1033 times
- Been thanked: 2222 times
Re: Backtesting bug throughout Multicharts ?
You have misunderstood the definition of Maxbarsback.Don't want to sidetrack the thread. There are two variables when multiplied mean that I need x mbb set. it's a highest high, lowest low xnb thing. That's all. 16kMBB is far in excess of what I need. I use this so that when I add everything to the portfolio backtester I can share the same data start state and have all intruments, timeframes and signals have their first trades at the same approx date.You might have misunderstood the definition of Maxbarsback
Are we focusing on the mbb setting because that could be a setting that causes the described bug????????????????????????(or not)
Please look it up in the wiki.
Re: Backtesting bug throughout Multicharts ?
TJ mate, there is a near infinite amount of things I don't know, fact.
One thing I do know is that,
you have misunderstood the purpose of this thread.
Also you have added no value to this thread.
Unless I have missed something of course.
If you can prove otherwise then be my guest.
One thing I do know is that,
you have misunderstood the purpose of this thread.
Also you have added no value to this thread.
Unless I have missed something of course.
If you can prove otherwise then be my guest.
Re: Backtesting bug throughout Multicharts ?
I don't see how Maxbarsback is relevant to receiving different performance while using copy and paste.
Wlkinsw, do you have IOG turned on? I suspect that there's something off in the code.
Wlkinsw, do you have IOG turned on? I suspect that there's something off in the code.
- Andrew MultiCharts
- Posts: 1587
- Joined: 11 Oct 2011
- Has thanked: 931 times
- Been thanked: 559 times
Re: Backtesting bug throughout Multicharts ?
Yes, Sir:Before I go into further details can I ask anyone reading this post to carry out the following and report your findings:
Open a new workspace and chart - Any timeframe any symbol (I used 30min CL from 02/06/2003 to 26/02/2014).
Load a signal / strategy - Any settings (I used BM=OFF and 16k maxbarsback with >800 trades)
Note the results - for simplicity I note the return on max drawdown
Now copy that chart and paste 5 times.
Do all charts have the same return on max drawdown figure?
-
- Posts: 199
- Joined: 25 Jan 2008
- Has thanked: 2 times
- Been thanked: 7 times
Re: Backtesting bug throughout Multicharts ?
Using limit and stop orders in the enhanced mode is likely to impact results the way you say see it showing up. Classic mode with orders based on close, I would guess do not push the envelope enough to cause the issue.
Re: Backtesting bug throughout Multicharts ?
I promised I would come back with more info and haven't. I will not have time to either for another week or so.
Feel free to close this thread and I can restart it at another time when I have gathered the required info
Thanks for your replies.
Feel free to close this thread and I can restart it at another time when I have gathered the required info
Thanks for your replies.
Re: Backtesting bug throughout Multicharts ? [SOLVED]
Ok, I found the problem.
Apologies but it was a rookie mistake, the solution to which, at least I hope will help the newbies.
Basically there was an inappropriate use of a series function. I should have been using a simple function instead.
On one line of code I was using Highestfc( , ) instead of Highest( , ). The use of the former was creating variance in backtesting results. Strangely, not all the time.
Changing this means that backtest results are now always the same. The backtest results are the same with Highest as the majority of Highestfc backtests. I can't get my head around why Highestfc will cause a varying but not peristent difference in backtest results.
Thanks for the replies.
Apologies but it was a rookie mistake, the solution to which, at least I hope will help the newbies.
Basically there was an inappropriate use of a series function. I should have been using a simple function instead.
On one line of code I was using Highestfc( , ) instead of Highest( , ). The use of the former was creating variance in backtesting results. Strangely, not all the time.
Changing this means that backtest results are now always the same. The backtest results are the same with Highest as the majority of Highestfc backtests. I can't get my head around why Highestfc will cause a varying but not peristent difference in backtest results.
Thanks for the replies.