Playback results different from backtested results

Questions about MultiCharts and user contributed studies.
paulc
Posts: 43
Joined: 26 Sep 2012
Has thanked: 7 times
Been thanked: 2 times

Playback results different from backtested results

Postby paulc » 01 Nov 2012

i have IOG turned on for my signals. i get a certain result on backtesting.

when i use playback to see the strategy in action i see a different result. i suspect this is due to how the bar is formed. i'm using candlesticks 1min resolution with no magnifier on the backtesting precision.

i want to get a consistent result between backtesting and playback which is most realistic to market results.

1. why does backtesting and playback give different results?
2. is playback always more accurate in simulating market realtime results?
3. other than them giving the same result, how do i know whether playback and backtesting have been setup the same?

Thanks.

User avatar
TJ
Posts: 6518
Joined: 29 Aug 2006
Location: Global Citizen
Has thanked: 957 times
Been thanked: 1883 times

Re: Playback results different from backtested results

Postby TJ » 01 Nov 2012

paulc wrote:i have IOG turned on for my signals. i get a certain result on backtesting.

when i use playback to see the strategy in action i see a different result. i suspect this is due to how the bar is formed. i'm using candlesticks 1min resolution with no magnifier on the backtesting precision.

i want to get a consistent result between backtesting and playback which is most realistic to market results.

1. why does backtesting and playback give different results?
2. is playback always more accurate in simulating market realtime results?
3. other than them giving the same result, how do i know whether playback and backtesting have been setup the same?

Thanks.


You can start here:
viewtopic.php?f=16&t=10811

post #10 might address some of your questions.

paulc
Posts: 43
Joined: 26 Sep 2012
Has thanked: 7 times
Been thanked: 2 times

Re: Playback results different from backtested results

Postby paulc » 01 Nov 2012

ok so i found the below page

http://www.multicharts.com/trading-software/index.php/Why_is_Data_Playback_strategy_performance_different_from_Backtesting_results%3F

i dont really understand it but i think i should have barmagnifier on to get the most accurate backtesting result?

User avatar
TJ
Posts: 6518
Joined: 29 Aug 2006
Location: Global Citizen
Has thanked: 957 times
Been thanked: 1883 times

Re: Playback results different from backtested results

Postby TJ » 01 Nov 2012

paulc wrote:ok so i found the below page

http://www.multicharts.com/trading-software/index.php/Why_is_Data_Playback_strategy_performance_different_from_Backtesting_results%3F

i dont really understand it but i think i should have barmagnifier on to get the most accurate backtesting result?


That's right... you need Precision Backtesting with Barmagnifier turned on.

more readings:
without Barmagnifier
https://www.multicharts.com/trading-soft ... ssumptions
These users thanked the author TJ for the post (total 2):
Andrew MultiChartsJoshM

paulc
Posts: 43
Joined: 26 Sep 2012
Has thanked: 7 times
Been thanked: 2 times

Re: Playback results different from backtested results

Postby paulc » 03 Nov 2012

So i have IOG turned on and have backtesting precision turned on. My chart is 1 h resolution and backtesting precision is set to 1 minute.

I note my list of trades from the performence report is showing trades being timed on the hour ie not at minute intervals.

does this suggest i should also have 1 minute data hidden on my chart to allow the backtesting precsiion to use the 1 minute data?

what is strange is that i get a different result when i turn off backtesting precision. this suggests that it is having some effect.

can you tell me why i am getting the different result and whether the trades are being conducted on the hour data or minute data as per backtesting precision setting.

thanks.

User avatar
TJ
Posts: 6518
Joined: 29 Aug 2006
Location: Global Citizen
Has thanked: 957 times
Been thanked: 1883 times

Re: Playback results different from backtested results

Postby TJ » 03 Nov 2012

paulc wrote:So i have IOG turned on and have backtesting precision turned on. My chart is 1 h resolution and backtesting precision is set to 1 minute.

I note my list of trades from the performence report is showing trades being timed on the hour ie not at minute intervals.

does this suggest i should also have 1 minute data hidden on my chart to allow the backtesting precsiion to use the 1 minute data?

what is strange is that i get a different result when i turn off backtesting precision. this suggests that it is having some effect.

can you tell me why i am getting the different result and whether the trades are being conducted on the hour data or minute data as per backtesting precision setting.

thanks.


Yes, you will need the required data to do the required backtest.
If you have hourly data, usually that should be ok because hourly charts are built from minute data.
To be sure, you should open your data to check for integrity and completeness.

Go to your QuoteManager, highlight the symbol, then
Edit > Edit Data
You can load the data on the screen and check for its validity.


For more information:
Understanding Precise Backtesting
https://www.multicharts.com/trading-soft ... acktesting
These users thanked the author TJ for the post:
paulc

paulc
Posts: 43
Joined: 26 Sep 2012
Has thanked: 7 times
Been thanked: 2 times

Re: Playback results different from backtested results

Postby paulc » 03 Nov 2012

thanks for that. i actually had tick data from another query i'd done.

so i dont need to add the 1m data as an instrument on the chart. the backtest will test on 1m data, provided i have it, if the magnifier is set to 1m data.

paulc
Posts: 43
Joined: 26 Sep 2012
Has thanked: 7 times
Been thanked: 2 times

Re: Playback results different from backtested results

Postby paulc » 03 Nov 2012

actually what about the trade times in the performance report. those show trade times on the hour but presumably these trades are not necessarily on the hour but within the hour.

how can one check the trade price without the actual trade time?

should i actually load 1m data in the chart?

User avatar
TJ
Posts: 6518
Joined: 29 Aug 2006
Location: Global Citizen
Has thanked: 957 times
Been thanked: 1883 times

Re: Playback results different from backtested results

Postby TJ » 03 Nov 2012

paulc wrote:actually what about the trade times in the performance report. those show trade times on the hour but presumably these trades are not necessarily on the hour but within the hour.

how can one check the trade price without the actual trade time?

should i actually load 1m data in the chart?

You will have 10,000 questions in backtesting. After answering those 10k questions, you will have another 10,000 questions to ask. Many questions have written answers, much more can only be answered by trying.

My suggestion would be to start reading; MultiCharts has gone to great lengths to prepare articles on often encountered questions. They are a gold mine of knowledge; investments in readings will save you much time and headache in your project.

A good place to start:
viewtopic.php?f=16&t=10811


ps. please sign up as a contributor to Wiki, your experience will be much appreciated.

paulc
Posts: 43
Joined: 26 Sep 2012
Has thanked: 7 times
Been thanked: 2 times

Re: Playback results different from backtested results

Postby paulc » 03 Nov 2012

thanks for your time. there is so much to understand that all questions can probably not be answered by a manageable amount of documentation.

i cannot find the answer to the question about the trade time. perhaps MC supprt can help.

thanks again.

User avatar
Andrew MultiCharts
Posts: 1587
Joined: 11 Oct 2011
Has thanked: 931 times
Been thanked: 557 times

Re: Playback results different from backtested results

Postby Andrew MultiCharts » 05 Nov 2012

Hello paulc,

There are several variants of strategy calculation:
1. Auto trading.
2. Forwardtesting (calculation in real-time).
3. Backtesting.
4. Backtesting in data playback.
Each of them can have IOG= ON/OFF. For #3 and #4 Bar Magnifier set to a different precision is also avilable. For #2, #3 and #4 one can turn on Extended Backtesting Mode. Everything also depends on chart type.
Even with all these tools for precise calculation of the strategy one should not expect 100% match in results between any of calculation variats mentioned above.

I also recommend you reading this:
1. Price movement assumptions for backtesting when IOG = OFF.
2. Why backtesting and auto trading provides different results.
3. You have already read this one, so if you have any questions, please let me know.

paulc
Posts: 43
Joined: 26 Sep 2012
Has thanked: 7 times
Been thanked: 2 times

Re: Playback results different from backtested results

Postby paulc » 06 Nov 2012

Hi Andrew,

Thanks. The reference material is excellent. It just takes time to digest and thoroughly understand.

When running live, i'm finding that my open and close orders are triggered far too frequently. As the signal oscillates up and down across the trigger value an order is fired each time.

I know i can control this via the 'Enable IOG' in the format signal setting. I can limit the order to once per bar using this control function.

However, if I chose not to use IOG on (i.e. its off) then I understand the strategy engine will only look at the signal at the end of the bar and place an order, or not, according to the applied strategy once at the end of each bar.

My question is, if I have IOG set to off, in backtesting i assume i should not choose bar magnifier since the signal will not fire an order until the close of the bar. Am i correct in this thinking?

Thanks.

User avatar
TJ
Posts: 6518
Joined: 29 Aug 2006
Location: Global Citizen
Has thanked: 957 times
Been thanked: 1883 times

Re: Playback results different from backtested results

Postby TJ » 06 Nov 2012

paulc wrote:Hi Andrew,

Thanks. The reference material is excellent. It just takes time to digest and thoroughly understand.

When running live, i'm finding that my open and close orders are triggered far too frequently. As the signal oscillates up and down across the trigger value an order is fired each time.

I know i can control this via the 'Enable IOG' in the format signal setting. I can limit the order to once per bar using this control function.

However, if I chose not to use IOG on (i.e. its off) then I understand the strategy engine will only look at the signal at the end of the bar and place an order, or not, according to the applied strategy once at the end of each bar.

My question is, if I have IOG set to off, in backtesting i assume i should not choose bar magnifier since the signal will not fire an order until the close of the bar. Am i correct in this thinking?

Thanks.


please see post #3 and post #4
viewtopic.php?f=16&t=10811
These users thanked the author TJ for the post:
Andrew MultiCharts

paulc
Posts: 43
Joined: 26 Sep 2012
Has thanked: 7 times
Been thanked: 2 times

Re: Playback results different from backtested results

Postby paulc » 06 Nov 2012

Scenario A: Calculation on historical data. Regular mode. No IOG. No Bar Magnifier.
The script is calculated on the bar close. It is considered that all prices were within the bar (Price movement assumption is used). Order filled on any price within the bar.


I'm trying to understand the 'order filled on any price within the bar'.

If the strategy only calculates on the bar close isn't the closing price the only relevant price? ie would the closing price not be the correct price to use in any trade which is triggered? If the order is filled on 'any price', what determines which price is used?

User avatar
TJ
Posts: 6518
Joined: 29 Aug 2006
Location: Global Citizen
Has thanked: 957 times
Been thanked: 1883 times

Re: Playback results different from backtested results

Postby TJ » 06 Nov 2012

paulc wrote:
Scenario A: Calculation on historical data. Regular mode. No IOG. No Bar Magnifier.
The script is calculated on the bar close. It is considered that all prices were within the bar (Price movement assumption is used). Order filled on any price within the bar.


I'm trying to understand the 'order filled on any price within the bar'.

If the strategy only calculates on the bar close isn't the closing price the only relevant price? ie would the closing price not be the correct price to use in any trade which is triggered? If the order is filled on 'any price', what determines which price is used?

see post #4 in this thread.
Intra-bar Price Movement Assumptions
These users thanked the author TJ for the post:
Andrew MultiCharts

paulc
Posts: 43
Joined: 26 Sep 2012
Has thanked: 7 times
Been thanked: 2 times

Re: Playback results different from backtested results

Postby paulc » 06 Nov 2012

Whilst a stop or limit order could be hit if those prices were contained within a bar, I see an order to open can be executed either at OHLC, ie 'not at any price'. Or am i still missing something?

User avatar
TJ
Posts: 6518
Joined: 29 Aug 2006
Location: Global Citizen
Has thanked: 957 times
Been thanked: 1883 times

Re: Playback results different from backtested results

Postby TJ » 06 Nov 2012

paulc wrote:Whilst a stop or limit order could be hit if those prices were contained within a bar, I see an order to open can be executed either at OHLC, ie 'not at any price'. Or am i still missing something?


Your order will be triggered if it is within the High/Low range, thus any price within the bar.
These users thanked the author TJ for the post:
Andrew MultiCharts

paulc
Posts: 43
Joined: 26 Sep 2012
Has thanked: 7 times
Been thanked: 2 times

Re: Playback results different from backtested results

Postby paulc » 06 Nov 2012

but if the strategy only calculates on the bar close then which price does it use within the bar?

for example, if there is a signal to buy at market in effect on the bar close then what price within the bar is assumed for the order to have taken place at in backtesting?

Scenario A: Calculation on historical data. Regular mode. No IOG. No Bar Magnifier.
The script is calculated on the bar close. It is considered that all prices were within the bar (Price movement assumption is used). Order filled on any price within the bar.


im sure the penny will drop soon!

User avatar
Andrew MultiCharts
Posts: 1587
Joined: 11 Oct 2011
Has thanked: 931 times
Been thanked: 557 times

Re: Playback results different from backtested results

Postby Andrew MultiCharts » 08 Nov 2012

paulc wrote:My question is, if I have IOG set to off, in backtesting i assume i should not choose bar magnifier since the signal will not fire an order until the close of the bar. Am i correct in this thinking?

If IOG is off for backtesting, script is calculated only once per bar - on its close. If Bar Magnifier is off for backtesting, then MultiCharts assumes all prices within each bar were available, so an order can be executed anywhere within a bar. If Bar Magnifier is on for backtesting, MC uses only prices available in detailed data series (depends on your Bar Magnifier setup) for an order to be executed with a bar. You can find good explanation here, as TJ recommended.

paulc wrote:I'm trying to understand the 'order filled on any price within the bar'.
If the strategy only calculates on the bar close isn't the closing price the only relevant price? ie would the closing price not be the correct price to use in any trade which is triggered? If the order is filled on 'any price', what determines which price is used?

Script is calculated only once, but it places orders with a specific price (limit, stop orders), so the generated order can get filled anywhere, not only on close. You should not mix the moment of script calculation and moment of order execution.
P.S. Market order will be always executed at open of next bar if strategy is calculated only on closes.


Return to “MultiCharts”