Realtime vs backtest mismatch problem  [SOLVED]

Questions about MultiCharts and user contributed studies.
glam_100
Posts: 157
Joined: 14 Jun 2006
Has thanked: 2 times
Been thanked: 9 times

Strategy won't calculate until open of next Data2 bar

Postby glam_100 » 02 Sep 2013

I've reported this problem 3 years ago and worked around it at the time. This is a very fundamental yet commonly encountered problem I cannot understand why after all these years it is still here.

What it is:

- The calculation of studies in MC happens at the open of "next bar" which is what it should be no problem here.

- However, if a strategy has any references to more than one data series, the whole strategy won't calculate until a new bar *of that series* is created. For example if a strategy has referenced Data2 the whole strategy won't calculate until there is a new Data2 bar created.

- That means if Data1 has a session end time of 15:15 and Data2 has a session end time of 15:00, the whole study won't calculate from 15:00-15:15 *everyday* in real-time trading until the next morning when there is a new Data2 bar created.

- Under back-testing this happens only on the last session end on chart because we have a new Data2 bar at every session end in *historical* back-testing except for the last session on chart. However in real time this is a daily problem and a very serious one! The strategy simply stops calculating until the next day!!!

- This creates a huge problem when we have multiple data series on a chart because many times we have different session end time among those data series.

- On TS the implementation is correct and we don't have this problem. Not so on MC. I've attached screenshots for both TS and MC.

I have provided with sample data and code and screen shot for TSS to reproduce this problem. Which is very straight forward to reproduce:

Data1: ES future 1 min intraday day session with session end time 15:00
Data2: S&P 500 index 1 min intraday session with session end time 15:15

Strategy Code:

if date data2 <> date[1] data2 then buy this bar close;
if time >= 1505 then sell this bar close;

What to expect:
- On the last session on chart the strategy won't sell at 1505 because no new Data2 bar has been created and the strategy stopped calculating since 1500

What it should be:
- The strategy should continue to calculate since there is still new data bars creating on Data1!!!


IMHO this is a very high priority bug as it affects *basic calculations* whenever the study uses more than 1 data series with different session end time!!! I'm totally surprised this bug is still here since I have reported it in full detail since MC 5.5 or 6.0.
Attachments
Data.zip
Data used for this test
(167.19 KiB) Downloaded 380 times
MC.jpg
MC implementation not correct
(115.79 KiB) Downloaded 2011 times
TS.jpg
TS implementation correct
(83.27 KiB) Downloaded 2005 times

evdl
Posts: 401
Joined: 19 Jan 2011
Location: Netherlands
Has thanked: 85 times
Been thanked: 124 times

Re: [Bug] Strategy won't calculate until open of next Data2

Postby evdl » 02 Sep 2013

Please try this:

Go to format study (your signal or indicator). Select properties tab and unselect "Realtime history matching".

glam_100
Posts: 157
Joined: 14 Jun 2006
Has thanked: 2 times
Been thanked: 9 times

Re: [Bug] Strategy won't calculate until open of next Data2

Postby glam_100 » 02 Sep 2013

Hi evdl,

Thanks for the reply. Un-checking that option doesn't solve the fact that during real-time trading the signal will stop calculating until the next bar of Data2 appears, which is tomorrow.

The correct implementation should be as long as Data1 (or the longest data series in the chart) has new bars the signal should continue to calculate and use the last Data2 bar value for study calculation should the signal reference any data2 values.

This is how TS implements and IMHO the only sensible way to implement. The current MC implementation makes it impossible to continue trading once the shortest data series session has ended. This is unacceptable for example one would trade a 24 hour forex symbol (say the USD/JPY) and may take other data series as input (say the S&P index) which only has a 8 hour session. In this case the study will stop calculating once the S&P session has closed despite what we want to trade is the 24 hour forex symbol.

glam_100
Posts: 157
Joined: 14 Jun 2006
Has thanked: 2 times
Been thanked: 9 times

Re: [Bug] Strategy won't calculate until open of next Data2

Postby glam_100 » 02 Sep 2013

The correct implementation should be this:

For indicators/studies:
- The calculation should continue as long as there is a new bar created on the series that indicator/studies is based on.

For strategies/signals:
- The calculation should continue as long as there is a new bar created on Data1. This is because a strategy/signal calculation is always based on Data1.

For data series that have ended early in the session the value of the last bar should be used for the calculation.

It should be an easy logical fix on the condition of when to stop calculation but a VERY IMPORTANT fix for those who use more than one data series in their studies.

User avatar
Henry MultiСharts
Posts: 9165
Joined: 25 Aug 2011
Has thanked: 1264 times
Been thanked: 2957 times

Re: Strategy won't calculate until open of next Data2 bar

Postby Henry MultiСharts » 18 Sep 2013

We are analyzing this behavior.

User avatar
Andrew MultiCharts
Posts: 1587
Joined: 11 Oct 2011
Has thanked: 931 times
Been thanked: 559 times

Re: Strategy won't calculate until open of next Data2 bar

Postby Andrew MultiCharts » 21 Oct 2013

As we have already discussed with glam_100 in Project Management, the "Real-time history matching" feature works as expected(all the following examples are true if script/study/signal/strategy is applied to/based on data series 1):
  • “Real-Time History Matching” is turned off (the box is cleared), the data series 1 continues updating, the data series 2 stopped updating:
    a script will be calculated on data series 1 using the updating values from data series 1 and the latest static available values from data series 2.
  • “Real-Time History Matching” is turned off (the box is cleared), the data series 1 stopped updating, the data series 2 continues updating:
    a script will be calculated on the latest bar of data series 1 and will not be calculated until new ticks/bars come in on data series 1, no matter if data series 2 is updating or not. At the moment of this last calculation of the script the latest values of data series 2 are used.
  • “Real-Time History Matching” is turned on (the box is checked), the data series 1 continues updating, the data series 2 stopped updating:
    a script will be calculated on a historical bar of data series 1 and will not be calculated until new ticks/bars come in on data series 2, no matter if data series 1 is updating or not.
  • “Real-Time History Matching” is turned on (the box is checked), the data series 1 stopped updating, the data series 2 continues updating:
    a script will be calculated on the latest bar of data series 1 and will not be calculated until new ticks/bars come in on data series 1, no matter if data series 2 is updating or not. At the moment of this last calculation of the script the corresponding (not the latest) values of data series 2 are used.
Glam_100 expects the following behavior: not to use values from bar B and use values from bar A, if the bar B is opened, but not yet closed. Our customers are welcome to vote for the respective feature request.

glam_100
Posts: 157
Joined: 14 Jun 2006
Has thanked: 2 times
Been thanked: 9 times

Realtime vs backtest mismatch problem

Postby glam_100 » 22 Oct 2013

The following behavior has been confirmed by MC (ie. not imagined by me) but they don't view it as a bug. Thus it would only be fixed if enough people would vote for it. I believe realtime vs backtest mismatch is a serious issue (whether we call it a bug or expected behavior). Please vote for it here if you find it relevant to you:

https://www.multicharts.com/pm/viewissu ... no=MC-1439



Details below:
---------------------------------------------------------------------------------------------------

There is an advanced option in MC called "Realtime-history matching" or RHM (default: on). As the name implies, turning it off *will* actually create realtime-history *mismatch*. It is important for anyone who uses more than one data series in their code to have a full understanding of this RHM option (explained below).

Currently there are problems with both turning RHM On and turning it Off:

Consider this example:

Chart:
Data1: 1 min ES future (session time: 24 hour)
Data2: 15 min SP500 cash index data (session time 8:30-1500)

Code:
Line1: if close data2 > 1700 then buy this bar close; // Buy if S&P passes 1700 level
Line2: if time = 1515 then sell this bar close; // End of day (EOD) exit at 1515

Problem #1 (RHM on):

In realtime Line2 simply won't run. This is because with RHM turned on (by default) MC would simply stops calculating at the earliest session-end of all data series referenced (in this case data2 stops at 1500). In backtest we don't see this problem because we have tomorrow's data and the data2 series continue. However in realtime Line2 just won't run.

Problem #2 (RHM off):

When RHM is off, MC will now continue calculation even when data2 stops. Problem #1 is now gone. This is the sole purpose of why this RHM option exist.

However a new and worse problem arises -- realtime backtest mismatch. Turning RHM off changes how MC sample bar data in real-time. In backtest data2 value is sampled on the bar interval of data2 on the chart (in this case 15 min bar) just the way it should be. However in real-time data2 value is now being sampled on the latest real-time tick-by-tick value of data2. The bar interval of data2 is completely ignored and real-time results becomes very different from backtest.

The name of this option says it all -- "Realtime history matching" (or allow mismatching). IMHO no one in their right mind would trade a signal/indicator that mis-matches in realtime vs backtest. Allowing realtime mismatch (RHM off) is not an option at all. Now, without turning RHM off, we are left with Problem #1 unsolved again.

We need a solution that continues to calculate as long as data1 updates (Problem #1) but does not change the way data2 value is sampled to avoid realtime mismatch (Problem #2).

If you currently use or plan to use more than one data series in your indicator/signals and find this issue may be relevant to you please vote for it here:

https://www.multicharts.com/pm/viewissu ... no=MC-1439

Thanks so much!

User avatar
Andrew MultiCharts
Posts: 1587
Joined: 11 Oct 2011
Has thanked: 931 times
Been thanked: 559 times

Re: Realtime vs backtest mismatch problem

Postby Andrew MultiCharts » 22 Oct 2013

Problem #1 (RHM on):

In realtime Line2 simply won't run. This is because with RHM turned on (by default) MC would simply stops calculating at the earliest session-end of all data series referenced (in this case data2 stops at 1500). In backtest we don't see this problem because we have tomorrow's data and the data2 series continue. However in realtime Line2 just won't run.
Doesn't look like a problem to me, simply because you resolved it in the next sentence:
When RHM is off, MC will now continue calculation even when data2 stops. Problem #1 is now gone. This is the sole purpose of why this RHM option exist.
;-)
However a new and worse problem arises -- realtime backtest mismatch. Turning RHM off changes how MC sample bar data in real-time. In backtest data2 value is sampled on the bar interval of data2 on the chart (in this case 15 min bar) just the way it should be. However in real-time data2 value is now being sampled on the latest real-time tick-by-tick value of data2. The bar interval of data2 is completely ignored and real-time results becomes very different from backtest.

The name of this option says it all -- "Realtime history matching" (or allow mismatching). IMHO no one in their right mind would trade a signal/indicator that mis-matches in realtime vs backtest. Allowing realtime mismatch (RHM off) is not an option at all. Now, without turning RHM off, we are left with Problem #1 unsolved again.

We need a solution that continues to calculate as long as data1 updates (Problem #1) but does not change the way data2 value is sampled to avoid realtime mismatch (Problem #2).
From what we discussed previously i understood that all you need is to reference data2 values only if bar of data is closed. I recommended this solution before and in order to let other customers know about it i have to do it one more time:
Checking the barstatus of data2 from your script would be the solution that is available right now.

User avatar
TJ
Posts: 7740
Joined: 29 Aug 2006
Location: Global Citizen
Has thanked: 1033 times
Been thanked: 2221 times

Re: Realtime vs backtest mismatch problem

Postby TJ » 22 Oct 2013

Checking BARSTATUS is a standard practice for sync critical analysis in multi-data studies.

glam_100
Posts: 157
Joined: 14 Jun 2006
Has thanked: 2 times
Been thanked: 9 times

Re: Realtime vs backtest mismatch problem

Postby glam_100 » 22 Oct 2013

Problem #1 (RHM on):

In realtime Line2 simply won't run. This is because with RHM turned on (by default) MC would simply stops calculating at the earliest session-end of all data series referenced (in this case data2 stops at 1500). In backtest we don't see this problem because we have tomorrow's data and the data2 series continue. However in realtime Line2 just won't run.
Doesn't look like a problem to me, simply because you resolved it in the next sentence:
When RHM is off, MC will now continue calculation even when data2 stops. Problem #1 is now gone. This is the sole purpose of why this RHM option exist.
;-)
However a new and worse problem arises -- realtime backtest mismatch. Turning RHM off changes how MC sample bar data in real-time. In backtest data2 value is sampled on the bar interval of data2 on the chart (in this case 15 min bar) just the way it should be. However in real-time data2 value is now being sampled on the latest real-time tick-by-tick value of data2. The bar interval of data2 is completely ignored and real-time results becomes very different from backtest.

The name of this option says it all -- "Realtime history matching" (or allow mismatching). IMHO no one in their right mind would trade a signal/indicator that mis-matches in realtime vs backtest. Allowing realtime mismatch (RHM off) is not an option at all. Now, without turning RHM off, we are left with Problem #1 unsolved again.

We need a solution that continues to calculate as long as data1 updates (Problem #1) but does not change the way data2 value is sampled to avoid realtime mismatch (Problem #2).
From what we discussed previously i understood that all you need is to reference data2 values only if bar of data is closed. I recommended this solution before and in order to let other customers know about it i have to do it one more time:
Checking the barstatus of data2 from your script would be the solution that is available right now.

Hi Andrew,

I have tried the solution you suggested with check barstatus and it doesn't work at all.

For example:
Data1: ES 1 min
Data2: S&P 15 min

Code:
sumHighData2 = sumHighData2 + high data2;

This piece of code will run 15 times before a new data2 bar would create which is what we want.

if I check barstatus:
if barstatus(2) = 2 then sumHighData2 = sumHighData2 + high data2;

This piece of code will only run once on data2 bar close which is not what we want.

The only half workable solution is to do this:
if barstatus(2) <> 2 then sumHighData2 = sumHighData2 + high[1] data2 else sumHighData2 = sumHighData2 + high data2;

Meaning that I use only closed bar value for calculation. However in this case I will have to change *every line* of code to this... not just a check barstatus bracket but actually changing every single line of code. Also all the backtest result will be different because actual code is changed not just a check barstatus. There is no way to know if the behavior in realtime would match backtest.

In short the problem we're facing under multiple data series situation is fundamentally different from intrabar order generation and is not a simple matter of checking barstatus. There is no reasonable work around to this realtime vs backtest mismatch problem created by turning RHM off.
Attachments
2013-10-04_110626.jpg
(147.35 KiB) Downloaded 1981 times
Last edited by glam_100 on 22 Oct 2013, edited 3 times in total.

glam_100
Posts: 157
Joined: 14 Jun 2006
Has thanked: 2 times
Been thanked: 9 times

Re: Realtime vs backtest mismatch problem

Postby glam_100 » 22 Oct 2013

Checking BARSTATUS is a standard practice for sync critical analysis in multi-data studies.
Hi TJ thanks for the reply.

The "sync problem" in Problem #2 above is *created* by MC and shouldn't have happened in the first place. For example we don't have this problem when RHM is turned on (default).

This problem is created by using "in-progress bar" (ie. tick by tick) on data2 to handle different session lengths. The better solution to handle this problem is to simply use the latest closed data2 bar value even if it stops updating. This is how TS handles this problem and IMHO the correct way to handle multiple data series with different session lengths.

User avatar
piranhaxp
Posts: 241
Joined: 18 Oct 2005
Has thanked: 4 times
Been thanked: 30 times

Re: Realtime vs backtest mismatch problem

Postby piranhaxp » 23 Oct 2013

I see you exporting TS-Data via script to an ASCII-file and mapping it in MC.

That's ok till you don't sync time for data1, here @ES.D with CT > 8.30-1515, and time for data2, here SPX.X with ET > 9.30-1600. You can do a "barstatus" checkup as long as you want, but you won't fix it in already mapped and not synced ASCII symbols and intraday timeframes.

May this is part of the problem I already experienced ....

If I'm totally wrong just give me a sign

User avatar
Andrew MultiCharts
Posts: 1587
Joined: 11 Oct 2011
Has thanked: 931 times
Been thanked: 559 times

Re: Realtime vs backtest mismatch problem

Postby Andrew MultiCharts » 23 Oct 2013

Data1: ES 1 min
Data2: S&P 15 min

Code:
sumHighData2 = sumHighData2 + high data2;
What do you want to do with this code? Please formulate your requirements the way i did in one of my posts above:

“Real-Time History Matching” is turned off (the box is cleared), the data series 1 continues updating, the data series 2 stopped updating: a script will be calculated on data series 1 using the updating values from data series 1 and the latest static available values from data series 2.

“Real-Time History Matching” is turned off (the box is cleared), the data series 1 stopped updating, the data series 2 continues updating: a script will be calculated on the latest bar of data series 1 and will not be calculated until new ticks/bars come in on data series 1, no matter if data series 2 is updating or not. At the moment of this last calculation of the script the latest values of data series 2 are used.

“Real-Time History Matching” is turned on (the box is checked), the data series 1 continues updating, the data series 2 stopped updating: a script will be calculated on a historical bar of data series 1 and will not be calculated until new ticks/bars come in on data series 2, no matter if data series 1 is updating or not.

“Real-Time History Matching” is turned on (the box is checked), the data series 1 stopped updating, the data series 2 continues updating: a script will be calculated on the latest bar of data series 1 and will not be calculated until new ticks/bars come in on data series 1, no matter if data series 2 is updating or not. At the moment of this last calculation of the script the corresponding (not the latest) values of data series 2 are used.
=====================================================
Also all the backtest result will be different because actual code is changed not just a check barstatus. There is no way to know if the behavior in realtime would match backtest.
It does behave in backtesting the same way as in real-time and we have already discussed it. If you want to get values from a bar of data2 only when the bar of data = closed bar, then this is the feature request. It is not like it works in real-time now.

glam_100
Posts: 157
Joined: 14 Jun 2006
Has thanked: 2 times
Been thanked: 9 times

Re: Realtime vs backtest mismatch problem

Postby glam_100 » 23 Oct 2013

Data1: ES 1 min
Data2: S&P 15 min

Code:
sumHighData2 = sumHighData2 + high data2;
>>What do you want to do with this code?

The code is to demonstrate that checking barstatus on data2 does not work as a workaround.

New code that may makes more sense:

Data1: ES 1min
Data2: S&P 15 min

Code #1:
if time = 1001 and close data2 > close[1] data2 then buy this bar close;

Code #2 (check barstatus) :
if barstatus(2) = 2 then
begin
if time = 1001 and close data2 > close[1] data2 then buy this bar close;
end;

Code #1 will run.
Code #2 will not run. Check barstatus = 2 forces it to run at data2 bar close (15 min interval) and skip the 10:01 bar.


So I am unable to workaround the realtime history unmatch problem (RHM unchecked) with simiple barstatus checking.

User avatar
Andrew MultiCharts
Posts: 1587
Joined: 11 Oct 2011
Has thanked: 931 times
Been thanked: 559 times

Re: Realtime vs backtest mismatch problem

Postby Andrew MultiCharts » 24 Oct 2013

[:
if barstatus(2) = 2 then
begin
if time = 1001 and close data2 > close[1] data2 then buy this bar close;
end;
I believe the code should be the following:

Code: Select all

var: data2price(0);

data2price = close data2;
if barstatus(2) <> 2 then
data2price = close[1] data2;

if time = 1001 and data2price > data2price[1] then buy this bar close;

glam_100
Posts: 157
Joined: 14 Jun 2006
Has thanked: 2 times
Been thanked: 9 times

Re: Realtime vs backtest mismatch problem  [SOLVED]

Postby glam_100 » 24 Oct 2013

[:
if barstatus(2) = 2 then
begin
if time = 1001 and close data2 > close[1] data2 then buy this bar close;
end;
I believe the code should be the following:

Code: Select all

var: data2price(0);

data2price = close data2;
if barstatus(2) <> 2 then
data2price = close[1] data2;

if time = 1001 and data2price > data2price[1] then buy this bar close;
That's much smarter than my code! Thanks a lot Andrew! I will try to work around with this kind of checking before the issue is addressed on the platform level.

Every line of data2 reference has to be checked this way individually. It is different from the normal Intrabar order generation barstatus checking where just one check for the whole piece of code is needed.

I hope platform level solution will come soon :) Thanks!

glam_100
Posts: 157
Joined: 14 Jun 2006
Has thanked: 2 times
Been thanked: 9 times

Re: Realtime vs backtest mismatch problem

Postby glam_100 » 11 Feb 2014

Hi Andrew,

Is there any update with the problem described in this thread?

As a reminder, the problem in this thread can be summarized as one line below:

In realtime, calculations from a bar of data2 (data3...etc) should only used when the bar of data2 is closed.

The above suggested work around to check barstatus is very tedious for people working with multiple data series.

User avatar
Andrew MultiCharts
Posts: 1587
Joined: 11 Oct 2011
Has thanked: 931 times
Been thanked: 559 times

Re: Realtime vs backtest mismatch problem

Postby Andrew MultiCharts » 11 Feb 2014

Hello glam_100,

At the moment the improvement feature is not scheduled for any particular version.

glam_100
Posts: 157
Joined: 14 Jun 2006
Has thanked: 2 times
Been thanked: 9 times

Re: Realtime vs backtest mismatch problem

Postby glam_100 » 25 Mar 2014

Andrew, how much would it cost if I would like to pay for this feature as custom development?

Thanks!

User avatar
Andrew MultiCharts
Posts: 1587
Joined: 11 Oct 2011
Has thanked: 931 times
Been thanked: 559 times

Re: Realtime vs backtest mismatch problem

Postby Andrew MultiCharts » 25 Mar 2014

Andrew, how much would it cost if I would like to pay for this feature as custom development?
Please contact us at support@multicharts.com to know the details.

glam_100
Posts: 157
Joined: 14 Jun 2006
Has thanked: 2 times
Been thanked: 9 times

Re: Realtime vs backtest mismatch problem

Postby glam_100 » 25 Mar 2014

Thanks Andrew!

capitaldog
Posts: 3
Joined: 29 May 2021
Has thanked: 1 time
Been thanked: 1 time

Re: Realtime vs backtest mismatch problem

Postby capitaldog » 25 Apr 2022

This thread is so valuable and worth thousands of dollars. I am using data2 in a different time zone and the real-time trade always doesn't match my backtest and it is so difficult to reproduce the issue. It made me lose money and I can't fix it even though I tried many times.

Thanks for @glam_100 finding the root cause and thanks for @TJ providing the solution. This really should be posted right next to data2 usage and real-time match wiki with a big red title.

Code: Select all

if BarStatus(2) = 2 then Print("Without checking BarStatus while real-time match off will make you lose money and debug for days");

Tibouss
Posts: 30
Joined: 26 May 2022
Has thanked: 7 times
Been thanked: 1 time

Re: Realtime vs backtest mismatch problem

Postby Tibouss » 30 May 2022

Hello
I have the same problem, i'm using IB Data for a strategy with data 1 = 1 minute and data2 = 5 minutes.
I have some trades with live data in work forward testing (not live trading) and when i open the platform again the trades are totally different.
Adding "if barstatus(2) = 2 then" to my strategy is supposed to resolve this issue?
Can you tell me if i have well understand the solution?
Many thanks in advance.


Return to “MultiCharts”