I am having problems with matching the backtest results with realtime results.
When using multi timeframe strategies with IOG and barmagnifier it is not possible to match the backtest results in realtime if you want to calculate all timeframes on the barmagnifier timeframe. After some timeconsuming investigation, I suspect, it is due to the fact that barmagnifier only works on data1 and not on all the data streams when backtesting. But in realtime, all the timeframes are calculated realtime.
The whole purpose of comparing different timeframes is to see how an indicator is developping on different time settings. But in backtest it is only possible to use the closing values of multiple timeframes. Otherwise you can never match the backtest results in real life, and that is the purpose of backtesting.
In backtest you will at best be able to get the last close of a value of data2, or data3 etc per barmagnifier interval. So to get the same results realtime, you have to build in a delay of calculating the other datastreams with the appropriate barstatus. This is in my opinion a bit strange, because you always want to get the least delay on calculating values unless you want to delay it.
There are more posts about the barmagnifier only working on data1 on the forum, and also there are a lot of feature request about this or related to this. So I only can hope that MC will get to this in the near future.
In the meantime I am looking for solving this problem. I am looking in to converting every indicator value I need in to a signal and calculate this on the timeframe I need, with barmagnifier set to the desired interval (1 tick). And save these values in a text file. And read this textfile in to the strategy also with IOG and barmagnifier on. In this way all the data is calculated realtime in backtest and will be the same as trading in realtime.
Now the question after this long story. Is it possible to read textfiles, all of the same symbol but calculated on different timeframes with the same barmagnifier setting of 1 tick, in a signal, and get this in sync with the chart, on which the signal is applied (also with barmagnifier set to 1 tick).
The textfiles will have the date and time of every tick, and also the calculated value (for example an ATR or moving average).
To read files you can use the Elcollection. But I am not sure how to read the textfiles and match the date and time of every bar in the textfile to every bar on the chart.
Matching backtest results and realtime results
-
evdl
- Posts: 401
- Joined: 19 Jan 2011
- Location: Netherlands
- Has thanked: 85 times
- Been thanked: 124 times
Re: Matching backtest results and realtime results
Instead of textfiles, I thought of a different way to get realtime in backtest for all datastreams.
With ADE, it is possible to send data of other timeframes to the trading chart. Usually I use an indicator to send the data. But an indicator can not look intrabar. So I decided to convert the indicator to an signal and use barmagnifier on 1 tick and send this data to the trading chart. In this way I want to create an barmagnifier situation on every datastream. Like realtime trading without creating an delay just to compensate for the lack of barmagnifier on datastreams other then data1.
This is the situation:
- Trading chart 30 tick (IOG on and BM on 1 tick) with strategy and ADE receiver in the code
- ADE Sending chart 240 tick with a signal (IOG on and BM on 1 tick).
- All variabels in the strategy and sender signal have intrabarpersist
- Symbol and data are the same for both charts
First I will run the sender signal to get all values in the ADE map. And then I run the strategy with the ADE receiver to get the values in my strategy.
This is where is gets confusing, because for some reason, when I look at the print outs. The ADE values received in the strategy are faster then the values in the ADE sender. Was expecting some delay instead of looking in the future. Although both the charts are on BM 1 tick and have the same ticks printed.
For example an ATR calculated on the sender chart is 4.5 at 9:27:51 and this same value is already in the strategy at 9:27:22.
Anyone else who knows what can cause this?
With ADE, it is possible to send data of other timeframes to the trading chart. Usually I use an indicator to send the data. But an indicator can not look intrabar. So I decided to convert the indicator to an signal and use barmagnifier on 1 tick and send this data to the trading chart. In this way I want to create an barmagnifier situation on every datastream. Like realtime trading without creating an delay just to compensate for the lack of barmagnifier on datastreams other then data1.
This is the situation:
- Trading chart 30 tick (IOG on and BM on 1 tick) with strategy and ADE receiver in the code
- ADE Sending chart 240 tick with a signal (IOG on and BM on 1 tick).
- All variabels in the strategy and sender signal have intrabarpersist
- Symbol and data are the same for both charts
First I will run the sender signal to get all values in the ADE map. And then I run the strategy with the ADE receiver to get the values in my strategy.
This is where is gets confusing, because for some reason, when I look at the print outs. The ADE values received in the strategy are faster then the values in the ADE sender. Was expecting some delay instead of looking in the future. Although both the charts are on BM 1 tick and have the same ticks printed.
For example an ATR calculated on the sender chart is 4.5 at 9:27:51 and this same value is already in the strategy at 9:27:22.
Anyone else who knows what can cause this?
-
evdl
- Posts: 401
- Joined: 19 Jan 2011
- Location: Netherlands
- Has thanked: 85 times
- Been thanked: 124 times
Re: Matching backtest results and realtime results
ADE can not be used when working with the BM on 1 tick. The reason why you receive the values faster in the receiver chart is because with tick data there is no way of telling when the barclose will be, so the assumption for the receiving chart is to take the closing value of the sending chart at the open of a bar on the receiver chart. And that is looking in to the future. And therefore not useful.
Back to the text files and almost done. Backtesting with all datastreams on 1 tick barmagnifier settings. The only way to get the closest to the results when using the code in real life trading. Without delaying data2, data3. This workaround is only possible when you use the same symbol and same data but with different timeframes. Because you have to synchronise the bars in your code.
Without barmagnifier working on data2, data3 etc and also indicators not working intrabar, it is good to know that you will have to delay these datastreams in your code, otherwise the results will be very different when trading realtime. These subtile details are easily lost when users expect that some features will work in a certain way. As with BM, you would expect this to work on all datastreams on the chart. Without this, the multi time frame feature on one chart is for a lot of situations not usable if you want realistic backtests. And you need to convert back to GV, ADE, or workarounds.
MC does a great job with their software. So I hope BM will be improved soon to work on all datastreams. Also indicators calculating intrabar in backtest would be great.
Back to the text files and almost done. Backtesting with all datastreams on 1 tick barmagnifier settings. The only way to get the closest to the results when using the code in real life trading. Without delaying data2, data3. This workaround is only possible when you use the same symbol and same data but with different timeframes. Because you have to synchronise the bars in your code.
Without barmagnifier working on data2, data3 etc and also indicators not working intrabar, it is good to know that you will have to delay these datastreams in your code, otherwise the results will be very different when trading realtime. These subtile details are easily lost when users expect that some features will work in a certain way. As with BM, you would expect this to work on all datastreams on the chart. Without this, the multi time frame feature on one chart is for a lot of situations not usable if you want realistic backtests. And you need to convert back to GV, ADE, or workarounds.
MC does a great job with their software. So I hope BM will be improved soon to work on all datastreams. Also indicators calculating intrabar in backtest would be great.
Re: Matching backtest results and realtime results
Hi evdl,
Agreed, multidata strats using IOG is a challenge to ensure synchronisation and that is why I'm taking static backtests 'with bar magnifier' switched on as 95% of the time totally worthless to any anticipated RT result - so we are not just talking about a pinch of salt..
Have you tried using data playback with strategy attached using RTHM 'off' and 'on' - when 'off' the strat doesn't wait for tertiary data to 'close' so there is no delay in your signal (even though you may have tried various synchronisation methodologies) the results will likely be considerably more like RT than any 'static' back test could give you.
FWIW I'm finding RTHM 'off' in data playback gives me a far more realistic picture of an 'expected' live performance and comparing same with RTHM 'on' shows whether or not a) more synchronisation is required or b) the signal logic needs tinkering. By contrast with RTHM 'on' I always receive a bar of slip, ie: the signal entry on the chart appears at the Open of the previous bar NOT the open of the new bar - in live trading set to convert a limit order to market a full bar of slip is experienced and (you can detect this visually using dataplayback too) but of course the playback results look artificially better when the slip is not factored in...
Must give credit to MC here though - dataplayback is a quantum leap shortening the time to gain confidence in the expected vs actual performance of a strat - or put another way demo trading on steroids.
Cheers,
Michael
Agreed, multidata strats using IOG is a challenge to ensure synchronisation and that is why I'm taking static backtests 'with bar magnifier' switched on as 95% of the time totally worthless to any anticipated RT result - so we are not just talking about a pinch of salt..
Have you tried using data playback with strategy attached using RTHM 'off' and 'on' - when 'off' the strat doesn't wait for tertiary data to 'close' so there is no delay in your signal (even though you may have tried various synchronisation methodologies) the results will likely be considerably more like RT than any 'static' back test could give you.
FWIW I'm finding RTHM 'off' in data playback gives me a far more realistic picture of an 'expected' live performance and comparing same with RTHM 'on' shows whether or not a) more synchronisation is required or b) the signal logic needs tinkering. By contrast with RTHM 'on' I always receive a bar of slip, ie: the signal entry on the chart appears at the Open of the previous bar NOT the open of the new bar - in live trading set to convert a limit order to market a full bar of slip is experienced and (you can detect this visually using dataplayback too) but of course the playback results look artificially better when the slip is not factored in...
Must give credit to MC here though - dataplayback is a quantum leap shortening the time to gain confidence in the expected vs actual performance of a strat - or put another way demo trading on steroids.
Cheers,
Michael
-
evdl
- Posts: 401
- Joined: 19 Jan 2011
- Location: Netherlands
- Has thanked: 85 times
- Been thanked: 124 times
Re: Matching backtest results and realtime results
Need some help with reading textfiles.
With the following code I read the textfile in my strategy
This works, but for testing I made a 6mb txt file and when reading this in the strategy, it will use 300mb of ram and is slow. The files that I will use for backtesting are around 800mb for every datastream in the strategy. I use 4 datastreams.
textfile looks like this:
With the following code I read the textfile in my strategy
Code: Select all
// get values data2
If Currentbar = 1 then begin
ListC_ID = ListC.New;
If ELC.PathExists(FileName_D2) = false then begin // in case file is not found
FileMissing_data2 = ELC.RaiseError("LSI A20", "Data 2 file missing");
end
else begin
ReadMap_ID_D2 = ListC.ReadFile(ListC_ID, Filename_D2);
BN_List_D2 = ListC.Get(ListC_ID,1); // barnumbers data2
ATR_List_D2 = ListC.Get(ListC_ID,2); // atr value data2
BR_List_D2 = ListC.Get(ListC_ID,3); // barrange value data2
Maxbars_data2 = ListN.Count(BN_list_D2); // end of file, last row
end; // end of If ELC.PathExists(FileName_D2) = false then begin
end; // end of If Currentbar = 1 then begin
// match barnumbers strategy with barnumbers data2
If (Count_barnumber > 0) // barnumbers strategy
and (Count_barnumber <= Maxbars_data2)
then begin
// get value of data2 at matching barnumbers
CountBarnumber_data2 = ListN.get(BN_list_D2, Count_barnumber);
Atr_data2 = ListN.get(ATR_list_D2, Count_barnumber);
Barrange_data2 = ListN.get(BR_list_D2, Count_barnumber);
end; // end of If (Count_barnumber > 0)
textfile looks like this:
- 1, 6.5, 10.00
2, 6.6, 8.00
3, 6.7, 7.00
-
evdl
- Posts: 401
- Joined: 19 Jan 2011
- Location: Netherlands
- Has thanked: 85 times
- Been thanked: 124 times
Re: Matching backtest results and realtime results
I resolved the problem with the ram usage. I use BM on 1 tick and was under the impression that the if currentbar statement, would be the first tick. But currentbar is one of those reserved words that will not change behaviour if you use IOG or BM. (like High or Low, this will also always give the high of the bar of the chart timeframe and not of the BM setting). So it would read my textfiles every tick of the first bar. In my case 30 times.
With so many different behaviours of reserved words when using IOG and BM. It is not that hard to make many mistakes, that you will only found out after a couple of weeks of backtesting. And makes all this work obsolete. The solution for the ram usage, is to always use If barstatus(1) = 2 for code that needs to only work on currentbar = 1. When you ever decide to use IOG or BM, you will not run in to problems.
So code should be:
But there is another issue. And this is a much bigger problem. Earlier on I had notice that with the use of ADE, I could look into the future. But it has nothing to do with ADE. It has to do with the use of multi data series.
If you use data1 and data2 for example. And you want the close of data2 and BM 1 tick on data1.
You will get the future close value of data2 instead of the last known close value of data2, at the open of a new bar on data1.
Data1 = 30 tick
Data2 = 60 tick
With BM or IOG you look intrabar and MC should give you the realtime close of other data series (BM on all data series). But that is not in place at the moment I know. What it should do now, is give the value of the previous closed bar on data2. And keep repeating this on data1 until data2 bar closes. This can not be done with RTHM because it will stop calculating data1 until a bar closes on data2.
Other user (on the .net forum) has noticed the same thing and there are more post about this I found out. http://www.multicharts.com/discussion/v ... 19&t=14815
I am sorry to say and prove me wrong, but the way it is now, you can't use multi data series for backtesting. Because it will give complete fictious results. Please fix this and save the users of MC a lot of time testing bogus strategies.
Update:
User Glam_100 has described in detail what the problem is. There is even an entry in the PM for it.
http://www.multicharts.com/pm/viewissue ... no=MC-1439.
http://www.multicharts.com/discussion/v ... 053#p99053
Probably many user who use multi data series in their strategy, are not aware of this problem. And keep going on, based on something that will never work in realtime.
With so many different behaviours of reserved words when using IOG and BM. It is not that hard to make many mistakes, that you will only found out after a couple of weeks of backtesting. And makes all this work obsolete. The solution for the ram usage, is to always use If barstatus(1) = 2 for code that needs to only work on currentbar = 1. When you ever decide to use IOG or BM, you will not run in to problems.
So code should be:
Code: Select all
// get values data2
If Currentbar = 1 then begin
If barstatus(1) = 2 then begin
ListC_ID = ListC.New;
If ELC.PathExists(FileName_D2) = false then begin // in case file is not found
FileMissing_data2 = ELC.RaiseError("LSI A20", "Data 2 file missing");
end
else begin
ReadMap_ID_D2 = ListC.ReadFile(ListC_ID, Filename_D2);
BN_List_D2 = ListC.Get(ListC_ID,1); // barnumbers data2
ATR_List_D2 = ListC.Get(ListC_ID,2); // atr value data2
BR_List_D2 = ListC.Get(ListC_ID,3); // barrange value data2
Maxbars_data2 = ListN.Count(BN_list_D2); // end of file, last row
end; // end of If ELC.PathExists(FileName_D2) = false then begin
end; // end of If barstatus(1) = 2 then begin
end; // end of If Currentbar = 1 then begin
But there is another issue. And this is a much bigger problem. Earlier on I had notice that with the use of ADE, I could look into the future. But it has nothing to do with ADE. It has to do with the use of multi data series.
If you use data1 and data2 for example. And you want the close of data2 and BM 1 tick on data1.
You will get the future close value of data2 instead of the last known close value of data2, at the open of a new bar on data1.
Data1 = 30 tick
Data2 = 60 tick
With BM or IOG you look intrabar and MC should give you the realtime close of other data series (BM on all data series). But that is not in place at the moment I know. What it should do now, is give the value of the previous closed bar on data2. And keep repeating this on data1 until data2 bar closes. This can not be done with RTHM because it will stop calculating data1 until a bar closes on data2.
Other user (on the .net forum) has noticed the same thing and there are more post about this I found out. http://www.multicharts.com/discussion/v ... 19&t=14815
I am sorry to say and prove me wrong, but the way it is now, you can't use multi data series for backtesting. Because it will give complete fictious results. Please fix this and save the users of MC a lot of time testing bogus strategies.
Update:
User Glam_100 has described in detail what the problem is. There is even an entry in the PM for it.
http://www.multicharts.com/pm/viewissue ... no=MC-1439.
http://www.multicharts.com/discussion/v ... 053#p99053
Probably many user who use multi data series in their strategy, are not aware of this problem. And keep going on, based on something that will never work in realtime.
Re: Matching backtest results and realtime results
I have no desire to prove anyone wrong but I will just pass along my experience and perspective. I use multidata for backtest and realtime and I find MC behavior to be fine. You might have noticed that MC staff did not agree with glam_100 about the RHM issue. I don't think there is a "problem" that needs fixing.I am sorry to say and prove me wrong, but the way it is now, you can't use multi data series for backtesting. Because it will give complete fictious results. Please fix this and save the users of MC a lot of time testing bogus strategies.
Update:
User Glam_100 has described in detail what the problem is. There is even an entry in the PM for it.
Probably many user who use multi data series in their strategy, are not aware of this problem. And keep going on, based on something that will never work in realtime.
-
evdl
- Posts: 401
- Joined: 19 Jan 2011
- Location: Netherlands
- Has thanked: 85 times
- Been thanked: 124 times
Re: Matching backtest results and realtime results
Orion,
thanks for your observation.
Do you also use IOG and BM with your testing? It then becomes very difficult to code your strategy. The workaround (if possible for all situations) at the end of the thread of glam_100 is not the right approach to this. Certainly with 5000 lines of code, that you will need to give a barstatus. And need to test with print statements, is IMO just crazy.
What glam_100 is describing is that MC looks intrabar at a future close of other data series. Why would you ever want to reference a future value? Realtime mc will look at the current value of other data series. Historical it will look in a bar of data2 for example, that is still in progress and not closed yet. But you will get the future close value as result. That is not normal, right?
For most users this is not expected behaviour. That's why I think it should be fixed.
thanks for your observation.
Do you also use IOG and BM with your testing? It then becomes very difficult to code your strategy. The workaround (if possible for all situations) at the end of the thread of glam_100 is not the right approach to this. Certainly with 5000 lines of code, that you will need to give a barstatus. And need to test with print statements, is IMO just crazy.
What glam_100 is describing is that MC looks intrabar at a future close of other data series. Why would you ever want to reference a future value? Realtime mc will look at the current value of other data series. Historical it will look in a bar of data2 for example, that is still in progress and not closed yet. But you will get the future close value as result. That is not normal, right?
For most users this is not expected behaviour. That's why I think it should be fixed.
Re: Matching backtest results and realtime results
Hi Edwin, have you tried
* data2 onwards as 1 tick (if d2 is same symbol then why not average d1 instead and lose d2 ?)
* testing your strat using data playback which accelerates the test process dramatically
* doing so with RTHM 'off' and then RTHM 'on'?
* adjust code and synchronisation accordingly since even tick needs matching
Cheers,
Michael
* data2 onwards as 1 tick (if d2 is same symbol then why not average d1 instead and lose d2 ?)
* testing your strat using data playback which accelerates the test process dramatically
* doing so with RTHM 'off' and then RTHM 'on'?
* adjust code and synchronisation accordingly since even tick needs matching
Cheers,
Michael
-
evdl
- Posts: 401
- Joined: 19 Jan 2011
- Location: Netherlands
- Has thanked: 85 times
- Been thanked: 124 times
Re: Matching backtest results and realtime results
Hi Michael,
I used data playback for forward testing or debugging when markets were closed. Never for backtesting. You say it speeds up testing. I did some quick tests. And it seems much slower then normal backtesting. Which is not strange I think, because of MC needs to display everything and take up more resources.
So for backtesting, say 2 years, data playback is not the tool to use I guess but I can be mistaken, because I do not have worked with playback much. Thanks for the tip.
I used data playback for forward testing or debugging when markets were closed. Never for backtesting. You say it speeds up testing. I did some quick tests. And it seems much slower then normal backtesting. Which is not strange I think, because of MC needs to display everything and take up more resources.
So for backtesting, say 2 years, data playback is not the tool to use I guess but I can be mistaken, because I do not have worked with playback much. Thanks for the tip.
-
evdl
- Posts: 401
- Joined: 19 Jan 2011
- Location: Netherlands
- Has thanked: 85 times
- Been thanked: 124 times
Re: Matching backtest results and realtime results
To get back on using data playback.
Unless I am mistaken. The data playback is not looking intrabar on dataN, tick by tick as according to the tick values in QM.
What I saw in some tests with data playback.
For example:
Data1: 30 tick
Data2: 60 tick
1) with IOG on, RTHM off, BM not available, and tick by tick playback, the same "issue" (in my opinion) of getting the future close of dataN (data2 or data3 etc) is used. So without the use of barstatus (see the workaround in the mentioned thread) to get the last know close of dataN. You will get the future close of the bar that is still in progress, just like in backtesting. Even if you use tick by tick playback. Which is inlogical to me, because for safety reasons, you should never get a future value in backtesting. Although you can use the barstatus workaround ofcourse. However this is not that obvious, because you have to expect this behaviour to use it. But that is my opinion.
2) with IOG on, RTHM on, BM not available, and tick by tick playback. From the simultanious open of data1 and data2, you will get the previous close of data2 every tick intrabar (same value every tick). That is expected. Until the next open of data1. From that point, you will get the close value tick by tick (possible new value every tick) of the data that is in QM (this could be data1 or data2, because they are the same in my test). Until the simultanious open of both data series again. This means intrabar the second half of the bar of data2, you are working with realtime close values of data1 or 2. The first half of the bar of data2 is the previous close of data2, as it should. The second half it is not. Again this can be solved with the barstatus workaround.
Trying to get the best user friendly solution. But will leave it at this.
Unless I am mistaken. The data playback is not looking intrabar on dataN, tick by tick as according to the tick values in QM.
What I saw in some tests with data playback.
For example:
Data1: 30 tick
Data2: 60 tick
1) with IOG on, RTHM off, BM not available, and tick by tick playback, the same "issue" (in my opinion) of getting the future close of dataN (data2 or data3 etc) is used. So without the use of barstatus (see the workaround in the mentioned thread) to get the last know close of dataN. You will get the future close of the bar that is still in progress, just like in backtesting. Even if you use tick by tick playback. Which is inlogical to me, because for safety reasons, you should never get a future value in backtesting. Although you can use the barstatus workaround ofcourse. However this is not that obvious, because you have to expect this behaviour to use it. But that is my opinion.
2) with IOG on, RTHM on, BM not available, and tick by tick playback. From the simultanious open of data1 and data2, you will get the previous close of data2 every tick intrabar (same value every tick). That is expected. Until the next open of data1. From that point, you will get the close value tick by tick (possible new value every tick) of the data that is in QM (this could be data1 or data2, because they are the same in my test). Until the simultanious open of both data series again. This means intrabar the second half of the bar of data2, you are working with realtime close values of data1 or 2. The first half of the bar of data2 is the previous close of data2, as it should. The second half it is not. Again this can be solved with the barstatus workaround.
Trying to get the best user friendly solution. But will leave it at this.
Re: Matching backtest results and realtime results
Assume there is stratey with only 1 symbol, based on
Data1 = 1 tick
Data2 = 16 tick
Following MichaelS answer (lose Data2), I think without Data2 the "story"
would become much easier with all the parameter like Barstatus=2, IOG, BM etc.
Therefore the question: is there any solution to build a framework for
(a) Data2 (or any type of bar) and
(b) indicators, based on Data2
Regards,
Fredi
Data1 = 1 tick
Data2 = 16 tick
Following MichaelS answer (lose Data2), I think without Data2 the "story"
would become much easier with all the parameter like Barstatus=2, IOG, BM etc.
Therefore the question: is there any solution to build a framework for
(a) Data2 (or any type of bar) and
(b) indicators, based on Data2
Regards,
Fredi