datafeed and bar formation

Questions about MultiCharts and user contributed studies.
orion
Posts: 250
Joined: 01 Oct 2014
Has thanked: 65 times
Been thanked: 104 times

datafeed and bar formation

Postby orion » 31 Oct 2014

I have a few questions regarding the datafeed handling and bar formation. I use TS as datafeed provider. I am planning to use Portfolio Trader (PT) with 10 second bars for signals. I am also thinking of setting CloseBarTimeout in the registry to 2 seconds. Since PT does not have IOG, some of following are important:

1) Does MC use my local computer time or some timestamp in the tick for bar formation?

2) If it is my local computer time, does that mean that in a perfect world with no datafeed glitches, all bars would close within 2 seconds of my computerDateTime being past the 10 second mark?

3) Does it also mean that some symbols may have bars close at (say for example) 9:53:20, others at 9:53:21, and the rest at 9:53:22 based on market activity and CloseBarTimeout?

4) Does the above introduce asynchrony in PT that my strategy has to be aware of since strategy signals on some symbols arrive at 9:53:20 on bar close while others do not get to run until 9:53:22?

5) Now lets understand the case with datafeed glitches. I have noticed in QM (Quote Manager) the logs have frequent 1 or 2 second glitches which say "Lost connection to Data Network" followed by "Connection is OK". Are these a result of loss of connection between QM and my local TS process or due to loss of connection between my local TS process and TS servers?

6) How do these 2 second glitches affect bar closure? Say a 2 second glitch is detected by QM at 9:53:19. Since we are the middle of a glitch, does the bar close for any of the symbols at 9:53:20?

7) I have also noticed occasional but less frequent 10-15 glitches in my local TS where it has a popup message saying it has lost connection to the servers. These glitches are less frequent than the 1-2 second glitches mentioned above but they occur nonetheless and they occur even when my ISP connection is fine which means the problem is at the TS server end. How would such a longer glitch affect bar closure in PT?

8) I understand that for historical data, MC uses 3 separate data series: tick, minute, and daily data. Is that also the case for realtime data?

9) If answer to last question is yes, then does that mean that a 10 second bar may be waiting for closure based on a CloseBarTimeout while the minute bar may have closed since it is being received directly from TS?

Thanks,
orion

hilbert
Posts: 224
Joined: 17 Aug 2011
Has thanked: 76 times
Been thanked: 64 times

Re: datafeed and bar formation

Postby hilbert » 01 Nov 2014

viewtopic.php?f=1&t=47403

Its weekend, so MC Support will revert to you after 2 days. Meanwhile, this thread might answer some of your questions.

User avatar
Henry MultiСharts
Posts: 9165
Joined: 25 Aug 2011
Has thanked: 1264 times
Been thanked: 2957 times

Re: datafeed and bar formation

Postby Henry MultiСharts » 05 Nov 2014

Hello Orion,

Let me answer your questions in order:
1. TS timestamps are used for both realtime and historical data (coming from TS).
2. In a perfect world all bars would close with resolution increment, CloseBarTimeout won’t be required.
3. 4. CloseBarTimeout is not included into the bar timestamp. If your chart resolution is 10 seconds then bars close with 10 secs increment starting from the Session Start time. Session Start time is specified in hh:mm format. So 9:53:22 is not possible for a 10 second chart.
5. Please provide logs for such occurrence and send specify the exact time such messages appeared.
6. 7. The bar will close at 9:53:20. Either itself or with the help of CloseBarTimeout. If the data source was unavailable then the data in the bar can be incomplete. TS reloads the data itself if there were connection issues. So if TS backfills the missing data – you need to copy/paste your chart or reload it in MultiCharts to backfill the data.
8.TS provides the following data:
Historical data: Trade(Tick,Min,Day);
Realtime: Trade(Tick, Day).
9. Tick-based and minute-based charts should be updating the same way.

orion
Posts: 250
Joined: 01 Oct 2014
Has thanked: 65 times
Been thanked: 104 times

Re: datafeed and bar formation

Postby orion » 05 Nov 2014

Thanks Henry.
5. Please provide logs for such occurrence and send specify the exact time such messages appeared.
I looked into the issue and found that the QM messages occur when my local TS process loses connectivity to TS servers. So the reliability issue is with TS server connectivity. MC is working flawlessly. Issue closed on MC side.
3. 4. CloseBarTimeout is not included into the bar timestamp. If your chart resolution is 10 seconds then bars close with 10 secs increment starting from the Session Start time. Session Start time is specified in hh:mm format. So 9:53:22 is not possible for a 10 second chart.
6. 7. The bar will close at 9:53:20. Either itself or with the help of CloseBarTimeout.
One more question and that will close this thread for me.

There are two possible definitions of "bar close" for realtime bars: (a) the bar time assigned to the bar by MC, and (b) the time at which barStatus = 2 event occurs for strategy code. When I said bar will close at 9:53:22 due to glitch in the datafeed, I was referring to barStatus = 2 event for the bar coming in at 9:53:22 given the CloseBarTimeout setting. I expect the assigned bar time to still be 9:53:20 for such a 10 sec bar; it would be madness to expect anything else. I presume that is what you mean as well?

User avatar
Henry MultiСharts
Posts: 9165
Joined: 25 Aug 2011
Has thanked: 1264 times
Been thanked: 2957 times

Re: datafeed and bar formation

Postby Henry MultiСharts » 07 Nov 2014

One more question and that will close this thread for me.

There are two possible definitions of "bar close" for realtime bars: (a) the bar time assigned to the bar by MC, and (b) the time at which barStatus = 2 event occurs for strategy code. When I said bar will close at 9:53:22 due to glitch in the datafeed, I was referring to barStatus = 2 event for the bar coming in at 9:53:22 given the CloseBarTimeout setting. I expect the assigned bar time to still be 9:53:20 for such a 10 sec bar; it would be madness to expect anything else. I presume that is what you mean as well?
That is correct.

orion
Posts: 250
Joined: 01 Oct 2014
Has thanked: 65 times
Been thanked: 104 times

Re: datafeed and bar formation

Postby orion » 07 Nov 2014

Thanks Henry.

There will be bars that close at 9:53:22 even without data glitches (close in the sense that the barStatus = 2 event occurs at 9:53:22). Example may be an equity which did not tick for a few seconds. How that works when you have one chart trading one symbol is well understood.

I want to understand what happens with the PT where you have this process shown in http://www.multicharts.com/trading-soft ... on_Diagram. Does that run through all the N symbols again at 9:53:22 even though it ran at 9:53:20, and if so, then during each round-robin of N symbols, whether the round-robin happens at 9:53:20 or 9:53:22, it skips over symbols that are not currently at barStatus = 2?

User avatar
Henry MultiСharts
Posts: 9165
Joined: 25 Aug 2011
Has thanked: 1264 times
Been thanked: 2957 times

Re: datafeed and bar formation

Postby Henry MultiСharts » 10 Nov 2014

I want to understand what happens with the PT where you have this process shown in http://www.multicharts.com/trading-soft ... on_Diagram. Does that run through all the N symbols again at 9:53:22 even though it ran at 9:53:20, and if so, then during each round-robin of N symbols, whether the round-robin happens at 9:53:20 or 9:53:22, it skips over symbols that are not currently at barStatus = 2?
Each strategy is calculated individually in realtime. The signals of a particular strategy run only when the data1 of this series is closed.

orion
Posts: 250
Joined: 01 Oct 2014
Has thanked: 65 times
Been thanked: 104 times

Re: datafeed and bar formation

Postby orion » 10 Nov 2014

I want to understand what happens with the PT where you have this process shown in http://www.multicharts.com/trading-soft ... on_Diagram. Does that run through all the N symbols again at 9:53:22 even though it ran at 9:53:20, and if so, then during each round-robin of N symbols, whether the round-robin happens at 9:53:20 or 9:53:22, it skips over symbols that are not currently at barStatus = 2?
Each strategy is calculated individually in realtime. The signals of a particular strategy run only when the data1 of this series is closed.
Henry, I understand that behavior in context of chart trading. I am looking to confirm what happens in case of PT. Assume data1 series is 10 second bars with 100 equities in PT. Since barStatus=2 event can occur at different times based on market ticks and CloseBarTimeout, does that mean that PMM signal can run at 9:53:20 when some symbols get barStatus=2, then at 9:53:21 when some more symbols get barStatus=2 due to market ticks, and then at 9:53:22 due to CloseBarTimeout of all symbols? Thanks.

User avatar
Henry MultiСharts
Posts: 9165
Joined: 25 Aug 2011
Has thanked: 1264 times
Been thanked: 2957 times

Re: datafeed and bar formation

Postby Henry MultiСharts » 11 Nov 2014

PMM signal is calculated in IOG, so each close of each data1 triggers it's calculation. It is also recalculated based on broker events.

orion
Posts: 250
Joined: 01 Oct 2014
Has thanked: 65 times
Been thanked: 104 times

Re: datafeed and bar formation

Postby orion » 14 Feb 2015

Resolved and deleted.
Last edited by orion on 18 Feb 2015, edited 1 time in total.

tony
Posts: 420
Joined: 14 Jun 2013
Has thanked: 30 times
Been thanked: 81 times
Contact:

Re: datafeed and bar formation

Postby tony » 16 Feb 2015

I thought bars formed based on computer time, NOT the time stamp on the tick. But reading above, it sounds like it is in fact the time stamp. Did this change with the most version of MC? I sync my PC clock every day because of time getting out of sync and thus messing up bar formations.

orion
Posts: 250
Joined: 01 Oct 2014
Has thanked: 65 times
Been thanked: 104 times

Re: datafeed and bar formation

Postby orion » 16 Feb 2015

I thought bars formed based on computer time, NOT the time stamp on the tick. But reading above, it sounds like it is in fact the time stamp. Did this change with the most version of MC?
You are misunderstanding what is being said in the thread. Bars do indeed form based on computer time for realtime data.

User avatar
Henry MultiСharts
Posts: 9165
Joined: 25 Aug 2011
Has thanked: 1264 times
Been thanked: 2957 times

Re: datafeed and bar formation

Postby Henry MultiСharts » 18 Feb 2015

I thought bars formed based on computer time, NOT the time stamp on the tick. But reading above, it sounds like it is in fact the time stamp. Did this change with the most version of MC?
You are misunderstanding what is being said in the thread. Bars do indeed form based on computer time for realtime data.
orion, I have stated the opposite above:
1. TS timestamps are used for both realtime and historical data (coming from TS).

orion
Posts: 250
Joined: 01 Oct 2014
Has thanked: 65 times
Been thanked: 104 times

Re: datafeed and bar formation

Postby orion » 18 Feb 2015

Henry, I apologize if I caused confusion.

I think tony was seeing differences in his trading strategies on two computers if their clocks were not synced and that is what motivated his question. The reason for him seeing differences is because bar formation (as determined by CloseBarTimeout setting in the registry) is indeed dependent on local computer time. The timestamps for both realtime and historical data do indeed come from TS as you very clearly stated.

It is a combination of the datafeed timestamp and the local computer time that determines the bar formation process. There is actually a third factor that also enters the equation. And that is the exchange timezone setting in the symbol dictionary! For example, if you change the exchange timezone for CME to New York instead of Chicago, you will cease to get bars on your ES chart. So in summary, there are 3 things that go into the bar formation process:

1. datafeed timestamps
2. computer clock time
3. exchange timezone setting in the symbol dictionary

Please let me know if the above is not correct.


Return to “MultiCharts”